Source Themes

Probability Distributions Computed by Hard-Attention Transformers
We show some divergences in expressive power when transformers are used as probabilistic and autoregressive sequence models.
Andy Yang, Anej Svete, Jiaoda Li, Anthony Widjaja Lin, Jon Rawski, Ryan Cotterell, David Chiang
ArXiv Preprint
Probability Distributions Computed by Hard-Attention Transformers
Transformers as Transducers
We characterize transformer sequence models as transducers and show some expressivity bounds for several variants.
Lena Strobl, Dana Angluin, David Chiang, Jon Rawski, Ashish Sabharwal
PDF Open-access Publication
Transformers as Transducers
The Problem-Ladenness of Theory
We give a pragmatic problem-centered account of theories’ epistemic virtues, like coherence, depth, and parsimony.
Daniel Levenstein, Aniello De Santo, Saskia Heijnen, M. Narayan, Freek Oude Maatman, Jon Rawski, Cory wright
Official Pub OSF link
The Problem-Ladenness of Theory