Autoregressive Language Modeling

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 81

All topics

Developmental approach reveals the statistical learning of Neural Language Models: Transformers generalize from the most abstract statistical patterns
Jun 25, 2026Wang Bojun, Holly Jenkins, Elizabeth WonnacottAutoregressive Language Modeling Computational Psycholinguistics
Department of Education, University of Oxford
Structure Before Collapse: Transient semantic geometry in next-token prediction
Jun 25, 2026Yize Zhao, Isabel Papadimitriou, Christos ThrampoulidisAutoregressive Language Modeling Next-Token Prediction
Department of Electrical and Computer Engineering, The University of British Columbia · Department of Linguistics, The University of British Columbia
Tapered Language Models
Jun 22, 2026Reza Bayat, Ali Behrouz, Aaron CourvilleAutoregressive Language Modeling
1Mila · 2Cornell University · 3Université de Montréal +1
Attention as Frustrated Synchronization
Jun 17, 2026Joshua NunleyOscillatory Neural Networks Autoregressive Language Modeling
Cognitive Science Program Luddy School of Informatics, Computing, and Engineering Indiana University Bloomington
Why SWAVE May Not Be All You Need:A Concept-Evolution Retrospective on Complex-Valued Recurrent Language Models
Jun 16, 2026Ramprasath Ganesaraja, Swathika N, Sahil Dilip PanseRecurrent Neural Networks Complex-Valued Neural Networks
EdgeVerve Systems Limited
Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining
Jun 15, 2026Michael K. Chen, Xikun Zhang, Fan Bai +2Language Model Pretraining Data Augmentation
1UC San Diego · 2RMIT University · 3Bloomberg AI
Circuit Tracing in Autoregressive Protein Language Models
Jun 14, 2026Darin Tsui, William Deinzer, Daniel Saeedi +1Circuit Discovery Protein Fitness Prediction
School of Electrical and Computer Engineering, Georgia Institute of Technology, Atltanta, GA.
Simplifying the Modeling of Arbitrary Conditionals in Natural Language
Jun 12, 2026Yinhan Lu, Eric Elmoznino, Léo Gagnon +3Transformer Autoregressive Language Modeling
Mila — Quebec AI Institute · McGill University · Université de Montréal
K-Forcing: Joint Next-K-Token Decoding via Push-Forward Language Modeling
Jun 9, 2026Zhiwei Tang, Yuanyu He, Yizheng Han +4LLM Serving Language Model Decoding
1DAMO Academy, Alibaba Group · 2Hupan Lab · 3Zhejiang University +1
Parallel Causal Associative Fields: Gated Sparse Memory for Long-Context Language Modeling
Jun 9, 2026Muhammad AhmedMemory-Augmented Language Models Long-Context Language Model Inference
Independent Researcher
CART: Context-Anchored Recurrent Transformer -- A Parameter-Efficient Architecture with Learned Stability
May 31, 2026Chad A. CappsRecurrent Neural Networks Autoregressive Language Modeling
Independent Researcher
Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing
May 29, 2026Erwan Fagnou, Paul Caillon, Blaise Delattre +1Transformer Expressivity Autoregressive Language Modeling
Miles Team, LAMSADE, Université Paris Dauphine-PSL, Paris, France · Department of Computer Science, School of Computing, Institute of Science Tokyo, Tokyo, Japan · ESPCI PSL, Paris, France
Cognitive Fatigue in Autoregressive Transformers: Formalization and Measurement
May 29, 2026Riju Marwah, Ritvik Garimella, Vishal Pallagani +3Autoregressive Language Modeling
Guru Gobind Singh Indraprastha University, India · Artificial Intelligence Institute, University of South Carolina, USA · Indian Institute of Technology, Kanpur, India +1
MELD: Mel-Spectrogram-Based Speech Language Modeling with Discrete Latent Variables
May 28, 2026Sung-Lin Yeh, Wei Zhou, Gil Keren +6Audio Representation Learning Autoregressive Language Modeling
University of Edinburgh · *Work done at Meta · 2Google DeepMind +1
EvoMD-LLM: Learning the Language of Species Evolution in Reactive Molecular Dynamics
May 28, 2026Zhichen Tang, Zhengzheng Dang, Yulin Chen +3Autoregressive Language Modeling Language Modeling
Global College, Shanghai Jiao Tong University, Shanghai, China · Global Institute of Future Technology, Shanghai Jiao Tong University, Shanghai, China
Kan Extension Transformers: A Categorical Unification of Attention, Diffusion, and Predict-Detach Self-Conditioning
May 26, 2026Sridhar MahadevanTransformer Autoregressive Language Modeling
Adobe Research San Jose, CA · Academic affiliation: Research Professor, University of Massachusetts, Amherst
Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
May 26, 2026Zeyi Huang, Xuehai He, LiLiang Ren +8Language Model Pretraining Transformer Inference
Microsoft · University of Washington · University of Wisconsin-Madison
Polynomial Context-Truncation Sensitivity in Autoregressive Language Models: Sequential Wyner-Ziv Bounds for KV Cache Compression
May 24, 2026Munsik KimKV Caching Rate-Distortion Theory
Independent Researcher.
TUBE: Tangent Upper Bound on Evidence for Discrete Diffusion Language Models
May 22, 2026Arseny Ivanov, Sergei Kholkin, Vladislav Gromadskii +3Masked Diffusion Models Autoregressive Language Modeling
AXXX, Russia · Applied AI Institute, Moscow, Russia · MIRAI, Russia
When Is Next-Token Prediction Useful? Marginalization, Ergodicity, Mixture Identifiability, Local Sufficiency, RAG, Tools, and Programming
May 22, 2026Francesco CorielliAutoregressive Language Modeling Next-Token Prediction
MiniGPT: Rebuilding GPT from First Principles
May 17, 2026Jibin JosephLanguage Model Pretraining Decoder-Only Language Models
Department of Computer Science The University of Texas at Austin Austin, TX 78712
Modeling Music as a Time-Frequency Image: A 2D Tokenizer for Music Generation
May 15, 2026Yuqing Cheng, Xingyu Ma, Guochen Yu +1Neural Audio Codecs Autoregressive Language Modeling
Department of Music AI and Information Technology, Central Conservatory of Music, Beijing, China · Zhipu AI, Beijing, China
Conditional Attribute Estimation with Autoregressive Sequence Models
May 13, 2026Erica Stutz, Giacomo Marino, Daniella Meeker +2Counterfactual Reasoning in Language Models Controllable Text Generation
Department of Biomedical Informatics and Data Science Yale University New Haven, CT 06510 · Department of Biostatistics Yale University New Haven, CT 06510 · Department of Biomedical Informatics and Data Science, Department of Pediatrics Yale University New Haven, CT 06510
A Hierarchical Language Model with Predictable Scaling Laws and Provable Benefits of Reasoning
May 13, 2026Jason Gaitonde, Frederic Koehler, Elchanan Mossel +2Long-Context Language Modeling Language Model Scaling Laws
Duke University · University of Chicago · Massachusetts Institute of Technology +1
The Expressivity Boundary of Probabilistic Circuits: A Comparison with Large Language Models
May 13, 2026Zhiyu Zhao, Xuejie Liu, Muhan Zhang +1Probabilistic Circuits Autoregressive Language Modeling
School of Computing, National University of Singapore · School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University
The Efficiency Gap in Byte Modeling
May 13, 2026Celine Lee, Jing Nathan Yan, Chen Liang +9Language Model Scaling Laws Autoregressive Language Modeling
1Google DeepMind · 2Work done while at Google DeepMind · Department of Computer Science, Cornell University.
A Causal Language Modeling Detour Improves Encoder Continued Pretraining
May 12, 2026Rian Touchent, Eric de la ClergerieLanguage Model Pretraining Masked Language Modeling
Sorbonne Université / INRIA Paris · ALMAnaCH Team · INRIA Paris
BitLM: Unlocking Multi-Token Language Generation with Bitwise Continuous Diffusion
May 12, 2026Shaobin Zhuang, Yuang Ai, Jiaming Han +7Multi-Token Prediction Autoregressive Language Modeling
Shanghai Jiao Tong University · MMLab, The Chinese University of Hong Kong · Institute of Automation, Chinese Academy of Sciences +1
A Single-Layer Model Can Do Language Modeling
May 11, 2026Zanmin WangLanguage Model Pretraining Recurrent Neural Networks
Self-Attention as a Covariance Readout: A Unified View of In-Context Learning and Repetition
May 11, 2026Haoren Xu, Guanhua FangSoftmax Attention Self-Attention
Fudan University

Jun 25, 2026, cs.CLJ/K move · Enter open · S save

Developmental approach reveals the statistical learning of Neural Language Models: Transformers generalize from the most abstract statistical patterns

Wang Bojun, Holly Jenkins, Elizabeth Wonnacott

Department of Education, University of Oxford

Autoregressive Language Modeling Computational Psycholinguistics

Open paper AskPDF arXiv

In this study, we use a developmental approach to investigate the statistical learning and mental representation of neural language models (NLM). A series of Generative Transformer models are trained on a synthetic grammar. The model states are saved at multiple stages in the course of training. Through analyzing how the internal representations of these models change in the developmental path, we found that NLMs acquire the most abstract global statistical knowledge at the beginning of learning and later acquire the relatively local statistical dependencies. This learning path contains many over-generalizations from the very beginning and these over-generalizations are gradually constrained in the later stage of learning. Based on this observation, we propose a new framework to explain the statistical learning and language cognition of NLMs.

Momentum

Related topics

Latest papers 81