Autoregressive Language Modeling

Follow this topic Ask about this topic

Autoregressive language models learn to predict each next token from the preceding context, a simple objective that supports both generation and tasks such as speech or protein modeling. Transformer attention remains a common approach, while recurrent and state-space models, sparse memory, and alternative tokenizations aim to reduce the cost of long contexts and large vocabularies. Current work also explores faster generation through multi-token prediction, latent lookahead, and combinations of causal and diffusion methods, evaluated on measures such as perplexity, retrieval, and downstream reasoning benchmarks. A central tension is how to gain speed and long-range capability without losing the quality, reliability, and generalization associated with left-to-right prediction.

Latest papers

132 papers15 papers in the last four weeks, up 25% on the four weeks before. 0.1% of all new papers.

A Comparative analysis of Layer-wise Representational Capacity in AR and Diffusion LLMs
Mar 8, 2026Raghavv Goel, Risheek Garrepalli, Sudhanshu Agrawal +3Diffusion Language Models Representation Geometry in Language Models
Qualcomm AI Research, Qualcomm Technologies Inc.
WavSLM: Single-Stream Speech Language Modeling via WavLM Distillation
Mar 5, 2026Luca Della Libera, Cem Subakan, Mirco RavanelliSpeech Language Models Autoregressive Language Modeling
Concordia University · Mila-Quebec AI Institute · Universit´e Laval
Thinking into the Future: Latent Lookahead Training for Transformers
Mar 3, 2026Lorenzo Noci, Gregor Bachmann, Seyed-Mohsen Moosavi-Dezfooli +1Autoregressive Language Modeling Sequence Prediction
Apple
Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning
Feb 24, 2026Justin Lovelace, Christian Belardi, Sofian Zalouk +3Autoregressive Language Modeling Diffusion Language Models
Department of Computer Science, Cornell University
Discrete Diffusion Models Exploit Asymmetry to Solve Lookahead Planning Tasks
Feb 23, 2026Itamar Trainin, Shauli Ravfogel, Omri Abend +1Diffusion Language Models Autoregressive Language Modeling
Hebrew University of Jerusalem · New York University
Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models
Jan 21, 2026Injin Kong, Hyoungjoon Lee, Yohan JoMasked Diffusion Models Autoregressive Diffusion
Graduate School of Data Science, Seoul National University · Department of Biosystems & Biomaterials Science and Engineering, Seoul National University
On the Runway Cascade of Transformers for Language Modeling
Jan 20, 2026Hunjae Lee, Corey ClarkTransformer Attention Decoder-Only Language Models
Department of Computer Science, Southern Methodist University, Dallas TX USA.
Testing Transformer Learnability on the Arithmetic Sequence of Rooted Trees
Dec 1, 2025Alessandro Breccia, Federica Gerace, Marco Lippi +2Transformer Autoregressive Language Modeling
Department of Mathematics, University of Bologna, Italy · Gatsby Computational Neuroscience Unit, UCL, UK · Department of Information Engineering, University of Florence, Italy
Selective Rotary Position Embedding
Nov 21, 2025Sajad Movahedi, Timur Carstensen, Arshia Afzal +3Softmax Attention Rotary Positional Embeddings
ELLIS Institute Tübingen1, LIONS, EPFL2, University of Freiburg3, Max-Planck-Institute for Intelligent Systems4, Prior Labs5
Training LLMs Beyond Next Token Prediction - Filling the Mutual Information Gap
Oct 31, 2025Chun-Hao Yang, Bo-Han Feng, Tzu-Yuan Lai +3Autoregressive Language Modeling Sequence Prediction
National Taiwan University
Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation
Sep 4, 2025Peter RacioppoSelf-Attention Long-Context Language Modeling
Independent Researcher, Los Angeles, CA, USA.
Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture
Jun 24, 2025Shuchen Xue, Tianyu Xie, Tianyang Hu +5Masked Diffusion Models Decoder-Only Language Models
University of Chinese Academy of Sciences · Academy of Mathematics and Systems Science, Chinese Academy of Sciences · School of Mathematical Sciences, Peking University +2

Mar 8, 2026J/K move · Enter open · S save

A Comparative analysis of Layer-wise Representational Capacity in AR and Diffusion LLMs

Raghavv Goel, Risheek Garrepalli, Sudhanshu Agrawal, Chris Lott+2

Qualcomm AI Research, Qualcomm Technologies Inc.

Diffusion Language Models Representation Geometry in Language Models Autoregressive Language Modeling LLM Compression LLM Pruning

Open paper AskPDF arXiv

Autoregressive (AR) language models build representations incrementally via left-to-right prediction, while diffusion language models (dLLMs) are trained through full-sequence denoising. Although recent dLLMs match AR performance, whether diffusion objectives fundamentally reshape internal representations remains unclear. We perform the first layer- and token-wise representational analysis comparing native dLLMs (LLaDA), native AR models (Qwen2.5), and AR-initialized dLLMs (Dream-7B), using cosine similarity across layers and tokens alongside static inference-time layer-skipping as an analytical probe of redundancy. We find that diffusion objectives produce more global representations with substantial early-layer redundancy and reduced recency bias, while AR objectives yield tightly coupled, locally structured representations. AR-initialized dLLMs retain AR-like dynamics despite diffusion training, revealing persistent initialization bias. Leveraging this redundancy, native dLLMs absorb up to 18.75% FLOPs reduction while retaining over 90% performance on math-reasoning and coding benchmarks, whereas AR models collapse under identical skipping, revealing that diffusion objectives, rather than architecture alone, induce depth redundancy that enables principled compression.