Autoregressive Language Modeling

Overview

Autoregressive language models learn to predict each next token from the preceding context, a simple objective that supports both generation and tasks such as speech or protein modeling. Transformer attention remains a common approach, while recurrent and state-space models, sparse memory, and alternative tokenizations aim to reduce the cost of long contexts and large vocabularies. Current work also explores faster generation through multi-token prediction, latent lookahead, and combinations of causal and diffusion methods, evaluated on measures such as perplexity, retrieval, and downstream reasoning benchmarks. A central tension is how to gain speed and long-range capability without losing the quality, reliability, and generalization associated with left-to-right prediction.

Latest papers

CardsList

142 papers7 papers in the last four weeks, down 56% on the four weeks before. 0.1% of all new papers.

  1. Kernel Ridge Attention: Nonlinear Memory with Selective Forgetting

    Nov 18, 2026Haowei Chen, Yichen Wang, Congfu XuAttention MechanismsMemory-Augmented Language Models

  2. Structure--Residual Inductive Biases for Language Model Pretraining

    Nov 17, 2026Hai Huang, Bartosz Żuk, Rassul Magauin +2Language Model PretrainingAutoregressive Language Modeling

  3. What's in a Tree ? Hierarchical nn-gram models as controlled approximators of language

    Nov 17, 2026Aritra Das, Francesco Cagnetta, Darshil Doshi +3Language ModelingAutoregressive Language Modeling

  4. Optimal Autoregressive Chain-of-Thought Learning via Bagging

    Nov 16, 2026Awni Altabaa, John Lafferty, Omar MontasserAutoregressive Language ModelingStructured Reasoning in Language Models

  5. On the Difficulty of Sequence-Level Supervised Learning for Autoregressive Language Models

    Nov 12, 2026Mathieu Blondel, Michaël E. Sander, Germain Vivier-Ardisson +2Moment MatchingAutoregressive Language Modeling

  6. Stochastic Dynamics of Generative AI: From Autoregressive Models to Interacting LLM Networks and Neural SDEs

    Nov 4, 2026Vikram Krishnamurthy, Yiming Zhang, Luke SnowAutoregressive Language ModelingStochastic Differential Equations

  7. When can Optimally Trained Transformers Achieve Near-Lossless Token Prediction for non-Markovian Data?

    Oct 29, 2026Kağan Akman, Jake Feldman Starosta, Naci Saldi +1Autoregressive TransformerSequence Modeling

  8. Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for Portuguese

    Oct 29, 2026Pedro Carvalho BromLanguage ModelingSmall Language Models

  9. Dual-Slot: From Serial Loops to Parallel Slots in Language Models

    Oct 18, 2026Yiqin Hao, Zitong Wang, Boyi Zeng +7Autoregressive Language ModelingRecurrent Language Models

  10. Causal Edit Serialization

    Oct 16, 2026Louis RobinsonPointer NetworksLanguage Model Editing

  11. The Geometry of Language Model Logits: Generic Structure and Learned Departures

    Oct 13, 2026Swathi Shree Narashiman, Aravind Ramana Venkatramanan, Nambi SeshadriAutoregressive Language ModelingRepresentation Geometry in Language Models

  12. Autoregressive Decoding as a Martingale: Characterizing the Stochastic Dynamics of LLM Generation

    Oct 12, 2026Xiahui Zhu, Zhenyi Qi, Wanwei Liu +4Autoregressive DecodingAutoregressive Language Modeling

  13. Greedy Local Learning for Language Model Pretraining: Gaps and Objective Design

    Oct 4, 2026Jihwan Moon, Sheir A. Zaheer, Jinmyoung Lee +2Local LearningLanguage Model Pretraining

  14. Learning Latent Protein Languages for Autoregressive Generation

    Oct 2, 2026Mahdi Pourmirzaei, Farzaneh Esmaili, Amir Ziashahabi +2Protein Structure GenerationProtein Language Models

  15. LS-AR: Future-Predictive Latent Steering in Autoregressive LLMs

    Oct 2, 2026Anubha Gupta, Eduardo PignatelliAutoregressive Language ModelingLanguage Model-Based Control

  16. PassGPT+: Leveraging Linguistic Priors for Password Modeling

    Sep 30, 2026Rajneesh Anand, Neeraj Lakshmanan, Masoud YariAutoregressive Language ModelingDiffusion Language Models

  17. CyFA: Linear Sequence Modeling with Relative-Time-Partitioned Memory

    Sep 28, 2026Yixiao Chen, Shuojin Yang, Shi-Min HuLinear RNNsLinear Attention

  18. Decoupling Token Roles in Autoregressive Pretraining

    Sep 27, 2026Suqin Yuan, Runqi Lin, Kevin Qinghong Lin +4Language Model PretrainingAutoregressive Language Modeling

  19. Log-Depth Recurrent Language Modeling

    Sep 23, 2026Yiqin Wang, Nuri Cingillioglu, Charles PertRecurrent Language ModelsLong-Context Language Modeling

  20. Variational Quantum Transformer Architecture for Synthetic Language Generation

    Sep 16, 2026Julian Hager, Michael Kölle, Gerhard Stenzel +3Quantum NLPQuantum Machine Learning

  21. Structural priors for data-efficient language learning

    Sep 10, 2026Yana Veitsman, Jonas Mayer Martins, Jonathan Lautenschlager +1Language Model PretrainingNeural Network Initialization