Autoregressive Language Modeling

Ask about this topic

Overview

Autoregressive language models learn to predict each next token from the preceding context, a simple objective that supports both generation and tasks such as speech or protein modeling. Transformer attention remains a common approach, while recurrent and state-space models, sparse memory, and alternative tokenizations aim to reduce the cost of long contexts and large vocabularies. Current work also explores faster generation through multi-token prediction, latent lookahead, and combinations of causal and diffusion methods, evaluated on measures such as perplexity, retrieval, and downstream reasoning benchmarks. A central tension is how to gain speed and long-range capability without losing the quality, reliability, and generalization associated with left-to-right prediction.

Latest papers

CardsList

132 papers15 papers in the last four weeks, up 25% on the four weeks before. 0.1% of all new papers.

  1. Emergence of the transformer architecture as the score of entropic laws of random measures

    Oct 29, 2026Giacomo GrecoTransformer AttentionAutoregressive Language Modeling

  2. Predictive Latents for Autoregressive Language Models: Possibility, Gauge, and Obstruction

    Oct 20, 2026Yuzhou HongAutoregressive Language ModelingLatent Dynamics Modeling

    Zhejiang Sci-Tech University

  3. Dual-Slot: From Serial Loops to Parallel Slots in Language Models

    Oct 18, 2026Yiqin Hao, Zitong Wang, Boyi Zeng +7Autoregressive Language ModelingRecurrent Language Models

    LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai AI Laboratory

  4. Causal Edit Serialization

    Oct 16, 2026Louis RobinsonPointer NetworksLanguage Model Editing

    Independent Researcher

  5. Causal Latent Workspaces for Looped Transformers

    Oct 16, 2026Ilia ZaznovAutoregressive Language ModelingMemory-Augmented Language Models

  6. D-Composer: Language Modeling for Simultaneous Drum Transcription and Sound Event Decomposition

    Oct 15, 2026Patrick O'Reilly, Saumya Pailwan, Annie Chu +2Automatic Music TranscriptionAutoregressive Language Modeling

    Northwestern University, Evanston, USA

  7. TurboMamba: Faster Inference, Faster Training, and Better Prediction with Lexical Memory

    Oct 14, 2026Hanxin ZhangHybrid Transformer-State Space Model ArchitecturesAutoregressive Language Modeling

    Department of Neurology, University of Pittsburgh

  8. The Geometry of Language Model Logits: Generic Structure and Learned Departures

    Oct 13, 2026Swathi Shree Narashiman, Aravind Ramana Venkatramanan, Nambi SeshadriAutoregressive Language ModelingRepresentation Geometry in Language Models

    Stanford University · Indian Institute of Technology Madras · University of California San Diego (UCSD)

  9. Autoregressive Decoding as a Martingale: Characterizing the Stochastic Dynamics of LLM Generation

    Oct 12, 2026Xiahui Zhu, Zhenyi Qi, Wanwei Liu +4Autoregressive DecodingAutoregressive Language Modeling

    College of Computer, National University of Defense Technology, Changsha, Hunan, China

  10. CNet: A Complex-Valued Deep Learning Framework with Wirtinger Autodifferentiation and FFT--Hadamard Convolution

    Oct 6, 2026Marcel CrasmaruComplex-Valued Neural NetworksAutoregressive Language Modeling

  11. Greedy Local Learning for Language Model Pretraining: Gaps and Objective Design

    Oct 4, 2026Jihwan Moon, Sheir A. Zaheer, Jinmyoung Lee +2Local LearningLanguage Model Pretraining

    INFOCZ Inc. · Seoul National University · KC ML2

  12. Learning Latent Protein Languages for Autoregressive Generation

    Oct 2, 2026Mahdi Pourmirzaei, Farzaneh Esmaili, Amir Ziashahabi +2Protein Structure GenerationProtein Language Models

    University of Missouri · University of Southern California · Independent Researcher

  13. LS-AR: Future-Predictive Latent Steering in Autoregressive LLMs

    Oct 2, 2026Anubha Gupta, Eduardo PignatelliAutoregressive Language ModelingLanguage Model-Based Control

    Electronic & Electrical Engineering University College London London, UK

  14. Generative modeling of intrinsically disordered protein regions by reinforcing sparse autoencoder features

    Oct 1, 2026Jason X. Liu, Sebastian Ibarraran, Frank Hu +7Autoregressive Language ModelingSparse Autoencoders

    Stanford University

  15. PassGPT+: Leveraging Linguistic Priors for Password Modeling

    Sep 30, 2026Rajneesh Anand, Neeraj Lakshmanan, Masoud YariAutoregressive Language ModelingDiffusion Language Models

    Department of Chemical and Biomolecular Engineering, Lehigh University, Bethlehem, PA, 18015, USA · Department of Computer Science & Design, Singapore University of Technology & Design, Singapore · Department of Computer Science & Engineering, Lehigh University, Bethlehem, PA, 18015, USA

  16. CyFA: Linear Sequence Modeling with Relative-Time-Partitioned Memory

    Sep 28, 2026Yixiao Chen, Shuojin Yang, Shi-Min HuLinear RNNsLinear Attention

    Tsinghua University

  17. Decoupling Token Roles in Autoregressive Pretraining

    Sep 27, 2026Suqin Yuan, Runqi Lin, Kevin Qinghong Lin +4Language Model PretrainingAutoregressive Language Modeling

    University of Sydney · University of Oxford · Southeast University

  18. Log-Depth Recurrent Language Modeling

    Sep 23, 2026Yiqin Wang, Nuri Cingillioglu, Charles PertRecurrent Language ModelsLong-Context Language Modeling

    Department of Computing Imperial College London London, United Kingdom

  19. The Probabilistic Structure of Large Language Models

    Sep 20, 2026Adnan AboulalaâLanguage ModelingAutoregressive Language Modeling

  20. Variational Quantum Transformer Architecture for Synthetic Language Generation

    Sep 16, 2026Julian Hager, Michael Kölle, Gerhard Stenzel +3Quantum NLPQuantum Machine Learning

    Institute of Informatics, LMU Munich, Oettingenstraße 67, 80538 Munich, Germany

  21. Structural priors for data-efficient language learning

    Sep 10, 2026Yana Veitsman, Jonas Mayer Martins, Jonathan Lautenschlager +1Language Model PretrainingNeural Network Initialization

    University of Göttingen, Germany

  22. NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction

    Sep 9, 2026The Intern-NCP Team, :, Jiaqi Cao +26Language Model PretrainingAutoregressive Language Modeling

  23. RiLM: Parameter-Efficient Language Modeling via Geodesic Decoding

    Sep 9, 2026Fang LiAutoregressive Language ModelingLanguage Modeling

    Department of Computer Science Oklahoma Christian University Edmond, USA

  24. Line-Coupled Language Model

    Sep 7, 2026Shiyuan Li, Shaorong Zhang, Zhaorui Yang +3Multi-Token PredictionLanguage Model Decoding

    University of California, Riverside Riverside, CA, USA

  25. ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Models

    Sep 3, 2026Quang Hoang Trung, Quang Huu Hieu, Nguyen Van Hoang Phuc +1Language Model PretrainingAutoregressive Language Modeling

    VJ Technologies, Da Nang, Vietnam · AJ Technologies, Nagoya, Japan · Vietnam National University, Ho Chi Minh City, Vietnam +1

  26. Graph Machine: Towards Better Pretraining via Edges

    Sep 2, 2026Lintai HouTransformerAutoregressive Language Modeling

    Iter Labs

  27. Subword Segmental BabyLMs: Learning to Tokenise for Sample-Efficient Pretraining

    Sep 1, 2026Francois MeyerLanguage Model PretrainingAutoregressive Language Modeling

    Department of Computer Science University of Cape Town

  28. Probabilistic Model Checking of Autoregressive Neural Sequence Models

    Sep 1, 2026Helge Spieker, Dennis Gross, Arnaud GotliebNeural Network VerificationAutoregressive Language Modeling

    Simula Research Laboratory, Oslo, Norway

  29. SpikeOPD: Stable On-Policy Distillation for Autoregressive Spiking Language Models

    Aug 28, 2026Enqiao Lu, Xingrui Yu, Yiwei Fu +7Spiking Language ModelsSpiking Neural Networks

    The Chinese University of Hong Kong, Shenzhen · Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University, Singapore +3

  30. Fast Weight Attention for Continual Learning

    Aug 27, 2026Yifan Zhang, Steve Ta, Jasper Zhang +8Recurrent Language ModelsMemory-Augmented Language Models

    Bytedance Seed · Princeton University · Hyperbolic Labs +1