Next-Token Prediction

Ask about this topic

Overview

Next-token prediction asks a model to estimate the next token given the preceding context, providing a self-supervised way to learn language and other sequential data. Most systems use autoregressive Transformers trained with cross-entropy, while alternatives add recurrent or fast-weight memory, multi-token forecasts, semantic targets, or sequence-level reinforcement learning. Researchers evaluate models with perplexity and held-out prediction, as well as long-context retrieval and question answering, reasoning benchmarks, and downstream tasks such as classification. A central tension is that accurate token predictions do not always yield semantic, robust, or well-calibrated representations, and longer contexts or richer objectives can bring gains while increasing training cost or complicating evaluation.

Latest papers

CardsList

41 papers4 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

  1. Theoretical Analysis of Muon on Transformers for Next-Token Prediction

    Nov 3, 2026Shengwei Wu, Yixuan Yang, Gao Huang +1Transformer Training DynamicsMuon Optimizer

    School of Mathematical Sciences, Zhejiang University, Hangzhou, Zhejiang, China · Department of Mathematics, National University of Singapore, Singapore

  2. EventGPT: Next-Event Pretraining on Banking Customer Histories

    Nov 2, 2026Denis BurakovNext-Event PredictionFinancial Services

  3. HieraNTP: Structured Risk Representations for Time-Series Anomaly Detection via Hierarchical Next-Token Prediction

    Oct 31, 2026Yujie Li, Zezhi Shao, Boyu Diao +4Time-Series Anomaly DetectionAnomaly Detection

    State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

  4. Specialization, Diversity, and Scaling in Child-Scale Language Models

    Oct 27, 2026Steven Y. Feng, Michael C. FrankSmall Language ModelsLLM Training

    Stanford University

  5. Predictive Latents for Autoregressive Language Models: Possibility, Gauge, and Obstruction

    Oct 20, 2026Yuzhou HongAutoregressive Language ModelingLatent Dynamics Modeling

    Zhejiang Sci-Tech University

  6. Language Modeling is Monotone Compression

    Oct 8, 2026Noam Mazor, Andrew Morgan, Rafael PassLLM CompressionNext-Token Prediction

    New York University. · Cornell Tech. · Cornell Tech, Technion and Tel-Aviv University.

  7. Information-Theoretic Analysis of Next-Token Prediction under Markovian Data

    Sep 28, 2026Masoud Kavian, Abdellatif Zaidi, Milad SefidgaranInformation-Theoretic Generalization BoundsNext-Token Prediction

    Mathematical and Algorithmic Science Laboratory, Huawei Paris Research Center, 92100 Boulogne-Billancourt, France · Laboratoire d’Informatique Gaspard Monge, Université Gustave Eiffel, 77420 Champs-sur-Marne, France

  8. Single-Layer MeMo as a Randomized Hamming-Kernel Classifier

    Sep 28, 2026Alessandro StraziotaMemory-Augmented Language ModelsNext-Token Prediction

    Department of Enterprise Engineering University of Rome “Tor Vergata” Rome, Italy

  9. The Price of Token Boundaries: Compression Certificates and Prediction

    Sep 26, 2026Yuhao Du, Shunian ChenToken CompressionToken Efficiency

    The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute

  10. Next-token functional estimation

    Sep 17, 2026Milind Nakul, Vidya Muthukumar, Ashwin PananjadyTime Series ReasoningNext-Token Prediction

    Schools of Industrial and Systems Engineering and Electrical and Computer Engineering Georgia Institute of Technology

  11. Almost Free State Prediction Separation

    Sep 3, 2026John Langford, Nathan Godey, Giovanni Monea +5Language Model PretrainingEfficient Language Model Training

    Microsoft · Cornell University

  12. Learning Representations through Token Prediction: Geometry, Approximation, and Downstream Guarantees

    Aug 30, 2026Shulei WangLanguage Model PretrainingNext-Token Prediction

    Department of Statistics, University of Illinois at Urbana-Champaign, 605 E. Springfield Ave., Champaign, IL 61820

  13. Language models suffer from a curse of ambiguity

    Aug 15, 2026Nicolas Zucchet, Hyun Dong Lee, Scott LindermanLLM TrainingLLM Reliability

    Stanford University

  14. Decoupling Is Not Identification: Supervised Evidential Learning in Next-Token Prediction

    Aug 14, 2026Ge WangEvidential Deep LearningNext-Token Prediction

    Biomedical Imaging Center, Center for Biotechnology and Interdisciplinary Studies, Department of Biomedical Engineering, Rensselaer Polytechnic Institute, Troy, NY, USA

  15. Maglev: Sliding Recurrent Memory

    Aug 3, 2026Bo Liu, Qiang LiuMemory-Augmented Language ModelsNext-Token Prediction

    The University of Texas at Austin

  16. PTP: Previous-Token Prediction based LLM Inversion for Near-Exact Prompt Reconstruction

    Jul 31, 2026Pirzada Suhail, Nagasai Saketh Naidu, Atanu R Sinha +1LLM PromptingNext-Token Prediction

    IIT Bombay · Adobe Research

  17. Dzongkha Next Word Prediction System

    Jul 13, 2026Prerna Chhetri, Tenzin Yoezer, Phuntsho Wangmo +1Gated Recurrent UnitNext-Token Prediction

    Information Technology Department, College of Science and Technology, Royal University of Bhutan

  18. The State-Prediction Separation Hypothesis

    Jul 1, 2026Giovanni Monea, Nathan Godey, Kianté Brantley +1Language Model PretrainingTransformer

    †Cornell University · ⋄Harvard University

  19. How Token Influence Decays with Distance: A Green-Function View of Trained Language Models

    Jun 28, 2026Matthias Brändel, Stephan Köhler, Oliver RheinbachTransformer InterpretabilityAutoregressive Language Modeling

    Faculty of Mathematics and Computer Science Technische Universität Bergakademie Freiberg, Freiberg, Germany

  20. From Tokens to States: LLMs as a Special Case of World Models and the Continuous Path Beyond

    Jun 26, 2026Paul DuboisLLM World ModelsWorld Models

  21. Structure Before Collapse: Transient semantic geometry in next-token prediction

    Jun 25, 2026Yize Zhao, Isabel Papadimitriou, Christos ThrampoulidisAutoregressive Language ModelingNext-Token Prediction

    Department of Electrical and Computer Engineering, The University of British Columbia · Department of Linguistics, The University of British Columbia

  22. Test-Time Training with Next-Token Prediction

    Jun 19, 2026Xuan Ouyang, Zefan Cai, Junjie HuLong-Context Language ModelingNext-Token Prediction

    University of Wisconsin–Madison

  23. CLP: Collocation-Length Prediction for Zero-Loss Adaptive Multi-Token Inference

    Jun 9, 2026Xuezhen Xie, Zhiqiang ZhouMulti-Token PredictionLanguage Model Decoding

  24. Next-Token Prediction Learns Generalisable Representations of Sleep Physiology

    Jun 8, 2026Jonathan F. Carter, Lionel TarassenkoSleep Stage ClassificationTime Series Classification

    Institute of Biomedical Engineering University of Oxford

  25. NITP: Next Implicit Token Prediction for LLM Pre-training

    May 24, 2026Xiangdong Zhang, Debing Zhang, Shaofeng Zhang +3Language Model PretrainingRepresentation Geometry

    School of AI, Shanghai Jiao Tong University · Dots Studio, Xiaohongshu Inc. · University of Science and Technology of China +1

  26. When Is Next-Token Prediction Useful? Marginalization, Ergodicity, Mixture Identifiability, Local Sufficiency, RAG, Tools, and Programming

    May 22, 2026Francesco CorielliAutoregressive Language ModelingNext-Token Prediction

  27. Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Framework

    May 17, 2026Yumeng Yao, Jingzhi Dong, Haowen Gu +4Point Cloud Learning3D ViTs

    Nanjing University of Science and Technology · Hangzhou City University · East China Normal University +1

  28. Conditional Attribute Estimation with Autoregressive Sequence Models

    May 13, 2026Erica Stutz, Giacomo Marino, Daniella Meeker +2Next-Token PredictionControllable Text Generation

    Department of Biomedical Informatics and Data Science Yale University New Haven, CT 06510 · Department of Biostatistics Yale University New Haven, CT 06510 · Department of Biomedical Informatics and Data Science, Department of Pediatrics Yale University New Haven, CT 06510

  29. Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilities

    May 11, 2026Daniel RanardLLM EvaluationNext-Token Prediction

    Department of Physics, California Institute of Technology, Pasadena, CA 91125, USA

  30. A Single-Layer Model Can Do Language Modeling

    May 11, 2026Zanmin WangLanguage Model PretrainingRecurrent Language Models