Next-Token Prediction

Follow this topic Ask about this topic

Latest papers 36

5 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

All topics

Language Modeling is Monotone Compression
Oct 8, 2026Noam Mazor, Andrew Morgan, Rafael PassLLM Compression Next-Token Prediction
New York University. · Cornell Tech. · Cornell Tech, Technion and Tel-Aviv University.
Information-Theoretic Analysis of Next-Token Prediction under Markovian Data
Sep 28, 2026Masoud Kavian, Abdellatif Zaidi, Milad SefidgaranInformation-Theoretic Generalization Bounds Next-Token Prediction
Mathematical and Algorithmic Science Laboratory, Huawei Paris Research Center, 92100 Boulogne-Billancourt, France · Laboratoire d’Informatique Gaspard Monge, Université Gustave Eiffel, 77420 Champs-sur-Marne, France
Single-Layer MeMo as a Randomized Hamming-Kernel Classifier
Sep 28, 2026Alessandro StraziotaMemory-Augmented Language Models Next-Token Prediction
Department of Enterprise Engineering University of Rome “Tor Vergata” Rome, Italy
The Price of Token Boundaries: Compression Certificates and Prediction
Sep 26, 2026Yuhao Du, Shunian ChenToken Efficiency Subword Tokenization
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute
Next-token functional estimation
Sep 17, 2026Milind Nakul, Vidya Muthukumar, Ashwin PananjadyTime Series Reasoning Next-Token Prediction
Schools of Industrial and Systems Engineering and Electrical and Computer Engineering Georgia Institute of Technology
Almost Free State Prediction Separation
Sep 3, 2026John Langford, Nathan Godey, Giovanni Monea +5Language Model Pretraining Efficient Language Model Training
Microsoft · Cornell University
Learning Representations through Token Prediction: Geometry, Approximation, and Downstream Guarantees
Aug 30, 2026Shulei WangLanguage Model Pretraining Next-Token Prediction
Department of Statistics, University of Illinois at Urbana-Champaign, 605 E. Springfield Ave., Champaign, IL 61820
Language models suffer from a curse of ambiguity
Aug 15, 2026Nicolas Zucchet, Hyun Dong Lee, Scott LindermanLLM Training LLM Reliability
Stanford University
Decoupling Is Not Identification: Supervised Evidential Learning in Next-Token Prediction
Aug 14, 2026Ge WangEvidential Deep Learning Next-Token Prediction
Biomedical Imaging Center, Center for Biotechnology and Interdisciplinary Studies, Department of Biomedical Engineering, Rensselaer Polytechnic Institute, Troy, NY, USA
Maglev: Sliding Recurrent Memory
Aug 3, 2026Bo Liu, Qiang LiuMemory-Augmented Language Models Next-Token Prediction
The University of Texas at Austin
PTP: Previous-Token Prediction based LLM Inversion for Near-Exact Prompt Reconstruction
Jul 31, 2026Pirzada Suhail, Nagasai Saketh Naidu, Atanu R Sinha +1LLM Prompting Next-Token Prediction
IIT Bombay · Adobe Research
Dzongkha Next Word Prediction System
Jul 13, 2026Prerna Chhetri, Tenzin Yoezer, Phuntsho Wangmo +1Next-Token Prediction Language Modeling
Information Technology Department, College of Science and Technology, Royal University of Bhutan
The State-Prediction Separation Hypothesis
Jul 1, 2026Giovanni Monea, Nathan Godey, Kianté Brantley +1Language Model Pretraining Transformer
†Cornell University · ⋄Harvard University
How Token Influence Decays with Distance: A Green-Function View of Trained Language Models
Jun 28, 2026Matthias Brändel, Stephan Köhler, Oliver RheinbachTransformer Interpretability Autoregressive Language Modeling
Faculty of Mathematics and Computer Science Technische Universität Bergakademie Freiberg, Freiberg, Germany
From Tokens to States: LLMs as a Special Case of World Models and the Continuous Path Beyond
Jun 26, 2026Paul DuboisLLM World Models World Models
Structure Before Collapse: Transient semantic geometry in next-token prediction
Jun 25, 2026Yize Zhao, Isabel Papadimitriou, Christos ThrampoulidisAutoregressive Language Modeling Next-Token Prediction
Department of Electrical and Computer Engineering, The University of British Columbia · Department of Linguistics, The University of British Columbia
Test-Time Training with Next-Token Prediction
Jun 19, 2026Xuan Ouyang, Zefan Cai, Junjie HuLong-Context Language Modeling Next-Token Prediction
University of Wisconsin–Madison
CLP: Collocation-Length Prediction for Zero-Loss Adaptive Multi-Token Inference
Jun 9, 2026Xuezhen Xie, Zhiqiang ZhouMulti-Token Prediction Language Model Decoding
Next-Token Prediction Learns Generalisable Representations of Sleep Physiology
Jun 8, 2026Jonathan F. Carter, Lionel TarassenkoSleep Stage Classification Time Series Classification
Institute of Biomedical Engineering University of Oxford
NITP: Next Implicit Token Prediction for LLM Pre-training
May 24, 2026Xiangdong Zhang, Debing Zhang, Shaofeng Zhang +3Language Model Pretraining Representation Geometry
School of AI, Shanghai Jiao Tong University · Dots Studio, Xiaohongshu Inc. · University of Science and Technology of China +1
When Is Next-Token Prediction Useful? Marginalization, Ergodicity, Mixture Identifiability, Local Sufficiency, RAG, Tools, and Programming
May 22, 2026Francesco CorielliAutoregressive Language Modeling Next-Token Prediction
Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Framework
May 17, 2026Yumeng Yao, Jingzhi Dong, Haowen Gu +43D ViTs Self-Supervised Pre-Training
Nanjing University of Science and Technology · Hangzhou City University · East China Normal University +1
Conditional Attribute Estimation with Autoregressive Sequence Models
May 13, 2026Erica Stutz, Giacomo Marino, Daniella Meeker +2Counterfactual Reasoning in Language Models Controllable Text Generation
Department of Biomedical Informatics and Data Science Yale University New Haven, CT 06510 · Department of Biostatistics Yale University New Haven, CT 06510 · Department of Biomedical Informatics and Data Science, Department of Pediatrics Yale University New Haven, CT 06510
Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilities
May 11, 2026Daniel RanardLLM Evaluation Next-Token Prediction
Department of Physics, California Institute of Technology, Pasadena, CA 91125, USA
A Single-Layer Model Can Do Language Modeling
May 11, 2026Zanmin WangLanguage Model Pretraining Recurrent Neural Networks
A Geometric Perspective on Next-Token Prediction in Large Language Models: Three Emerging Phases
May 9, 2026Gianfranco Lombardo, Giuseppe Trimigno, Stefano CagnoniNeural Representation Geometry LLM Interpretability
Department of Engineering and Architecture University of Parma Parma, IT 43125
The Convergence Gap: Instruction-Tuned Language Models Stabilize Later in the Forward Pass
May 8, 2026Yifan ZhouLLM Interpretability Next-Token Prediction
University of California, Los Angeles
Perturbation is All You Need for Extrapolating Language Models
May 5, 2026Zetai Cen, Jin Zhu, Xinwei Shen +1OOD Generalization Autoregressive Language Modeling
School of Mathematics, University of Bristol · School of Mathematics, University of Birmingham · Department of Statistics, University of Washington +1
LLM-Codec: Neural Audio Codec Meets Language Model Objectives
Apr 20, 2026Ho-Lam Chung, Yiming Chen, Hung-yi LeeNeural Audio Codecs Multi-Token Prediction
Graduate Institute of Communication Engineering, National Taiwan University · ASUS Intelligent Cloud Services · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)
Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Models
Mar 31, 2026Christine Zhang, Dan Jurafsky, Chen ShaniLanguage Model Pretraining LLM Alignment
Stanford University

Oct 8, 2026J/K move · Enter open · S save

Language Modeling is Monotone Compression

Noam Mazor, Andrew Morgan, Rafael Pass

New York University. · Cornell Tech. · Cornell Tech, Technion and Tel-Aviv University.

LLM Compression Next-Token Prediction

Open paper AskPDF arXiv

A long-standing hypothesis in artificial intelligence and neuroscience posits that intelligence is closely related to compression: the ability to compress information efficiently intuitively reflects capacities associated with intelligence and learning. Indeed, recent experimental works verify this intuition by showing connections between the capabilities of large language models (LLMs) and their ability as compressors: for instance, Deletang et al. (ICLR'24) demonstrate that LLMs can be used as powerful compressors, and Huang et al. (COLM'24) show that the compression ability of LLMs is highly correlated with their performance on benchmarks for knowledge and reasoning. In this work, we initiate a theoretical study of this connection. Our main result is that LLMs (formally modeled as next-token predictors) are equivalent to monotone (a.k.a. order-preserving) compression algorithms---namely, compression algorithms where the encoding process preserves the ordering of the inputs---in the sense that the one can be constructed from the other while preserving the same error up to an additive gap of 2. We next show that the monotonicity is required for this equivalence to hold if and only if cryptographic (infinitely-often) one-way functions exist. As a direct corollary, we get a cryptographic result of independent interest: the notion of next-bit pseudoentropy (a computational analogue of entropy) of a distribution is equivalent to monotone incompressibility of the distribution. (Previously, it was only known (Haitner et al., ITCS'23) that incompressibility implies next-bit pseudoentropy.)