Multi-Token Prediction

Also known as MTP

Multi-token prediction studies how language and other sequence models can learn or generate several future tokens in one step rather than predicting only the next token. Approaches include auxiliary prediction heads, latent-space and blockwise generation, adaptive prediction horizons, and speculative decoding, where a target model verifies proposed tokens to preserve output quality. The goal is to improve training signals, reasoning or planning, and inference throughput across settings such as text generation, speech, document parsing, and recommendation. A central tension is that deeper or more parallel predictions can accelerate generation but are harder to get right, so current work focuses on balancing acceptance, accuracy, training cost, and memory use.

Latest papers

42 papers11 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Multi-Token Prediction and Alternative Tokenization Improve Learning Latent Multi-Hop Graph Search at Scale
Oct 22, 2026Arvid Frydenlund, FuTe Wong, Katherine Lambert +1LLM Reasoning with Graphs Graph-Based Planning
University of Toronto, Vector Institute, Manulife Applied AI Research · University of Toronto, Vector Institute · University of Toronto +1
Adaptive Multi-Token Prediction: How far should we go?
Oct 20, 2026Emile Anand, Mohammed Rehan Parwani, Xinyuan CaoMulti-Token Prediction LLM Inference Acceleration
Georgia Institute of Technology
BitMTP: When Multi-Token Prediction Meets Low-Bit Large Language Models
Oct 20, 2026Ning Zhang, Shihao Wang, Jinrui Zhang +2LLM Inference Acceleration Multi-Token Prediction
Department of Computing, The Hong Kong Polytechnic University
ECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction
Oct 17, 2026Yuxuan Zhou, Junyuan Shang, Tianyu Wang +5Multi-Token Prediction Efficient Language Model Inference
ERNIE Team, Baidu · Independent Researcher
Next Token or Vector? Learning How Far to Decode
Oct 16, 2026Sai Sreenivas Chintha, Chidera Biringa, Rishab Sharma +6LLM Inference Acceleration Multi-Token Prediction
Microsoft · University of Massachusetts Amherst
HyCo: Hybrid Drafting for Cost-Efficient Mobile-Cloud Speculative Decoding
Oct 15, 2026Hanbin Bae, Heejin Choi, Joohyung Lee +3Speculative Decoding LLM Inference Acceleration
AI Solution Team, Samsung Research, Seoul, Republic of Korea
Personalized LLM Recommendation with Semantic IDs: Autoregressive Beam Search vs. Multi-Token Prediction
Oct 11, 2026Sejoon Oh, Fernando Amat Gil, Evan Smothers +11Large Language Model-Based Recommendation Semantic ID Recommendation
Netflix Personalization Foundations Team
LightMTP: Lightweight Latent Multi-Token Prediction
Oct 5, 2026Tamara Czinczoll, Julie Kallini, Gerard de Melo +1Language Model Pretraining Multi-Token Prediction
Hasso Plattner Institute / University of Potsdam · Stanford University · Tel-Aviv University
Match the Distribution, Not the Compute: Post-Training Multi-Token Prediction Heads
Oct 1, 2026Prachi Badarayani, Aidan Jay, Chenghui Zhou +8Multi-Token Prediction Speculative Decoding
Microsoft
IronLLM: Forging Compact Edge-Native Language Models for Real-Time Embodied Intelligence
Sep 29, 2026Changdi Yang, Fengquan Jiao, Haochih Lin +14Multi-Token Prediction On-Device Language Model Inference
Robotics Foundation Model Team, Xpeng Inc.
Beneath the Tokens: A Performance Engineering Study of Multi-Token Prediction in GPU-Accelerated LLM Inference
Sep 28, 2026Suwesh Prasad SahLLM Inference Efficiency Multi-Token Prediction
Line-Coupled Language Model
Sep 7, 2026Shiyuan Li, Shaorong Zhang, Zhaorui Yang +3Multi-Token Prediction Language Model Decoding
University of California, Riverside Riverside, CA, USA
Efficient Training with Foresight: Multi-Token Auxiliary Supervision for Autoregressive Image Generation
Aug 26, 2026Guo Niu, Xiongfei Yao, Teng Wang +1Autoregressive Image Generation Multi-Token Prediction
Foshan University Foshan, China · The University of Hong Kong Hong Kong, China · Sun Yat-sen University Guangzhou, China
Dynamic Multi-Byte Prediction With Hierarchical Language Models
Aug 16, 2026Abraham Toluwase Owodunni, Chibuzor Okocha, Christan Grant +2Language Modeling LLM Inference Acceleration
The Ohio State University · University of Florida · University of Washington
EchoRec: Multi-Item Prediction-Empowered Generative Recommendation via Cycle-Consistent Preference Alignment
Aug 14, 2026Haokai Ma, Aoqi Hu, Yueao Xing +5Recommender Systems Generative Recommendation
National University of Singapore Singapore, Singapore · Beijing University of Posts and Telecommunications Beijing, China · Tencent Beijing, China +1
Hierarchical Latent Prediction for Language Models
Aug 6, 2026Chang Shi, Tim Pearce, Manan Tomar +2Language Model Pretraining Multi-Token Prediction
University of Texas at Austin · Microsoft Research
LoopMTP: A looped transformer guided by latent multi-token prediction
Aug 4, 2026Behzad Shomali, Markus Frey, David Berghaus +2Multi-Token Prediction Recurrent Transformers
Lamarr Institute · University of Bonn · Fraunhofer IAIS
AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction
Aug 1, 2026Ziqiang Cui, Han Shi, Bowei He +8Multi-Token Prediction LLM Inference Acceleration
City University of Hong Kong · Huawei Technologies · Mohamed bin Zayed University of Artificial Intelligence +2
ParaASR: Multi-Token Prediction for Fast and Long-Context LLM-Based Speech Recognition
Jul 31, 2026Qingjian Lin, Yuxin Li, Haoyang Zhang +14Multi-Token Prediction Automatic Speech Recognition
StepFun · NTU · PKU +3
Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context
Jul 23, 2026Alagappan ValliappanMulti-Token Prediction Long-Context Language Model Inference
NVIDIA
EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction
Jun 25, 2026Carrie ChenMulti-Token Prediction LLM Inference Scheduling
Cornell University
P-MTP: Efficient Document Parsing via Multi-Token Prediction with Progressive Depth Scaling
Jun 23, 2026Le Xiang, Chenxi Zhai, Shu Wei +5Multi-Token Prediction Efficient VLM Inference
Department of Computer Vision Technology (VIS), Baidu Inc., China · Tsinghua University, Shenzhen International Graduate School, China
SuperThoughts: Reasoning Tokens in Superposition
Jun 11, 2026Zheyang Xiong, Shivam Garg, Max Yu +4Multi-Token Prediction LLM Inference Acceleration
mMicrosoft Research · iIndependent · pPrinceton University +2
Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling
Jun 10, 2026Yucheng Li, Huiqiang Jiang, Yang Xu +14Multi-Token Prediction Speculative Decoding
Qwen Team, Alibaba Inc.
CLP: Collocation-Length Prediction for Zero-Loss Adaptive Multi-Token Inference
Jun 9, 2026Xuezhen Xie, Zhiqiang ZhouMulti-Token Prediction Language Model Decoding
When Do Autoregressive Sequence Models Forecast Physical Wavefields? A Controlled Study on Synthetic Seismograms
Jun 9, 2026Waleed Esmail, Stuart Russell, Jana Klinge +2Multi-Token Prediction Seismology
Institut für Kernphysik, Universität Münster Wilhelm-Klemm-Straße 9, 48149 Münster, Germany · Institut für Geophysik, Universität Münster Corrensstraße 24, 48149 Münster, Germany · James Cook University, 1 James Cook Drive, Douglas, QLD 4814, Australia +1
FlashTTS: Fast Streaming TTS with MTP Acceleration and X-pred Mean Flow Distillation
Jun 8, 2026Hanke Xie, Xiaming Ren, Dake Guo +10Flow Matching Multi-Token Prediction
Huawei Technologies Co., Ltd
Joint Training of Multi-Token Prediction in Reinforcement Learning via Optimal Coefficient Calibration
May 27, 2026Zili Wang, Jiajun Chai, Lin Chen +3Multi-Token Prediction RL for Language Model Reasoning
School of Artificial Intelligence, University of Chinese Academy of Sciences, China · MAIS, Institute of Automation, Chinese Academy of Sciences, China · Meituan
Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs
May 26, 2026Wenhui Tan, Minghao Li, Xiaoqian Ma +5Multi-Token Prediction LLM Compression
Gaoling School of Artificial Intelligence, Renmin University of China · AI Platform, Xiaohongshu Inc. · University of Electronic Science and Technology of China +1
Multi-Token Residual Prediction
May 12, 2026Yufeng Xu, Zishuo Bao, Qian Wang +6Masked Diffusion Models Multi-Token Prediction
New York University · New York University Shanghai · Nous Research +1

Oct 22, 2026J/K move · Enter open · S save

Multi-Token Prediction and Alternative Tokenization Improve Learning Latent Multi-Hop Graph Search at Scale

Arvid Frydenlund, FuTe Wong, Katherine Lambert, Frank Rudzicz

University of Toronto, Vector Institute, Manulife Applied AI Research · University of Toronto, Vector Institute · University of Toronto+1

LLM Reasoning with Graphs Graph-Based Planning Multi-Token Prediction Tokenization in Language Models

Open paper AskPDF arXiv

Given an unseen graph, a start node,

s

, and an end node,

t

, can a language model generate the shortest path from

s

t

? Prior work has found that models fail to learn this task as the graphs scale. We show that this failure has two causes: the supervision signal and the tokenization. First, we theoretically establish and empirically demonstrate the effectiveness of multi-token prediction (MTP), and show that the benefits of MTP are topologically agnostic. Second, we introduce alternative tokenizations and show that the two causes are orthogonal. Using these methods, we break through the scaling wall observed by prior work, consistently and stably learning to solve the task at scales far beyond previous limits.