cs.CLNov 19, 2026

Efficient and Robust Drafting with Multi-Token Prediction

Authors: Filippo Galgani, Ziwei Ji, Omri Homburger, Ganesh Jawahar, Michelle Ramanovich, Ivan Korotkov, Luciano Martins, Utku Evci, 

Organizations: Google

Abstract

Speculative decoding has emerged as a critical technique for accelerating Large Language Model (LLM) inference by decoupling generation into a drafting phase and a parallel verification phase. A crucial challenge is therefore designing a drafter that is both lightweight and accurate enough to approximate the predictions of the target model. We present a highly optimized Multi-Token Prediction (MTP) architecture that extends existing backbone LLMs to efficiently produce high-quality drafts. Built around a single shared drafting head, our approach features three core architectural optimizations: target activation injection to anchor semantic planning, zero-copy KV-cache cross-attention to bypass prefill compute and reduce memory overhead, and an efficient clustered vocabulary projection to reduce the cost of full-vocabulary decoding. Testing these improvements with the Gemma 4 model family, our G-MTP shows out-of-the-box speedups for both large server-side setups (up to 3.1×\times) and smaller models targeting memory-constrained on-device environments (up to 2.3×\times on H100 and up to 2.2×\times on a smartphone GPU). In addition, we present parameter-efficient fine-tuning recipes for G-MTP that keep the drafter aligned with the backbone LLM under domain shift, requiring up to 1.83×\times fewer target forward passes than tuning only the backbone.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Multi-Drafter Speculative Decoding with Alignment Feedback

    Apr 7, 2026Taehyeon Kim, Hojung Jung, Se-Young YunSpeculative DecodingLLM Inference Acceleration

  2. Adaptive Multi-Token Prediction: How far should we go?

    Oct 20, 2026Emile Anand, Mohammed Rehan Parwani, Xinyuan CaoMulti-Token PredictionLLM Inference Acceleration