BitMTP: When Multi-Token Prediction Meets Low-Bit Large Language Models

Ning Zhang; Shihao Wang; Jinrui Zhang; Chaodong Xiao; Lei Zhang

cs.CLOct 20, 2026

BitMTP: When Multi-Token Prediction Meets Low-Bit Large Language Models

Authors: Ning Zhang, Shihao Wang, Jinrui Zhang, Chaodong Xiao, Lei Zhang

Organizations: Department of Computing, The Hong Kong Polytechnic University

Abstract

For efficient deployment of large language models (LLMs), methods of ultra-low-bit backbones reduce weight storage, memory traffic, and linear-layer execution cost, while multi-token prediction (MTP) methods reduce sequential decoding steps by predicting multiple future tokens in parallel. Although complementary, the direct combination of these methods cannot accelerate decoding at comparable quality: masked drafting lacks intermediate token input supplied in next-token training, while wider computation and causal key-value (KV) updates can offset the savings from fewer steps. To address these challenges, we propose BitMTP, an algorithm-system co-design framework for efficient MTP on ultra-low-bit LLM backbones. A joint next-token prediction and MTP objective equips a single model with both autoregressive (AR) generation and block-causal drafting. Confidence-only decoding, speculative-only decoding, and hybrid decoding provide different quality-throughput operating points without using separate draft and verifier models. Accepted-prefix alignment keeps the committed prefix, materialized causal KV boundary, and CUDA Graph replay state consistent for continued generation. BitMTP uses this contract to select width-specialized low-bit execution and reuse captured graphs over shared KV storage. On NVIDIA H20, BitMTP with hybrid decoding achieves five-task mean speedups of 2.2x on BitNet b1.58-2B and 2.1-2.4x on 1.7B-4B Bonsai models at comparable task quality. On RTX 3090 and Jetson Orin, it further achieves up to 2.0x in five-task mean throughput and 3.3x on MBPP+, respectively. These results demonstrate that BitMTP accelerates ultra-low-bit LLM generation across backbones, model sizes, and GPU types. The source code is released at https://github.com/PolyU-VCLab/BitMTP

Figures & tables

Figure 1: Five-task mean generation quality (%) and decode throughput (tokens/s) for BitNet b1.58-2B-SFT AR, Naive BitNet $+$ MTP, and the proposed BitMTP on NVIDIA H20. The left panel shows the results by using BF16 reference execution, while the right panel uses low-bit W2A8 execution (2-bit weights and 8-bit activations). We see that the naive combination on BF16 improves throughput, but it loses both throughput and quality in the low-bit W2A8 setting. In contrast, our BitMTP substantially increases throughput while retaining task quality comparable to SFT AR.

Figure 2: The BitMTP framework. (a) Joint NTP $+$ MTP training supports mask-conditioned block-causal drafting and an AR path in one model. (b) Confidence-only decoding commits a confidence-selected prefix. Speculative-only decoding verifies each block, while hybrid decoding verifies when the confidence prefix is too short. Verifier-based steps add a correction or bonus token. (c) The runtime executes accepted-prefix transitions with width-specialized low-bit operators and captured CUDA Graphs. Graphs reuse causal KV history through shared storage. In hybrid decoding, verifier inputs are prepared before the host branch decision.

Figure 3: Runtime mechanisms M1–M3. M1 selects packed operators by replay width $M$ and input/output dimensions $(K,N)$ , while numerical role $r$ selects the output projection $\pi(r)$ . M2 indexes captured graphs by $(M,\sigma,r)$ and shares one KV buffer $B_{\sigma}$ across widths and roles within each capacity segment. M3 moves verifier-input preparation ahead of the hybrid branch decision.

Model	W/A	Type	GSM8K		MATH-500		IFEval		HumanEval+		MBPP+		Average
Model	W/A	Type	Sc.	Rate	Sc.	Rate	Sc.	Rate	Sc.	Rate	Sc.	Rate	Sc.	Rate
Compact, quantized, and speculative baselines
MiniCPM-2B	BF16/BF16	AR	47.84	273.83	11.00	301.18	43.65	306.92	40.24	290.12	43.65	252.08	37.28	284.82
Qwen2.5-1.5B	BF16/BF16	AR	57.54	441.38	49.00	470.88	51.08	467.00	37.20	439.50	51.32	395.61	49.23	442.88
Llama-3.2-3B	BF16/BF16	AR	66.79	274.71	45.80	303.84	78.90	303.35	44.51	284.95	51.06	256.08	57.41	284.59
Qwen3-1.7B	BF16/BF16	AR	68.16	452.38	67.40	454.19	75.30	457.90	37.20	456.47	43.12	464.42	58.23	457.07

Table 1: Generation quality (Sc., %) and decode throughput (Rate, tokens/s) on NVIDIA H20. Dark-red values in parentheses give speedups over AR baselines. Average is the unweighted five-task mean. W/A denotes weight/activation precision. Conf., Spec., and Hybr. denote confidence-only, speculative-only, and hybrid decoding, respectively.

Configuration	GSM8K		MATH-500		IFEval		HumanEval+		MBPP+		Average
Configuration	Sc.	Rate	Sc.	Rate	Sc.	Rate	Sc.	Rate	Sc.	Rate	Sc.	Rate
BitNet b1.58-2B-SFT	58.38	275.54	35.00	281.68	56.35	277.36	39.02	284.19	40.74	282.94	45.90	280.34
BitMTP ( $D{=}4$ )	62.77	510.62 (1.9 $\times$ )	35.00	492.56 (1.7 $\times$ )	54.08	392.57 (1.4 $\times$ )	35.37	572.04 (2.0 $\times$ )	33.60	553.23 (2.0 $\times$ )	44.16	504.20 (1.8 $\times$ )
BitMTP ( $D{=}8$ )	59.89	524.16 (1.9 $\times$ )	33.00	515.08 (1.8 $\times$ )	55.76	364.84 (1.3 $\times$ )	35.37	661.16 (2.3 $\times$ )	33.33	626.82 (2.2 $\times$ )	43.47	538.41 (1.9 $\times$ )
Bonsai-1.7B-SFT	61.18	244.51	42.00	245.31	68.94	244.81	35.37	245.08	36.51	244.64	48.80	244.87
BitMTP ( $D{=}4$ )	57.54	430.60 (1.8 $\times$ )	42.60	472.35 (1.9 $\times$ )	67.39	376.04 (1.5 $\times$ )	34.15	516.57 (2.1 $\times$ )	37.57	474.59 (1.9 $\times$ )	47.85	454.03 (1.9 $\times$ )

Table 2: Five-task quality (Sc., %) and decode throughput (Rate, tokens/s) of hybrid decoding on NVIDIA RTX 3090. Parentheses give speedups over SFT AR; Average is the unweighted task mean.

Backbone	SFT AR	BitMTP ( $D{=}4$ )			BitMTP ( $D{=}8$ )
Backbone	SFT AR	Conf.	Spec.	Hybr.	Conf.	Spec.	Hybr.
BitNet b1.58-2B	19.35 (1.0 $\times$ )	35.38 (1.8 $\times$ )	34.38 (1.8 $\times$ )	40.92 (2.1 $\times$ )	44.74 (2.3 $\times$ )	54.55 (2.8 $\times$ )	61.76 (3.2 $\times$ )
Bonsai-1.7B	15.95 (1.0 $\times$ )	30.80 (1.9 $\times$ )	28.83 (1.8 $\times$ )	36.20 (2.3 $\times$ )	40.42 (2.5 $\times$ )	42.47 (2.7 $\times$ )	52.05 (3.3 $\times$ )

Table 3: MBPP+ decode throughput (tokens/s) on Jetson Orin. Parentheses give speedups over SFT AR; decoding-policy abbreviations follow Table 1 .

Model	W/A	MMLU	Wino	Hella	ARC-C	ARC-E	BoolQ	OBQA	PIQA	Trivia	CSQA	TQA	Avg.
MiniCPM-2B	BF16/BF16	52.4	65.5	71.4	43.8	68.9	77.6	42.6	76.1	34.2	73.1	43.1	59.0
Qwen2.5-1.5B	BF16/BF16	60.2	63.4	68.3	46.6	75.9	78.1	40.4	75.9	39.0	76.5	46.7	61.0
Llama-3.2-3B	BF16/BF16	60.7	68.6	71.6	46.1	71.0	78.8	39.2	76.9	58.6	72.5	51.5	63.2
Qwen2.5-1.5B-AWQ	INT4/FP16	57.5	63.5	66.4	44.3	72.5	74.4	40.4	75.6	33.9	74.0	45.4	58.9
Llama3-8B-1.58 †	1.58/BF16	35.2	59.2	68.7	43.4	69.8	68.6	39.4	75.9	20.0	30.0	39.1	49.9
Falcon3-7B-1.58bit †	1.58/BF16	43.3	60.1	59.5	37.6	64.8	72.3	38.6	72.8	0.0	67.3	43.2	50.9

Table 4: Eleven-task model capability scores (%). Avg. denotes the unweighted mean; TQA is TruthfulQA-mc2. W/A denotes weight/activation precision. TriviaQA scores generated answers; the remaining tasks use likelihood scoring.

Runtime configuration	$D{=}4$ (tokens/s)						$D{=}8$ (tokens/s)
Runtime configuration	GSM8K	MATH-500	IFEval	HE+	MBPP+	Avg.	GSM8K	MATH-500	IFEval	HE+	MBPP+	Avg.
SFT AR	365.0	365.0	368.7	393.7	374.8	373.4 (1.0 $\times$ )	365.0	365.0	368.7	393.7	374.8	373.4 (1.0 $\times$ )
Generic	158.0	133.0	103.8	157.5	162.9	143.0 (0.4 $\times$ )	268.9	215.9	144.2	290.5	301.3	244.2 (0.7 $\times$ )
$+\,$ M1	724.2	658.2	515.1	739.2	758.0	678.9 (1.8 $\times$ )	759.1	663.7	429.6	793.7	840.6	697.4 (1.9 $\times$ )
$+\,$ M1 $+$ M2	735.2	664.8	529.1	746.0	772.5	689.5 (1.8 $\times$ )	784.5	684.3	452.4	825.3	879.6	725.2 (1.9 $\times$ )
$+\,$ M1 $+$ M2 $+$ M3 (Full)	761.4	707.1	580.4	768.7	793.0	722.1 (1.9 $\times$ )	819.4	777.0	543.6	923.6	964.2	805.6 (2.2 $\times$ )

Table 5: BitNet hybrid-decoding configurations at

D{\in}\{4,8\}

. Rates are decode tokens/s; Avg. is the unweighted task mean. Parentheses give speedups over SFT AR; HE+ denotes HumanEval+.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Examples	Domain
ScaleQuest-Math	1,003,467	Math
SmolTalk	1,038,618	General
OPC-SFT-Stage2	435,930	Code
Tulu-3-SFT-Mixture	923,278	General
SciRIFF	70,520	Science
Table-GPT	13,214	Table

Appendix

Table 6: Composition of the processed training mixture used in the training-cost study.

Configuration	Wall time (h)	GPU-hours	Relative cost
SFT	71.29	570.3	$1.00\times$
BitMTP ( $D{=}4$ )	103.10	824.8	$1.45\times$
BitMTP ( $D{=}8$ )	112.88	903.0	$1.58\times$

Appendix

Table 7: One-epoch training cost on Bonsai-1.7B using eight NVIDIA H20 GPUs. All configurations use the same corpus, global batch size, and optimizer-step count. Relative cost uses the one-epoch SFT run as its baseline.

Backbone	SFT AR Decode	Forced rejection, $D{=}4$			Forced rejection, $D{=}8$
Backbone	SFT AR Decode	Decode	Accepted	Com.	Decode	Accepted	Com.
BitNet b1.58-2B	373.42 (1.0 $\times$ )	142.91 (0.4 $\times$ )	0	0.9975	103.40 (0.3 $\times$ )	0	0.9974
Bonsai-1.7B	469.26 (1.0 $\times$ )	188.57 (0.4 $\times$ )	0	0.9978	160.28 (0.3 $\times$ )	0	0.9978

Appendix

Table 8: Forced-rejection stress test on H20. Decode gives five-task mean throughput (tokens/s); dark-red values in parentheses show speedups over SFT AR. Accepted and Com. report per-iteration means of accepted draft tokens and committed output tokens.

Configuration	Graphs	Emb./head	BF16 prefill	Packed decode	Model total	Shared KV	Static	Other allocated	Peak
BitNet b1.58-2B
SFT AR	1	2505.0	6563.3	881.6	9950.0	300.1	0.5	132.4	11028
BitMTP ( $D{=}4$ )	7	2818.1	6563.3	881.6	10263.1	300.6	13.7	326.8	11714
BitMTP ( $D{=}8$ )	7	2818.1	6563.3	881.6	10263.1	301.2	26.9	327.6	11742
Bonsai-1.7B
SFT AR	1	671.3	0.0	651.2	1322.5	448.1	0.6	0.0	5118

Appendix

Table 9: H20 model and runtime memory at a 4,096-token segment, in MiB. Model total includes Emb./head. Static contains per-graph token, position, and mask buffers. Other allocated is CUDA-allocated memory after graph capture minus Model total, Shared KV, and Static. Peak is the median of three runs’ maximum device-memory readings across the recorded stages.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,080 papers

EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction
Jun 25, 2026Carrie ChenMulti-Token Prediction LLM Inference Scheduling
Cornell University
Beneath the Tokens: A Performance Engineering Study of Multi-Token Prediction in GPU-Accelerated LLM Inference
Sep 28, 2026Suwesh Prasad SahLLM Inference Efficiency Multi-Token Prediction
Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context
Jul 23, 2026Alagappan ValliappanMulti-Token Prediction Long-Context Language Model Inference
NVIDIA

Jun 25, 2026J/K move · Enter open · S save

EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction

Carrie Chen

Cornell University

Multi-Token Prediction LLM Inference Scheduling Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Multi-token prediction has been shown to increase data density during training, improve downstream text-generation quality, and serves as the defacto approach for self-speculative decoding. Existing foundation and open source models that use MTP heads commit to a static tree-based attention topology throughout the entire generation sequence, meaning the speculation depth, and thus the compute required during verification, stays constant regardless of the context. This is fundamentally misaligned with the entropy patterns of natural language where low-entropy regions often support reliable multi-step drafting, while high-entropy regions require more conservative speculation. To address this, we propose Entropy-guided Multi-Token Prediction (EntMTP), a training-free scheduler that toggles between tree-based attention topologies from a set of task-specific pareto-optimal trees conditioned on a running estimate of local generation entropy. By matching speculation depth to context predictability, EntMTP maximizes expected accepted-token throughput across the full distribution of generated text without sacrificing generation quality. When evaluated across Humaneval, ShareGPT, GSM8k, and Litbench benchmarks, EntMTP consistently achieves a 1.15x speedup against Hydra and peak speedup of 1.36x against Medusa baselines respectively.

Abstract

LLM Inference Acceleration

Explore similar work

EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction

Beneath the Tokens: A Performance Engineering Study of Multi-Token Prediction in GPU-Accelerated LLM Inference

Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context