BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration

Chence Yang; Ningxi Cheng; Arash Akbari; Qitao Tan; Qingchan Zhu; Ci Zhang; Changdi Yang; Yanzhi Wang

cs.AIOct 2, 2026

BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration

Authors: Chence Yang, Ningxi Cheng, Arash Akbari, Qitao Tan, Qingchan Zhu, Ci Zhang, Changdi Yang, Yanzhi Wang,

Organizations: University of Georgia · Northeastern University · XPeng Motors · The University of Alabama

Abstract

Speculative decoding accelerates autoregressive generation by using a lightweight draft to propose multiple tokens for parallel verification. However, existing methods often require an additional draft model or weight representation, introducing non-negligible memory overhead on resource-constrained devices. Self-speculative approaches reduce this overhead, yet still face trade-offs between draft quality, target quality, and storage efficiency. We propose BitNest, a bit-nested speculative decoding framework that embeds a low-precision draft directly into the higher-precision target representation. Instead of deriving a draft from a predefined target, BitNest first constructs a strong low-precision base and then recovers the higher-precision target through residual refinement, enabling both models to share a single physical weight representation. BitNest further extends this progressive-precision design to the KV cache for long-context inference. Across multiple 7B--8B edge-friendly LLMs and diverse workloads, BitNest achieves an average speculative acceptance rate of 95.2% while closely preserving higher-precision model quality, and delivers 1.48--1.61x end-to-end speedup over FP16 autoregressive decoding. On the LLaMA models supported by all representative self-speculative baselines, BitNest also achieves consistently competitive or higher decoding speedup.

Figures & tables

Figure 1: End-to-end decoding speedup across six workloads on LLaMA-2-7B and LLaMA-3-8B. All results are normalized to the corresponding 16-bit autoregressive baseline (1.0

\times

Figure 2: Overview of BitNest. BitNest first applies a function-preserving rotation-based transformation to obtain an equivalent model representation, then quantizes the transformed weights into a $b$ -bit base and the remaining error into an $r$ -bit refinement. The two are organized using Dual-Plane Weight Storage . During speculative decoding, drafting reads only the base, while verification reads both planes to recover the $(b+r)$ -bit target. BitNest physically nests the low-precision draft within the higher-precision target without requiring a separate draft model.

Figure 3: Dual-plane weight storage in BitNest. Base and refinement codes are packed into two independent 4-bit streams, each occupying

N/2

bytes for

N

weights. The draft accesses only the base plane, while the target accesses both planes to recover the full 8-bit representation.

Figure 4: KV nesting during speculative decoding. The draft stage only reads the low-bit base view, while verification reads the full $b_{kv}+r_{kv}$ -bit view from the same nested cache. Accepted KV entries are retained while the rejected suffix is discarded.

Model	Method	Draft $\rightarrow$ Target	Wiki	GSM8K	Code	ShareGPT	LongDoc	PG19
LLaMA-2-7B	QSpec	W4A4 $\rightarrow$ W4A16	1.35 $\times$	1.43 $\times$	1.46 $\times$	1.46 $\times$	1.44 $\times$	1.40 $\times$
	QuantSpec	W4A16 $\rightarrow$ W16A16	1.17 $\times$	1.19 $\times$	1.23 $\times$	1.22 $\times$	1.17 $\times$	1.12 $\times$
	Draft & Verify	W16A16 $\rightarrow$ W16A16	0.96 $\times$	1.14 $\times$	1.16 $\times$	1.20 $\times$	1.16 $\times$	1.01 $\times$
	BitNest	W4A8 $\rightarrow$ W8A8	1.50 $\times$	1.59 $\times$	1.59 $\times$	1.61 $\times$	1.60 $\times$	1.57 $\times$
LLaMA-3-8B	QSpec	W4A4 $\rightarrow$ W4A16	1.38 $\times$	1.56 $\times$	1.55 $\times$	1.52 $\times$	1.51 $\times$	1.41 $\times$
	QuantSpec	W4A16 $\rightarrow$ W16A16	1.13 $\times$	1.21 $\times$	1.18 $\times$	1.30 $\times$	1.07 $\times$	1.13 $\times$

Table 1: End-to-end decoding speedup across six workloads. Speedup is measured over the corresponding 16-bit autoregressive baseline in the same inference engine. N/A denotes configurations not supported by the official implementations for the evaluated Qwen models.

Model	Method	WikiText-2 $\downarrow$	GSM8K $\uparrow$	Code $\downarrow$	ShareGPT $\downarrow$	LongDoc $\downarrow$	PG19 $\downarrow$
LLaMA-2-7B	FP16	5.4706	13.6	2.6069	2.2206	4.3336	9.6594
	W8A8	5.4762	13.4	2.6101	2.2229	4.3374	9.6686
	QSpec (W4A16)	5.650	12.5	2.642	2.389	4.640	10.052
	BitNest	5.5123	13.8	2.6123	2.2321	4.3540	9.7616
LLaMA-3-8B	FP16	6.1389	50.1	2.7131	2.6462	5.2592	11.8364
	W8A8	6.1617	50.0	2.7192	2.6551	5.2755	11.8722

Table 2: Model performance across six evaluation domains. GSM8K reports accuracy (

\uparrow

), while all other datasets report perplexity (

\downarrow

). QSpec uses a W4A16 target and is evaluated with its corresponding inference engine.

Figure 5: Long-context efficiency on LLaMA-2-7B-32K. (a) Decoding speedup over the corresponding 16-bit autoregressive baseline in the same inference engine. Reading only the KV4 base during drafting increasingly benefits BitNest as the context grows. (b) Peak GPU memory across context lengths. Weight quantization reduces the persistent model footprint, while the nested KV representation further limits memory growth at long context lengths.

Method	Model	1K	2K	4K
QSpec	LLaMA-2-7B	4.60 (13.42)	5.35 (14.09)	6.10 (14.84)
QSpec	LLaMA-3-8B	5.90 (15.40)	6.34 (15.73)	6.67 (16.40)
QuantSpec	LLaMA-2-7B	13.21 (13.32)	13.47 (13.82)	13.93 (14.57)
QuantSpec	LLaMA-3-8B	15.34 (15.33)	15.64 (15.46)	16.27 (15.71)
Draft & Verify	LLaMA-2-7B	14.21 (13.87)	15.47 (14.94)	18.33 (18.33)
Draft & Verify	LLaMA-3-8B	15.96 (15.96)	16.82 (16.82)	20.81 (20.81)

Table 3: Peak GPU memory (GiB) under different context lengths. Values in parentheses denote the corresponding 16-bit autoregressive baseline in the same inference engine.

Model (Device)	Method	Throughput	Speedup	Energy
		(tok/s)		(J/token)
LLaMA-2-7B-32K (NX16)	FP16 AR	6.0	1.00 $\times$	1.52
	W8A8 AR	7.8	$\sim$ 1.30 $\times$	1.15
	BitNest	9.0–9.3	$\sim$ 1.50 $\times$	0.95
Llama-3.2-3B (NX8)	FP16 AR	10.6	1.00 $\times$	0.768
	W8A8 AR	14.0	1.32 $\times$	0.54

Table 4: Edge-device decoding performance on NVIDIA Jetson Orin NX. Energy is averaged over the decode phase.

Configuration	GSM8K $\uparrow$	LongBench $\uparrow$
KV8 target AR	82.4	41.4
KV8 draft $\rightarrow$ KV8	82.2	41.2
KV4 draft $\rightarrow$ KV8	81.6	41.3
KV4 target AR	80.2	41.5
KV4 draft $\rightarrow$ KV4	78.4	41.4

Table 5: Why higher-precision representations are retained for the target. Low-precision representations provide effective drafts, while KV8 and W8 preserve the fidelity required for final generation.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Precision	Success Rate (%)
FP16	FP16	86
W8A8	W8A8	87
W4A8	W4A8	80
BitNest	W4A8 $\rightarrow$ W8A8	89

Appendix

Table 6: OpenVLA-7B performance on LIBERO Spatial. Success rate is measured using the same evaluation protocol for all precision configurations.

Method	Prefill (ms)	Decode (ms)	Action Total (ms)	Decode tok/s
FP16 AR	65.9	135.6	201.5	44.2
BitNest W8A8 AR	111.7	105.3	216.9	57.0 ( $1.29\times$ )
BitNest Spec. ( $\gamma=7$ )	111.7	100.8	212.7	59.5 ( $1.35\times$ )

Appendix

Table 7: Per-action latency breakdown on OpenVLA-7B. Numbers are median latencies over 100 frames on an NVIDIA A6000. Prefill includes the 288-token multimodal prefix and the first action token; decode generates the remaining six action tokens. Visual encoding and projection add 9.9 ms equally to all methods and are excluded.

Model	Method	$\gamma$	WikiText-2	GSM8K	Code	ShareGPT	LongDoc	PG19	Avg.
LLaMA-2-7B	QSpec	3	95.4	97.4	97.9	98.1	97.0	94.5	96.7
	QuantSpec	6	87.9	89.6	93.5	95.1	90.2	81.5	89.6
	Draft & Verify	Adaptive	62.6	82.7	82.8	84.3	83.0	72.2	77.9
	BitNest	4	92.5	95.9	96.9	97.8	96.7	94.2	95.7
LLaMA-3-8B	QSpec	3	92.0	97.0	97.1	97.5	94.5	90.8	94.8
	QuantSpec	6	77.5	88.9	87.2	94.6	80.9	77.7	84.5

Appendix

Table 8: Generation-time draft acceptance rate (%) across workloads on an NVIDIA RTX A6000 with 2K prefixes, batch size 1, and 256 generated tokens. Results use each method’s native speculation setting. Acceptance follows the native definition used by each method.

Context	$\gamma=2$	$\gamma=3$	$\gamma=4$	$\gamma=5$	$\gamma=6$
4K	57.2 (.913)	56.8 (.897)	59.3 (.856)	56.1 (.820)	55.6 (.793)
31K	33.0 (.901)	33.6 (.881)	34.3 (.875)	34.0 (.834)	33.0 (.797)

Appendix

Table 9: Effect of speculation length

\gamma

on LLaMA-2-7B-32K with KV4 drafting. Each entry reports decode throughput in tok/s, with the draft-slot acceptance rate in parentheses.

Model	Attention	BF16 KV	KV8	KV4
LLaMA-2-7B-32K	MHA	23.4	28.5	34.3
Qwen2.5-7B	GQA	50.7	51.1	51.4

Appendix

Table 10: Effect of draft KV precision at 32K context across different attention architectures. All entries report decode throughput in tok/s.

Model	Configuration	GSM8K $\uparrow$	Qasper $\uparrow$	HotpotQA $\uparrow$	MultiFieldQA $\uparrow$	SAMSum $\uparrow$	GovReport $\uparrow$	TREC $\uparrow$	Avg.	Accept $\uparrow$
LLaMA-2-7B-32K	FP16 AR	–	34.6	47.9	39.8	36.8	–	71.0	46.0	–
	W8A8+KV8 AR	–	34.1	47.9	40.3	36.8	20.8	69.5	45.7	–
	KV8 draft $\rightarrow$ KV8 target	–	33.9	47.9	40.3	37.1	20.7	69.5	45.7	0.970
	KV4 draft $\rightarrow$ KV8 target	–	33.7	47.9	40.3	36.9	20.7	69.5	45.7	0.946
	W8A8+KV4 AR	–	33.5	47.2	39.9	36.7	20.6	69.5	45.4	–
	KV4 draft $\rightarrow$ KV4 target	–	32.9	46.7	40.0	37.1	20.7	69.5	45.3	0.965

Appendix

Table 11: Detailed ablation on draft and target KV precision. The target weights are fixed to W8A8 and speculative decoding uses

\gamma=4

. GSM8K reports 8-shot accuracy on 500 examples. For LongBench, Qasper, HotpotQA, and MultiFieldQA report F1; SAMSum and GovReport report ROUGE-L; TREC reports accuracy.

Model	FP16	W8A8	W4A16	W4A8	BitNest target	W4A4
LLaMA-2-7B	5.471 / 13.6%	5.476 / 13.4%	5.599 / 11.9%	5.591 / 12.2%	5.512 / 13.8%	6.717 / 5.8%
LLaMA-3-8B	6.139/ 50.1%	6.162 / 50.0%	6.470 / 41.8%	6.476 / 41.4%	6.223 / 49.8%	8.660 / 26.6%
Qwen2-7B	7.134 / 79.4%	7.152 / 78.9%	7.238 / 75.5%	7.244 / 75.5%	7.176 / 77.6%	8.079 / 63.3%
Qwen2.5-7B	6.851 / 82.9%	6.872 / 83.2%	6.998 / 80.1%	7.017 / 79.3%	6.906 / 82.0%	8.085 / 66.7%

Appendix

Table 12: Detailed ablation on target weight precision. For language models, each entry reports WikiText-2 perplexity (

\downarrow

) / GSM8K 5-shot strict accuracy (

\uparrow

). W4A16 isolates the effect of 4-bit weights from activation quantization.

Model	Device	FP16 AR	W8A8 AR	BitNest
LLaMA-2-7B-32K	NX16	6.0	7.8	9.0–9.3
Qwen2.5-7B	NX16	OOM	7.37	8.37–8.50
Llama-3.2-3B	NX8	10.6	14.0	15.5–16.6
Qwen2.5-3B	NX8	11.0	14.1	15.2

Appendix

Table 13: Edge-device decoding performance across the evaluated models. Throughput is measured over the decode phase. OOM denotes configurations that exceed the available device memory.

Model	Method	Peak Memory (GiB)	Max. Context
LLaMA-2-7B-32K	FP16 AR	13.9 / 15.4	2K
	W8A8 AR	8.2 / 11.4	6K
	BitNest	7.7 / 10.8	10K
Qwen2.5-7B	FP16 AR	OOM	–
	W8A8 AR	9.0 / 13.7	–
	BitNest	8.9 / 13.5	–

Appendix

Table 14: Edge-device memory usage and context capacity on Orin NX16. Peak memory is measured at 2K context and reported as PyTorch peak allocation / system-wide peak memory in GiB.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 314 papers

SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration
Apr 14, 2026Zhuofan Wen, Yang FengSpeculative Decoding LLM Inference Acceleration
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
Jul 30, 2026Yuesong Liu, Yuan Zeng, Min Lyu +3Long-Context Language Model Inference Speculative Decoding
University of Science and Technology of China
ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference
Sep 16, 2026Amir Ziashahabi, Hossein Entezari Zarch, Lei Gao +2Long-Context Language Model Inference LLM Inference Scheduling
University of Southern California, Los Angeles, CA, USA

Apr 14, 2026J/K move · Enter open · S save

SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration

Zhuofan Wen, Yang Feng

Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Speculative decoding has emerged as a promising approach to accelerate autoregressive inference in large language models (LLMs). Self-draft methods, which leverage the base LLM itself for speculation, avoid the overhead of auxiliary draft models but face limitations: shallow layers often produce overconfident yet incorrect token predictions, and the presence of difficult tokens in a draft sequence forces redundant computation through deeper layers, undermining both draft acceptance and overall speedup. To address these issues, we propose a novel self-draft framework that suppresses spurious confidence via layer-wise temperature annealing in early-exit decision and adaptively bounds speculation length based on token-wise decoding difficulty. By reprocessing the hidden states of draft tokens in a unified parallel pass through deep layers, our method maintains exact output equivalence with the original model while maximizing computational efficiency. It requires no modifications to the base LLM parameters and achieves up to 2.33x wall-time speedup over standard autoregressive decoding across diverse long-form generation tasks and multiple model architectures.

Abstract

Speculative Decoding

Explore similar work

SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration

A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference