RECAP: Calibrating Low-Bit Reasoning Models on the History They Read

Chi Ruan; Yujie Chen; Zixiang Zhao; Jingcai Guo; Haotong Qin

cs.AIOct 11, 2026

RECAP: Calibrating Low-Bit Reasoning Models on the History They Read

Authors: Chi Ruan, Yujie Chen, Zixiang Zhao, Jingcai Guo, Haotong Qin

Organizations: SpinEntropy · ETH Zurich · PolyU

Abstract

Low-bit quantization reduces the cost of serving reasoning models but can substantially degrade long chains of thought. Each step reads cached keys and values spanning thousands of tokens, allowing quantization errors to accumulate along the trajectory. Standard calibration operates on short segments that omit the earlier context needed by later reasoning steps. Even when this context is retained, cache reconstruction error alone does not capture how quantization errors affect subsequent attention. We present RECAP, a post-training quantization method that calibrates reasoning models on the history they read. Dynamic-prefix calibration separates long historical context from short-block reconstruction. It regenerates prefix keys and values with the current quantizer and rotates the active block to supervise every position along the trajectory. Prefix read geometry further supervises historical cache errors according to their effects on attention outputs, weighting key and value errors asymmetrically. With both components applied only during calibration, RECAP outperforms the baseline across multiple reasoning models and quantization settings. Under W4A4KV4, RECAP improves accuracy over the baseline by 13.3 percentage points on MATH500 with DeepSeek-R1-Distill-Qwen-1.5B and 15.8 points on AIME-120 with Qwen3-4B. These results highlight the importance of preserving historical dependencies during calibration for low-bit reasoning.

Figures & tables

Figure 1: Attention error and reasoning performance of DeepSeek-R1-Distill-Qwen-1.5B. (a) Cumulative normalized attention error with a 12K-token INT4 KV prefix. (b,c) MATH500 accuracy and re-tokenized output-length distributions under W4A4KV4, with BF16 as the reference.

Figure 2: Quantization sensitivity and calibration coverage. (a) A rule that finds the important cached step in one problem misses the important step in another. (b) Short calibration windows omit historical dependencies, while reconstructing every position in a long sequence incurs a higher computational cost per update. (c) The current step reads its cached history with unequal weight, so the same-size error on different steps has unequal influence.

Figure 3: RECAP overview. (a) Full-precision attention magnitudes alongside quantization errors for the baseline, prefix-only calibration, and RECAP in one reader window; bars give W4A4KV4 AIME-120 accuracy from Table 1 on R1-1.5B, R1-7B, and Qwen3-4B. (b) The calibration workflow rebuilds historical keys and values with the current quantizer and combines block reconstruction with a read-geometry penalty. (c) Dynamic prefixes retain historical context as the active block rotates along the reasoning trajectory. (d) Read geometry weights key errors by direction and value errors by read strength, as in Equation 10 ; prefix and head indices are omitted.

		W4A4KV4						W4A8KV4
Model	Method	MATH	AIME	GSM	GPQA	LCB	Avg.	MATH	AIME	GSM	GPQA	LCB	Avg.
R1-1.5B	BF16	84.7	23.3	84.5	36.2	16.4	49.0	84.7	23.3	84.5	36.2	16.4	49.0
	QuaRot	1.4	0.0	0.9	12.0	0.0	2.9	0.6	0.0	0.7	22.2	0.0	4.7
	FlatQuant	66.1	8.6	76.8	32.0	7.5	38.2	67.1	9.6	80.4	32.8	7.8	39.5
	RECAP	79.4	18.3	79.9	32.8	12.1	44.5	82.1	22.1	80.9	34.3	11.9	46.3
R1-7B	BF16	93.9	46.1	91.2	51.0	36.7	63.8	93.9	46.1	91.2	51.0	36.7	63.8

Table 1: Comparison of quantization methods on three reasoning models under W4A4KV4 and W4A8KV4. Scores are percentages, with BF16 (gray) as the reference. MATH, AIME, GSM, GPQA, and LCB denote MATH500, AIME-120, GSM8K, GPQA-Diamond, and LiveCodeBench; Avg. is their unweighted mean. Bold marks the best quantized method per model and setting. R1-1.5B/7B: DeepSeek-R1-Distill-Qwen-1.5B/7B.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Max prefix	Total context	Fit samples	Epochs	Updates/layer
Historical ablation	8K	12K	128	15	480
Default	12K	16K	64	32	512

Appendix

Table 3: Calibration budgets. Both configurations use a 4K active block, optimizer batch 4, and microbatch 1. Each position is reconstructed five times at 8K prefix and eight times at 12K.

Score	AUC	AP	P@10%
Codeword shrinkage	0.495	0.109	0.096
Residual-direction cluster	0.504	0.108	0.111
Position-bin shrinkage	0.429	0.091	0.053
Future-read count	0.495	0.107	0.091
Local V error	0.504	0.103	0.067

Appendix

Table 4: Importance-score transfer, averaged over four held-out trajectories. AP is average precision; P@10% is precision among the selected 10%. Future-read count uses information unavailable during calibration and is computed from a separate trace; it serves as a reference.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 381 papers

Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories
Oct 21, 2026Boyuan Zheng, Jing Xiong, Chenwei Wu +8LLM Quantization Quantization-Aware Training
University of Michigan, Ann Arbor · University of California, Santa Barbara · Meta AI
Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models
Jun 24, 2026Xinyu Lian, Walid Krichene, Beichen Huang +4LLM Inference Efficiency LLM Quantization
University of Illinois Urbana-Champaign · Microsoft · Anyscale +1
Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery
Jun 1, 2026Ekaterina Alimaskina, Darya Rudas, Denis Shveykin +3LLM Quantization Repetition in Language Generation
BRAIn Lab, Moscow, Russia

Oct 21, 2026J/K move · Enter open · S save

Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories

Boyuan Zheng, Jing Xiong, Chenwei Wu, Hui Shen+7

University of Michigan, Ann Arbor · University of California, Santa Barbara · Meta AI

LLM Quantization Quantization-Aware Training Quantization-Aware Distillation Long-Horizon LLM Reasoning

Open paper AskPDF arXiv

Quantization-aware training (QAT) aims to recover reasoning capabilities in low-bit large language models (LLMs). Existing recipes typically supervise the quantized student with the original full-precision model on fixed response sequences. However, the resulting low-bit quantized models still struggle on long-horizon reasoning tasks and often fail to terminate within a token budget that suffices for the full-precision model. We identify two challenges in closing this gap. First, fixed-response supervision leaves a mismatch between training prefixes and those encountered during student generation, and this trajectory divergence grows as responses lengthen. Second, on its own prefixes, the student underpredicts some next-token candidates and overpredicts others, and this distributional misalignment becomes more frequent at high-entropy prefixes. We present OnQ, a low-bit recovery pipeline that combines on-policy distillation and candidate-wise Dynamic KL. Starting from block-wise calibration, the quantized student generates responses and receives full-precision teacher supervision on its own prefixes. Dynamic KL weights forward and reverse KL by each candidate's teacher-student probability ratio to balance correction of under- and overprediction. Across Qwen3-1.7B, 4B, and 8B at 2- and 3-bit precision, OnQ achieves the highest nine-benchmark average, with the largest gains on long-horizon reasoning benchmarks. At 2 bits, it outperforms every baseline on every benchmark at every model size, improving the Qwen3-8B average by 10.26 points over the strongest baseline. It also produces shorter responses with fewer truncations and repetitive errors.

Abstract

LLM Quantization

Explore similar work

Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories

Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models

Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery