RATIO: Reasoning Analysis and Token-level Inference Optimization for Quantized Reasoning Models

Chengzhu Bao; Xianglong Yan; Tianao Zhang; Jiaqi Chen; Shaoqiu Zhang; Yulun Zhang

cs.LGSep 30, 2026

RATIO: Reasoning Analysis and Token-level Inference Optimization for Quantized Reasoning Models

Authors: Chengzhu Bao, Xianglong Yan, Tianao Zhang, Jiaqi Chen, Shaoqiu Zhang, Yulun Zhang

Organizations: Shanghai Jiao Tong University

Abstract

Post-training quantization (PTQ) has become a widely adopted technique for reducing the memory footprint and inference cost of large language models (LLMs). However, recent studies reveal that when applied to reasoning models, PTQ not only degrades reasoning performance but also exacerbates overthinking, leading to longer reasoning trajectories. These issues may offset the efficiency gains expected from lower-precision inference. Existing approaches mainly rely on complex optimization procedures. More recent lightweight inference strategies instead use predefined overthinking markers, limiting their adaptability across quantized models. To address these issues, we propose Reasoning Analysis and Token-level Inference Optimization (RATIO), a framework that identifies model-specific overthinking tokens and assigns each a tailored penalty. RATIO first introduces Quantization-aware Reasoning Behavior Analysis (QRBA) to identify overthinking tokens by analyzing discrepancies between full-precision and quantized models. It then adopts Token-Specific Penalty Determination (TSPD), which leverages full-precision guidance to derive token-specific penalties without additional training. Extensive experiments show that RATIO achieves a better accuracy-efficiency trade-off than existing token-level interventions. Specifically, RATIO achieves up to 9.8 points accuracy improvement and reduces chain-of-thought (CoT) length by up to 51.3% compared with quantized baselines. The code will be available at https://github.com/steven-bao1/RATIO.

Figures & tables

Figure 1: Performance comparison of different strategies across three quantized reasoning models. RATIO consistently achieves higher accuracy gains and larger CoT length reductions.

Panel of Figure 2 — Figure 2: Target tokens and calibrated penalties for Qwen-1.5B. (a) Word cloud of the 21 selected tokens, repeated for visualization. (b) Token-specific logit penalties in descending order.

Figure 3: Overview of RATIO. The two panels on the left illustrate QRBA, which discovers model-specific quantization-sensitive reasoning tokens by combining token-level discrepancies with reasoning-context information. The rightmost panel presents TSPD, which leverages full-precision guidance to derive token-specific penalties for inference-time calibration.

Figure 4: Numerical evidence construction in RTV. (A) Quantitative behavioral validation compares token probabilities under the same quantized prefixes. (B) Trajectory-level validation examines associations with incorrect answers and repetitive reasoning. Here, $Q$ and $F$ denote the quantized and full-precision models, respectively.

	AIME	MATH	GSM8K	GPQA	HumanEval	Avg.	$\Delta$ Acc	$\Delta$ Len%
	Acc (%) / Len (k)					Acc / Len	vs. baseline
BF16	20.83 / 25.11	85.60 / 6.06	84.69 / 2.81	36.36 / 9.97	73.17 / 6.50	60.13 / 10.09	–	–
GPTQ	4.17 / 51.53	54.00 / 24.27	68.76 / 9.54	31.82 / 23.67	10.37 / 39.31	33.82 / 29.66	–	–
+ Lotfi et al. (2026)	4.17 / 47.53	58.20 / 18.48	70.81 / 6.22	27.27 / 21.14	20.73 / 32.16	36.24 / 25.11	$+2.42$	$-15.34$
+RATIO	5.00 / 42.55	61.40 / 14.06	71.49 / 3.11	30.81 / 20.64	18.29 / 29.90	37.40 / 22.05	$+3.58$	$-25.66$
AWQ	7.50 / 55.53	44.80 / 36.13	61.18 / 22.16	23.23 / 36.56	17.68 / 39.81	30.88 / 38.04	–	–

Table 1: Accuracy (%) and CoT length (k tokens) for BF16 and W3-quantized models using GPTQ or AWQ, with shared-penalty decoding (+ Lotfi et al. (2026) ) or our method (+RATIO). The

\Delta

columns report the accuracy change in percentage points and the relative CoT length change (%) compared with the corresponding quantized model without intervention. For each model, the fixed-penalty baseline uses the penalty strength that achieves the highest mean accuracy across AWQ and GPTQ, averaged over the five benchmarks.

Panel of Figure 5 — Figure 5: Accuracy–efficiency trade-off and selected-token occurrence analysis. (a) Changes in average accuracy and CoT length for fixed-penalty decoding with different penalty strengths and RATIO on Qwen-7B, measured relative to the corresponding uncalibrated AWQ-W3 and GPTQ-W3 baselines. RATIO occupies the upper-left region (shorter CoT and higher accuracy) under both quantizers. (b) Benchmark-wise occurrence rates of QRBA-selected tokens on Qwen-1.5B with AWQ-W3 under RATIO, its component ablations, and the fixed-penalty baseline. RATIO consistently produces lower occurrence rates across benchmarks, showing that it effectively suppresses the selected target tokens during reasoning.

	GPQA	AIME	GSM8K	MATH	HumanEval	Avg.	$\Delta$ Acc	$\Delta$ Len%
	Acc (%) / Len (k)					Acc / Len	vs. baseline
AWQ	23.23 / 36.56	7.50 / 55.53	61.18 / 22.16	44.80 / 36.13	17.68 / 39.81	30.88 / 38.04	–	–
QRBA ( $\lambda=0.5$ )	29.80 / 32.37	4.17 / 48.73	69.29 / 9.17	59.40 / 20.21	34.76 / 29.86	39.48 / 28.07	$+8.60$	$-26.21$
QRBA ( $\lambda=1.0$ )	29.29 / 26.25	4.17 / 42.11	70.36 / 4.08	62.60 / 14.10	31.10 / 26.39	39.50 / 22.59	$+8.62$	$-40.62$
TSPD	20.71 / 31.74	7.50 / 47.74	66.19 / 10.20	59.00 / 20.21	33.54 / 28.25	37.39 / 27.63	$+6.51$	$-27.37$
RATIO	26.26 / 19.87	6.67 / 32.52	72.18 / 3.63	62.80 / 13.86	35.37 / 22.68	40.66 / 18.51	$+9.78$	$-51.34$

Table 2: Effect of QRBA and TSPD. Component-wise ablation on Qwen-1.5B under AWQ-W3 and GPTQ-W3. QRBA applies uniform penalties to the identified tokens, while TSPD applies token-specific penalties to the manually selected markers from prior work. Each entry reports accuracy (%) and CoT length (k tokens), with

\Delta

columns showing changes relative to the corresponding quantized baseline.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

		GPQA	AIME	GSM8K	MATH	HumanEval	Avg.	$\Delta$ Acc	$\Delta$ Len%
		Acc (%) / Len (k)					Acc / Len	vs. baseline
	FlatQuant	34.34 / 25.68	8.33 / 37.20	74.37 / 4.06	65.40 / 16.47	43.90 / 14.64	45.27 / 19.61	–	–
	+ Lotfi et al. (2026)	30.30 / 20.81	11.67 / 39.26	75.66 / 2.83	69.80 / 13.30	48.17 / 11.75	47.12 / 17.59	$+1.85$	$-10.30$
Qwen-1.5B	+RATIO	30.81 / 22.16	11.67 / 37.77	75.51 / 1.69	73.20 / 11.87	51.22 / 11.90	48.48 / 17.08	$\mathbf{+3.21}$	$\mathbf{-12.90}$

Appendix

Table 3: Performance of RATIO on Qwen-1.5B with FlatQuant W4A4KV4. Accuracy (%) and CoT length (k tokens) are reported as Acc/Len. The

\Delta

columns report changes relative to uncalibrated FlatQuant. For each metric, the highest accuracy and shortest CoT length are in bold.

Qwen-1.5B		Qwen-7B		Llama-8B		Qwen-14B
Token	$\bm{\lambda_{k}}$	Token	$\bm{\lambda_{k}}$	Token	$\bm{\lambda_{k}}$	Token	$\bm{\lambda_{k}}$
_but	1.22	_but	1.06	_back	1.00	_but	1.00
_what	1.21	_back	0.58	_try	0.90	_what	1.04
_try	0.73	_check	0.48	_well	0.86	_think	0.71
_again	1.16	_But	1.15	_check	0.93	_But	1.23
_well	0.88	_something	1.09	_But	1.12	_something	1.57

Appendix

Table 4: Model-specific tokens and normalized penalties used by RATIO. A leading underscore denotes a whitespace prefix in the decoded token.

Token	$\bm{\lambda_{k}}$	Token	$\bm{\lambda_{k}}$
_but	1.11	_question	1.11
_what	1.03	But	1.06
_back	0.83	_However	0.94
_think	0.81	but	1.00
_But	1.05	_perhaps	0.84
_something	1.00	Wait	1.02

Appendix

Table 5: Selected tokens and normalized penalties for Qwen3-4B in thinking mode. A leading underscore denotes a whitespace prefix in the decoded token.

Case 1: Repetitive reasoning after reaching the correct result
Prompt	From her science class study, Brady learned that each whale has 40 gallons of blood and that a shark has three times as much blood as a whale. How many gallons of blood do ten sharks have?
GPTQ-W3	Each shark has $3\times 40=120$ gallons, so ten sharks have $10\times 120=1200$ gallons. Wait, let me double-check that. $\ldots$ Wait, that seems correct. Let me think if I interpreted the question correctly. $\ldots$ Alternatively, perhaps I can think of the problem as ten sharks each having three times as much as a whale, so $10\times(3\times W)=3\times(10\times W)$ . Therefore, ten sharks have $3\times 10\times 40=1200$ gallons. Alternatively, perhaps I can think of it as $10\times(3\times W)=3\times(10\times W)$ . $\ldots$ Wait, 3 W. $\ldots$ 3 W3 W3 W.
RATIO	A shark has three times as much blood as a whale. Therefore, each shark has $3\times 40=120$ gallons of blood. Ten such sharks have $10\times 120=1200$ gallons of blood. The final answer is $\boxed{1200}$ .
Case 2: Repetition followed by an incorrect reasoning trajectory
Prompt	Compute $99^{2}+99+1$ in your head.
GPTQ-W3	First, $99^{2}=(100-1)^{2}=10{,}000-200+1=9{,}801$ . Adding 99 gives $9{,}900$ , and adding 1 gives $9{,}901$ . $\ldots$ Let me verify each step again. $99^{2}$ is 9,801, then $9{,}801+99=9{,}900$ , and adding 1 gives 9,901. $\ldots$ Alternatively, maybe I can factor it as $99\times(99+1)+1$ . That gives $99\times 100+1=9{,}901$ . Alternatively, if I think of $99^{2}+99+1$ as $(99+1)^{2}-1$ , that is $100^{2}-1=9{,}999$ . Wait, no, that is incorrect. $\ldots$ So, 9999. $\ldots$ 1099999999999999999999999999999999999999.

Appendix

Table 6: Dialogue examples on Qwen-1.5B with GPTQ-W3. Red marks repetitive or erroneous reasoning; green marks correct RATIO calculations and answers. Ellipses indicate omissions.

Explore similar work

Jun 24, 2026cs.AI

Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models

Quantization is widely used to reduce the inference cost of large language models, but its effect on reasoning models is not fully captured by final-answer accuracy or per-token latency. We show that low-bit post-training quantization can introduce a hidden test-time compute cost: quantized reasoning models often generate longer chains of thought even when they still answer correctly. Across mathematical reasoning, code generation, scientific question answering, and agentic tool-use benchmarks, we find that INT4/INT3 quantization can preserve accuracy but increase reasoning-token usage, offsetting the expected per-token speedup. To measure this effect, we introduce the CoT Token Inflation Ratio, which compares reasoning length between quantized and full-precision models averaged across all evaluation benchmarks. We further show that token inflation is accompanied by behavioral changes in the reasoning trace, including more intermediate steps and greater semantic repetition. These changes translate into measurable end-to-end real-world serving penalties. Finally, we evaluate mitigation strategies and find that prompting and decoding-time sampling offer inconsistent accuracy-length trade-offs, while quantization-aware training shows more promise in reducing both accuracy degradation and token inflation. Our results suggest that reasoning-token usage should be reported alongside accuracy when evaluating quantized reasoning models.

Xinyu Lian, Walid Krichene, Beichen Huang +4

University of Illinois Urbana-Champaign · Microsoft · Anyscale +1

Large Language Model Quantization Quantization-Aware Training Inference Cost Repetition

Ask PDF

May 29, 2026cs.LG

Quantized Reasoning Models Think They Need to Think Longer, but They Do Not

Post-training quantization (PTQ) is widely used to deploy large language models efficiently, but its effect on reasoning models is not well understood. Across math, coding, and science QA, we find that aggressive PTQ reduces accuracy while increasing chain-of-thought (CoT) length. Surprisingly, we show that in up to 52% of the quantized models' failures, models reach the right answer in intermediate reasoning steps but do not output it as a final answer. To understand why quantization leads to this increase in overthinking errors, we measure the token-level KL divergence between quantized and full-precision output distributions. Positions with high KL divergence correlate strongly with high next-token entropy, and at these positions quantized models disproportionately sample overthinking markers such as "wait", "but", and "alternatively". We show that simply introducing a training-free logit penalty on a curated set of overthinking markers can reduce CoT length by 12--23% while preserving or improving accuracy across 5 models (1.5B-32B parameters), 3 quantization methods, and 5 benchmarks, yielding a favorable Pareto frontier of accuracy against reasoning cost compared to penalizing other token sets. Overthinking errors produced by quantized models are particularly reduced by up to 58%.

Sanae Lotfi, Polina Kirichenko, Steven Li +1

1FAIR at Meta · 2Meta AI

Large Reasoning Models Chain-of-Thought Reasoning Kullback-Leibler Divergence

Ask PDF

Sep 28, 2026cs.AI

Efficient Reasoning via Constrained Optimization in Latent Space

Large Reasoning Models (LRMs) have shown remarkable reasoning capabilities, yet they still suffer from overthinking, generating redundant reasoning steps which incur substantial token consumption. Existing methods, such as suppressing reflective keywords or forcing shorter reasoning lengths, attempt to mitigate this issue but inevitably truncate necessary steps and induce underthinking, thereby compromising performance. To address this dilemma, we investigate the latent representations and observe that efficient reasoning steps naturally cluster into a concentrated region in latent space, while those deviating from this region tend to produce verbose sequences. To leverage this, we keep reasoning focused within this region via a quadratic program which projects deviating hidden states back into the region. Then we propose a novel training-free framework to achieve efficient reasoning that reduces token generation costs without sacrificing performance. Extensive experiments conducted on four models ranging from 1.5B to 14B, and across six benchmarks in math reasoning, coding, and scientific QA, validate the effectiveness of our method, up to a 12.1% improvement in accuracy while reducing generated tokens by 11.8% to 52.8%. Codes are available at https://github.com/hzn18/Opt4Reasoning.

Zhinan Hou, XingChen Li, Keyou You

Tsinghua University

Large Reasoning Models Reasoning Traces Latent Space Constrained Optimization

Ask PDF