Stable Adaptive Thinking via Advantage Shaping and Length-Aware Gradient Regulation

Zihang Xu; Haozhi Xie; Ziqi Miao; Wuxuan Gong; Chen Qian; Lijun Li

cs.LGFeb 26, 2026

Stable Adaptive Thinking via Advantage Shaping and Length-Aware Gradient Regulation

Authors: Zihang Xu, Haozhi Xie, Ziqi Miao, Wuxuan Gong, Chen Qian, Lijun Li

Organizations: Renmin University of China

Abstract

Large reasoning models (LRMs) achieve strong performance through extended reasoning traces, but they often exhibit overthinking behavior for low-complexity queries. Existing efforts to mitigate this issue are fundamentally limited by unstable accuracy-efficiency trade-offs and poor robustness to heterogeneous reasoning behaviors. To address these challenges, we propose a two-stage framework for stable adaptive thinking in LRMs. The framework first applies Hybrid Fine-Tuning to expose the model to both thinking and no-thinking behaviors, establishing well-conditioned initialization. It then performs adaptive reinforcement learning with Correctness-Preserving Advantage Shaping (CPAS) to avoid suppressing correct long-chain reasoning, and Length-Aware Gradient Regulation (LAGR) to stabilize optimization under severe reasoning-length heterogeneity. Extensive experiments on Qwen2.5-1.5B and 7B show consistent improvements over strong baselines, achieving up to +3.7/+3.6 accuracy points while reducing generated tokens by 40.6%/43.9%. Further analyses across varying problem difficulties and out-of-distribution tasks confirm the robustness and generalization of our approach.

Figures & tables

Figure 1: Illustration of efficient reasoning methods.

Figure 2: Overview of our two-stage training pipeline. Stage 1 performs Hybrid Fine-Tuning (HFT) on paired thinking and no-thinking formats to initialize a unified policy. Stage 2 applies GRPO-style reinforcement learning with correctness-preserving advantage shaping (CPAS) and length-aware gradient regulation (LAGR) to stabilize optimization under extreme length heterogeneity and to learn when to think.

Method		MATH-500			AIME-2024			AIME-2025			Average
Method		Acc	Len	$\text{Ratio}_{NT}$	Acc	Len	$\text{Ratio}_{NT}$	Acc	Len	$\text{Ratio}_{NT}$	$\Delta$ Acc	$\Delta$ Len
Qwen-2.5-1.5B
	Thinking	80.4	10441	0.0%	25.7	35829	0.0%	20.6	36977	0.0%	0.0	+0.0%
	No-Thinking	60.4	2320	100.0%	9.1	10751	100.0%	7.5	11150	100.0%	-16.6	-70.9%
	HFT	72.3	7106	42.4%	16.8	23673	54.2%	14.5	25376	46.2%	-7.4	-32.5%
	O1-Pruner	77.1	7043	0.0%	20.3	33417	0.0%	16.7	32185	0.0%	-4.2	-12.7%

Table 1: Overall performance on MATH-500, AIME-2024, and AIME-2025 for Qwen2.5 series models.

Len

is the average number of generated tokens per example.

Ratio_{NT}

denotes the fraction of examples solved in NoThinking mode.

\Delta Acc

and

\Delta Len

are computed relative to the always-Thinking baseline of the same base model.

Figure 3: Difficulty-aware mode selection and performance. (a) Performance of Two Think Mode on MATH-500, AIME-2024, and AIME-2025. (b) Mode ratio across MATH-500 difficulty levels. (c) Accuracy across difficulty levels, comparing our adaptive policy with always-Thinking and always-No-Thinking baselines.

Figure 4: Ablation and sensitivity analysis. (a) Training dynamics with/without CPAS: mean response length (left) and AIME-2024 accuracy (right). (b) Effect of the LAGR length-weight parameter

\beta

(left) and the control-token boost factor

\lambda

(right) on accuracy and the no-thinking ratio.

Method	GPQA
Method	Acc	Len	Ratio NT	$\Delta$ Len
Thinking	47.5	24135	0.0%	–
No-Thinking	35.3	8031	100.0%	-66.7%
HFT	41.8	16849	44.1%	-30.2%
Ours	50.4	11826	31.3%	-51.0%

Table 2: OOD Performance on GPQA Dataset.

Method		MATH-500			AIME-2024			AIME-2025
Method		Acc	Len	$\text{Ratio}_{NT}$	Acc	Len	$\text{Ratio}_{NT}$	Acc	Len	$\text{Ratio}_{NT}$
Qwen-2.5-1.5B
	Thinking	80.4	10441	0.0%	25.7	35829	0.0%	20.6	36977	0.0%
	No-Thinking	60.4	2320	100.0%	9.1	10751	100.0%	7.5	11150	100.0%
	HFT	72.3	7106	42.4%	16.8	23673	54.2%	14.5	25376	46.2%
	Ours	84.6	4176	69.8%	28.1	23258	3.7%	25.2	22062	5.4%

Table 3: Appendix Results on Diverse Model Sizes and Architectures.

Panel of Figure 5 — Figure 5: For an example from MATH-500, the Thinking baseline generates a long chain-of-thought with redundant intermediate steps. In contrast, our method chooses NoThinking and directly produces a concise final solution, using 713 tokens in total.

Panel of Figure 6 — Figure 6: For an example from AIME, the Thinking baseline generates a long chain-of-thought with redundant intermediate steps. In contrast, our method chooses NoThinking and directly produces a concise final solution, using 2977 tokens in total.

Panel of Figure 7 — Figure 7: For an example from GPQA, the Thinking baseline generates a long chain-of-thought with redundant intermediate steps. In contrast, our method chooses NoThinking and directly produces a concise final solution, using 2197 tokens in total.

Overthinking in Language Models

Overthinking in language models studies when extra reasoning becomes redundant or harmful, and how to control it without losing accuracy.

See all 68 papers

Efficiently Learning To Reason or Not to Reason: Root-token Policy Optimization for Adaptive Thinking
Oct 13, 2026Taehyeon Kim, Hyunsoo Lee, Youngsoo Jang +1Adaptive Reasoning in Language Models Efficient Language Model Reasoning
LG AI Research · UNIST · University of Illinois Chicago
Stop When Further Reasoning Won't Help: Attention-State Adaptive Generation in Reasoning Models
Jun 13, 2026Jiakai Li, Ke Qin, Rongzheng Wang +4Early Stopping for Language Model Reasoning Efficient Language Model Inference
University of Electronic Science and Technology of China, Chengdu, China · Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province, Chengdu, China
SmartThinker: Progressive Chain-of-Thought Length Calibration for Efficient Large Language Model Reasoning
Mar 9, 2026Chenzhi Hu, Qinzhe Hu, Yuhang Xu +6Efficient Language Model Reasoning Large Reasoning Models
Shanghai Jiao Tong University · Beihang University

Oct 13, 2026J/K move · Enter open · S save

Efficiently Learning To Reason or Not to Reason: Root-token Policy Optimization for Adaptive Thinking

Taehyeon Kim, Hyunsoo Lee, Youngsoo Jang, Moontae Lee

LG AI Research · UNIST · University of Illinois Chicago

Adaptive Reasoning in Language Models Efficient Language Model Reasoning Large Reasoning Models Overthinking in Language Models RL for Language Model Reasoning

Open paper AskPDF arXiv

Large reasoning models (LRMs) achieve strong performance by externalizing explicit reasoning traces before producing the answer, yet suffer from overthinking challenge that allocates uniformly heavy computation to queries of varying difficulty. While proprietary models mitigate this via opaque routing, open-source LRMs still lack an efficient mechanism to internalize adaptive reasoning due to both expensive training cost and limited disclosure of training recipes. In response, we introduce RPO (Root-token Policy Optimization), a framework that enables LRMs to self-determine when to reason by training only the initial root token (e.g., whether to invoke the <think> tag) via group relative reward and group-wise advantages. By focusing on this pivotal branching point, RPO drastically reduces training overhead and VRAM usage. Across multiple model families and scales, RPO learns difficulty-aware adaptive thinking at just ~2% of the training compute of prior adaptive-reasoning methods.

Abstract

Overthinking in Language Models

Explore similar work

Efficiently Learning To Reason or Not to Reason: Root-token Policy Optimization for Adaptive Thinking

Stop When Further Reasoning Won't Help: Attention-State Adaptive Generation in Reasoning Models

SmartThinker: Progressive Chain-of-Thought Length Calibration for Efficient Large Language Model Reasoning