SteadyThought: Mitigating LLM Under-Thinking via Thought-Level Preference Optimization

Zhenyue Peng; Lin Yang; Jie Wang; Xiaoqi Ni; Hanzhu Chen; Xize Liang; Yanjun Wu; Rujing Wang

cs.AIOct 18, 2026

SteadyThought: Mitigating LLM Under-Thinking via Thought-Level Preference Optimization

Authors: Zhenyue Peng, Lin Yang, Jie Wang, Xiaoqi Ni, Hanzhu Chen, Xize Liang, Yanjun Wu, Rujing Wang,

Organizations: Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China · University of Science and Technology of China, Hefei 230026, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China ·

Abstract

Flexible switching between reasoning trajectories (i.e., thought switching) has significantly enhanced the reasoning capabilities of Large Reasoning Models (LRMs). However, existing models often switch excessively yet fail to sustain promising reasoning thoughts, a phenomenon termed "under-thinking". While recent efforts suppress switching to mitigate this, such over-correction may discard valuable trajectories. To address this challenge, we propose Steady Thought (ST), a novel thought-level preference optimization framework. ST formalizes under-thinking as a preference issue at switching points, forcing single-path continuations from identified points to yield high-quality reasoning. Then, ST performs thought-level preference optimization by treating the newly generated response as preferred and the original one as dis-preferred. Experiments across multiple models and datasets show that ST effectively reduces token consumption, achieving reductions of up to 63.8%, while maintaining strong reasoning accuracy. Further analysis suggests that ST helps models acquire a generalizable ability to reduce redundant switching across domains and languages. Code is available at https://github.com/MIRALab-USTC/SteadyThought.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Rank of the first correct thought position for DeepSeek-R1-Distill-Qwen-1.5B on MATH-500. (b) Rank of the first correct thought position for Qwen3-8B on AIME2024. (c) Overview of the ST framework, which identifies thought switching points using two complementary signals: explicit linguistic markers and concurrent entropy spikes around step boundaries. Starting from these identified junctions, the model is guided via logit control to generate persistent reasoning continuations, effectively verifying the potential of the current thought. These synthesized paths are then used to construct fine-grained preference pairs for the STPO objective, which reinforces trajectory commitment at the exact point of divergence to effectively mitigate under-thinking and improve reasoning efficiency.

Panel of Figure 2 — Figure 2: Comparison of switching behavior across four metrics: average response length and number of switches on all responses (first two columns) and on the common correct subset (last two columns). Response length is measured in words. The first row shows DeepSeek-R1-Distill-Qwen-1.5B, and the second row shows Qwen3-8B. Blue bars represent the base model, and orange bars represent ST.

Table 3

Training Method	MATH-500		AIME 2024
Training Method	Acc(%) $\uparrow$	Tokens $\downarrow$	Acc(%) $\uparrow$	Tokens $\downarrow$
DeepSeek-R1-Distill-Qwen-1.5B
Vanilla	82.0	4385	27.5	11273
+ SFT-standard	80.4	2650	22.9	7169
+ SFT-thought	78.0	2661	20.8	6650
+ DPO-standard	82.6	4273	30.8	10701

Table 5: Comparison of different training methods.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

alternatively	another way	another thought
another approach	another method	another solution
another technique	think differently	another strategy

Appendix

Table 6: Explicit linguistic markers

Method	MATH-500		AIME 2024		GSM8K		LiveCode
Method	Acc(%) $\uparrow$	Tokens $\downarrow$	Acc(%) $\uparrow$	Tokens $\downarrow$	Acc(%) $\uparrow$	Tokens $\downarrow$	Acc(%) $\uparrow$	Tokens $\downarrow$
Vanilla	82.8	3369	41.6	9638	88.1	1112	56.4	8398
Steady Thought	85.8	2653	44.2	9100	87.4	874	56.8	7447

Appendix

Table 8: Results on DeepSeek-R1-Distill-Llama-8B across different benchmarks.

Panel of Figure 3 — Figure 3: Comparison of switching behavior across four metrics: average response length and number of switches on all responses (first two columns) and on the common correct subset (last two columns). Response length is measured in words. The first row shows DeepSeek-R1-Distill-Qwen-14B, and the second row shows Qwen3-32B. Blue bars represent the base model, and orange bars represent ST.

Method	MATH-500		AIME 2024
Method	Acc(%) $\uparrow$	Tokens $\downarrow$	Acc(%) $\uparrow$	Tokens $\downarrow$
DeepSeek-R1-Distill-Qwen-1.5B	82.0	4385	27.5	11273
+ Steady Thought	83.6	2691	30.8	7577
w/o explicit linguistic markers	82.2	2609	29.2	8316
w/o boundary entropy spikes	83.4	3554	28.7	8199
Qwen3-8B	91.4	4724	62.1	10895

Appendix

Table 9: Ablation study on switching-point signals.

Model	Total Tokens Consumed	Average Tokens Consumed
DeepSeek-R1-Distill-Qwen-1.5B	49165922	3117
Qwen3-8B	104958704	6912
DeepSeek-R1-Distill-Qwen-14B	45182779	3374
Qwen3-32B	84345884	5981

Appendix

Table 11: Token consumption overview for trajectory synthesis stage

Model	Initial Samples	Number of Thoughts	Converted Pairs
DeepSeek-R1-Distill-Qwen-1.5B	1051	16177	2054
Qwen3-8B	1051	15185	1757
DeepSeek-R1-Distill-Qwen-14B	1051	13389	1907
Qwen3-32B	1051	15500	1877

Appendix

Table 12: Data utilization statistics in the Trajectory Synthesis stage.

Problem :

The polynomial

x^{3}-3x^{2}+4x-1

is a factor of

x^{9}+px^{6}+qx^{3}+r.

Enter the ordered triple

(p,q,r).

Reference Answer : (6,31,-1)

Before ST (14336 output tokens) :

Thought 1: Okay, so I need to find the ordered triple

(p,q,r)

such that the polynomial

x^{3}-3x^{2}+4x-1

is a factor of

x^{9}+px^{6}+qx^{3}+r.

… Let me check if this makes sense. Thought 2: Wait, let me verify this result by another method to be sure. Maybe by polynomial division or by considering that

x^{9}\dots

So perhaps my earlier approach is correct. Thought 3: But let me check with the values I found. If

p=6,\dots

Let me check if

x^{3}-3x^{2}+4x-1

divides this. Thought 4: Alternatively, let me try to perform the division …So there is a discrepancy in the coefficient of

p

. Thought 5: Let me check the first approach again. …Wait, but this seems conflicting with the previous result. Thought 6: Wait, actually, no. The original polynomial is

x^{9}\dots

Let me check my calculations again. Thought 7: First, let me verify the first approach where I substituted

\alpha

into the polynomial. …So the equations are correct. Thought 8: But then why does the second approach give a different equation for the

x^{2}

coefficient? Wait, maybe the second approach is not correct. …This seems correct. Thought 9: The second approach, however, might have an error in the expansion or reduction. Let me recheck …there is a mistake in one of the approaches. Thought 10: Wait, let me compute

\alpha^{9}+p\alpha^{6}+q\alpha^{3}+r

with

p=6

q=31

r=-1

…Hence, this approach might not be straightforward. Thought 11: Alternatively, if I substitute

y=x^{3}

, then

P(x)=y^{3}+py^{2}+qy+r

. But since

x^{3}=3x^{2}-4x+1

, then

y=3x^{2}-4x+1

. Therefore,

P(x)=(3x^{2}-4x+1)^{3}+\cdots

After ST (7413 output tokens) :

Thought 1: Okay, so I need to find the ordered triple

(p,q,r)

such that the polynomial

x^{3}-3x^{2}+4x-1

is a factor of

x^{9}+px^{6}+qx^{3}+r.

…This seems complicated, but maybe there’s a smarter way. Thought 2: Alternatively, since the given polynomial is a factor, then all roots …Let me check if indeed this is divisible by the cubic. Thought 3: Alternatively, perhaps there is a better way. …Let me try this approach. Thought 4: Let me assume that

Q(x)=x^{6}+ax^{3}+b.\dots

Final Answer The ordered triple is

\boxed{(6,31,-1)}

Appendix

Table 13: An example trajectory before and after ST.

Efficient Language Model Reasoning

Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.

See all 371 papers

Draft-Thinking: Learning Efficient Reasoning in Long Chain-of-Thought LLMs
Feb 28, 2026Jie Cao, Tianwei Lin, Juekai Lin +7Efficient Language Model Reasoning CoT Reasoning
Zhejiang University · Tencent
Efficient Reasoning with Balanced Thinking
Mar 12, 2026Yulin Li, Tengyao Tu, Li Ding +5Efficient Language Model Reasoning Large Reasoning Models
Harbin Institute of Technology (Shenzhen) · Zhongguancun Academy · Huawei Noah’s Ark Lab +3
CALM: Consistency-Aware Logit Masking in Decoding Space for Mitigating Overthinking of Large Reasoning Models
Oct 19, 2026Shengjing Liu, Feiyang Li, Yuxuan Yang +2Overthinking in Language Models Efficient Language Model Reasoning
College of Computer Science and Software Engineering, Shenzhen University

Feb 28, 2026J/K move · Enter open · S save

Draft-Thinking: Learning Efficient Reasoning in Long Chain-of-Thought LLMs

Jie Cao, Tianwei Lin, Juekai Lin, Zhenxuan Fan+6

Zhejiang University · Tencent

Efficient Language Model Reasoning CoT Reasoning Large Reasoning Models Overthinking in Language Models

Open paper AskPDF arXiv

Long chain-of-thought~(CoT) reasoning improves the problem-solving capability of large reasoning models~(LRMs), but often incurs substantial inference cost and overthinking. Existing efficiency methods primarily rely on prompting, post-hoc compression, or length-regularized optimization; these approaches can be sensitive to model scale, depend on external estimators, or under-allocate reasoning to difficult instances. We propose Draft-Thinking, a lightweight adaptation framework that learns a model-adapted draft CoT: a compact, self-contained working derivation that directly supports the final answer. We identify a Cross-Scale Draft Mismatch: concise traces generated by a strong teacher can be too sparse for a smaller target model. To address this mismatch, Progressive Draft Adaptation first transfers a concise reasoning-form prior through supervised fine-tuning (SFT) and then applies two-stage reinforcement learning (RL) with progressively harder data and relaxed response budgets to calibrate the reasoning granularity to the target model. The method requires neither post-hoc trajectory compression nor an explicit length-penalty reward. At inference time, draft, adaptive, and step-by-step prompts provide efficiency-oriented, balanced, and detailed reasoning modes, respectively. The step-by-step mode largely preserves the original model's accuracy while using fewer tokens. With only 847 training problems and 120 RL optimization steps, Draft-Thinking substantially reduces reasoning cost across six mathematical and scientific reasoning benchmarks. On MATH500, its draft mode reduces the average response length from 5,668 to 986 tokens, an 82.6% reduction, while achieving 90.6% accuracy compared with 93.0% for the original long CoT model.

Abstract

Efficient Language Model Reasoning

Explore similar work

Draft-Thinking: Learning Efficient Reasoning in Long Chain-of-Thought LLMs

Efficient Reasoning with Balanced Thinking

CALM: Consistency-Aware Logit Masking in Decoding Space for Mitigating Overthinking of Large Reasoning Models