SAT: Balancing Reasoning Accuracy and Efficiency with Stepwise Adaptive Thinking

Weiyang Huang; Xuefeng Bai; Kehai Chen; Xinyang Chen; Yibin Chen; Weili Guan; Min Zhang

cs.AIApr 9, 2026

SAT: Balancing Reasoning Accuracy and Efficiency with Stepwise Adaptive Thinking

Authors: Weiyang Huang, Xuefeng Bai, Kehai Chen, Xinyang Chen, Yibin Chen, Weili Guan, Min Zhang

Organizations: Harbin Institute of Technology (Shenzhen), Shenzhen, China · Huawei Technologies

Abstract

Large Reasoning Models (LRMs) have revolutionized complex problem-solving, yet they exhibit a pervasive "overthinking", generating unnecessarily long reasoning chains. While current solutions improve token efficiency, they often sacrifice fine-grained control or risk disrupting the logical integrity of the reasoning process. To address this, we introduce Stepwise Adaptive Thinking (SAT), a framework that performs step-level, difficulty-aware pruning while preserving the core reasoning structure. SAT formulates reasoning as a Finite-State Machine (FSM) with distinct thinking modes (Slow, Normal, Fast, Skip). It navigates these states dynamically using a lightweight Process Reward Model (PRM), compressing easy steps while preserving depth for hard ones. Experiments across 9 LRMs and 7 benchmarks show that SAT achieves up to 40% reduction in reasoning tokens while generally maintaining or improving accuracy.

Figures & tables

Figure 1: Baselines vs. SAT. COT spends redundant checks for easy steps, Early-Stopping halts after a high-confidence first-pass answer and fails, while SAT skips redundancy on easy steps but preserves verification on hard steps, achieving the correct answer efficiently.

Figure 2: Overview of the proposed framework that models LRM reasoning as a Finite-State Machine (FSM). The left panel provides a detailed view of the specific state transition dynamics and stability rules of the proposed reasoning FSM. The right panel illustrates the closed-loop control flow: the Pilot perceives the difficulty ( $r_{t}$ ) of the previous step ( $y_{t-1}$ ), driving the State Transition Logic to update the thinking state ( $s_{t}$ ). This state is then mapped to a control tag ( $c_{t}$ ) via In-Context Steering to modulate the next generation step ( $y_{t}$ ).

Figure 3: Lightweight Pilot: A GRU-based framework fusing semantic embeddings and uncertainty features to capture stepwise difficulty trajectories.

Model	Method	GSM8K		MATH500		AIME 2024		AIME 2025		AMC
Model	Method	Acc ( $\uparrow$ , %)	Tokens ( $\downarrow$ )	Acc ( $\uparrow$ , %)	Tokens ( $\downarrow$ )	Acc ( $\uparrow$ , %)	Tokens ( $\downarrow$ )	Acc ( $\uparrow$ , %)	Tokens ( $\downarrow$ )	Acc ( $\uparrow$ , %)	Tokens ( $\downarrow$ )
	COT	95.1	2136	95.0	4823	60.0	12662	50.0	12720	92.5	7408
	DEER	94.5 ( -0.6 )	1250 ( -41% )	92.6 ( -2.4 )	3214 ( -33% )	63.3 ( +3.3 )	9327 ( -26% )	55.0 ( +5.0 )	12084 ( -5% )	87.5 ( -5.0 )	4906 ( -34% )
	CGRS	–	–	91.3 ( -3.7 )	2704 ( -44% )	56.7 ( -3.3 )	7893 ( -37% )	–	–	86.7 ( -5.8 )	4351 ( -41% )
Qwen3-4B	SAT	95.1 ( +0.0 )	845 ( -60% )	95.6 ( +0.6 )	2833 ( -41% )	60.0 ( +0.0 )	9462 ( -25% )	53.3 ( +3.3 )	9907 ( -22% )	92.5 ( +0.0 )	5255 ( -29% )
	COT	95.8	2152	95.6	5166	66.7	12393	60.0	12835	90.0	7920

Table 1: Performance on mathematical reasoning benchmarks. Metrics include Acc (

\uparrow

) and Tokens (

\downarrow

). Changes relative to the COT are highlighted in orange for Acc and blue for Tokens. Best results within each group are bolded.

Figure 4: Accuracy and token usage on GPQA and HumanEval across different model scales.

Figure 5: Ablation study results. Left: training and test loss curves for GRU-based Pilot under different inputs. Right: test-set correlation (Pearson/Spearman) between predicted step scores and supervision targets.

Figure 6: Strategy ablation results on MATH500 (Qwen3-14B). Accuracy and token usage of COT , w/o FAST , w/o SLOW , and SAT.

Figure 7: Attribution analysis of token savings on MATH500 (Qwen3-14B). Left: decomposition of SAT’s total token savings into reflection-related and branching-related components. Right: Pearson correlation between per-sample total token savings and each component.

Figure 8: Length-limit failures on aggregated math benchmarks. Number of incorrect predictions caused by exceeding maximum length, aggregated over GSM8K, MATH500, AIME 2024, AIME 2025, and AMC.

Figure 9: Performance analysis on MATH500 (Qwen3-14B). Left: Absolute accuracy comparison. Right: Normalized computational costs (Tokens/s, Time, VRAM) relative to the COT baseline (

1.0\times

Figure 10: Hyperparameter robustness on MATH500 (Qwen3-14B). Sensitivity of SAT to

\tau_{\texttt{fast}}

and

\tau_{\texttt{slow}}

Figure 11: State allocation across dataset difficulty and model scales. We report the step-level thinking mode ratios on GSM8K (easy), MATH500 (medium), and AIME 2025 (hard), for Qwen3-14B and Qwen3-4B.

Figure 12: Outcome-conditioned token savings on AIME2025 (Qwen3-8B). SAT saves substantially more tokens on correct instances than on failed ones, where many runs hit the max-token limit.

Category	Feature Name	Description & Definition
Semantic	Step Embedding	The 384-dimensional dense vector encoded by gte-small representing the semantic content of the current reasoning step text.
Uncertainty (Static)	canonical_logprobs	The log-probability of the sampled token: $\log P(x_{t}\mid x_{<t})$ .
	canonical_selected_rank	The rank of the sampled token in the vocabulary distribution (1-based).
	canonical_entropy	The Shannon entropy of the local probability distribution (truncated to Top- $K$ ).
	canonical_logit_gap	The difference between the largest logit (Top-1) and the second largest logit (Top-2).
	canonical_margin	The probability margin between the Top-1 and Top-2 tokens: $P(x_{\text{top1}})-P(x_{\text{top2}})$ .

Table 2: List of Input Features for the Lightweight Pilot. The features are categorized into Semantic inputs and Uncertainty inputs (subdivided into Static and Dynamic metrics).

Control Tag	Instruction / Semantics
[Fast_Step]	Indicates the current step appears easy. The model is instructed to keep reasoning brief and avoid unnecessary details.
[Slow_Step]	Indicates the current step appears difficult. The model is instructed to perform detailed, expansive reasoning.
[Normal_Step]	Indicates moderate difficulty. The model is instructed to resume standard step-by-step reasoning. (Corresponds to the Normal state).
[Skip_Step]	Indicates the step is excessively difficult and further elaboration is unlikely to be helpful. The model is instructed to summarize existing reasoning, make a reasonable guess, and quickly output the final answer.

Table 3: Definitions of Control Tags. These tags are inserted into the context to modulate the reasoning style of the subsequent step.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 13: Baselines vs. SAT. Early-Stopping halts after a high-confidence first-pass answer and fails; Suppressing Reflective eliminates all verification steps, preventing necessary self-correction and leading to the same error. In contrast, SAT skips redundancy on easy steps but preserves verification on hard steps, achieving the correct answer efficiently.

Model	Method	GSM8K		MATH500		AIME 2024		AIME 2025		AMC
Model	Method	Acc ( $\uparrow$ , %)	Tokens ( $\downarrow$ )	Acc ( $\uparrow$ , %)	Tokens ( $\downarrow$ )	Acc ( $\uparrow$ , %)	Tokens ( $\downarrow$ )	Acc ( $\uparrow$ , %)	Tokens ( $\downarrow$ )	Acc ( $\uparrow$ , %)	Tokens ( $\downarrow$ )
	COT	95.1	2136	95.0	4823	60.0	12662	50.0	12720	92.5	7408
	TALE	–	–	89.1 ( -5.9 )	2657 ( -45% )	48.9 ( -11.1 )	9727 ( -23% )	–	–	86.7 ( -5.8 )	5107 ( -31% )
	NoThinking	–	–	84.9 ( -10.1 )	988 ( -80% )	24.4 ( -35.6 )	4504 ( -64% )	–	–	70.0 ( -22.5 )	1710 ( -77% )
	Dynasor	–	–	90.1 ( -4.9 )	3877 ( -20% )	54.3 ( -5.7 )	9912 ( -22% )	–	–	86.7 ( -5.8 )	6233 ( -16% )
	DEER	94.5 ( -0.6 )	1250 ( -41% )	92.6 ( -2.4 )	3214 ( -33% )	63.3 ( +3.3 )	9327 ( -26% )	55.0 ( +5.0 )	12039 ( -5% )	87.5 ( -5.0 )	4906 ( -34% )

Appendix

Table 4: Comparisons with additional baselines. Metrics include Acc (

\uparrow

) and Tokens (

\downarrow

). Changes relative to the COT are highlighted in orange for Acc and blue for Tokens. Best results within each group are bolded.

Figure 14: Process behavior analysis on MATH500. Average reflect_steps and branch_steps (lower is better) for COT , w/o FAST , w/o SLOW , and SAT, computed from the thought traces using the forward window scheme (

W{=}1

Figure 15: Window-size sensitivity for token-savings attribution. We report (i) contribution ratios of reflection-/branch-related savings and (ii) Pearson correlations between per-sample total savings and each component, under window sizes

W\in\{0,1,2\}

Figure 16: Generation Throughput Comparison. The average number of tokens generated per second by COT and SAT using Qwen3-32B. The consistent throughput confirms that SAT introduces negligible per-step latency overhead.

Explore similar work

Efficient Reasoning via Constrained Optimization in Latent Space
Sep 28, 2026Zhinan Hou, Xingchen Li, Keyou YouOverthinking in Language Models Efficient Language Model Reasoning
Tsinghua University
CAT: Confidence-Adaptive Thinking for Efficient Reasoning of Large Reasoning Models
Jul 1, 2026Qizhi Jiang, Shuo Wang, Pei Ke +2Overthinking in Language Models Efficient Language Model Inference
Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China, Chengdu, China · Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province
EvoThink: Evolving Thinking in Large Reasoning Models via Self-Pruning and Aha-Moment Preference Optimization
Jul 22, 2026Xinbang Dai, Zheyu Xin, Huikang Hu +7Overthinking in Language Models Self-Training for Language Models
Southeast University · Noah’s Ark Lab

Sep 28, 2026J/K move · Enter open · S save

Efficient Reasoning via Constrained Optimization in Latent Space

Zhinan Hou, Xingchen Li, Keyou You

Tsinghua University

Overthinking in Language Models Efficient Language Model Reasoning Quadratic Programming Efficient Language Model Inference Latent Space Optimization Large Reasoning Models

Open paper AskPDF arXiv

Large Reasoning Models (LRMs) have shown remarkable reasoning capabilities, yet they still suffer from overthinking, generating redundant reasoning steps which incur substantial token consumption. Existing methods, such as suppressing reflective keywords or forcing shorter reasoning lengths, attempt to mitigate this issue but inevitably truncate necessary steps and induce underthinking, thereby compromising performance. To address this dilemma, we investigate the latent representations and observe that efficient reasoning steps naturally cluster into a concentrated region in latent space, while those deviating from this region tend to produce verbose sequences. To leverage this, we keep reasoning focused within this region via a quadratic program which projects deviating hidden states back into the region. Then we propose a novel training-free framework to achieve efficient reasoning that reduces token generation costs without sacrificing performance. Extensive experiments conducted on four models ranging from 1.5B to 14B, and across six benchmarks in math reasoning, coding, and scientific QA, validate the effectiveness of our method, up to a 12.1% improvement in accuracy while reducing generated tokens by 11.8% to 52.8%. Codes are available at https://github.com/hzn18/Opt4Reasoning.

Abstract

Explore similar work

Efficient Reasoning via Constrained Optimization in Latent Space

CAT: Confidence-Adaptive Thinking for Efficient Reasoning of Large Reasoning Models

EvoThink: Evolving Thinking in Large Reasoning Models via Self-Pruning and Aha-Moment Preference Optimization