Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation

Gijs Kassenaar; Zhao Yang; Vincent François-Lavet

cs.AIAug 20, 2026

Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation

Authors: Gijs Kassenaar, Zhao Yang, Vincent François-Lavet

Organizations: Vrije Universiteit Amsterdam

Abstract

Reasoning language models trained with reinforcement learning typically operate under a fixed token budget rather than an explicitly adaptive one, which can lead to over-computation on easy problems and insufficient computation on difficult ones. We study whether a model can learn to allocate its own reasoning effort by choosing, as the first token of its response, one of three modes: NoThink (answer as quickly as possible), Short (brief reasoning), or Long (extended reasoning). The choice is learned inside Group Relative Policy Optimization (GRPO) with no separate router, through a shaped reward that makes each mode worthwhile at a different response length, together with hard per-mode token caps that keep the modes distinct. On a 1.5B distilled model trained on MATH, the three modes emerge without collapsing to a single choice, and the brief modes end up more accurate than Long, which shows that the router sorts problems by difficulty rather than at random. Averaged over three seeds, the resulting policy stays close to the base model's accuracy on the held-out MATH500 ( $0.782$ vs.\ $0.796$ ) while cutting the mean response length from $4{,}796$ to $2{,}811$ tokens (a $41\%$ reduction). Interestingly, it also transfers to other benchmarks without retraining, with the largest savings where problems are easier, with for instance 76% token reduction on GSM8K and at higher accuracy than the baselines at similar response length. In short, we build a reasoning model that adaptively chooses how much to reason for each problem.

Figures & tables

Figure 1: Three-mode self-routing GRPO. The model emits a routing token ( NoThink / Short / Long ) as the first token of its response and then reasons under that mode’s shaped reward and hard token cap; the routing token is part of the main policy and is trained end-to-end by GRPO, with no separate router.

$L$	NoThink	Short	Long
0	1.30	1.20	1.00
256	1.25	1.18	1.00
800	1.14	1.14	1.00
3000	0.80	1.00	1.00
8192	0.35	0.73	1.00

Table 1: Correct-answer reward by mode at selected response lengths

L

, under default bases and caps. Crossovers occur where two modes tie (bold).

Figure 2: Correct-answer reward for each mode as a function of the number of tokens

L

generated after the routing token, under the bases and caps of Table 4 . NoThink (

1.3\,\gamma_{\text{NT}}^{L}

) and Short (

1.2\,\gamma_{\text{S}}^{L}

) decay with length while Long is flat at

1.0

. The discounts place the crossovers at

L_{\text{NT}}^{*}\!\approx\!800

( NoThink / Short ) and

L_{\text{S}}^{*}\!\approx\!3000

( Short / Long ), at or below the mode caps. Each mode is reward-optimal in its own length band.

Figure 3: Free-rollout mode shares over training, mean over three seeds (the forced warmup runs to step 45). After warmup all three modes persist without collapsing to any single mode, with Long remaining the plurality.

Figure 4: Per-mode accuracy over training (mean, with std bands, over three seeds). The forced base-model ordering (

\textsc{Long}>\textsc{Short}>\textsc{NoThink}

) inverts around step 50, evidence that the router assigns problems by difficulty: brief modes get easier problems and Long concentrates the hard ones.

Figure 5: Free-validation routing split ( NoThink / Short / Long ) by MATH-500 difficulty level, with per-level accuracy above each bar. Mass shifts from NoThink on easy levels to Long on hard ones, with Short most used at intermediate difficulty.

Figure 6: Accuracy versus response length on MATH-500 (mean over three seeds; error bars: std on both axes). The dashed line traces the Pareto frontier of the single-mode baselines (shaded: on or below the frontier). Free routing (ours) lands strictly above it. Up and to the left is better.

Step	0	30	60	90
Accuracy	$0.807$	$0.727$	$0.764$	$0.783$
	$\pm 0.003$	$\pm 0.012$	$\pm 0.016$	$\pm 0.015$
$\Delta$ vs. step 0		$-0.080$	$-0.043$	$-0.024$

Table 2: MATH validation accuracy over training (free routing), mean

\pm

std over three seeds, at the checkpoints where all three seeds were evaluated. Step 0 is the untrained base under the routing prompt: accuracy dips while routing is being established, then recovers.

Figure 7: Out-of-distribution accuracy versus response length on GSM8K (top) and AIME 2024+2025 (bottom); error bars: std on both axes. Dashed line and shading as in Figure 6 . The above-frontier behavior seen in-distribution transfers zero-shot: free routing exceeds the frontier on GSM8K and matches it on AIME, where nearly all problems demand long reasoning.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Schematic two-mode Countdown reward surface. The realised Short length (

\approx\!1{,}000

, grey band) sat well past the designed crossover

L^{*}

, so Short and Long produced near-identical traces and the crossover never biased routing.

$L$	NoThink	Short	Long
0	1.30	1.20	1.00
256	1.25	1.18	1.00
800	1.14	1.14	1.00
3000	0.80	1.00	1.00
8192	0.35	0.73	1.00

Appendix

Table 3: Correct-answer reward by mode at selected lengths

L

. Crossovers (bold) mark where the optimal mode changes.

Figure 9: Reward surface with the run gammas. Dashed verticals mark the analytic crossovers; dotted verticals the per-mode caps.

Parameter	Value
Bases $b_{NT}/b_{S}/b_{L}$ (phase 1)	$1.3/1.2/1.0$
Discounts $\gamma_{NT}/\gamma_{S}/\gamma_{L}$	$0.99984/0.99994/1.0$
Caps $C_{NT}/C_{S}$ / context	$1024/3000/16384$
Wrong / absent-routing penalty	$0.0$ / $-0.5$
Std-normalization	disabled
Balance coef. / target	$1.0$ / $1/3$

Appendix

Table 4: Hyperparameters for three-mode self-routing GRPO. Phase 2 flattens the reward; caps and balance keep the modes distinct.

	Accuracy				Mean length (tokens)
	MATH-500	GSM8K	AIME’24	AIME’25	MATH-500	GSM8K	AIME (comb.)
Base (no routing)	$0.796{\scriptstyle\pm.002}$	$0.831{\scriptstyle\pm.006}$	$0.265{\scriptstyle\pm.008}$	$0.231{\scriptstyle\pm.013}$	$4743{\scriptstyle\pm 38}$	$1930{\scriptstyle\pm 18}$	$12266{\scriptstyle\pm 26}$
NoThink -only	$0.668{\scriptstyle\pm.013}$	$0.757{\scriptstyle\pm.001}$	$0.137{\scriptstyle\pm.010}$	$0.105{\scriptstyle\pm.018}$	$1096{\scriptstyle\pm 141}$	$336{\scriptstyle\pm 52}$	$3768{\scriptstyle\pm 453}$
Short -only	$0.727{\scriptstyle\pm.007}$	$0.756{\scriptstyle\pm.004}$	$0.218{\scriptstyle\pm.013}$	$0.166{\scriptstyle\pm.028}$	$2030{\scriptstyle\pm 149}$	$440{\scriptstyle\pm 13}$	$7259{\scriptstyle\pm 470}$
Long -only	$0.802{\scriptstyle\pm.004}$	$0.837{\scriptstyle\pm.007}$	$0.297{\scriptstyle\pm.007}$	$0.230{\scriptstyle\pm.011}$	$4507{\scriptstyle\pm 88}$	$1665{\scriptstyle\pm 54}$	$11881{\scriptstyle\pm 56}$
Free routing (ours)	$0.782{\scriptstyle\pm.006}$	$0.781{\scriptstyle\pm.005}$	$0.276{\scriptstyle\pm.010}$	$0.222{\scriptstyle\pm.005}$	$2810{\scriptstyle\pm 59}$	$459{\scriptstyle\pm 18}$	$10716{\scriptstyle\pm 74}$

Appendix

Table 5: Final accuracy and mean response length (tokens), mean

\pm

std over three seeds. MATH-500 in-distribution; GSM8K/AIME out-of-distribution. Base is the untrained model under the routing prompt.

	Base	Router	Reduction
GSM8K	$1930$	$459$	$76\%$ ( $4.2\times$ )
MATH-500	$4743$	$2810$	$41\%$ ( $1.7\times$ )
AIME	$12266$	$10716$	$13\%$ ( $1.1\times$ )

Appendix

Table 6: Token reduction of free routing vs. the base model.

Figure 10: Router mean response length per benchmark (black diamond,

\pm

std) against the three single-mode anchors (coloured ticks), benchmarks ordered easy

\to

hard (log scale). The router hugs the NoThink anchor on GSM8K and the Long anchor on AIME — realised length is a proxy for the routing split, which the released logs do not record per OOD benchmark.

Figure 11: Free-rollout training dynamics (mean over three seeds, std bands). (a) Mode shares emerge after warmup and persist. (b) Routing entropy approaches

\ln 3

. (c) Long lengthens while the brief modes stay capped.

Figure 12: Free-validation routing share by MATH-500 difficulty level at three checkpoints. The easy

\to

NoThink , hard

\to

Long gradient emerges over training.

Explore similar work

Aug 30, 2026

To Think or Not to Think: Allocating Reasoning Where It Helps

Reinforcement learning (RL) has proven effective in enhancing the reasoning performance of large language models (LLMs), particularly in complex mathematical and programming tasks. However, this capability comes with systematic length misallocation, in which models devote excessive reasoning to simple questions while terminating prematurely on harder ones, degrading inference efficiency with negligible accuracy improvement. Many length-adaptive methods mitigate this issue by allocating token budgets according to question difficulty, under the implicit assumption that harder questions benefit monotonically from extended reasoning. In contrast, we find that the effect of reasoning length on accuracy is concentrated on partially solvable questions. Our further analysis reveals that explicit length rewards can produce unintended training dynamics. Motivated by these findings, we propose CARE---Contrastive Accuracy Reward Estimation---which compares the beneficial length adjustment per question from online sampled responses and applies adaptive length rewards within Group Relative Policy Optimization, with no extra hyperparameters or additional inference cost. Experiments across multiple reasoning benchmarks demonstrate that our method improves Pass@1 by up to

4\%

while simultaneously reducing reasoning length by

37\%

, achieving higher token efficiency. Code will be available upon the acceptance of this paper.

Zhengdong He, Yunfan Zhou, Jianguo Yao +2

Shanghai Key Laboratory of Scalable Computing and Systems · School of Computer Science, Shanghai Jiao Tong University

Efficient Language Model Reasoning RL for Language Model Reasoning

Ask PDF

Oct 13, 2026

Efficiently Learning To Reason or Not to Reason: Root-token Policy Optimization for Adaptive Thinking

Large reasoning models (LRMs) achieve strong performance by externalizing explicit reasoning traces before producing the answer, yet suffer from overthinking challenge that allocates uniformly heavy computation to queries of varying difficulty. While proprietary models mitigate this via opaque routing, open-source LRMs still lack an efficient mechanism to internalize adaptive reasoning due to both expensive training cost and limited disclosure of training recipes. In response, we introduce RPO (Root-token Policy Optimization), a framework that enables LRMs to self-determine when to reason by training only the initial root token (e.g., whether to invoke the <think> tag) via group relative reward and group-wise advantages. By focusing on this pivotal branching point, RPO drastically reduces training overhead and VRAM usage. Across multiple model families and scales, RPO learns difficulty-aware adaptive thinking at just ~2% of the training compute of prior adaptive-reasoning methods.

Taehyeon Kim, Hyunsoo Lee, Youngsoo Jang +1

LG AI Research · UNIST · University of Illinois Chicago

Efficient Language Model Reasoning Large Reasoning Models Overthinking in Language Models RL for Language Model Reasoning

Ask PDF

Sep 17, 2026

When2Think: Learning When and How Much to Reason

Large Reasoning Models (LRMs) often overthink easy problems and underthink hard ones, leading to inefficient computation allocation. Existing methods regulate generated computation or select between direct answering and explicit reasoning, but do not jointly control whether}to reason and how much computation to allocate within reasoning. We call the resulting difficulty-dependent loss in accuracy under computation reduction the efficiency tax. We propose When2Think, an RLVR-based post-training framework for instance-adaptive computation allocation. Its core mechanism, Instance-level Difficulty-Aware Control (IDAC), uses cached reference statistics of success and token cost to modulate a correctness-gated efficiency bonus based on generated token count. Importance sampling supports exploration of Think and NoThink, while Batch-Wise Standardization constructs standardized advantages for critic-free optimization. The framework requires neither a learned reward model nor a learned critic, and offline reference caching avoids online reference-model queries during policy updates. On AIME24, When2Think improves Pass@3 by 10.0 percentage points while reducing token usage by 27.9% relative to the backbone.

Jaejun Shim, HyunJin Kim, Young Jin Kim +1

Sungkyunkwan University · Microsoft

Overthinking in Language Models RL for Language Model Reasoning Efficient Language Model Reasoning Efficient Language Model Inference Adaptive Computation

Ask PDF