CounterRoute: Self-Routed Reasoning via Hierarchical Counterfactual Credit Assignment

Ruochen Jiao; Besnik Fetahu; Zhenyu Shi; Priyanka Nigam

cs.AISep 24, 2026

CounterRoute: Self-Routed Reasoning via Hierarchical Counterfactual Credit Assignment

Authors: Ruochen Jiao, Besnik Fetahu, Zhenyu Shi, Priyanka Nigam

Abstract

Reasoning-capable language models often produce long chains of thought when direct answers suffice, wasting inference compute. Many dual-mode models leave this choice to users. Automating it is challenging because routing targets evolve with the policy, initial mode preferences destabilize exploration, and sequence-level objectives entangle routing with response learning. We introduce CounterRoute, an online reinforcement-learning framework that jointly learns routing and modeconditioned responses in one shared policy directly from a native dual-mode checkpoint, without method-specific SFT warm-up. Paired current-policy counterfactual rollouts assign cross-mode credit only to the routing token, while within-mode GRPO trains response tokens. A paired-to-self-routed curriculum stabilizes early training with forced rollouts from both modes, then increases self-routed updates to improve autonomous routing. Across nine benchmarks, CounterRoute better balances accuracy and efficiency than heuristic and learned adaptive-routing methods. Relative to always-thinking checkpoints, it improves macro-average accuracy while reducing mean generated tokens by 51% for Qwen3-8B and 41% for Qwen3-14B. On instruction-following and commonsense benchmarks where direct answering is strong, think rates fall as low as 1% while response quality improves. Despite training only on math and instruction following, its routing behavior and response quality generalize to held-out coding, science, knowledge, and commonsense benchmarks.

Figures & tables

Figure 1: CounterRoute trains and serves one self-routed checkpoint. The upper panel sends self-routed outputs to inference or self-routed training. The lower panel uses paired current-policy groups and assigns routing and response credit to disjoint token positions. Both rollout types update the shared policy under a curriculum. TH and NT denote thinking and no-thinking.

Panel of Figure 2 — Figure 2: CounterRoute pipeline. Paired counterfactual rollouts compute per-prompt utility gaps $\Delta_{i}$ , which are transformed into bounded routing advantages $\pm s\tanh(\Delta_{i})$ at position 0. Subsequent response tokens train under standard GRPO. Think, no-think, and self-routed rollout groups normalize advantages separately.

	Accuracy (%)										Efficiency
Method (Qwen3-8B)	GSM8K	MATH500	GPQA	HumanEval	MMLU-Pro	MMLU	HellaSwag	IFEval	IFBench	Avg	Mean Tok.	Saving
Dual-mode post-trained checkpoints (no additional RL)
Always Think	95.8	92.0	57.6	90.2	65.8	82.7	84.0	90.2	34.7	77.0	2770	0%
Always No-Think	89.2	81.8	46.7	86.0	52.4	77.3	80.4	89.7	28.2	70.2	488	82%
Fixed-mode GRPO (same data, no routing)
GRPO Think-Only	95.6	92.8	57.9	92.7	73.8	82.4	83.9	90.9	34.9	78.3	1571	43%

Table 1: Qwen3-8B results. Parentheses give self-routed think rates (%) for learned routers; savings are relative to Always Think. AdaptThink and HGPO are our reproductions.

	Accuracy (%)										Efficiency
Method (Qwen3-14B)	GSM8K	MATH500	GPQA	HumanEval	MMLU-Pro	MMLU	HellaSwag	IFEval	IFBench	Avg	Mean Tok.	Saving
Dual-mode post-trained checkpoints (no additional RL)
Always Think	96.1	93.1	62.7	93.3	74.9	84.5	87.8	91.1	37.5	80.1	2211	0%
Always No-Think	90.1	85.6	50.4	90.9	60.6	79.9	86.1	90.4	31.4	73.9	396	82%
Fixed-mode GRPO (same data, no routing)
GRPO Think-Only	96.3	94.4	61.2	96.3	76.6	84.9	87.5	91.4	37.8	80.7	1416	36%

Table 2: Qwen3-14B results. Parentheses give self-routed think rates (%); savings are relative to Always Think. AdaptThink and HGPO are our reproductions.

Figure 3: Same-checkpoint mode values and autonomous think rates for CounterRoute with Qwen3-8B, ordered by the forced-mode accuracy gap. Numerical results are in Appendix D.2 .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Shared Qwen3 setting
Initialization	Corresponding dual-mode Qwen3 post-trained checkpoint
Prompt template	Fixed system prompt; one direct and one reasoning demonstration
Training data	Math and synthetic instruction following, each weighted 0.5
Batch and duration	Batch 32; 300 optimizer steps; 4 epochs; $K=4$ rollouts
Lengths	4096 prompt tokens; 8192 response tokens
Sampling	temperature 1.0; top- $p$ 0.8; top- $k=-1$

Appendix

Table 3: Principal training settings shared by Qwen3-8B and Qwen3-14B.

Figure 4: Qwen3-8B training dynamics on math data. The plotted accuracy is the mean correctness reward used as the task reward during training, rather than held-out evaluation accuracy. The self-routed think fraction remains non-collapsed after early exploration.

Benchmark	Acc (%)	Med Words	Acc (%)	Med Words	$\Delta$ (pp)	Acc (%)	Med Words	Think (%)
	Forced Think		Forced No-Think			CounterRoute Self-Routed
GPQA	57.07	1444	35.91	2	$+21.16$	56.9	1383	90.0
MMLU-Pro	73.24	501	53.86	2	$+19.38$	73.3	467	93.0
MMLU	81.87	299	71.95	2	$+9.92$	81.2	268	81.0
MATH500	92.20	443	56.30	43	$+35.90$	92.5	440	100.0
GSM8K	96.13	210	91.58	61	$+4.55$	95.8	212	100.0

Appendix

Table 4: Same-checkpoint routing analysis for CounterRoute with Qwen3-8B.

\Delta

is forced-think minus forced-no-think accuracy in percentage points. Forced modes estimate the benchmark-level value of thinking; self-routed accuracy, median words, and think rate report the learned policy’s actual behavior. Macro values equally weight the nine benchmarks.

Variant	Accuracy (%)	Mean Words	Think (%)
Control method : scheduled $b$ -decay (later ckpt)	78.58	877.1	68.77
Control method : fixed $b=0.05$	77.41	1015.4	69.02
No counterfactual pairing	77.42	1214.5	98.96
Always paired	77.25	1409.0	100.00
Trajectory-wide routing broadcast	51.78	2645.6	63.32
Shared GRPO group	72.16	595.8	42.19

Appendix

Table 5: Controlled Qwen3-8B component ablations. Component and fixed-margin controls are reported at step 60; scheduled

b

-decay is reported at a later checkpoint. Lengths are macro-average mean output words. The trajectory-wide arm is a broadcast diagnostic and does not isolate additive versus overwrite at position 0.

Fixed margin	Accuracy (%)	Med Words	Think (%)
$b=0.00$	78.28	803	99.79
$b=0.05$	77.41	603.8	69.02
$b=0.10$	72.40	132	17.54
$b=0.15$	71.36	120	19.79

Appendix

Table 6: Fixed-margin controls for Qwen3-8B. Lengths are macro-average median output words, the available statistic for these runs.

Model	Method	Avg Acc (%)	Think Rate	Mean Tok.	Saving
Qwen3-8B	Always Think	77.0	100%	2770	0%
	Thinkless-style (DeGRPO)	79.4	100%	2499	10%
	CounterRoute	79.1	65.6%	1347	51%
Qwen3-14B	Always Think	80.1	100%	2211	0%
	Thinkless-style (DeGRPO)	81.7	97%	1848	16%
	CounterRoute	82.4	68.3%	1296	41%

Appendix

Table 7: Controlled Qwen3 adaptation of Thinkless-style DeGRPO under the matched sample budget.

Explore similar work

From Rollouts to Recipes: Self-Contained Post-Training for LLMs
Sep 1, 2026Yifei Li, Lingling Zhang, Muye Huang +3Post-Training Reinforcement Learning Post-Training
School of Computer Science and Technology, Xi’an Jiaotong University, China · Shaanxi Province Key Laboratory of Big Data Knowledge Engineering · Zhongguancun Academy, Beijing, China +1
Policy-Guided Stepwise Model Routing for Cost-Effective Reasoning
May 7, 2026Wenwen Si, Insup Lee, Osbert BastaniInference Cost Chain-of-Thought Reasoning
Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104, USA
When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
May 8, 2026Youngsik Yoon, Siwei Wang, Wei Chen +1Mixture-Of-Experts Experts
Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea

Sep 1, 2026, cs.CLJ/K move · Enter open · S save

From Rollouts to Recipes: Self-Contained Post-Training for LLMs

Yifei Li, Lingling Zhang, Muye Huang, Zihan Ma+2

School of Computer Science and Technology, Xi’an Jiaotong University, China · Shaanxi Province Key Laboratory of Big Data Knowledge Engineering · Zhongguancun Academy, Beijing, China+1

Post-Training Reinforcement Learning Post-Training Large Language Model Training Unsupervised On-Policy Self-Distillation Rollout Recipe

Open paper AskPDF arXiv

Post-training large language models usually applies a single training recipe to all samples, even though the model's own rollouts reveal different sample-level learning states. We propose Self-Routing, a behavior-conditioned post-training framework that uses rollout correctness and confidence to decide how each sample should be optimized. Depending on its behavior state, a sample is routed to GRPO, on-policy self-distillation, regularization, or skipping, allowing training to adapt without external teachers, extra annotations, or additional sampling. Experiments on mathematical reasoning across Qwen3 and Qwen3.5 backbones show that Self-Routing consistently improves over uniform GRPO, uniform OPSD, fixed mixtures, and simpler routing baselines. Further analyses show that the routing distribution changes over training and reduces unnecessary updates on low-signal or already stable samples.

Abstract

Explore similar work

From Rollouts to Recipes: Self-Contained Post-Training for LLMs

Policy-Guided Stepwise Model Routing for Cost-Effective Reasoning

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models