CounterRoute: Self-Routed Reasoning via Hierarchical Counterfactual Credit Assignment

Ruochen Jiao; Besnik Fetahu; Zhenyu Shi; Priyanka Nigam

cs.AISep 24, 2026

CounterRoute: Self-Routed Reasoning via Hierarchical Counterfactual Credit Assignment

Authors: Ruochen Jiao, Besnik Fetahu, Zhenyu Shi, Priyanka Nigam

Abstract

Reasoning-capable language models often produce long chains of thought when direct answers suffice, wasting inference compute. Many dual-mode models leave this choice to users. Automating it is challenging because routing targets evolve with the policy, initial mode preferences destabilize exploration, and sequence-level objectives entangle routing with response learning. We introduce CounterRoute, an online reinforcement-learning framework that jointly learns routing and modeconditioned responses in one shared policy directly from a native dual-mode checkpoint, without method-specific SFT warm-up. Paired current-policy counterfactual rollouts assign cross-mode credit only to the routing token, while within-mode GRPO trains response tokens. A paired-to-self-routed curriculum stabilizes early training with forced rollouts from both modes, then increases self-routed updates to improve autonomous routing. Across nine benchmarks, CounterRoute better balances accuracy and efficiency than heuristic and learned adaptive-routing methods. Relative to always-thinking checkpoints, it improves macro-average accuracy while reducing mean generated tokens by 51% for Qwen3-8B and 41% for Qwen3-14B. On instruction-following and commonsense benchmarks where direct answering is strong, think rates fall as low as 1% while response quality improves. Despite training only on math and instruction following, its routing behavior and response quality generalize to held-out coding, science, knowledge, and commonsense benchmarks.

Figures & tables

Figure 1: CounterRoute trains and serves one self-routed checkpoint. The upper panel sends self-routed outputs to inference or self-routed training. The lower panel uses paired current-policy groups and assigns routing and response credit to disjoint token positions. Both rollout types update the shared policy under a curriculum. TH and NT denote thinking and no-thinking.

Panel of Figure 2 — Figure 2: CounterRoute pipeline. Paired counterfactual rollouts compute per-prompt utility gaps $\Delta_{i}$ , which are transformed into bounded routing advantages $\pm s\tanh(\Delta_{i})$ at position 0. Subsequent response tokens train under standard GRPO. Think, no-think, and self-routed rollout groups normalize advantages separately.

	Accuracy (%)										Efficiency
Method (Qwen3-8B)	GSM8K	MATH500	GPQA	HumanEval	MMLU-Pro	MMLU	HellaSwag	IFEval	IFBench	Avg	Mean Tok.	Saving
Dual-mode post-trained checkpoints (no additional RL)
Always Think	95.8	92.0	57.6	90.2	65.8	82.7	84.0	90.2	34.7	77.0	2770	0%
Always No-Think	89.2	81.8	46.7	86.0	52.4	77.3	80.4	89.7	28.2	70.2	488	82%
Fixed-mode GRPO (same data, no routing)
GRPO Think-Only	95.6	92.8	57.9	92.7	73.8	82.4	83.9	90.9	34.9	78.3	1571	43%

Table 1: Qwen3-8B results. Parentheses give self-routed think rates (%) for learned routers; savings are relative to Always Think. AdaptThink and HGPO are our reproductions.

	Accuracy (%)										Efficiency
Method (Qwen3-14B)	GSM8K	MATH500	GPQA	HumanEval	MMLU-Pro	MMLU	HellaSwag	IFEval	IFBench	Avg	Mean Tok.	Saving
Dual-mode post-trained checkpoints (no additional RL)
Always Think	96.1	93.1	62.7	93.3	74.9	84.5	87.8	91.1	37.5	80.1	2211	0%
Always No-Think	90.1	85.6	50.4	90.9	60.6	79.9	86.1	90.4	31.4	73.9	396	82%
Fixed-mode GRPO (same data, no routing)
GRPO Think-Only	96.3	94.4	61.2	96.3	76.6	84.9	87.5	91.4	37.8	80.7	1416	36%

Table 2: Qwen3-14B results. Parentheses give self-routed think rates (%); savings are relative to Always Think. AdaptThink and HGPO are our reproductions.

Figure 3: Same-checkpoint mode values and autonomous think rates for CounterRoute with Qwen3-8B, ordered by the forced-mode accuracy gap. Numerical results are in Appendix D.2 .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Shared Qwen3 setting
Initialization	Corresponding dual-mode Qwen3 post-trained checkpoint
Prompt template	Fixed system prompt; one direct and one reasoning demonstration
Training data	Math and synthetic instruction following, each weighted 0.5
Batch and duration	Batch 32; 300 optimizer steps; 4 epochs; $K=4$ rollouts
Lengths	4096 prompt tokens; 8192 response tokens
Sampling	temperature 1.0; top- $p$ 0.8; top- $k=-1$

Appendix

Table 3: Principal training settings shared by Qwen3-8B and Qwen3-14B.

Figure 4: Qwen3-8B training dynamics on math data. The plotted accuracy is the mean correctness reward used as the task reward during training, rather than held-out evaluation accuracy. The self-routed think fraction remains non-collapsed after early exploration.

Benchmark	Acc (%)	Med Words	Acc (%)	Med Words	$\Delta$ (pp)	Acc (%)	Med Words	Think (%)
	Forced Think		Forced No-Think			CounterRoute Self-Routed
GPQA	57.07	1444	35.91	2	$+21.16$	56.9	1383	90.0
MMLU-Pro	73.24	501	53.86	2	$+19.38$	73.3	467	93.0
MMLU	81.87	299	71.95	2	$+9.92$	81.2	268	81.0
MATH500	92.20	443	56.30	43	$+35.90$	92.5	440	100.0
GSM8K	96.13	210	91.58	61	$+4.55$	95.8	212	100.0

Appendix

Table 4: Same-checkpoint routing analysis for CounterRoute with Qwen3-8B.

\Delta

is forced-think minus forced-no-think accuracy in percentage points. Forced modes estimate the benchmark-level value of thinking; self-routed accuracy, median words, and think rate report the learned policy’s actual behavior. Macro values equally weight the nine benchmarks.

Variant	Accuracy (%)	Mean Words	Think (%)
Control method : scheduled $b$ -decay (later ckpt)	78.58	877.1	68.77
Control method : fixed $b=0.05$	77.41	1015.4	69.02
No counterfactual pairing	77.42	1214.5	98.96
Always paired	77.25	1409.0	100.00
Trajectory-wide routing broadcast	51.78	2645.6	63.32
Shared GRPO group	72.16	595.8	42.19

Appendix

Table 5: Controlled Qwen3-8B component ablations. Component and fixed-margin controls are reported at step 60; scheduled

b

-decay is reported at a later checkpoint. Lengths are macro-average mean output words. The trajectory-wide arm is a broadcast diagnostic and does not isolate additive versus overwrite at position 0.

Fixed margin	Accuracy (%)	Med Words	Think (%)
$b=0.00$	78.28	803	99.79
$b=0.05$	77.41	603.8	69.02
$b=0.10$	72.40	132	17.54
$b=0.15$	71.36	120	19.79

Appendix

Table 6: Fixed-margin controls for Qwen3-8B. Lengths are macro-average median output words, the available statistic for these runs.

Model	Method	Avg Acc (%)	Think Rate	Mean Tok.	Saving
Qwen3-8B	Always Think	77.0	100%	2770	0%
	Thinkless-style (DeGRPO)	79.4	100%	2499	10%
	CounterRoute	79.1	65.6%	1347	51%
Qwen3-14B	Always Think	80.1	100%	2211	0%
	Thinkless-style (DeGRPO)	81.7	97%	1848	16%
	CounterRoute	82.4	68.3%	1296	41%

Appendix

Table 7: Controlled Qwen3 adaptation of Thinkless-style DeGRPO under the matched sample budget.

Explore similar work

Sep 1, 2026cs.CL

From Rollouts to Recipes: Self-Contained Post-Training for LLMs

Post-training large language models usually applies a single training recipe to all samples, even though the model's own rollouts reveal different sample-level learning states. We propose Self-Routing, a behavior-conditioned post-training framework that uses rollout correctness and confidence to decide how each sample should be optimized. Depending on its behavior state, a sample is routed to GRPO, on-policy self-distillation, regularization, or skipping, allowing training to adapt without external teachers, extra annotations, or additional sampling. Experiments on mathematical reasoning across Qwen3 and Qwen3.5 backbones show that Self-Routing consistently improves over uniform GRPO, uniform OPSD, fixed mixtures, and simpler routing baselines. Further analyses show that the routing distribution changes over training and reduces unnecessary updates on low-signal or already stable samples.

Yifei Li, Lingling Zhang, Muye Huang +3

School of Computer Science and Technology, Xi’an Jiaotong University, China · Shaanxi Province Key Laboratory of Big Data Knowledge Engineering · Zhongguancun Academy, Beijing, China +1

Post-Training Reinforcement Learning Post-Training Large Language Model Training Unsupervised On-Policy Self-Distillation Rollout Recipe

Ask PDF

May 7, 2026cs.AI

Policy-Guided Stepwise Model Routing for Cost-Effective Reasoning

Inference-time computation has greatly enhanced the performance of large language models (LLMs) on challenging reasoning tasks, but this strategy can incur high inference costs. One solution is to route intermediate chain-of-thought (CoT) states to language models of different sizes; however, existing approaches rely on handcrafted routing strategies that limit performance, or on training large process reward models that may be infeasible in many applications. We formulate stepwise model routing as a constrained decision-making problem, which we solve by training a small control policy using reinforcement learning in conjunction with threshold calibration to tune the performance-efficiency tradeoff. We validate our method on three math benchmarks (GSM8K, MATH500, and OmniMath) on both open and closed models. Our method consistently improves the accuracy-cost tradeoff compared to handcrafted approaches, while achieving a comparable tradeoff to methods that require training large process reward models.

Wenwen Si, Insup Lee, Osbert Bastani

Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104, USA

Inference Cost Chain-of-Thought Reasoning

Ask PDF

May 8, 2026cs.LG

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models

Mixture-of-Experts (MoE) language models route each token to a small subset of experts, but whether the routes selected by a trained top-

k

router are good ones is rarely evaluated directly. Holding the model fixed, we compare each standard route against sampled equal-compute alternatives for the same token and score each by the next-token probability it assigns to the realized token in a verified reasoning trajectory. The result is sharply token-conditional: the standard router is well-aligned with route utility on confident tokens but uninformative on the fragile tokens that drive hard reasoning, where lower-loss equal-compute routes consistently exist inside the frozen model but are not selected. The same pattern holds across Qwen3-30B-A3B, GPT-OSS-20B, DeepSeek-V2-Lite, and OLMoE-1B-7B, and follows structurally from how standard top-

k

training evaluates routing decisions: the language modeling loss scores only the executed route, and load balancing depends only on aggregate routing statistics. A minimal router-only update to the final-layer router, leaving every expert and every other router frozen, is sufficient to shift pass@K on AIME 2024+2025 and HMMT 2025 for both Qwen3-30B-A3B and GPT-OSS-20B, suggesting that at least part of the failure reflects router-reachable misallocation rather than expert capacity alone.

Youngsik Yoon, Siwei Wang, Wei Chen +1

Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea

Mixture-Of-Experts Experts Large Language Model Routing Deepseek Loss Function

Ask PDF