Routes of Reasoning: Reading and Reshaping Test-Time Compute in MoE Models

Shengjie Wang; Mohsen Fayyaz; Nanyun Peng

cs.AINov 4, 2026

Routes of Reasoning: Reading and Reshaping Test-Time Compute in MoE Models

Authors: Shengjie Wang, Mohsen Fayyaz, Nanyun Peng

Organizations: University of California, Los Angeles

Abstract

Reasoning models allocate test-time compute across distinct behaviors such as planning, deriving, and verifying. However, models frequently "overthink," continuing to verify and re-derive long after finding a correct answer. This wastes expensive compute and can even degrade performance. Existing interventions, such as token budgets or steering, primarily regulate reasoning duration rather than targeting specific behaviors. In this work, we demonstrate that Mixture-of-Experts (MoE) routers provide a natural, training-free interface for behavioral control. We first split reasoning traces into sentence-level segments and label each with one of six reasoning behaviors (planning, execution, restatement, fact, reflection, and conclusion), so that every token carries the behavior of its segment. Linear probes on router logits predict this label nearly as well as probes on the residual stream, the model's full hidden state and the standard target for probing, while using about half as many features. These signals generalize: experts associated with a reasoning behavior on one dataset predict the same behavior on the others, across six datasets spanning five domains (math, code, science, knowledge, and commonsense). Building on this, we selectively edit expert router scores during inference to curb overthinking. Across four models and four benchmarks, this intervention reduces the number of generated tokens by a median of 30.4% (and up to 63.3%). Crucially, it encourages the model to stop once the problem is solved, rather than skipping essential reasoning steps. Accuracy never drops by more than 0.5 percentage points, and frequently improves. Ultimately, MoE routing offers both an interpretable lens into reasoning states and a lightweight mechanism for optimizing test-time compute.

Figures & tables

Figure 1: Overview. Read : router logits reveal the behavior of each reasoning step (Section 4 ). Identify : behavior-associated experts recur across domains (Section 5 ). Steer : editing behavior-associated experts reduces generated tokens while largely preserving accuracy (Section 6 ).

Behavior	What the segment does
planning	proposes how to proceed
execution	derives a new result
restatement	repeats without new inference
fact	invokes established knowledge
reflection	checks or revises earlier work
conclusion	states a result or closes a subgoal

Table 1: Six reasoning behaviors.

Figure 2: Linear behavior readout. Curves use one layer; circles mark dev-selected layers. The dashed line concatenates router L8–L16. Bands: 95% problem-bootstrap CIs.

Figure 3: Expert-set transfer at layer 36. Rose: overall AUC and four reported behaviors on full segments. Blue-gray: the same ranking after dropping the first five tokens of each segment. Dashed line: chance.

Figure 4: Efficiency. Token change relative to the unmodified model; the table reports baseline accuracy and accuracy changes, with baseline tokens in gray. Rows above give each edit: behavior ( P planning, Rf reflection, and Rs restatement), direction, and number of layer–expert pairs.

Figure 5: Cut at the first answer.

Panel of Figure 6 — Figure 6: Same prefix and seed, different next plan. More in Appendix E.2.3 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: One annotated trace (Qwen3-30B-A3B, NuminaMath problem 00245; spans 7–27 and 85–88 of 108).

Source	Train	Dev	Test
GSM8K training split	100	22	28
NuminaMath training split	98	23	29
AIME 2023	21	4	5
Problems	219	49	62
Segments	52,008	10,315	16,632

Appendix

Table 2: Problem-grouped corpus splits, stratified by source.

Layers	Width	Token test	Segment test
L12 only	128	0.608	0.708
L8–L16 ( $\pm 4$ )	1,152	0.659	0.731
L7–L17 ( $\pm 5$ )	1,408	0.656	0.731
L6–L18 ( $\pm 6$ )	1,664	0.655	0.733
L5–L19 ( $\pm 7$ )	1,920	0.653	0.732

Appendix

Table 3: Router layer windows centered on L12 (macro-F1).

Readout	Width	Macro-F1
Token majority vote	128	0.608
Token probability average	128	0.613
Final token	128	0.702
Max pooling	128	0.682
Mean pooling	128	0.708
Mean + max pooling	256	0.716

Appendix

Table 4: Segment readout from router L12.

Source	Domain	Segments
Mathematical mixture	Mathematics	78,955
MATH500 ( Hendrycks et al., 2021 )	Mathematics	79,816
LiveCodeBench ( Jain et al., 2024 )	Code	187,068
GPQA (non-Diamond) ( Rein et al., 2024 )	Science	112,825
MMLU-Pro ( Wang et al., 2024 )	Knowledge	53,770
CommonsenseQA ( Talmor et al., 2019 )	Commonsense	17,123

Appendix

Table 5: Cross-dataset association corpus, with 330 problems per source.

Model	Layers	Total	Active	Routed	Top- $k$ & shared
Gemma-4-26B-A4B-it	30	26B	4B	128	8
Qwen3-30B-A3B	48	30B	3B	128	8
Qwen3.6-35B-A3B	40	35B	3B	256	8 $+$ 1 shared
GPT-OSS-120B	36	120B	$\sim$ 5.1B	128	4

Appendix

Table 6: MoE models and routing configurations used for steering.

		$\Delta$ tokens (%)		$\Delta$ accuracy (pp)
Model	Benchmark	Prompt	Ours	Prompt	Ours
Gemma-4	MATH500	$-27.3$	$-32.2$	$-0.6$	$+0.0$
	AIME	$+0.6$	$-24.9$	$-7.3$	$+2.2$
	GPQA-D	$-9.1$	$-21.2$	$+0.3$	$-0.5$
	AICrypto	$-18.6$	$-44.9$	$-0.7$	$+2.2$
Qwen3	MATH500	$-17.4$	$-23.1$	$-0.6$	$+0.6$

Appendix

Table 7: Prompt baseline versus behavior-targeted steering for all models. Changes are relative to the unmodified model; token changes average the evaluated seeds, as in Figure 4 .

Suppress	Tokens	Answer at	Re-checks	Doubts	Openers	Correct
none	1,494	[31]	4	7	10	yes
planning ( $\downarrow$ 200)	1,076	[30]	2	3	7	yes
reflection ( $\downarrow$ 100)	1,579	[38]	4	3	15	yes
restatement ( $\downarrow$ 100)	1,866	[43]	5	7	15	yes
execution ( $\downarrow$ 30)	1,758	[64]	4	5	14	yes
execution ( $\downarrow$ 100)	16,384	–	–	19	40	no

Appendix

Table 8: One problem under each suppression. Parentheses give the number of suppressed layer–expert pairs. Re-checks: rounds of checking after the answer; Doubts and Openers: reflection and planning sentences in the trace.

Planning experts	Wrap-up	New angle	Thinking ends
unmodified	23	23	5%
suppressed throughout	68	11	17%
suppressed for one window	43	21	6%

Appendix

Table 9: First plan after the cut (Qwen3.6, MATH500, 314 cut points). The last column is the share of continuations that end thinking within

{\sim}170

tokens.

Efficient Language Model Reasoning

Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.

See all 388 papers

TempoMoE: Identifying and Steering Fast-Slow Thinking Experts for Efficient Multimodal Reasoning
Oct 19, 2026Xiyue Zhang, Keru Zhou, Ziyi Wang +2Mixture-of-Experts Inference Efficient Multimodal Inference
Zhejiang University · Tsinghua University · Wenzhou Business College
Beyond the Trace: Coupling an Interpretable Reasoning-State Readout to Native MoE Routing
Aug 18, 2026Kang Chen, Sihan Zhao, Yixin Cao +1Mixture-of-Experts Interpretability LLM Interpretability
Fudan University · Shanghai Innovation Institute
From Concentration to Differentiation and Back: Routing Effective Rank in MoE Reasoning Cohorts
Sep 6, 2026Kang Chen, Sihan Zhao, Yixin Cao +1Mixture-of-Experts Models Mixture of Experts
Fudan University · Shanghai Innovation Institute

Oct 19, 2026J/K move · Enter open · S save

TempoMoE: Identifying and Steering Fast-Slow Thinking Experts for Efficient Multimodal Reasoning

Xiyue Zhang, Keru Zhou, Ziyi Wang, Bing Hu+1

Zhejiang University · Tsinghua University · Wenzhou Business College

Mixture-of-Experts Inference Efficient Multimodal Inference VLM Reasoning Multimodal CoT Reasoning Mixture-of-Experts Interpretability

Open paper AskPDF arXiv

Multimodal Mixture-of-Experts (MoE) reasoning models rely on long chain-of-thought (CoT) for complex tasks, yet often overthink easy inputs, increasing token cost and latency, and sometimes degrading accuracy. Existing methods optimize reasoning length during training or impose external control at inference, but leave the underlying reasoning mechanism a black box. We find that fast and slow thinking exhibit distinct expert-routing patterns: holding the input fixed and varying only the prompted reasoning tempo produces stable routing shifts. Suppressing slow-thinking experts shortens the reasoning chain, whereas suppressing fast-thinking experts lengthens it, providing causal evidence that these experts influence reasoning depth. Based on this finding, we propose TempoMoE, a training-free, two-stage framework. It identifies fast- and slow-thinking experts via tempo-contrastive routing analysis and redistributes routing weights within the original top-k set according to sample difficulty. It amplifies fast-thinking experts and suppresses slow-thinking experts on easy inputs and reverses this adjustment on hard ones, enabling difficulty-adaptive control over reasoning depth. We evaluate TempoMoE on Kimi-VL, Qwen3-VL, and ERNIE-4.5-VL across six benchmarks. Compared with Vanilla, TempoMoE consistently improves accuracy while reducing output tokens, with accuracy gains of up to 2.6 points and token reductions of up to 35.4%.

Abstract

Efficient Language Model Reasoning

Explore similar work

TempoMoE: Identifying and Steering Fast-Slow Thinking Experts for Efficient Multimodal Reasoning

Beyond the Trace: Coupling an Interpretable Reasoning-State Readout to Native MoE Routing

From Concentration to Differentiation and Back: Routing Effective Rank in MoE Reasoning Cohorts