Dynamic Routing as a New Dimension for Test-time Versatility of LLMs

Michal Štefánik; Marek Kadlčík; Josef Kuchař; Michal Spiegel

cs.AIOct 3, 2026

Dynamic Routing as a New Dimension for Test-time Versatility of LLMs

Authors: Michal Štefánik, Marek Kadlčík, Josef Kuchař, Michal Spiegel

Organizations: TransformersClub, Faculty of Informatics, Masaryk University · R&D Centre for Large Language Models, National Institute of Informatics, Japan

Abstract

Beyond scaling their parameters and data, large language models currently gain versatility on new problems along a single axis: the tokens they spend on chain-of-thought (CoT). We investigate whether dynamic routing programs, which execute a subset of the model's layers or iterate some of them, can open a second axis of test-time adaptation, complementary to CoT and free of any gradient update. Prior work showed that such programs exist and bring accuracy and efficiency gains on problems similar to those they were trained on; we ask whether they can also be identified rapidly, from a handful of demonstrations (3 or 10), by a strategy that transfers across models and tasks without training. First, we find that strategies that select programs by the probability they assign to the demonstrations' labels, arbitrated by the model's own confidence, bring consistent gains: on average over the 49 tasks of MMLU and substantially on four of seven models, and most of all on far out-of-distribution tasks such as ARC-AGI, where programs double the accuracy of a 7B model whose CoT fails. Second, on MMLU across the seven post-trained models, routing complements CoT in practice: the two succeed on different queries, and their composition exceeds CoT alone. Despite these gains, our analyses show that confidence-based selection leaves much of the potential untapped, in two places in particular: (1) in surfacing the routing potential that is already present early in pre-training but becomes harder to select after post-training, and (2) in making models robust to the refinements routing introduces, since unsuccessful routes tend to drive the residual stream out of the distribution that the following layers expect. Together, our results point to dynamic routing as a paradigm for extending the plasticity of existing and future LLMs in rapid test-time adaptation.

Figures & tables

Panel of Figure 1 — Figure 1: Programs of layers for a 12-layer model. Boxes are layer applications in execution order; blue boxes re-apply a contiguous block (a cycle ), the dashed box is a skipped layer. Every program runs the unchanged pretrained weights and is read out in one forward pass.

	DeepSeek-R1	Llama-3.1	Ministral-3	Qwen3	Qwen3.5	Gemma-4	Gemma-4
Method	7B	8B	8B	8B	9B	26B-A4B	31B	avg
Direct (standard pass)	.242	.497	.618	.747	.795	.842	.882	.660
CoT (10k tokens)	.294	.629	.554	.724	.823	.895	.929	.692
MindSkip ( He et al., 2025 )	.082	.350	.405	.156	.618	.765	.740	.445
Dr.LLM ( Heakl et al., 2025 )	.249	.349	.621	.687	.786	.789	.882	.623
PoLar ( Li et al., 2026 )	.231	.501	.622	.747	.787	.850	.882	.660

Table 1: Accuracy on MMLU (49 tasks

\times

20 queries, macro accuracy). Direct : standard pass; CoT : 10k-token chain-of-thought; ConfSwitch : highest-confidence arm; Oracle : per-query best of the three arms. Bold/underline: best / second-best method.

Strategy on the unseen split	accuracy	vs. standard pass
random guessing	.200	–
standard pass	.271	–
CoT , best-performing thinking budget	.350	$+.03$ on the same 60 tasks; longer budgets fall to $.300$ and $.200$
Program fitted on the task’s own demonstrations (Eq. 1 )	.429	better on 34 tasks, worse on 6 ( $p{=}8{\times}10^{-6}$ )
+ Voting $R_{\pi}$ over programs from demonstrations	.540	better on 22, worse on 3 vs. Program
Oracle: best test score among the found programs	.605

Table 2: Textual ARC-AGI-1, DeepSeek-R1-Distill-Qwen-7B, unseen evaluation split (177 tasks),

k{=}3

demonstrations. Per-task adaptation uses only the task’s own demonstrations; fixed programs are selected on the training split.

Panel of Figure 2 — Figure 2: Finding 3: two training trajectories on the MMLU bed. Standard pass, single fitted program, on OLMo-3 the vote over 15 fitted programs, the hindsight per-query oracle over the standard pass and ten searched programs (first sampling draw), and the Transferable Oracle (program correct in both of two independent draws). The vote’s gain on OLMo-3 vanishes at SFT; the reproducible headroom persists at every stage, while the hindsight oracle’s inflation by sampling luck shrinks with post-training. SmolLM3’s stage-3 and base checkpoints run under the raw scaffold, which depresses their standard pass (answer-format compliance, not capability).

(a) Finding 4a: what helpful programs are and do
Question	Statistic	Value
Repairing program exists?	standard-pass misses corrected by some program (GSM8K)	61% (oracle .70 vs. .245)
Are repairs cycles?	repairing programs containing a repeat (GSM8K / ARC)	99% / 56 of 56 (pure skips 6/634, 0)
Where do valid programs cycle?	layers 1–2 / 11–18 / last two, valid vs. rejected (ARC)	.82/.31/.00 vs. .62/.15/.14
How many extra passes?	fixed cycle with 1 / 2 / 3 / 4 extra passes (GSM8K)	.258/.192/.115/.070 (standard .300)
Does the refinement stay close?	validity scorer over 31 label-free features	AUC .80; top: KL proximity

Table 3: Finding 4. (a) What helpful programs are and do (DeepSeek-R1-Distill-Qwen-7B; GSM8K search, matched fix/break pairs and fixed-cycle scan; verified vs. search-rejected ARC-AGI-1 programs). (b) The trace of a helpful cycle at the decision points of chain-of-thought (MMLU, Qwen3.5-9B, 213 split prefixes over 32 items; GSM1k, DeepSeek-R1-Distill-Qwen-7B, 74 problems); brackets: 95% bootstrap intervals over items; offset-0 excess: cosine between

C

and the chain’s update at the cycle’s own depth minus the permuted-item null (raw / shared components removed). AUC: area under the ROC curve; KL: Kullback–Leibler divergence to the full model’s output; full statistics in Appendix G .

Distance to the reference cloud	wrong	correct	paired $\Delta$ (wrong $-$ correct)	wrong further	AUC
cosine (direction; survives the pre-norm)	.261	.187	$+.069$ $[+.050,+.081]$	69%	.654
Mahalanobis (magnitude; divided out by the norm)	13.92	13.37	$+.46$ $[-.07,+1.13]$	54%	.554
control: cycle vs. no-cycle pairs	gap follows loop presence, not correctness (fix 0%, break 100% of pairs)
control: cycle-on states, not count-matched	AUC .52

Table 4: Do unsuccessful programs leave the distribution the following layers expect? DeepSeek-R1-Distill-Qwen-7B on DART-Math: per problem, one correct and one wrong program that both contain a cycle and are matched on total and repeated layer passes (419 pairs), so loop presence and compute are held fixed. Each program’s final state (the input to the final norm and head) is scored against the full-depth reference cloud in a 256-dimensional principal subspace. Brackets: 95% bootstrap intervals; AUC: area under the ROC curve for wrong vs. correct. Controls: the same scoring on unmatched pairs.

Panel of Figure 3 — Figure 3: Decision-point design of § 5.2 : minimal pairs on both sides. (a) Programs that differ only in one extra pass over a block, graded helpful or harmful by the flip of the answer. (b) Chains sampled on the same item that share every token up to the decision point, so that question, prompt and reasoning so far cancel exactly in the contrast.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	DeepSeek-R1 7B	Qwen3 8B
Direct (standard pass)	.242	.747
CoT (10k tokens)	.294	.724
Program (single fitted program)	.310	.614
soft- Vote 10 (one search’s top-10)	.298	.659
Vote 15 (bagged, hard majority)	.407	–
soft- Vote 15 (bagged, confidence-weighted)	.412	.719

Appendix

Table 5: Aggregation ablation on MMLU (macro accuracy) for the two models on which the 15-member bagged vote was run; the other rows repeat Table 1 for reference.

Rule	DeepSeek	Llama	Ministral	Qwen3	Qwen3.5	Gemma 26B	Gemma 31B	avg	$\Delta$ CoT	$+$ on
Margin , CoT default	.327	.625	.627	.750	.841	.893	.922	.712	$+.019$	5/7
rank-calibrated confidence	.325	.561	.637	.750	.838	.885	.905	.700	$+.008$	4/7
demonstration-gated regime	.292	.626	.618	.734	.823	.887	.909	.698	$+.006$	2/7
ConfSwitch { Direct , CoT , Program }	.329	.524	.628	.750	.837	.886	.915	.696	$+.003$	4/7
Margin , Direct default, CoT / Program	.332	.518	.627	.741	.833	.886	.915	.693	$+.001$	–
CoT (10k)	.294	.629	.554	.724	.823	.895	.929	.692	–	–

Appendix

Table 6: All arms and selection rules on MMLU (macro accuracy; margins and thresholds are cross-fitted; “

+

on” = models on which the rule beats always- CoT ), the two oracles, and the number of subjects on which the search preferred the standard pass over every program.

Method	DeepSeek	Llama	Ministral	Qwen3	Qwen3.5	Gemma-26B	Gemma-31B	avg
Direct (standard pass)	.242	.497	.618	.747	.795	.842	.882	.660
Margin ( Direct default; Program )	.309	.508	.632	.748	.795	.841	.883	.674
MindSkip, transfer (capacity)	.082 (.61)	.350 (.46)	.405 (.57)	.156 (.53)	.618 (.56)	.765 (.80) ‡	.740 (.77) ‡	.445
MindSkip, few-shot	.007	.350	.234	.038	.646	–	.838	–
Dr.LLM, few-shot	.249	.349	.621	.687	.786	.789 ∗	.882	.623
PoLar predictor, transfer	.231	.383	–	–	–	–	–	–

Appendix

Table 7: Trained depth-routing baselines on the same bed, in each paper’s transfer configuration and trained on our 490 fit items (few-shot), against the standard pass and our direct-regime margin rule (no CoT anywhere). MindSkip rows carry realized attention capacity; ‡ above the paper’s

\sim

.5 target, i.e. milder gating; ∗ partial arm; –: not run.

Rung	Direct	Program	s Vote 10	ConfSwitch	Margin	Vote 15	s Vote 15	$\Delta$ vote	conf.	identity	hinds.	transf.	luck / null
stage-1, 5%	.299	.293	.276	.299	.294	–	–	–	$\sim$ .6	46	.733	–	– / .870
stage-1, 15%	.337	.339	–	.345	.338	–	–	–	$\sim$ .6	–	.745	–	– / .860
stage-1, 25%	.392	.376	.387	.381	.386	.448	.448	$+.057$	.589	38	.803	.600	$+.203$ / .936
stage-1, 50%	.470	.412	.444	.458	.457	.478	.484	$+.016$	.597	23	.833	.672	$+.160$ / .964
stage-1, 100%	.487	.457	.411	.473	.479	.531	.531	$+.045$	.582	42	.855	.665	$+.190$ / .960
base	.533	.518	.560	.552	.553	.580	.592	$+.059$	.749	19	.866	.768	$+.098$ / .986

Appendix

Table 8: OLMo-3-7B trajectory on the MMLU bed (macro accuracy). Stage-1 percentages are of the pre-training token budget; “base” includes stages 2–3. s Vote 10 : soft vote over one search’s top-10; Vote 15 /s Vote 15 : (soft) vote of 15 members fitted on demonstration subsets;

\Delta

vote: best vote minus Direct . Bold: above the rung’s standard pass by more than

.01

. “conf.”: median answer confidence of the standard pass; “identity”: subjects (of 49) on which the search preferred the standard pass. Oracles over { Direct , top-10 programs}: hindsight (first draw), transferable (correct in both of two independent draws), luck (hindsight minus transferable), and the independence null (within-subject per-query shuffle, 20 repetitions).

Rung	Direct	Program	hindsight	transferable	luck	null	transferable $-$ Direct	identity
stage-1, 5%	.252	.224	.625	.370	$+.255$	.754	$+.118$	43
stage-1, 15%	.262	.242	.675	.439	$+.237$	.843	$+.176$	32
stage-1, 25%	.332	.312	.743	.518	$+.225$	.920	$+.187$	32
stage-1, 50%	.362	.310	.740	.525	$+.214$	.939	$+.163$	30
stage 3 r	.263	.266	.660	.456	$+.204$	.793	$+.193$	–
base r	.204	.236	.582	.366	$+.215$	.701	$+.162$	–

Appendix

Table 9: SmolLM3-3B trajectory on the MMLU bed (macro accuracy). Program : single fitted program; oracles over { Direct , top-10 programs} as in Table 8 : hindsight (first draw), transferable (correct in both draws), luck (hindsight minus transferable), independence null; “identity”: subjects (of 49) on which the search preferred the standard pass, where recorded. r : raw few-shot scaffold with depressed answer-format compliance.

Inference-Time Adaptation of Language Models

See all 24 papers

RouteWeaver: Weaving Mode Selection and Execution into Unified LLM Routing
Oct 14, 2026Xiaohan Wang, Haozhen Zhang, Qingyuan Liu +2LLM Routing RL for Language Models
Nanyang Technological University · Huazhong University of Science and Technology · University of Illinois Urbana-Champaign
MACRO: Markov Chain Routing of Transformer Layers
Aug 6, 2026Paweł Batorski, Abtin Pourhadi, Akylgali Aitaza +2Markov Models LLM Inference Acceleration
Heinrich Heine University Düsseldorf · Jagiellonian University · IDEAS Research Institute
Dynamic LLM Routers are Often Misguided
Oct 2, 2026Sam Wang, Julia White, Sahibzada Allahyar +3LLM Routing Cost-Aware Inference
Fastino Labs

Oct 14, 2026J/K move · Enter open · S save

RouteWeaver: Weaving Mode Selection and Execution into Unified LLM Routing

Xiaohan Wang, Haozhen Zhang, Qingyuan Liu, Tao Feng+1

Nanyang Technological University · Huazhong University of Science and Technology · University of Illinois Urbana-Champaign

LLM Routing RL for Language Models Group Relative Policy Optimization

Open paper AskPDF arXiv

LLM routing dynamically selects and coordinates heterogeneous language models across diverse tasks. However, existing approaches are largely built around a predefined execution paradigm, such as single-round, multi-round, or agentic routing, which fixes how model calls are organized and limits adaptation to different query needs. We propose RouteWeaver, a unified routing framework that integrates these routing modes within a single LLM policy, jointly learning which mode to use and how to select models within that mode. To jointly optimize mode selection and within-mode execution, we develop COMET-GRPO (COordinated Mode-Execution Training), which decouples the trajectory-level advantage into separate signals for mode selection and within-mode execution, enabling more targeted credit assignment to each decision level. We further introduce progressive exploration, which gradually shifts training from externally assigned modes to autonomous selection, so that the router first learns how to execute each mode before deciding when to use it. We evaluate RouteWeaver across mathematics, coding, knowledge, logical reasoning, and factual recall. RouteWeaver achieves an average accuracy of 61.4%, an 11.03% relative improvement over the strongest baseline, and outperforms all compared baselines in each of the five domains. Ablations show that both COMET-GRPO and progressive exploration are important for adaptive routing; without progressive exploration, the routing policy collapses entirely to single-round execution.

Abstract

Inference-Time Adaptation of Language Models

Explore similar work

RouteWeaver: Weaving Mode Selection and Execution into Unified LLM Routing

MACRO: Markov Chain Routing of Transformer Layers

Dynamic LLM Routers are Often Misguided