Dynamic Routing as a New Dimension for Test-time Versatility of LLMs

Michal Štefánik; Marek Kadlčík; Josef Kuchař; Michal Spiegel

cs.AIOct 3, 2026

Dynamic Routing as a New Dimension for Test-time Versatility of LLMs

Authors: Michal Štefánik, Marek Kadlčík, Josef Kuchař, Michal Spiegel

Organizations: TransformersClub, Faculty of Informatics, Masaryk University · R&D Centre for Large Language Models, National Institute of Informatics, Japan

Abstract

Beyond scaling their parameters and data, large language models currently gain versatility on new problems along a single axis: the tokens they spend on chain-of-thought (CoT). We investigate whether dynamic routing programs, which execute a subset of the model's layers or iterate some of them, can open a second axis of test-time adaptation, complementary to CoT and free of any gradient update. Prior work showed that such programs exist and bring accuracy and efficiency gains on problems similar to those they were trained on; we ask whether they can also be identified rapidly, from a handful of demonstrations (3 or 10), by a strategy that transfers across models and tasks without training. First, we find that strategies that select programs by the probability they assign to the demonstrations' labels, arbitrated by the model's own confidence, bring consistent gains: on average over the 49 tasks of MMLU and substantially on four of seven models, and most of all on far out-of-distribution tasks such as ARC-AGI, where programs double the accuracy of a 7B model whose CoT fails. Second, on MMLU across the seven post-trained models, routing complements CoT in practice: the two succeed on different queries, and their composition exceeds CoT alone. Despite these gains, our analyses show that confidence-based selection leaves much of the potential untapped, in two places in particular: (1) in surfacing the routing potential that is already present early in pre-training but becomes harder to select after post-training, and (2) in making models robust to the refinements routing introduces, since unsuccessful routes tend to drive the residual stream out of the distribution that the following layers expect. Together, our results point to dynamic routing as a paradigm for extending the plasticity of existing and future LLMs in rapid test-time adaptation.

Figures & tables

Panel of Figure 1 — Figure 1: Programs of layers for a 12-layer model. Boxes are layer applications in execution order; blue boxes re-apply a contiguous block (a cycle ), the dashed box is a skipped layer. Every program runs the unchanged pretrained weights and is read out in one forward pass.

	DeepSeek-R1	Llama-3.1	Ministral-3	Qwen3	Qwen3.5	Gemma-4	Gemma-4
Method	7B	8B	8B	8B	9B	26B-A4B	31B	avg
Direct (standard pass)	.242	.497	.618	.747	.795	.842	.882	.660
CoT (10k tokens)	.294	.629	.554	.724	.823	.895	.929	.692
MindSkip ( He et al., 2025 )	.082	.350	.405	.156	.618	.765	.740	.445
Dr.LLM ( Heakl et al., 2025 )	.249	.349	.621	.687	.786	.789	.882	.623
PoLar ( Li et al., 2026 )	.231	.501	.622	.747	.787	.850	.882	.660

Table 1: Accuracy on MMLU (49 tasks

\times

20 queries, macro accuracy). Direct : standard pass; CoT : 10k-token chain-of-thought; ConfSwitch : highest-confidence arm; Oracle : per-query best of the three arms. Bold/underline: best / second-best method.

Strategy on the unseen split	accuracy	vs. standard pass
random guessing	.200	–
standard pass	.271	–
CoT , best-performing thinking budget	.350	$+.03$ on the same 60 tasks; longer budgets fall to $.300$ and $.200$
Program fitted on the task’s own demonstrations (Eq. 1 )	.429	better on 34 tasks, worse on 6 ( $p{=}8{\times}10^{-6}$ )
+ Voting $R_{\pi}$ over programs from demonstrations	.540	better on 22, worse on 3 vs. Program
Oracle: best test score among the found programs	.605

Table 2: Textual ARC-AGI-1, DeepSeek-R1-Distill-Qwen-7B, unseen evaluation split (177 tasks),

k{=}3

demonstrations. Per-task adaptation uses only the task’s own demonstrations; fixed programs are selected on the training split.

Panel of Figure 2 — Figure 2: Finding 3: two training trajectories on the MMLU bed. Standard pass, single fitted program, on OLMo-3 the vote over 15 fitted programs, the hindsight per-query oracle over the standard pass and ten searched programs (first sampling draw), and the Transferable Oracle (program correct in both of two independent draws). The vote’s gain on OLMo-3 vanishes at SFT; the reproducible headroom persists at every stage, while the hindsight oracle’s inflation by sampling luck shrinks with post-training. SmolLM3’s stage-3 and base checkpoints run under the raw scaffold, which depresses their standard pass (answer-format compliance, not capability).

(a) Finding 4a: what helpful programs are and do
Question	Statistic	Value
Repairing program exists?	standard-pass misses corrected by some program (GSM8K)	61% (oracle .70 vs. .245)
Are repairs cycles?	repairing programs containing a repeat (GSM8K / ARC)	99% / 56 of 56 (pure skips 6/634, 0)
Where do valid programs cycle?	layers 1–2 / 11–18 / last two, valid vs. rejected (ARC)	.82/.31/.00 vs. .62/.15/.14
How many extra passes?	fixed cycle with 1 / 2 / 3 / 4 extra passes (GSM8K)	.258/.192/.115/.070 (standard .300)
Does the refinement stay close?	validity scorer over 31 label-free features	AUC .80; top: KL proximity

Table 3: Finding 4. (a) What helpful programs are and do (DeepSeek-R1-Distill-Qwen-7B; GSM8K search, matched fix/break pairs and fixed-cycle scan; verified vs. search-rejected ARC-AGI-1 programs). (b) The trace of a helpful cycle at the decision points of chain-of-thought (MMLU, Qwen3.5-9B, 213 split prefixes over 32 items; GSM1k, DeepSeek-R1-Distill-Qwen-7B, 74 problems); brackets: 95% bootstrap intervals over items; offset-0 excess: cosine between

C

and the chain’s update at the cycle’s own depth minus the permuted-item null (raw / shared components removed). AUC: area under the ROC curve; KL: Kullback–Leibler divergence to the full model’s output; full statistics in Appendix G .

Distance to the reference cloud	wrong	correct	paired $\Delta$ (wrong $-$ correct)	wrong further	AUC
cosine (direction; survives the pre-norm)	.261	.187	$+.069$ $[+.050,+.081]$	69%	.654
Mahalanobis (magnitude; divided out by the norm)	13.92	13.37	$+.46$ $[-.07,+1.13]$	54%	.554
control: cycle vs. no-cycle pairs	gap follows loop presence, not correctness (fix 0%, break 100% of pairs)
control: cycle-on states, not count-matched	AUC .52

Table 4: Do unsuccessful programs leave the distribution the following layers expect? DeepSeek-R1-Distill-Qwen-7B on DART-Math: per problem, one correct and one wrong program that both contain a cycle and are matched on total and repeated layer passes (419 pairs), so loop presence and compute are held fixed. Each program’s final state (the input to the final norm and head) is scored against the full-depth reference cloud in a 256-dimensional principal subspace. Brackets: 95% bootstrap intervals; AUC: area under the ROC curve for wrong vs. correct. Controls: the same scoring on unmatched pairs.

Panel of Figure 3 — Figure 3: Decision-point design of § 5.2 : minimal pairs on both sides. (a) Programs that differ only in one extra pass over a block, graded helpful or harmful by the flip of the answer. (b) Chains sampled on the same item that share every token up to the decision point, so that question, prompt and reasoning so far cancel exactly in the contrast.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	DeepSeek-R1 7B	Qwen3 8B
Direct (standard pass)	.242	.747
CoT (10k tokens)	.294	.724
Program (single fitted program)	.310	.614
soft- Vote 10 (one search’s top-10)	.298	.659
Vote 15 (bagged, hard majority)	.407	–
soft- Vote 15 (bagged, confidence-weighted)	.412	.719

Appendix

Table 5: Aggregation ablation on MMLU (macro accuracy) for the two models on which the 15-member bagged vote was run; the other rows repeat Table 1 for reference.

Rule	DeepSeek	Llama	Ministral	Qwen3	Qwen3.5	Gemma 26B	Gemma 31B	avg	$\Delta$ CoT	$+$ on
Margin , CoT default	.327	.625	.627	.750	.841	.893	.922	.712	$+.019$	5/7
rank-calibrated confidence	.325	.561	.637	.750	.838	.885	.905	.700	$+.008$	4/7
demonstration-gated regime	.292	.626	.618	.734	.823	.887	.909	.698	$+.006$	2/7
ConfSwitch { Direct , CoT , Program }	.329	.524	.628	.750	.837	.886	.915	.696	$+.003$	4/7
Margin , Direct default, CoT / Program	.332	.518	.627	.741	.833	.886	.915	.693	$+.001$	–
CoT (10k)	.294	.629	.554	.724	.823	.895	.929	.692	–	–

Appendix

Table 6: All arms and selection rules on MMLU (macro accuracy; margins and thresholds are cross-fitted; “

+

on” = models on which the rule beats always- CoT ), the two oracles, and the number of subjects on which the search preferred the standard pass over every program.

Method	DeepSeek	Llama	Ministral	Qwen3	Qwen3.5	Gemma-26B	Gemma-31B	avg
Direct (standard pass)	.242	.497	.618	.747	.795	.842	.882	.660
Margin ( Direct default; Program )	.309	.508	.632	.748	.795	.841	.883	.674
MindSkip, transfer (capacity)	.082 (.61)	.350 (.46)	.405 (.57)	.156 (.53)	.618 (.56)	.765 (.80) ‡	.740 (.77) ‡	.445
MindSkip, few-shot	.007	.350	.234	.038	.646	–	.838	–
Dr.LLM, few-shot	.249	.349	.621	.687	.786	.789 ∗	.882	.623
PoLar predictor, transfer	.231	.383	–	–	–	–	–	–

Appendix

Table 7: Trained depth-routing baselines on the same bed, in each paper’s transfer configuration and trained on our 490 fit items (few-shot), against the standard pass and our direct-regime margin rule (no CoT anywhere). MindSkip rows carry realized attention capacity; ‡ above the paper’s

\sim

.5 target, i.e. milder gating; ∗ partial arm; –: not run.

Rung	Direct	Program	s Vote 10	ConfSwitch	Margin	Vote 15	s Vote 15	$\Delta$ vote	conf.	identity	hinds.	transf.	luck / null
stage-1, 5%	.299	.293	.276	.299	.294	–	–	–	$\sim$ .6	46	.733	–	– / .870
stage-1, 15%	.337	.339	–	.345	.338	–	–	–	$\sim$ .6	–	.745	–	– / .860
stage-1, 25%	.392	.376	.387	.381	.386	.448	.448	$+.057$	.589	38	.803	.600	$+.203$ / .936
stage-1, 50%	.470	.412	.444	.458	.457	.478	.484	$+.016$	.597	23	.833	.672	$+.160$ / .964
stage-1, 100%	.487	.457	.411	.473	.479	.531	.531	$+.045$	.582	42	.855	.665	$+.190$ / .960
base	.533	.518	.560	.552	.553	.580	.592	$+.059$	.749	19	.866	.768	$+.098$ / .986

Appendix

Table 8: OLMo-3-7B trajectory on the MMLU bed (macro accuracy). Stage-1 percentages are of the pre-training token budget; “base” includes stages 2–3. s Vote 10 : soft vote over one search’s top-10; Vote 15 /s Vote 15 : (soft) vote of 15 members fitted on demonstration subsets;

\Delta

vote: best vote minus Direct . Bold: above the rung’s standard pass by more than

.01

. “conf.”: median answer confidence of the standard pass; “identity”: subjects (of 49) on which the search preferred the standard pass. Oracles over { Direct , top-10 programs}: hindsight (first draw), transferable (correct in both of two independent draws), luck (hindsight minus transferable), and the independence null (within-subject per-query shuffle, 20 repetitions).

Rung	Direct	Program	hindsight	transferable	luck	null	transferable $-$ Direct	identity
stage-1, 5%	.252	.224	.625	.370	$+.255$	.754	$+.118$	43
stage-1, 15%	.262	.242	.675	.439	$+.237$	.843	$+.176$	32
stage-1, 25%	.332	.312	.743	.518	$+.225$	.920	$+.187$	32
stage-1, 50%	.362	.310	.740	.525	$+.214$	.939	$+.163$	30
stage 3 r	.263	.266	.660	.456	$+.204$	.793	$+.193$	–
base r	.204	.236	.582	.366	$+.215$	.701	$+.162$	–

Appendix

Table 9: SmolLM3-3B trajectory on the MMLU bed (macro accuracy). Program : single fitted program; oracles over { Direct , top-10 programs} as in Table 8 : hindsight (first draw), transferable (correct in both draws), luck (hindsight minus transferable), independence null; “identity”: subjects (of 49) on which the search preferred the standard pass, where recorded. r : raw few-shot scaffold with depressed answer-format compliance.

Inference-Time Adaptation of Language Models

See all 25 papers

Oct 14, 2026

RouteWeaver: Weaving Mode Selection and Execution into Unified LLM Routing

LLM routing dynamically selects and coordinates heterogeneous language models across diverse tasks. However, existing approaches are largely built around a predefined execution paradigm, such as single-round, multi-round, or agentic routing, which fixes how model calls are organized and limits adaptation to different query needs. We propose RouteWeaver, a unified routing framework that integrates these routing modes within a single LLM policy, jointly learning which mode to use and how to select models within that mode. To jointly optimize mode selection and within-mode execution, we develop COMET-GRPO (COordinated Mode-Execution Training), which decouples the trajectory-level advantage into separate signals for mode selection and within-mode execution, enabling more targeted credit assignment to each decision level. We further introduce progressive exploration, which gradually shifts training from externally assigned modes to autonomous selection, so that the router first learns how to execute each mode before deciding when to use it. We evaluate RouteWeaver across mathematics, coding, knowledge, logical reasoning, and factual recall. RouteWeaver achieves an average accuracy of 61.4%, an 11.03% relative improvement over the strongest baseline, and outperforms all compared baselines in each of the five domains. Ablations show that both COMET-GRPO and progressive exploration are important for adaptive routing; without progressive exploration, the routing policy collapses entirely to single-round execution.

Xiaohan Wang, Haozhen Zhang, Qingyuan Liu +2

Nanyang Technological University · Huazhong University of Science and Technology · University of Illinois Urbana-Champaign

LLM Routing RL for Language Models Group Relative Policy Optimization

Ask PDF

Aug 6, 2026

MACRO: Markov Chain Routing of Transformer Layers

Standard Large Language Models (LLMs) execute layers sequentially. Dynamic layer routing, i.e. search for a different execution path through layers involving layer repetitions, skips and other moves, can improve performance. Existing routing approaches often require updating model weights, running expensive search loops per test instance, or demand ground-truth labels during inference. In this work, we propose Markov Chain Routing of Transformer Layers (MACRO), a framework that learns task-specific routes over LLM architectures without modifying underlying parameters. MACRO models layer routing as a context-dependent Markov policy conditioned on layer indices, computation budget phases, directional displacements, and operator context, supporting skip, repeat, and residual hidden-state addition operations. The Markov route distribution is updated via feedback on training data and decoded using a top-k Viterbi algorithm to isolate high-probability candidate programs. We evaluate MACRO across diverse reasoning and knowledge benchmarks on multiple open-weight LLMs. MACRO achieves a +5.0% average accuracy improvement over the unrouted baselines, with largest gains on small models. We outperform the best dynamic routing approach Dr. LLM by +7.2%, while reducing route-search time 9.4x (from 14.8 to 1.6 hours). Our code is publicly available at https://github.com/Batorskq/MACRO.

Paweł Batorski, Abtin Pourhadi, Akylgali Aitaza +2

Heinrich Heine University Düsseldorf · Jagiellonian University · IDEAS Research Institute

Markov Models LLM Inference Acceleration LLM Routing

Ask PDF

Oct 2, 2026

Dynamic LLM Routers are Often Misguided

Dynamic LLM routers promise to cut inference costs by sending each query to the cheapest model that can answer it correctly. We analyze six commercial routers across 14 settings on a diverse benchmark spanning eight task categories, finding that none of them outperforms a router that randomly selects between two well-chosen models at matched cost. Some underperform by more than 10 percentage points. We trace this gap to four patterns prevalent across routers: difficulty blindness, length reversal, semantic matching, and roster suboptimality. We show that the first three are what the standard objective rewards: cost-accuracy Pareto efficiency on realized costs favors escalating moderately hard queries over the hardest ones, shorter queries over longer ones, and routing by a query's source over its difficulty. We also argue that the two assumptions that would justify large rosters, model granularity and model specialization, do not hold empirically. We propose an alternative evaluation methodology that does not reward these patterns, and as a proof of concept, we design a simple two-model router that avoids all four. Nevertheless, its gain over random routing is limited, because a well-chosen roster leaves little to route.

Sam Wang, Julia White, Sahibzada Allahyar +3

Fastino Labs

LLM Routing Cost-Aware Inference LLM Evaluation

Ask PDF

Abstract

Inference-Time Adaptation of Language Models

Explore similar work

RouteWeaver: Weaving Mode Selection and Execution into Unified LLM Routing

MACRO: Markov Chain Routing of Transformer Layers

Dynamic LLM Routers are Often Misguided