Inference in LLMs is conventionally a fixed-depth, fixed-order forward pass through every layer, regardless of how difficult the input is. The human brain does not work this way: using the thalamus as a central hub, it routes information flexibly to all regions of the cortex according to demand. Li et al. (2026) recently showed, with a system they call program-of-layers (PoLar), that transformers can be given an analogous flexibility if their layers are treated as a library of functions rather than a fixed sequence. Performance improves over the standard forward pass when each input is dynamically routed through an adaptive sequence of skipped or repeated contiguous layer blocks. We reconstructed PoLar's diagnostic MCTS in more detail than the original paper and applied it across 5 models. We reproduced several of PoLar's findings: skipping outperformed the standard pass, repeating outperformed skipping, and combining both outperformed either alone. Shorter programs sufficed for easier questions, while harder questions required more layer repeats. However, we failed to replicate the main claim regarding their learned router for single-shot inference: its top-ranked prediction consistently collapsed back to the standard pass, even though its top-k predicted programs, taken together, did show a real accuracy gain. Beyond reproduction, we find that a small number of generic programs are enough to solve most of the questions. We also provide a much deeper analysis of these programs' structure and robustness: for example, we found that programs that correct errors are highly brittle: undoing even a single edit inside a program typically breaks the correction. Connecting this to the brain's routing mechanisms, PoLar mirrors principles of thalamo-cortical coordination between cortical-area-like transformer layers. We publicly release the code at https://datexis.github.io/RE-PoLar/
Figures & tables
Figure 1: A PoLar , predicted by a lightweight router, executes a schedule of skip , keep , and repeat operations over frozen, pretrained transformer layers (left). We read this coordinating role as functionally analogous to the thalamus routing computation across cortical areas (right). A transformer’s layer order is fixed only by training, not by necessity: a model trained to run its layers in order can nonetheless be usefully executed out of that order at inference time, a flexibility the brain already has by default.
Finding
Description
Status
\ftag F0
Reproducing Li et al. ’s baseline numbers
↯
\ftag F1
Skip & Repeat combined beats either alone
✓
\ftag F2
Occam’s razor: most samples have shorter valid programs than a standard forward pass
✓
\ftag F3
Harder inputs need larger programs (test-time scaling via recurrence)
✓
\ftag F4
Valid programs concentrate on short, mostly single-recurrence segments
↯
\ftag F5
A small menu of programs generalizes across many questions
★
Table 1: Every finding in section 4 , checked against Li et al. (2026) ’s own study. ✓ = reproduced, ↯ = not reproduced (we tested it and got a contradicting result), ★ = a new finding of ours with no equivalent in the original paper to check against. \ftag F1– \ftag F4 are Li et al. ’s own diagnostic Findings 1–4.
Figure 2: Simply repeating a well-chosen middle block (layers 18–22), no retraining, no other intervention, already improves Qwen3-8B’s accuracy on a 6-domain subset of MMLU-Pro ( Wang et al., 2024 ) . The representational pattern in (a) and the behavioral pattern in (b) agree on two boundaries: layer 6, where mixing layers from before and after collapses accuracy under both repeat and skip, and layer 25, past which layers grow mutually redundant, so skipping there turns from harmful to roughly neutral.
Figure 3: One simulation of algorithm 1 , traced end to end on a toy tree ( D=7 ). (a) N0 is already widening-capped, so selection descends by argmax UCB into child B rather than expanding N0 further. (b) B is still under its own cap ( 1 of 2 children allowed at v=1 ), so SelectExpand expands it directly, popping a random untried edit to create G . (c) x is scored under G ’s edit set ( {skip[5,6),repeat[2,4)×2} ), producing reward r . (d) r backpropagates along the traversed chain N0→B→G , and the global counter V increments once.
Model
DM-1
DM-2
DM-3
DM-4
DM-5
Qwen1.5-MoE-A2.7B
Base
τ=0
22.6
(-15.4)
24.2
(-0.4)
21.2
(+0.6)
18.0
(+2.8)
12.2
(-2.0)
p@1
21.6
(-13.8)
24.2
(+2.2)
21.2
(+6.6)
18.2
(+5.6)
11.4
(+4.8)
p@5
34.4
(-5.6)
36.2
(+10.6)
30.8
(+12.2)
28.6
(+13.6)
21.2
(+9.4)
MCTS
73.0
(-0.2)
71.0
(+11.3)
67.8
(+17.4)
63.2
(+25.0)
43.9
(+11.5)
Router
p@1
22.6
(-21.2)
24.2
(-2.0)
21.2
(-0.8)
18.0
(+0.8)
12.2
(-3.2)
p@5
35.6
(-26.4)
37.4
(-6.6)
34.2
(+1.2)
30.6
(+5.2)
19.0
(-4.2)
Table 2: \ftag F0 \ftag F8 The reproduction gap against Li et al. (2026) ’s own numbers is model-dependent (colored Δ= ours − their matching tables), and Router p@1 equals Base ( τ=0 ) exactly on every model and difficulty: the router always predicts identity. Accuracy (%) by DART-Math difficulty: Base τ=0 /p@1/p@5 and Router p@1/p@5 are pass@ k on the 500 -question held-out test split per difficulty; MCTS is the share of samples that have at least one valid program, computed over the full 2,000 -question pool.
Figure 4: \ftag F2 Occam’s razor on Qwen3-8B, by DART-Math difficulty: (a) the share of all questions admitting a program shorter than the standard forward pass; (b) that shortest program’s actual executed length, bar = mean, black dot = median, dotted line marks the identity length D . Same measurement as figs. 13 and 14 , all 5 models, in the appendix.
Figure 5: \ftag F3 Of every question that is solvable at all, the share that needed recurrence (solid) or needed skipping (dashed) to be solved. We are using the same per-model panel as Li et al. (2026) for our reproduction.
Figure 6: \ftag F3 Qwen3-32B: share of shortest-valid programs that skip (a) or repeat (b) each layer, by layer index and DART-Math difficulty. Easier difficulties concentrate more on skipped layers; harder difficulties spread repeats across more of the stack. Same measurement as figs. 17 and 18 , all 5 models, in the appendix.
Figure 7: \ftag F4 Program-editing structure by segment length, grouped by model.
Figure 8: \ftag F5 Coverage and visualization of the menu behavior.
Figure 11
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Variant
Grader
Accuracy
\errtag unboxed
\errtag prompt template echo
Qwen1.5-MoE-A2.7B
polar_default
PoLar
16.3%
0.3%
2.70%
drllm (faithful)
mathruler
2.0% ∗∗∗
89.2%
0.00%
drllm_chat_prefill
mathruler
15.6%
0.0%
0.00%
polar_oneshot
PoLar
19.3% ∗∗∗
2.0%
0.00%
drllm_chat_oneshot
mathruler
13.2% ∗∗∗
18.3%
0.00%
drllm_chat_oneshot_prefill
mathruler
15.3% ∗
0.0%
0.00%
Appendix
Table 4: Accuracy, \errtag unboxed rate, and \errtag prompt template echo rate, all 6×6=36 model/prompt-variant combinations, on our 10,000 -query DART-Math split ( section A.1 ). Bold = best accuracy for that model. Grader : polar_default / polar_oneshot use PoLar , Li et al. (2026) ’s own last- \boxed{} -match DART-Math extraction (byte-identical to their vendored copy); the four drllm variants use mathruler , Heakl et al. ’s own grading library ( Heakl et al., 2025 ) – each graded the way its own source protocol would grade it. polar_oneshot repeats the instruction-and-answer block twice inside the 50-token budget, risking a hallucinated third, incomplete \boxed{...} span that a naive last-match extraction would grab instead of the real answer; we truncate before grading at the first complete span, so it is, in effect, graded on the first \boxed{...} span rather than the last ( polar_default ’s single, unrepeated block never needs this). Superscripts are McNemar’s paired test against that model’s own polar_default row: ∗∗∗ p<0.001 , ∗ p<0.05 , unmarked = not significant. Table 5 gives the exact wording of every variant swept here.
Variant
Template
polar_default
“ Solve the following math problem and output ONLY the final answer directly, formatted strictly as \boxed{ANSWER}. ### Problem Start <q> ### Problem End Answer: ”
drllm (faithful) †
User: “ Question: <q> The final answer MUST BE put in \boxed{} and no explanation. ”
drllm_chat_prefill †
User: “ Question: <q> The final answer MUST BE put in \boxed{} and no explanation. ” Assistant: “ \boxed{ ”
polar_oneshot
“ Solve the following math problem and output ONLY the final answer directly, formatted strictly as \boxed{ANSWER}. ### Problem Start What is 1+1? ### Problem End Answer: \boxed{2} ” “ Solve the following math problem and output ONLY the final answer directly, formatted strictly as \boxed{ANSWER}. ### Problem Start <q> ### Problem End Answer: ”
drllm_chat_oneshot †
User: “ Question: What is 1+1? The final answer MUST BE put in \boxed{} and no explanation. ” Assistant: “ \boxed{2} ” User: “ Question: <q> The final answer MUST BE put in \boxed{} and no explanation. ”
drllm_chat_oneshot_prefill †
User: “ Question: What is 1+1? The final answer MUST BE put in \boxed{} and no explanation. ” Assistant: “ \boxed{2} ” User: “ Question: <q> The final answer MUST BE put in \boxed{} and no explanation. ” Assistant: “ \boxed{ ”
Appendix
Table 5: The 6 prompt variants swept in table 4 (question text abbreviated as <q> ), every message spelled out in full. Rows marked † are chat-templated ( section A.3 ); the literal chat-template markup itself (special tokens etc.) is tokenizer-specific and omitted, but every turn’s exact text content is shown.
Model
no chat template
chat template
Δ (pp)
Qwen1.5-MoE-A2.7B
22.5%
7.0%
−15.5
Qwen2.5-3B
22.0%
20.0%
−2.0
Qwen2.5-7B
44.0%
36.5%
−7.5
LLaMA-3.2-3B
20.5%
2.5%
−18.0
Qwen3-8B
34.0%
33.5%
−0.5
Qwen3-32B
40.0%
1.0%
−39.0
Appendix
Table 6: Chat template ablation: Base ( τ=0 ) accuracy with vs. without the model’s native chat template, prompt content held fixed at polar_oneshot , 200 -question pooled sample (seed 42 ) per model. “chat template” adds the model family’s own system message where Li et al. (2026) ’s code applies one (Qwen2.5-family, Qwen1.5-MoE-A2.7B); Qwen3-8B’s chat setting has no system message, matching Li et al. ’s own polar/eval.py found at ( https://github.com/tianyi-lab/PoLar/blob/main/polar/eval.py ).
Model
Metric
DM-1
DM-2
DM-3
DM-4
DM-5
LLaMA-3.2-3B
τ=0
18.8
(-23.6)
14.8
(-13.8)
17.2
(-10.0)
20.0
(-7.6)
24.4
(-4.2)
p@1
19.2
(-21.4)
14.8
(-13.8)
16.2
(-11.2)
19.0
(-9.0)
24.4
(-4.8)
p@2
22.6
(-21.4)
18.0
(-17.0)
20.2
(-9.4)
21.2
(-9.2)
28.2
(-4.6)
p@3
24.6
(-21.6)
21.2
(-17.6)
21.8
(-9.4)
25.0
(-6.4)
32.0
(-2.4)
p@4
25.2
(-21.8)
23.4
(-18.0)
23.4
(-8.8)
26.4
(-5.8)
34.6
(-0.2)
p@5
27.2
(-20.4)
25.0
(-18.2)
25.2
(-7.6)
28.0
(-4.8)
35.8
(+0.2)
Appendix
Table 7: Full Base ( τ=0 ) and Base (sampling) p@1–p@5 breakdown (%) on our reconstructed DART-Math split ( section A.1 ), 500 questions/difficulty, all 6 models, polar_oneshot , no chat template ( section A.3 ). Colored Δ= ours − Li et al. (2026) ’s matching table, where one exists; Qwen2.5-7B and Qwen3-32B are new here (see \ftag F0 for the reproduction-gap pattern).
Model
MMLU-Pro (overall)
ASDiv
MAWPS
LLaMA-3.2-3B
31.4
64.4
(-14.0)
70.6
(-0.9)
Qwen1.5-MoE-A2.7B
24.4
52.7
(-6.4)
62.3
(+20.6)
Qwen2.5-3B
35.2
68.1
(+18.6)
62.9
(+26.7)
Qwen2.5-7B
42.3
75.3
67.9
Qwen3-8B
42.1
78.7
(+11.6)
71.5
(+19.4)
Qwen3-32B
48.7
77.0
77.7
Appendix
Table 8: Base ( τ=0 ) accuracy (%): MMLU-Pro (overall, all 14 categories, forced-choice logit scoring) and OOD transfer targets ASDiv ( n=2,305 ) and MAWPS ( n=520 , generation-based, same prompt/mechanism as table 7 ), all 6 models. Colored Δ= ours − Li et al. (2026) ’s matching table, where one exists.
Model
pass@ 1
pass@ 2
pass@ 3
pass@ 4
pass@ 5
Qwen1.5-MoE-A2.7B
19.6%
21.4%
23.6%
27.9%
31.4%
Qwen2.5-3B
22.2%
25.2%
29.0%
32.1%
34.6%
Qwen2.5-7B
37.8%
40.6%
43.3%
47.4%
50.2%
Qwen3-8B
33.2%
36.3%
38.6%
41.2%
43.6%
Qwen3-32B
35.0%
40.8%
45.2%
48.9%
52.1%
Appendix
Table 9: Router ’s real online-execution pass@ 1 – 5 , Drop-CE recipe, averaged across all 5 DART-Math difficulty splits, all 5 models in this study; the intermediate pass@ 2 – 4 ranks here are not in table 2 , which reports only pass@ 1 /pass@ 5 per difficulty level rather than pooled across difficulty. Router ’s pass@ 1 equals Base ( τ=0 ) exactly: it always predicts identity. The pass@ 5 gap over pass@ 1 is the real, non-random rescue signal documented in section 4.4 , not something Router ’s own top- 1 choice can access.
Model
DM
Solvable
No-id %
Examples
Id %
Keep %
Qwen1.5-MoE-A2.7B
1
923
68.1%
4,196
7.0%
80.1%
2
883
68.2%
4,037
6.9%
79.9%
3
825
74.2%
3,380
6.3%
80.0%
4
801
67.8%
3,592
7.2%
80.2%
5
553
73.6%
1,946
7.5%
80.4%
Qwen2.5-3B
1
1,032
60.8%
8,538
4.5%
84.0%
Appendix
Table 10: Router training-data composition, Drop treatment, full per-difficulty breakdown, 1,250 -question train slice (data composition depends only on the data treatment, not the loss function). Solvable = questions with ≥1 MCTS -verified-correct program; no-id % = share of those with no valid identity program at all; examples = the resulting number of training program-examples; id % / Keep % as table 3 .
Validation loss
Pass@ 5
Model
Drop-CE
Crop-CE
Drop-CE
Crop-CE
Qwen1.5-MoE-A2.7B
1.071
1.086
31.4%
30.5%
Qwen2.5-3B
1.024
1.034
34.6%
32.7%
Qwen2.5-7B
1.088
1.105
50.2%
49.6%
Qwen3-8B
1.008
1.032
43.6%
44.9%
Qwen3-32B
0.892
0.915
52.1%
56.4%
Appendix
Table 11: Cross-model recipe comparison: validation loss (winning hyperparameter configuration) and real online-execution pass@ 1 /pass@ 5 , averaged across all 5 difficulty splits, for the four comparable variants (Drop/Crop × CE/focal). Drop-CE is the recipe used throughout the rest of this paper. Drop-CE(pass@ 5 ) below is corrected to the real full- 500 -question test split (same backfill as tables 9 and 2 ); Crop-CE(pass@ 5 ) still uses the original n=250 half, not yet backfilled, so that one column is not apples-to-apples with the rest of the table (verified against the repo’s own result files: no full- 500 Crop-CE backfill exists yet).
Model
Split
DM-1
DM-2
DM-3
DM-4
DM-5
Qwen1.5-MoE-A2.7B
train
8
10
9
5
9
val
8
9
8
5
9
test
8
9
9
5
9
Qwen2.5-3B
train
12
13
11
13
11
val
8
10
9
13
11
test
10
11
10
13
10
Appendix
Table 12: The \ftag F9 small-fixed-menu property holds on every split: the number of distinct programs appearing anywhere across all questions’ top- 5 decoded candidate sets, Drop-CE recipe, per model and DART-Math difficulty level, on training ( n=1,250 /difficulty), validation ( n=250 /difficulty), and test ( n=500 /difficulty) data.
Stage
GPU-hours
Architecture
DART-Math track
MCTS (5 difficulties × 5 models)
∼ 800
B200
Router training & recipe selection ( section A.8 ), incl. the test-split backfill above
∼ 865
A100
MMLU-Pro track
MCTS (14-domain, Qwen3-8B)
∼ 40
A100
Router training & recipe selection
∼ 150
A100
Appendix
Table 13: GPU-hours by what the compute went toward (rounded).
Model
\errtag wrong final answer
\errtag unboxed
\errtag prompt template echo
\errtag malformed boxed
Total
Qwen1.5-MoE-A2.7B
7,953 (98.7%)
87 (1.1%)
1 (0.0%)
16 (0.2%)
8,057
Qwen2.5-3B
7,668 (99.0%)
74 (1.0%)
0 (0.0%)
0 (0.0%)
7,742
Qwen2.5-7B
6,299 (100.0%)
0 (0.0%)
0 (0.0%)
3 (0.0%)
6,302
Qwen3-8B
6,599 (99.9%)
0 (0.0%)
2 (0.0%)
4 (0.1%)
6,605
Qwen3-32B
2,548 (39.1%)
3,857 (59.3%)
102 (1.6%)
2 (0.0%)
6,509
All models
31,067 (88.2%)
4,018 (11.4%)
105 (0.3%)
25 (0.1%)
35,215
Appendix
Table 14: Error-category breakdown for every identity-wrong record, all 5 DART-Math difficulties pooled, per model. Taxonomy and methodology above.
Model
Related n
Related rescued
Unrelated n
Unrelated rescued
Qwen1.5-MoE-A2.7B
6,653
57.9%
1,215
41.3%
Qwen2.5-3B
6,822
64.3%
784
36.9%
Qwen2.5-7B
5,661
66.3%
587
37.6%
Qwen3-8B
5,841
63.5%
700
43.6%
Qwen3-32B
2,355
83.1%
185
62.7%
All models
27,332
64.6%
3,471
41.3%
Appendix
Table 15: Rescue rate (share of \errtag wrong final answer records for which MCTS found at least one valid fixing program) by closeness label, all 5 difficulties pooled, per model. Closeness methodology above.
Figure 10: (a) Accuracy gained by allowing skip and repeat together, over the standard forward pass alone, at each DART-Math difficulty level (1 = easiest, 5 = hardest); same Gain column as table 16 above. (b) Share of questions with at least one valid program, as more extra latent execution steps (recurrence) are allowed beyond the standard forward pass depth, pooled over all difficulties, the recurrence-budget half of \ftag F3 . One line per model in both panels.
Model
Difficulty
Base
Skip
Repeat
Skip&Repeat
Gain
Qwen1.5-MoE-A2.7B
DM-1
22.7%
34.8%
67.1%
73.0%
+50.2
DM-2
22.8%
35.8%
64.8%
71.0%
+48.2
DM-3
19.6%
33.5%
61.4%
67.8%
+48.3
DM-4
20.0%
32.2%
58.4%
63.2%
+43.3
DM-5
12.2%
19.4%
39.1%
43.9%
+31.7
Qwen2.5-3B
DM-1
31.6%
54.8%
76.5%
82.7%
+51.0
Appendix
Table 16: Combining skip and repeat always beats either alone ( \ftag F1 ): Skip&Repeat’s gain over Base ranges from +31.7 to +68.7 accuracy points across all 5 models and every DART-Math difficulty. Base = identity already correct. Skip = identity or a skip-only valid program exists. Repeat = identity or a repeat-only valid program exists. Skip&Repeat = identity or any valid program exists. Gain = Skip&Repeat − Base .
Figure 11: The recurrence-budget evidence behind \ftag F3 , from a depth-budget angle: best accuracy achievable using only a program (identity or found) whose executed length is at most the given budget, as a percentage of the standard forward pass depth (DART-Math difficulty 1 = easiest to 5 = hardest). We are using the same per-model panel as Li et al. (2026) for our reproduction.
Model
n segments
Single-layer
At-most-two-layers
Qwen1.5-MoE-A2.7B
9,097
46.5%
73.8%
Qwen2.5-3B
12,575
32.6%
58.4%
Qwen2.5-7B
12,410
37.4%
63.1%
Qwen3-8B
13,503
32.7%
58.5%
Qwen3-32B
18,825
21.1%
42.7%
PoLar (reference)
–
54.5%
>66.7%
Appendix
Table 17: Segment-length share of the shortest valid program per solved question, pooled over all 5 DART-Math difficulties, per model: the \ftag F4 comparison. PoLar reports 54.5% single-layer and >66.7% (over two-thirds) at-most-two-layers.
Figure 12: The per-model, per-difficulty view behind \ftag F5 : menu generalization degrades with difficulty on every model, coverage still climbing at K=100 on the hardest difficulty but nearly flat by K=30 on the easiest. Menu coverage of solved questions as the menu size K grows, one line per DART-Math difficulty, one panel per model. K on a log scale.
Figure 13: The per-model, pooled-over-difficulty counterpart to \ftag F2 (Occam’s razor): share of all questions (not split by whether the standard forward pass already answered correctly) admitting a program shorter than it, by DART-Math difficulty. PoLar ’s F2 uses the C → C/W → C split of this measure instead, pooled over difficulty. The prose numbers for that split are in section 4.2 and section 4 . Qwen3-8B is omitted here since it already has its own dedicated figure in the main body ( fig. 4 ). One panel per remaining model, all four sharing the leftmost panel’s y-axis (0–100%, same scale throughout).
Figure 14: The \ftag F2 split’s program-length counterpart. Same split as fig. 13 , but showing the shortest valid program’s actual executed length rather than just whether it beats identity. The dotted line marks the identity (standard forward pass) length D for reference. Qwen3-8B again omitted ( fig. 4 in the main body). One panel per remaining model, by DART-Math difficulty, all four sharing the leftmost panel’s y-axis.
Figure 15: The baseline/rescued/unsolved split behind \ftag F5 ’s menu coverage. Of every question: the share already answered correctly by the standard forward pass (baseline); the share the standard forward pass got wrong but for which the search found at least one valid program (rescued); the share for which nothing valid was found (unsolved). One panel per model, by DART-Math difficulty, all five sharing the leftmost panel’s y-axis.
Figure 16: The \ftag F1 solvability split by operation class. Of every solved question, whether the standard forward pass already works (identity), a program using only skips works, a program using only repeats works, and a program using both works (shortest-valid-program lens). These four checks are independent, so a single question can count toward more than one bar and the four bars do not need to sum to 100%. One panel per model, by DART-Math difficulty, all five sharing the leftmost panel’s y-axis.
Figure 17: The per-model layer-position evidence behind \ftag F3 : the earliest layers are almost never skipped in any model, consistent with the early load-bearing region we find independently in fig. 2 . This panel shows the share of shortest-valid programs that skip each layer, by layer index and DART-Math difficulty. Qwen3-32B is omitted here since it already has its own dedicated figure in the main body ( fig. 6 ).
Figure 18: The repeat-side counterpart to fig. 17 , also part of the \ftag F3 layer-position evidence. Share of shortest-valid programs that repeat each layer, by layer index and DART-Math difficulty. Qwen3-32B again omitted ( fig. 6 in the main body).
pass@ 1
pass@ 2
pass@ 3
pass@ 4
pass@ 5
Identity
43.9%
–
–
–
–
Random baseline
17.0%±9.7%
25.9%±8.8%
36.1%±6.4%
40.4%±7.8%
45.3%±8.8%
Router (Drop-CE)
43.9%
–
–
–
62.6% (identical, all 5)
Appendix
Table 18: The Drop-CE router beats a random-valid-program control at pass@ 5 ( 62.6% vs. 45.3%±8.8% ), but identically across all 5 independently seeded retrainings, a structurally fixed margin, not per-question adaptation. Qwen3-8B, 2,800 -question held-out test split, same online-execution grading protocol both rows. “Random baseline” draws 5 shared valid programs from the router’s action grammar, independent of training, repeated over 5 seeds (mean ± std). Drop-CE router retrains that recipe’s winning hyperparameter configuration 5 independent times (weight-init/data-order seed only); all 5 runs give the identical figure.
Transformer inference studies how trained Transformer models produce outputs efficiently, accurately, and securely across hardware and context lengths.
Large language models (LLMs) perform inference by following a fixed depth and order, non-recurrent execution of all layers. We reveal the wide existence of training-free, flexible, dynamic program-of-layers (PoLar), where pretrained layers can be packed as modules and then skipped or looped to form a customized program for each input. For most inputs, substantially shorter program executions can achieve the same or better accuracy, while incorrect predictions of the original LLM can be corrected by alternative programs with fewer layers. These observations indicate that inference admits multiple valid latent computations beyond the standard forward pass. To efficiently achieve PoLar in practice, we propose a lightweight PoLar prediction network, which learns to generate execution programs that dynamically skip or repeat pretrained layers for each input. Experiments on mathematical reasoning benchmarks demonstrate that PoLar consistently improves accuracy over standard inference and prior dynamic-depth methods, often while executing fewer layers, and that these gains persist under out-of-distribution evaluation. Our results suggest that fixed-depth execution captures only a narrow subset of an LLM's latent reasoning capacity.
Ziyue Li, Yang Li, Tianyi Zhou
University of Maryland, College Park, MD, USA · 2MBZUAI, Abu Dhabi, UAE.
We introduce LayerRoute, a parameter-efficient method for adaptive transformer layer-skipping that combines per-layer hard-gated routing (trained via a straight-through estimator) with joint LoRA fine-tuning. LayerRoute augments each of the 24 transformer blocks in Qwen2.5-0.5B-Instruct with a lightweight per-layer router (~21.5K parameters) and LoRA adapters (rank 8, ~1.08M parameters), training both jointly under a gate-regularized language-modeling objective. Across 10 independently-seeded training runs, LayerRoute converges to an identical skip-pattern structure in every run - a consistent set of 9 middle layers (8-16) becomes skip-eligible in all 10 seeds - and delivers genuine, verified wallclock speedup in every run (1.02x-1.06x, mean 1.04x). Quality is preserved or improved in every configuration tested: joint LoRA adaptation yields a perplexity improvement over the unmodified backbone in all 10 seeds (mean delta = -1.16 and -1.11 across the two evaluation splits used). We further verify the router performs genuine, non-trivial per-input computation: gate decisions in skip-eligible layers change the actual skip/run outcome for 87-100% of held-out samples, confirming real input-dependent routing rather than a fixed pruning pattern. LayerRoute trains in under 7 minutes on a single A100 and adds negligible overhead beyond the routing decision itself. We report our full reproducibility methodology, including a systematic diagnostic investigation into what determines the router's per-input decisions, as part of this work.
Standard Large Language Models (LLMs) execute layers sequentially. Dynamic layer routing, i.e. search for a different execution path through layers involving layer repetitions, skips and other moves, can improve performance. Existing routing approaches often require updating model weights, running expensive search loops per test instance, or demand ground-truth labels during inference. In this work, we propose Markov Chain Routing of Transformer Layers (MACRO), a framework that learns task-specific routes over LLM architectures without modifying underlying parameters. MACRO models layer routing as a context-dependent Markov policy conditioned on layer indices, computation budget phases, directional displacements, and operator context, supporting skip, repeat, and residual hidden-state addition operations. The Markov route distribution is updated via feedback on training data and decoded using a top-k Viterbi algorithm to isolate high-probability candidate programs. We evaluate MACRO across diverse reasoning and knowledge benchmarks on multiple open-weight LLMs. MACRO achieves a +5.0% average accuracy improvement over the unrouted baselines, with largest gains on small models. We outperform the best dynamic routing approach Dr. LLM by +7.2%, while reducing route-search time 9.4x (from 14.8 to 1.6 hours). Our code is publicly available at https://github.com/Batorskq/MACRO.
Paweł Batorski, Abtin Pourhadi, Akylgali Aitaza +2
Heinrich Heine University Düsseldorf · Jagiellonian University · IDEAS Research Institute