Large language models serve applications with diverse inference budgets, but training and maintaining separate models for each budget incurs substantial costs. Recent Looped Transformers enable flexible inference computation through parameter reuse, but maintaining strong performance across different inference budgets remains an open challenge. We introduce Accordion, a mixture-of-experts-based model that supports five inference budgets through fixed-prefix layer looping. Each budget follows a pre-defined execution path, while the minimum budget retains the unlooped model. Joint pretraining samples one budget per micro-batch and optimizes its next-token prediction loss without additional cross-budget consistency objectives. We further preserve these inference budgets through supervised fine-tuning and reinforcement learning, where parameter-group merging and budget-matched multi-teacher on-policy distillation integrate three domain experts into one model. After 3.6T tokens of pretraining, increasing inference depth improves Accordion-6B-A0.5B's five-category macro-average across 37 tasks by 3.01 percentage points, with the middle budget capturing 89% of this gain using half as many additional layer executions. The final model surpasses each domain expert on its corresponding task-and-budget average, while achieving strong instruction-following performance across all budgets and retaining depth-dependent gains in mathematical reasoning and code generation.
Figures & tables
Figure 2: Comparison of eight layer-selection strategies. Accordion’s fixed prefix over layers 7–18 is highlighted in blue. Strategies are compared at matched inference budgets.
Model
Training budgets
k=0
k=6
k=12
Mean
Five-budget model
{0,3,6,9,12}
2.5793
2.5703
2.5696
2.5744
Three-budget model
{0,6,12}
2.5810
2.5729
2.5721
2.5766
Four-budget model
{0,4,8,12}
2.5814
–
2.5719
2.5767
Table 1: Comparison of training-budget sets. Validation losses are in nats; the mean uses the common endpoints k=0,12 . The four-budget model was not trained at k=6 . Bold marks the lowest loss in each column.
Routing
k=0
k=3
k=6
k=9
k=12
Mean
Shared router
2.5793
2.5731
2.5703
2.5698
2.5696
2.5724
Separate router
2.5814
2.5746
2.5715
2.5705
2.5702
2.5737
Separate router + exclusion
2.5823
2.5764
2.5718
2.5704
2.5694
2.5741
Table 2: Second-pass routing variants. Configurations are rows and inference budgets are columns. Validation losses are in nats; bold identifies the lowest loss within each column.
Configuration
k=0
k=3
k=6
k=9
k=12
Mean
Budget-independent gates and norms
2.5793
2.5731
2.5703
2.5698
2.5696
2.5724
Budget-conditioned gates
2.5814
2.5761
2.5731
2.5723
2.5721
2.5750
Budget-conditioned norm scales
2.5830
2.5769
2.5735
2.5724
2.5722
2.5756
Table 3: Comparison of budget-conditioned parameters. Each row identifies which parameter type is budget-conditioned; the other type remains budget-independent. Validation loss is in nats (lower is better). Bold marks the lowest loss within each column.
Figure 3: Shared models and budget-specific specialists. Accordion uses uniform budget sampling. Comparisons match training tokens (80B, left) or training FLOPs (right). Each specialist is trained for a single budget. The left panel also includes two controls with increased unlooped-path exposure, defined below.
Figure 4: Training pipeline for Accordion-6B-A0.5B. Accordion-Base is exported after 3.6T tokens at 8K context; another 50B tokens produce Accordion-Base-32K. SFT and RL continue from this extended model and retain all five budget paths. RL combines three domain experts through parameter merging and budget-matched multi-teacher distillation.
Figure 5: Pretraining data composition and curriculum. Estimated token consumption by domain and source across the 3.6T-token curriculum, followed by 50B-token context extension. Stage bars are normalized to their token budgets; labels are rounded billions of tokens. Exact source quantities appear in Table 11 .
Stage
Tokens (B)
Cumulative
Context
Learning rate
Warmup / optimizer
General
3,000
3.00T
8K
5×10−4 , constant
2,000 / initialize
Mid-training
150
3.15T
8K
5×10−4 , constant
0 / continue
Annealing
450
3.60T
8K
5×10−4→5×10−5
0 / continue
Context extension
50
3.65T
32K
5×10−5→1.5×10−5
200 / reset
Table 4: Pretraining hyperparameters by stage. All stages use 8,388,608 tokens per global batch on 512 H800 GPUs. Token budgets are nominal, warmup is measured in optimizer steps, and arrows indicate linear learning-rate decay after warmup.
Setting
Value
Layers / hidden size / Q / KV / head dim
24 / 1536 / 12 / 4 / 128
Routed / shared / active experts; intermediate size
128 / 1 / 8; 384
Total / active parameters; vocabulary
6.03B / 0.54B; 128,000
Loopable window / budgets
7–18 / {0,3,6,9,12}
Training tokens / context / tokens per step
3.6T / 8K / 8.39M
Peak LR / final LR / schedule
5×10−4 / 10% of peak / WSD
Table 5: Architecture and training configuration of Accordion-6B-A0.5B. Active parameters refer to a single unlooped forward pass and exclude the input embedding and output head. Training settings describe the 3.6T-token, 8K curriculum; Table 4 details the stage schedules and subsequent 32K context extension. WSD denotes warmup–stable–decay.
Table 6: Base-model comparison across all 37 benchmarks. All models follow the protocol in Appendix B.2 . Accordion-Base uses the 3.6T-token, 8K checkpoint. Category and overall scores appear in Figure 6 .
Figure 6: Overall and category-level performance of Accordion-Base. All five budgets and four references follow Appendix B.2 ; Ouro’s corrected HumanEval scores enter both Code and Overall. Category scores average all benchmarks in their corresponding table group; Overall equally weights the five category means, following Equation 7 . Blue bars denote Accordion-Base at k=0,3,6,9,12 ; gray bars denote Qwen3-0.6B, Qwen3-1.7B, OLMoE-1B-7B-0125 and Ouro-1.4B, from left to right. Panels use separate y-axis ranges; break marks indicate omitted zero baselines. Table 6 retains the complete benchmark results.
Figure 7: Quality and inference cost across six workloads. The quality panel uses the 37-task five-category aggregate versus time per output token; other panels report time to first token, batch decoding throughput and peak allocated GPU memory. Runtime uses Accordion-RL on one H800, averaging two nodes’ measurements; quality is evaluated separately. TTFT and throughput use logarithmic axes; memory uses separate batch-1 and batch-8 ranges. Measurement details appear in Table 44 .
Figure 8: Benchmark gains across inference budgets. Accordion-Base gains over k=0 , in percentage points, for all 37 benchmarks. The baseline column is identically zero and omitted. All four non-baseline budgets share one linear color scale; † marks report-only tasks. Absolute scores appear in Table 6 ; category means appear in Figure 6 .
Figure 9: Performance and budget gains across training. 18-task core aggregates through the 8K curriculum (left) and gains over each checkpoint’s k=0 score (right). Checkpoints are equally spaced, with cumulative token counts labeled; horizontal distance does not represent token exposure. Gains use rounded scores.
Figure 10: Long-context performance of Accordion-Base-32K on RULER. Scores average 13 tasks with 500 examples per task and context length. Gains over k=0 use task-stratified paired bootstrap 95% confidence intervals (10,000 resamples). Full scores and endpoint intervals appear in Table 23 .
Figure 11: SFT data composition. Category and source shares use packed input tokens after applying training epochs, totaling 71.734B; 68.5B carry training loss. Validation is excluded. B/M denote billion/million tokens. Exact source counts appear in Table 32 .
Table 7: Accordion-Instruct comparison with public post-trained models (%). Reference checkpoints (left to right): Qwen3-0.6B/1.7B, OLMoE-1B-7B-0125-SFT and Ouro-1.4B-Thinking. Bold/underlining identify best/second-best among the nine settings in each row, including ties; category-colored fills highlight Accordion’s first/second places. All models use the common sampling and scoring protocol with native chat templates; OLMoE uses 4K context and the others 32K. AIME averages 16 responses per problem; code uses pass@1.
Figure 12: SFT gains over the unlooped budget. Score differences from k=0 under the sampled conversational protocol. AIME averages 16 responses per problem; other tasks use one. Absolute scores and paired intervals appear in Table 34 .
Domain
Expert pool (prompts)
MOPD sample (prompts)
MOPD share
Mathematics
181,405
19,200
1/3
Code
21,679
19,200
1/3
Instruction following
80,291
19,200
1/3
Total
283,375
57,600
1
Table 8: Expert training pools and balanced MOPD sampling. Counts are prompts. MOPD samples equally from the three final expert pools; the student generates its own responses under domain- and budget-matched expert supervision.
Figure 13: Expert training improves validation scores; distillation reduces teacher–student discrepancy. Top row: expert validation with two responses per problem at T=0.6 (512 IF/math problems; 387 code problems). Bottom row: IF training reward, MOPD task reward and distillation loss. For IF training reward, solid curves are trailing five-step means over faint raw observations. Dotted lines mark the selected IF/math/code steps 130/30/32 and final MOPD step 10. Each curve represents one run.
Table 9: RL comparison with public post-trained models (%). Reference checkpoints (left to right): Qwen3-0.6B/1.7B, OLMoE-1B-7B-0125-SFT and Ouro-1.4B-Thinking. Bold/underlining identify best/second-best among the nine settings in each row, including ties; category-colored fills highlight Accordion’s first/second places. All models use the common sampling and scoring protocol with native chat templates; OLMoE uses 4K context and the others 32K. AIME averages 16 responses per problem; code uses pass@1.
Figure 14: SFT and RL performance across inference budgets. Accordion-Instruct and Accordion-RL use matched sampling and scoring on all ten tasks. Exact scores appear in Appendix Table 38 .
Figure 15: One student exceeds each domain expert. Blue bars identify the final Accordion-RL model after grouped merging and MOPD. Gains average task differences relative to each domain expert, then average over budgets; error bars are paired bootstrap 95% intervals. IF, code and mathematics contain two, three and four tasks; MMLU-Redux is excluded.
Figure 16: Input-driven expert switching contributes to prediction. Left, mean overlap under actual routing, input-only and bias-only controls; independent uniform top-eight draws from 128 experts have expected overlap 0.5. Center, token-level overlap distributions for all twelve layers. Right, fixing first-pass expert sets versus removing second-pass MoE writes. Counts exclude the shared expert. Curves show point estimates; definitions and selected intervals are in Appendix C .
Figure 17: Shared parameters produce distinct residual updates. Left, second-pass write norms relative to first-pass norms, with second-pass gates applied; the dashed line denotes equal magnitude. Right, cross-pass cosines of attention writes, MoE writes and residual streams. Residual streams remain closely aligned while attention and MoE writes change direction. Measurements average per-token quantities within each document and then across documents. Accordion-Base-32K, k=12 .
Adjacent layers
Spaced layers
Removed layers
7–9
10–12
13–15
16–18
7, 11, 15
8, 12, 16
9, 13, 17
10, 14, 18
Sum of singles
24.65
19.79
8.00
0.86
16.32
15.27
12.15
9.56
Joint effect
8.07
3.61
1.23
0.25
16.49
15.81
12.19
9.13
Joint / sum
0.33
0.18
0.15
0.29
1.01
1.03
1.00
0.96
Table 10: Removing adjacent repetitions together costs less than the sum of separate removals. All eight tested three-layer sets. “Sum of singles” adds NLL increases from three separate interventions; “Joint effect” measures one intervention removing all three layers’ second-pass attention and MoE writes. Values are in millinats/token relative to the intact k=12 model. First-pass execution remains intact. A ratio of one means the two quantities agree. Complete results and uncertainty estimates are in Appendix C.3 .
Appendix figures & tables39 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 18: Execution structures for depth reuse. Execution proceeds bottom to top. Accordion repeats a middle-layer prefix ( k=6 shown); Huginn repeats a core; Ouro repeats the full stack; Loopie repeats each MoE layer. R denotes recurrence count. Blue denotes single execution and cream/gold repeated execution. Peer layer counts are schematic.
Table 11: Source-level mixture for the 3.6T/8K stages. Available pools, expected consumption and repetitions use the documented rounded source quantities. General/mid/anneal columns each sum to 100%; consumed tokens sum to 3,600B. The separate 50B context-extension mixture is excluded from consumption and repetition estimates.
Metric
Web
Synthetic web
Reasoning
Mathematics
Code
Long documents (B tokens)
8
0
5
5
7
Replay (B tokens)
4
3
6.25
4.25
7.5
Share (%)
24
6
22.5
18.5
29
Appendix
Table 12: Context-extension data allocation. Long-document and replay allocations each total 25B tokens. Domain shares combine both components; replay allocations follow the annealing mixture.
Category
Tasks and prompting Scoring
Knowledge
MMLU ( Hendrycks et al., 2021a ) (5-shot) Multiple-choice accuracy TriviaQA ( Joshi et al., 2017 ) Natural Questions ( Kwiatkowski et al., 2019 ) (5-shot) Closed-book exact match
Common sense
ARC-C / ARC-E ( Clark et al., 2018 ) (25-/8-shot) CommonsenseQA ( Talmor et al., 2019 ) HellaSwag ( Zellers et al., 2019 ) (10-shot) WinoGrande ( Sakaguchi et al., 2019 ) (5-shot) OpenBookQA ( Mihaylov et al., 2018 ) BoolQ ( Clark et al., 2019 ) , PIQA ( Bisk et al., 2019 ) (0-shot) Task-specific harness scoring
Reasoning
BBH ( Suzgun et al., 2022 ) (3-shot CoT) Strict answer matching aggregate over subtasks
Mathematics
GSM8K ( Cobbe et al., 2021 ) (3-shot CoT) Flexible answer extraction Full MATH (4-shot) Mathematical-equivalence checking
Code
HumanEval ( Chen et al., 2021 ) / HumanEval+ ( Liu et al., 2023 ) (0-shot) MBPP ( Austin et al., 2021 ) / MBPP+ (3-shot) Greedy pass@1 CPU subprocess test execution
Appendix
Table 13: Core evaluation protocol for Accordion-Base and public references. All five budgets use identical tasks, prompts and scoring. Shot counts specify in-context examples; CoT denotes chain-of-thought prompting. Evaluation includes full MATH ( Hendrycks et al., 2021b ) with 5,000 problems and a separately reported MATH500 ( Lightman et al., 2023 ) score.
Summary
Tasks
Category scope
Core aggregate
18
Five: knowledge, common sense, reasoning, mathematics and code.
Secondary aggregate
34
Six: the core categories plus reading, with 16 additional tasks overall.
Full-table summary
37
Five display groups; knowledge and reading are combined under Knowledge and QA. Every displayed task is included.
Appendix
Table 14: Task and category scopes of the pretraining summaries. All three use the equal-category averaging rule in Equation 7 . The core aggregate is used for training trajectories.
Domain
Accordion-Base
k=0
k=3
k=6
k=9
k=12
Web
2.3059
2.2976
2.2935
2.2921
2.2919
Math
1.3505
1.3419
1.3379
1.3367
1.3366
Code
0.8693
0.8629
0.8596
0.8586
0.8584
Appendix
Table 15: Held-out validation loss of Accordion-Base. Lower is better. Each domain uses identical validation tokens across budgets. Values are batch-mean point estimates rounded to four decimal places; sequence-level records for paired intervals are unavailable. The endpoint reductions reported in § 8 use unrounded records. Bold and underlining mark the lowest and second-lowest loss in each row.
Setting
Value
Setting
Value
Sequences per domain
1,400
Expert parallelism degree
1
Tokens per sequence
8,192
Micro-batch / global batch (sequences)
4 / 28
Loss tokens per domain
11,468,800
Evaluation iterations
50
GPUs
7
Random seed
1234
Appendix
Table 16: Held-out loss evaluation settings. The same settings and validation sequences are used for all five inference budgets in each domain.
Table 17: Base-model comparison specifications. Parameter counts are in billions and retain source conventions. Active shared weights are not multiplied by recurrence count. Accordion’s active count includes its output head (0.54B without it) and describes an unlooped pass. Training exposure excludes subsequent SFT/RL.
Task
Gain
95% CI
Task
Gain
95% CI
MMLU
+2.63
[+2.02,+3.21]
BoolQ
+0.98
[+0.03,+1.96]
TriviaQA
+0.55
[+0.19,+0.91]
PIQA
+0.49
[−0.33,+1.31]
Natural Questions
+0.50
[−0.11,+1.14]
BBH
+6.01
[+5.01,+7.06]
ARC-C
+1.45
[+0.09,+2.73]
GSM8K
+8.26
[+6.07,+10.61]
ARC-E
+0.59
[−0.04,+1.26]
Full MATH
+2.64
[+1.54,+3.74]
CommonsenseQA
+3.52
[+1.80,+5.41]
HumanEval
+4.27
[−1.22,+9.76]
Appendix
Table 18: Paired endpoint gains on the core suite. Accordion-Base differences between k=12 and k=0 (percentage points), with paired bootstrap 95% confidence intervals reflecting evaluation-sample uncertainty.
Setting
Value
Layers / hidden size / Q / KV / head dim
24 / 768 / 6 / 2 / 128
Routed / shared / active experts; intermediate size
128 / 1 / 8; 192
Total / active parameters; vocabulary
1.61B / 0.14B; 129,280
Loopable window / budgets
7–18 / {0,3,6,9,12}
Training tokens / context / tokens per step
80B / 8K / 2.1M
Peak LR / final LR / schedule
7.07×10−4 / 1% of peak / WSD
Appendix
Table 19: Configuration for the 1.61B-parameter design studies. Active parameters refer to a single unlooped forward pass and exclude the input embedding and output head. WSD denotes warmup–stable–decay.
Accordion-Base-32K
Qwen3
OLMoE
Ouro
Category
k=0
k=3
k=6
k=9
k=12
0.6B
1.7B
0125
1.4B
Knowledge and QA
49.27
49.94
50.34
50.43
50.55
41.80
49.65
50.44
51.36
Common sense
65.28
66.18
66.61
66.56
66.62
57.00
65.33
66.60
67.67
Reasoning
39.13
40.60
42.30
42.43
42.57
33.90
43.37
31.20
48.23
Mathematics
40.25
41.83
43.40
43.85
43.66
42.41
53.18
30.53
50.30
Code
40.87
43.52
44.20
44.95
44.65
31.67
46.13
17.67
45.28
Appendix
Table 20: Category-level performance of Accordion-Base-32K. Categories average the displayed scores in Table 21 , including the Ouro correction. Reference columns follow the main-text order.
Table 21: Base-model comparison across all 37 benchmarks. All models follow the protocol in Appendix B.2 . Accordion-Base-32K uses 3.65T training tokens, including 50B for context extension. Category scores appear in Table 20 .
Domain
k=0
k=3
k=6
k=9
k=12
Web
2.3007
2.2923
2.2883
2.2869
2.2867
Math
1.3436
1.3347
1.3308
1.3296
1.3295
Code
0.8681
0.8618
0.8586
0.8576
0.8574
Appendix
Table 22: Accordion-Base-32K held-out loss. The 8K evaluation settings match Table 15 ; lower is better.
Context
k=0
k=3
k=6
k=9
k=12
Δ12 [95% CI]
4K
86.52
86.29
86.87
87.27
87.23
+0.72[0.24,1.20]
8K
78.15
79.18
79.44
79.97
79.95
+1.79[1.17,2.40]
16K
70.10
73.94
74.38
74.15
74.03
+3.93[3.16,4.66]
32K
49.49
51.80
54.58
54.90
54.95
+5.47[4.69,6.23]
Appendix
Table 23: RULER for Accordion-Base-32K. Scores average 13 tasks (%), using 500 examples per task and context length. The last column gives the paired k=0 to k=12 gain [95% CI] in percentage points. Bold and underline mark the highest and second-highest scores within each row.
4K context
Task
k=0
k=3
k=6
k=9
k=12
S1
100.00
100.00
100.00
100.00
100.00
S2
100.00
100.00
100.00
100.00
100.00
S3
97.20
93.20
95.00
95.40
95.40
MK1
95.60
96.60
96.40
96.60
96.00
MK2
99.40
99.80
100.00
100.00
100.00
Appendix
Table 24: Complete RULER task scores at 4K and 8K. Scores are percentages over 500 examples per task and budget. S: single-needle; MK: multi-key; MQ: multi-query; MV: multi-value; VT: variable tracking; CWE/FWE: common/frequent-word extraction; QA1/2: SQuAD/HotpotQA. Bold/underline mark the highest/second-highest scores per row, including ties.
16K context
Task
k=0
k=3
k=6
k=9
k=12
S1
100.00
100.00
100.00
100.00
100.00
S2
100.00
100.00
100.00
100.00
100.00
S3
97.80
97.80
97.40
97.60
97.80
MK1
69.20
72.20
77.80
78.80
79.20
MK2
86.80
91.60
93.80
92.80
93.20
Appendix
Table 25: Complete RULER task scores at 16K and 32K. The protocol, task abbreviations and ranking marks follow Table 24 . Each task and budget uses 500 examples.
Figure 19: Attention readouts change despite similar target selection. Left, cross-pass cosines before gating: normalized input ( X ), queries ( Q ), keys ( K ), values ( V ), attention-weighted values ( PV ) and output projection ( O ). Right, top-eight target overlap for the same head across passes (Same head), and within-pass controls using another head in the same or a different KV group (Same KV / Other KV), at query positions at least 64.
Edit / layer
7
8
9
10
11
12
Remove attention
1.86
3.11
2.71
2.34
2.45
1.98
Remove MoE
5.32
10.23
6.73
3.03
7.80
5.62
Remove both writes
7.94
9.07
7.64
7.28
6.86
5.66
Fix expert set
0.57
3.23
6.08
0.08
1.93
0.21
Fix set and weights
1.03
4.08
5.95
0.21
2.13
0.26
Attention, 0.5
0.32
0.50
0.54
0.54
0.53
0.25
Appendix
Table 26: Complete layerwise intervention effects. Δ NLL in millinats/token. “Both” always refers to second-pass attention and MoE writes, with first-pass execution preserved. The lower block of each panel gives scaling factors.
Measurement
Base-32K
Instruct
RL
Baseline NLL
1.5674
1.6974
1.7046
Remove all writes
13.38 [12.92, 13.87]
9.66 [8.92, 10.41]
7.22 [6.44, 7.97]
Remove all MoE
38.23 [37.28, 39.18]
43.04 [41.76, 44.30]
47.20 [45.87, 48.53]
Remove all attention
16.94 [16.38, 17.52]
29.79 [28.82, 30.70]
29.65 [28.68, 30.56]
Fix all expert sets
12.58 [12.10, 13.06]
17.98 [17.22, 18.71]
19.56 [18.78, 20.33]
Appendix
Table 27: Second-pass computation remains useful across training stages. Baseline NLL is in nats/token; intervention effects and their 95% document bootstrap intervals are in millinats/token. Each stage uses its own intact baseline on the same raw documents, without chat templates.
Removed layers
16–18
13–15
10–12
7–9
7, 11, 15
Joint effect
0.25
1.23
3.61
8.07
16.49
95% interval
[0.18, 0.32]
[1.09, 1.37]
[3.37, 3.84]
[7.72, 8.46]
[15.98, 17.04]
Sum of singles
0.86
8.00
19.79
24.65
16.32
Joint / sum
0.29
0.15
0.18
0.33
1.01
Ratio 95% interval
[0.23, 0.36]
[0.14, 0.17]
[0.17, 0.19]
[0.32, 0.34]
[1.00, 1.03]
Appendix
Table 28: Joint effects for every tested three-, six- and nine-layer set. Effects and intervals are in millinats/token. Sums use unrounded single-layer estimates. Ratio intervals resample the joint and single-layer effects on the same documents.
Removed layers
7, 8
8, 9
8, 11
9, 10
9, 11
10, 11
12, 13
Joint effect
8.78
12.91
16.60
14.45
16.56
6.70
8.61
95% interval
[8.43, 9.16]
[12.45, 13.43]
[16.07, 17.18]
[13.97, 14.98]
[16.04, 17.13]
[6.40, 7.01]
[8.28, 8.93]
Sum of singles
17.02
16.71
15.93
14.91
14.49
14.13
9.94
Joint / sum
0.52
0.77
1.04
0.97
1.14
0.47
0.87
Ratio 95% interval
[0.50, 0.53]
[0.76, 0.79]
[1.03, 1.05]
[0.95, 0.98]
[1.13, 1.16]
[0.46, 0.49]
[0.85, 0.88]
Appendix
Table 29: Pair effects depend on layer identity. Seven layer-pair interventions quantify variation in joint effects, with 95% document bootstrap intervals. Joint Δ NLL is in millinats/token.
Measurement
k=3
k=6
k=9
k=12
Long / short gain
1.15
1.29
1.44
1.44
95% interval
[0.97, 1.43]
[1.10, 1.57]
[1.23, 1.73]
[1.23, 1.73]
Appendix
Table 30: Length preference across all nonzero budgets. Long/short gain ratios for Accordion-Base-32K. Long bins end at 512–4,096 tokens; short bins at 32–256 tokens. The ratio compares unweighted means of the four bin-level gains in each group. Intervals are 95% document bootstrap intervals.
Figure 20: Depth gains across input lengths and positions. Accordion-Base-32K, k=12 versus k=0 . All nine bins are shown; whiskers give 95% document bootstrap intervals. The final bin ends at 8,191 tokens.
Metric
k=0
k=3
k=6
k=9
k=12
Exact match (%)
8.50
10.00
11.75
12.00
12.00
Answer NLL change
0.000
-0.224
-0.259
-0.272
-0.259
Appendix
Table 31: Depth improves bracket completion with long demonstrations. All five budgets on the same 400 prompts with 32 demonstrations. Exact match is teacher-forced, and answer NLL changes are nats per answer relative to k=0 . At all tested positive demonstration counts below 32, paired exact-match gain intervals include zero.
Category
Dataset / retained subset
Single pass (B)
Epochs
Training (B)
Nemotron-SFT-Math-v3
18.098
1
18.098
OpenMathReasoning (CoT)
23.592
1
23.592
Mathematics
OpenR1-Math-220k
1.028
1
1.028
Nemotron-SFT-Comp.-Prog.-v2 (Python)
1.985
1
1.985
Nemotron-SFT-Comp.-Prog.-v2 (C++)
2.014
1
2.014
AceReason-1.1-SFT (code)
9.529
1
9.529
Appendix
Table 32: SFT training data by source. All 12 packed training components, with input tokens in billions before and after applying the indicated epochs. Category shading identifies mathematics, code, and instruction/other data. Validation is excluded; the 68.5B loss-bearing tokens are a subset of the 71.734B training input tokens. Totals use unrounded counts.
Tasks
Maximum output tokens
GSM8K, IFEval, IFBench, MMLU-Redux
8,192
MATH500, HumanEval+, MBPP+
16,384
AIME 2024, AIME 2025, LiveCodeBench
30,000
Appendix
Table 33: Output-token limits for the common post-training evaluation. The matched SFT and RL comparison uses the same limits. Public references request these limits, capped by their remaining context space as specified in Appendix D.4 .
Task
k=0
k=3
k=6
k=9
k=12
Δ12
95% CI
Greedy (one response per problem)
GSM8K
75.82
79.76
81.20
82.26
81.27
+5.46
[+3.26,+7.58]
MATH500
69.40
76.60
77.60
78.40
82.20
+12.80
[+8.80,+16.60]
HumanEval+
65.24
63.41
70.12
68.29
65.85
+0.61
[−6.10,+6.71]
MBPP+
60.05
62.17
66.40
66.14
66.67
+6.61
[+2.38,+10.85]
LiveCodeBench
18.94
23.79
27.97
26.65
26.21
+7.27
[+4.19,+10.35]
Appendix
Table 34: SFT decoding policies across all five budgets. vLLM scores (%), endpoint differences Δ12 and paired 95% intervals (percentage points). Differences are calculated before rounding. Intervals use NumPy default_rng(0) ; the resample count is unrecorded in the archived evaluation. Sampled scores serve as the SFT baseline in Table 38 ; the greedy comparison tests depth gains across decoding policies. AIME has one 16-response sampled evaluation. Ranks compare the five budgets within each task and decoding-policy row.
Figure 21: SFT budget response under greedy and sampled decoding. Sampling uses T=0.6 , top- p=0.95 and one response per problem. Output limits are held fixed within each task. Complete scores and paired intervals appear in Table 34 .
Domain
Public dataset / subset
Prompts
Instruction following
Nemotron-Cascade-RL-Instruction-Following
98,092
Nemotron-RL-instruction_following
45,079
Nemotron-RL instruction following (structured outputs)
9,437
Mathematics
Nemotron-RL-math-OpenMathReasoning
81,277
OpenR1-Math-220k
59,703
AceReason-Math
46,280
Appendix
Table 35: Public sources for RL. Counts describe the deduplicated and decontaminated source pools before difficulty and code-verifier filtering. The final expert pool sizes are reported in the text. Mathematics counts sum to 207,156, IF to 152,608 and code to 22,016. Dataset names link to their public releases where separately identifiable.
Expert
Selected step
Validation score (%)
Stopping step
Entropy at stopping
Entropy threshold
IF
130
81.11
133
1.811
1.8
Mathematics
30
47.89
45
1.533
1.5
Code
32
70.85
35
1.527
1.5
Appendix
Table 36: Expert selection and entropy-based stopping. Steps count domain-specific rollout/training iterations, each containing up to five per-budget optimizer updates; hardware allocations are reported below. The learning-rate scheduler advances once per training step. Validation scores average all five budgets. Training stops when entropy exceeds the domain-specific threshold. The selected experts are merged to initialize the student, selected after ten distillation training steps.
Quantity
Value or change
IF mean training reward, first 50 recorded steps
≈0.47→0.72
Mathematics validation gain over initial evaluation (pp)
+1.11
Code validation gain over initial evaluation (pp)
+6.59
MOPD task reward, ten-step run
0.466→0.649
MOPD token distillation loss, ten-step run
0.0708→0.0074
Mathematics validation at step 40 (%)
48.40
Appendix
Table 37: Expert learning, distillation progress and selection statistics. Expert validation averages the five budgets. The ten-step distillation objective uses teacher supervision; task reward is monitored separately. The lower block reports the paired mathematics checkpoint-selection comparison.
Table 38: Complete matched SFT and RL evaluation. Benchmark cells give Accordion-Instruct / Accordion-RL scores (%). Mean Δ rows average the displayed RL − SFT differences equally over tasks within each group, in percentage points. Bold and underlining identify the best and second-best of the ten model–budget scores per benchmark; group changes are not ranked.
Figure 22: Response completion and conditional accuracy on LiveCodeBench. RL and SFT use matched generation settings. Conditional scores characterize each model’s own nontruncated subset, with RL − SFT differences of −14.83 to −5.18 points across these different subsets. Overall accuracy in Table 38 uses the full paired problem set.
Task
Δ12
95% CI
Task
Δ12
95% CI
GSM8K
+5.00
[3.18,6.82]
HumanEval+
+4.27
–
MATH500
+6.00
[3.00,9.00]
MBPP+
+4.23
[0.53,7.94]
AIME 2024
+10.63
–
LiveCodeBench
+6.61
[3.08,10.35]
AIME 2025
+8.75
–
MMLU-Redux
+3.79
[2.57,4.99]
Appendix
Table 39: Accordion-RL gains from additional depth. Differences compare k=12 with k=0 , in percentage points, and are calculated before rounding. Paired 95% percentile intervals use 2,000 problem-level resamples with Python random.Random(0) ; a dash indicates that no interval is reported here.
Parameter group
Tensors
IF
Code
Math
Attention projections and attention normalization; shared MoE experts; loop gates; layer normalization, including the final norm
289
1
0.5
0.5
Routed expert projections; routers; embeddings; language-model head
74
1
1
1
Frozen balancing biases
48
Copied from SFT
Total
411
–
–
–
Appendix
Table 40: Parameter groups in the selected RL merge. Coefficients multiply expert deltas relative to SFT. The first row defines the 289 shared-component tensors referred to in the merge comparisons. Frozen balancing biases are copied, not merged.
Variant
Instruction following
Code
Mathematics
IF + code
+1.76 [+0.35,+3.11]
+0.47 [−0.77,+1.62]
–
IF + math
+1.93 [+0.51,+3.28]
+2.03 [+0.83,+3.21]
−0.52 [−1.34,+0.32]
Code + math
−24.67 [−26.44,−22.91]
+1.97 [+0.85,+3.13]
−0.06 [−0.84,+0.69]
Unit weights
+1.85 [+0.47,+3.21]
+0.44 [−0.76,+1.66]
−4.92 [−5.95,−3.93]
Math 0.5 throughout
+2.69 [+1.31,+4.11]
+1.17 [+0.12,+2.27]
−2.24 [−3.03,−1.44]
Shared code/math 0.5
+2.36 [+0.90,+3.79]
+1.78 [+0.64,+2.98]
−0.93 [−1.75,−0.10]
Appendix
Table 41: Merge variants relative to the corresponding domain expert. Entries are equal-task, equal-budget mean gains [paired 95% intervals], in percentage points. Omitted domains in pairwise merges have coefficient zero; unspecified included coefficients are one. “Shared” denotes the 289 tensors in Table 40 . A mathematics-domain interval is unavailable for the IF–code merge.
Comparison
Mean gain
95% CI
Student versus IF expert, instruction following
+1.65
[0.30,3.03]
Student versus mathematics expert, mathematics
+1.37
[0.65,2.11]
Student versus code expert, code
+2.04
[0.82,3.21]
Student versus grouped merge, mathematics
+2.30
[1.46,3.14]
Appendix
Table 42: Domain-level gains after merging and distillation. Differences are percentage points, averaging tasks equally within each budget and then averaging the five budgets. Intervals use paired problem-bootstrap resampling.
Setting
Value
Setting
Value
Training pool (prompts)
57,600
Warmup (training steps)
10
Prompts per rollout batch
576
Peak learning rate
10−6
Responses per prompt
8
Student nodes
36
Responses per rollout batch
4,608
Additional teacher nodes
4
Max. prompt tokens
2,048
GPUs per node
8 H800
Max. response tokens
28,672
Teacher inference instances
32
Appendix
Table 43: Multi-teacher on-policy distillation settings. The student generates its own responses, and each domain expert evaluates them at the student’s inference budget.
Calculated model costs at 8K
k=0
k=3
k=6
k=9
k=12
Layer executions
24
27
30
33
36
Rel. train FLOPs
1.000
1.101
1.202
1.303
1.404
KV/token (KiB)
48
54
60
66
72
Appendix
Table 44: Calculated model costs and measured inference cost. The first block is analytical. In the runtime block, each metric cell gives batch 1 / batch 8 , measured on one H800 with 128 output tokens per sequence. Throughput counts tokens across the batch; latency covers the batch. Prompt lengths group all five budgets, with 32,640 prompt tokens per sequence for the 32K workload. Measurement settings appear above.
Large language models (LLMs) incur high inference cost due to their depth and parameter scale. Depth pruning can reduce latency by skipping redundant Transformer blocks, but existing methods (i) provide limited control under user-specific compute budgets and (ii) typically fix the routing path, failing to adapt as the context grows during decoding. We propose Buddy, a budget-driven dynamic depth routing framework. Buddy uses a lightweight Decision Module to score intermediate layers conditioned on the input and deterministically executes the top-k layers to satisfy a given budget. To support decode-time adaptation, Buddy reuses the first-layer KV cache as a low-overhead global context source and pools it together with the newest token representation before each routing decision. When no explicit budget is provided, an optional Budget Predictor estimates an input-dependent compute level to balance quality and efficiency. Experiments on Llama-family and Qwen models show that Buddy is competitive with strong static pruning baselines and often improves the accuracy-compute trade-off, while uniquely supporting strict budget control, decode-time rerouting, and multiple budgets within a single trained model.
Yuhua Zhou, Shaoqi Yu, Shichao Weng +4
College of Computer Science and Technology, Zhejiang University · Zhejiang Lab · Ant Group +1
A main promise of looped language models is depth-adaptive inference. By looping a block of shared layers a variable number of times, the model can use less compute for "easy" tokens and more for "hard" ones. However, tokens with different numbers of loops cannot share a uniform forward pass and therefore cannot be handled by standard batching systems such as vLLM. The practical value of depth-adaptive inference thus hinges on whether batching can be made efficient. We introduce the first efficient method for depth-adaptive looped LMs via continuous depth batching (CDB), which forms new batches between loop steps. Our method dynamically schedules looped and non-looped parts of the architecture, manages looped KV-caching, and predicts which tokens will exit the loop in advance so it can prepare batches asynchronously. Experiments on Ouro 1.4B and Huginn 3.5B show that fully looped architectures are best suited to depth-adaptive inference, as large non-looped layers outside the recurrent core (e.g., token embedding, LM head, and unshared transformer blocks) slow down and complicate scheduling. Overall, CDB realizes up to 99% of the estimated maximum speedup available, leaving further gains primarily dependent on model architecture and exit behavior.
Kristian Schwethelm, Daniel Rueckert, Georgios Kaissis
Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM), Germany · Department of Computing, Imperial College London, UK · Munich Center for Machine Learning (MCML), Germany +1
Large language models (LLMs) perform inference by following a fixed depth and order, non-recurrent execution of all layers. We reveal the wide existence of training-free, flexible, dynamic program-of-layers (PoLar), where pretrained layers can be packed as modules and then skipped or looped to form a customized program for each input. For most inputs, substantially shorter program executions can achieve the same or better accuracy, while incorrect predictions of the original LLM can be corrected by alternative programs with fewer layers. These observations indicate that inference admits multiple valid latent computations beyond the standard forward pass. To efficiently achieve PoLar in practice, we propose a lightweight PoLar prediction network, which learns to generate execution programs that dynamically skip or repeat pretrained layers for each input. Experiments on mathematical reasoning benchmarks demonstrate that PoLar consistently improves accuracy over standard inference and prior dynamic-depth methods, often while executing fewer layers, and that these gains persist under out-of-distribution evaluation. Our results suggest that fixed-depth execution captures only a narrow subset of an LLM's latent reasoning capacity.
Ziyue Li, Yang Li, Tianyi Zhou
University of Maryland, College Park, MD, USA · 2MBZUAI, Abu Dhabi, UAE.