cs.LGOct 23, 2026
SaveAccordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping
Organizations: Tsinghua University · Baidu Inc.
Abstract
Large language models serve applications with diverse inference budgets, but training and maintaining separate models for each budget incurs substantial costs. Recent Looped Transformers enable flexible inference computation through parameter reuse, but maintaining strong performance across different inference budgets remains an open challenge. We introduce Accordion, a mixture-of-experts-based model that supports five inference budgets through fixed-prefix layer looping. Each budget follows a pre-defined execution path, while the minimum budget retains the unlooped model. Joint pretraining samples one budget per micro-batch and optimizes its next-token prediction loss without additional cross-budget consistency objectives. We further preserve these inference budgets through supervised fine-tuning and reinforcement learning, where parameter-group merging and budget-matched multi-teacher on-policy distillation integrate three domain experts into one model. After 3.6T tokens of pretraining, increasing inference depth improves Accordion-6B-A0.5B's five-category macro-average across 37 tasks by 3.01 percentage points, with the middle budget capturing 89% of this gain using half as many additional layer executions. The final model surpasses each domain expert on its corresponding task-and-budget average, while achieving strong instruction-following performance across all budgets and retaining depth-dependent gains in mathematical reasoning and code generation.
Figures & tables
Figure 2: Comparison of eight layer-selection strategies. Accordion’s fixed prefix over layers 7–18 is highlighted in blue. Strategies are compared at matched inference budgets.
| Model | Training budgets | Mean | |||
| Five-budget model | 2.5793 | 2.5703 | 2.5696 | 2.5744 | |
| Three-budget model | 2.5810 | 2.5729 | 2.5721 | 2.5766 | |
| Four-budget model | 2.5814 | – | 2.5719 | 2.5767 |
Table 1: Comparison of training-budget sets. Validation losses are in nats; the mean uses the common endpoints . The four-budget model was not trained at . Bold marks the lowest loss in each column.
| Routing | Mean | |||||
| Shared router | 2.5793 | 2.5731 | 2.5703 | 2.5698 | 2.5696 | 2.5724 |
| Separate router | 2.5814 | 2.5746 | 2.5715 | 2.5705 | 2.5702 | 2.5737 |
| Separate router + exclusion | 2.5823 | 2.5764 | 2.5718 | 2.5704 | 2.5694 | 2.5741 |
Table 2: Second-pass routing variants. Configurations are rows and inference budgets are columns. Validation losses are in nats; bold identifies the lowest loss within each column.
| Configuration | Mean | |||||
| Budget-independent gates and norms | 2.5793 | 2.5731 | 2.5703 | 2.5698 | 2.5696 | 2.5724 |
| Budget-conditioned gates | 2.5814 | 2.5761 | 2.5731 | 2.5723 | 2.5721 | 2.5750 |
| Budget-conditioned norm scales | 2.5830 | 2.5769 | 2.5735 | 2.5724 | 2.5722 | 2.5756 |
Table 3: Comparison of budget-conditioned parameters. Each row identifies which parameter type is budget-conditioned; the other type remains budget-independent. Validation loss is in nats (lower is better). Bold marks the lowest loss within each column.
Figure 3: Shared models and budget-specific specialists. Accordion uses uniform budget sampling. Comparisons match training tokens (80B, left) or training FLOPs (right). Each specialist is trained for a single budget. The left panel also includes two controls with increased unlooped-path exposure, defined below.
Figure 4: Training pipeline for Accordion-6B-A0.5B. Accordion-Base is exported after 3.6T tokens at 8K context; another 50B tokens produce Accordion-Base-32K. SFT and RL continue from this extended model and retain all five budget paths. RL combines three domain experts through parameter merging and budget-matched multi-teacher distillation.
Figure 5: Pretraining data composition and curriculum. Estimated token consumption by domain and source across the 3.6T-token curriculum, followed by 50B-token context extension. Stage bars are normalized to their token budgets; labels are rounded billions of tokens. Exact source quantities appear in Table 11 .
| Stage | Tokens (B) | Cumulative | Context | Learning rate | Warmup / optimizer |
| General | 3,000 | 3.00T | 8K | , constant | 2,000 / initialize |
| Mid-training | 150 | 3.15T | 8K | , constant | 0 / continue |
| Annealing | 450 | 3.60T | 8K | 0 / continue | |
| Context extension | 50 | 3.65T | 32K | 200 / reset |
Table 4: Pretraining hyperparameters by stage. All stages use 8,388,608 tokens per global batch on 512 H800 GPUs. Token budgets are nominal, warmup is measured in optimizer steps, and arrows indicate linear learning-rate decay after warmup.
| Setting | Value |
| Layers / hidden size / Q / KV / head dim | 24 / 1536 / 12 / 4 / 128 |
| Routed / shared / active experts; intermediate size | 128 / 1 / 8; 384 |
| Total / active parameters; vocabulary | 6.03B / 0.54B; 128,000 |
| Loopable window / budgets | 7–18 / |
| Training tokens / context / tokens per step | 3.6T / 8K / 8.39M |
| Peak LR / final LR / schedule | / 10% of peak / WSD |
Table 5: Architecture and training configuration of Accordion-6B-A0.5B. Active parameters refer to a single unlooped forward pass and exclude the input embedding and output head. Training settings describe the 3.6T-token, 8K curriculum; Table 4 details the stage schedules and subsequent 32K context extension. WSD denotes warmup–stable–decay.
Table 6: Base-model comparison across all 37 benchmarks. All models follow the protocol in Appendix B.2 . Accordion-Base uses the 3.6T-token, 8K checkpoint. Category and overall scores appear in Figure 6 .
Figure 6: Overall and category-level performance of Accordion-Base. All five budgets and four references follow Appendix B.2 ; Ouro’s corrected HumanEval scores enter both Code and Overall. Category scores average all benchmarks in their corresponding table group; Overall equally weights the five category means, following Equation 7 . Blue bars denote Accordion-Base at ; gray bars denote Qwen3-0.6B, Qwen3-1.7B, OLMoE-1B-7B-0125 and Ouro-1.4B, from left to right. Panels use separate y-axis ranges; break marks indicate omitted zero baselines. Table 6 retains the complete benchmark results.
Figure 7: Quality and inference cost across six workloads. The quality panel uses the 37-task five-category aggregate versus time per output token; other panels report time to first token, batch decoding throughput and peak allocated GPU memory. Runtime uses Accordion-RL on one H800, averaging two nodes’ measurements; quality is evaluated separately. TTFT and throughput use logarithmic axes; memory uses separate batch-1 and batch-8 ranges. Measurement details appear in Table 44 .
Figure 8: Benchmark gains across inference budgets. Accordion-Base gains over , in percentage points, for all 37 benchmarks. The baseline column is identically zero and omitted. All four non-baseline budgets share one linear color scale; marks report-only tasks. Absolute scores appear in Table 6 ; category means appear in Figure 6 .
Figure 9: Performance and budget gains across training. 18-task core aggregates through the 8K curriculum (left) and gains over each checkpoint’s score (right). Checkpoints are equally spaced, with cumulative token counts labeled; horizontal distance does not represent token exposure. Gains use rounded scores.
Figure 10: Long-context performance of Accordion-Base-32K on RULER. Scores average 13 tasks with 500 examples per task and context length. Gains over use task-stratified paired bootstrap 95% confidence intervals (10,000 resamples). Full scores and endpoint intervals appear in Table 23 .
Figure 11: SFT data composition. Category and source shares use packed input tokens after applying training epochs, totaling 71.734B; 68.5B carry training loss. Validation is excluded. B/M denote billion/million tokens. Exact source counts appear in Table 32 .
Table 7: Accordion-Instruct comparison with public post-trained models (%). Reference checkpoints (left to right): Qwen3-0.6B/1.7B, OLMoE-1B-7B-0125-SFT and Ouro-1.4B-Thinking. Bold/underlining identify best/second-best among the nine settings in each row, including ties; category-colored fills highlight Accordion’s first/second places. All models use the common sampling and scoring protocol with native chat templates; OLMoE uses 4K context and the others 32K. AIME averages 16 responses per problem; code uses pass@1.
Figure 12: SFT gains over the unlooped budget. Score differences from under the sampled conversational protocol. AIME averages 16 responses per problem; other tasks use one. Absolute scores and paired intervals appear in Table 34 .
| Domain | Expert pool (prompts) | MOPD sample (prompts) | MOPD share |
| Mathematics | 181,405 | 19,200 | |
| Code | 21,679 | 19,200 | |
| Instruction following | 80,291 | 19,200 | |
| Total | 283,375 | 57,600 |
Table 8: Expert training pools and balanced MOPD sampling. Counts are prompts. MOPD samples equally from the three final expert pools; the student generates its own responses under domain- and budget-matched expert supervision.
Figure 13: Expert training improves validation scores; distillation reduces teacher–student discrepancy. Top row: expert validation with two responses per problem at (512 IF/math problems; 387 code problems). Bottom row: IF training reward, MOPD task reward and distillation loss. For IF training reward, solid curves are trailing five-step means over faint raw observations. Dotted lines mark the selected IF/math/code steps 130/30/32 and final MOPD step 10. Each curve represents one run.
Table 9: RL comparison with public post-trained models (%). Reference checkpoints (left to right): Qwen3-0.6B/1.7B, OLMoE-1B-7B-0125-SFT and Ouro-1.4B-Thinking. Bold/underlining identify best/second-best among the nine settings in each row, including ties; category-colored fills highlight Accordion’s first/second places. All models use the common sampling and scoring protocol with native chat templates; OLMoE uses 4K context and the others 32K. AIME averages 16 responses per problem; code uses pass@1.
Figure 14: SFT and RL performance across inference budgets. Accordion-Instruct and Accordion-RL use matched sampling and scoring on all ten tasks. Exact scores appear in Appendix Table 38 .
Figure 15: One student exceeds each domain expert. Blue bars identify the final Accordion-RL model after grouped merging and MOPD. Gains average task differences relative to each domain expert, then average over budgets; error bars are paired bootstrap 95% intervals. IF, code and mathematics contain two, three and four tasks; MMLU-Redux is excluded.
Figure 16: Input-driven expert switching contributes to prediction. Left, mean overlap under actual routing, input-only and bias-only controls; independent uniform top-eight draws from 128 experts have expected overlap 0.5. Center, token-level overlap distributions for all twelve layers. Right, fixing first-pass expert sets versus removing second-pass MoE writes. Counts exclude the shared expert. Curves show point estimates; definitions and selected intervals are in Appendix C .
Figure 17: Shared parameters produce distinct residual updates. Left, second-pass write norms relative to first-pass norms, with second-pass gates applied; the dashed line denotes equal magnitude. Right, cross-pass cosines of attention writes, MoE writes and residual streams. Residual streams remain closely aligned while attention and MoE writes change direction. Measurements average per-token quantities within each document and then across documents. Accordion-Base-32K, .
| Adjacent layers | Spaced layers | |||||||
| Removed layers | 7–9 | 10–12 | 13–15 | 16–18 | 7, 11, 15 | 8, 12, 16 | 9, 13, 17 | 10, 14, 18 |
| Sum of singles | 24.65 | 19.79 | 8.00 | 0.86 | 16.32 | 15.27 | 12.15 | 9.56 |
| Joint effect | 8.07 | 3.61 | 1.23 | 0.25 | 16.49 | 15.81 | 12.19 | 9.13 |
| Joint / sum | 0.33 | 0.18 | 0.15 | 0.29 | 1.01 | 1.03 | 1.00 | 0.96 |
Table 10: Removing adjacent repetitions together costs less than the sum of separate removals. All eight tested three-layer sets. “Sum of singles” adds NLL increases from three separate interventions; “Joint effect” measures one intervention removing all three layers’ second-pass attention and MoE writes. Values are in millinats/token relative to the intact model. First-pass execution remains intact. A ratio of one means the two quantities agree. Complete results and uncertainty estimates are in Appendix C.3 .
Appendix figures & tables39 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 18: Execution structures for depth reuse. Execution proceeds bottom to top. Accordion repeats a middle-layer prefix ( shown); Huginn repeats a core; Ouro repeats the full stack; Loopie repeats each MoE layer. denotes recurrence count. Blue denotes single execution and cream/gold repeated execution. Peer layer counts are schematic.
| Source | Available (B tokens) | General 3,000B | Mid 150B | Anneal 450B | Expected used (B) | Expected repetitions |
| Nemotron-CC web, high quality (v2/v2.1) | 1,243 | 23% | 17% | 13% | 774 | 0.62 |
| Nemotron-CC web, medium quality (v2/v2.1) | 2,161 | 15.25% | 7% | 3% | 481.5 | 0.22 |
| Nemotron-CC synthetic rephrasings | 3,503 | 22% | 17% | 12% | 739.5 | 0.21 |
| Nemotron-CC Diverse-QA + HQ-DQA | 679 | 6.5% | 7% | 5% | 228 | 0.34 |
| Nemotron Specialized (RQA, STEM-SFT, InfiniByte) | 228 | 2.5% | 5% | 8% | 118.5 | 0.52 |
| Nemotron reasoning (Math-v2, Comp.-Prog., PrismMath) | 102 | 0% | 6% | 12% | 63 | 0.62 |
Table 11: Source-level mixture for the 3.6T/8K stages. Available pools, expected consumption and repetitions use the documented rounded source quantities. General/mid/anneal columns each sum to 100%; consumed tokens sum to 3,600B. The separate 50B context-extension mixture is excluded from consumption and repetition estimates.
| Metric | Web | Synthetic web | Reasoning | Mathematics | Code |
| Long documents (B tokens) | 8 | 0 | 5 | 5 | 7 |
| Replay (B tokens) | 4 | 3 | 6.25 | 4.25 | 7.5 |
| Share (%) | 24 | 6 | 22.5 | 18.5 | 29 |
Table 12: Context-extension data allocation. Long-document and replay allocations each total 25B tokens. Domain shares combine both components; replay allocations follow the annealing mixture.
| Category | Tasks and prompting Scoring |
| Knowledge | MMLU ( Hendrycks et al., 2021a ) (5-shot) Multiple-choice accuracy TriviaQA ( Joshi et al., 2017 ) Natural Questions ( Kwiatkowski et al., 2019 ) (5-shot) Closed-book exact match |
| Common sense | ARC-C / ARC-E ( Clark et al., 2018 ) (25-/8-shot) CommonsenseQA ( Talmor et al., 2019 ) HellaSwag ( Zellers et al., 2019 ) (10-shot) WinoGrande ( Sakaguchi et al., 2019 ) (5-shot) OpenBookQA ( Mihaylov et al., 2018 ) BoolQ ( Clark et al., 2019 ) , PIQA ( Bisk et al., 2019 ) (0-shot) Task-specific harness scoring |
| Reasoning | BBH ( Suzgun et al., 2022 ) (3-shot CoT) Strict answer matching aggregate over subtasks |
| Mathematics | GSM8K ( Cobbe et al., 2021 ) (3-shot CoT) Flexible answer extraction Full MATH (4-shot) Mathematical-equivalence checking |
| Code | HumanEval ( Chen et al., 2021 ) / HumanEval+ ( Liu et al., 2023 ) (0-shot) MBPP ( Austin et al., 2021 ) / MBPP+ (3-shot) Greedy pass@1 CPU subprocess test execution |
Table 13: Core evaluation protocol for Accordion-Base and public references. All five budgets use identical tasks, prompts and scoring. Shot counts specify in-context examples; CoT denotes chain-of-thought prompting. Evaluation includes full MATH ( Hendrycks et al., 2021b ) with 5,000 problems and a separately reported MATH500 ( Lightman et al., 2023 ) score.
| Summary | Tasks | Category scope |
| Core aggregate | 18 | Five: knowledge, common sense, reasoning, mathematics and code. |
| Secondary aggregate | 34 | Six: the core categories plus reading, with 16 additional tasks overall. |
| Full-table summary | 37 | Five display groups; knowledge and reading are combined under Knowledge and QA. Every displayed task is included. |
Table 14: Task and category scopes of the pretraining summaries. All three use the equal-category averaging rule in Equation 7 . The core aggregate is used for training trajectories.
| Domain | Accordion-Base | ||||
| Web | 2.3059 | 2.2976 | 2.2935 | 2.2921 | 2.2919 |
| Math | 1.3505 | 1.3419 | 1.3379 | 1.3367 | 1.3366 |
| Code | 0.8693 | 0.8629 | 0.8596 | 0.8586 | 0.8584 |
Table 15: Held-out validation loss of Accordion-Base. Lower is better. Each domain uses identical validation tokens across budgets. Values are batch-mean point estimates rounded to four decimal places; sequence-level records for paired intervals are unavailable. The endpoint reductions reported in § 8 use unrounded records. Bold and underlining mark the lowest and second-lowest loss in each row.
| Setting | Value | Setting | Value |
| Sequences per domain | 1,400 | Expert parallelism degree | 1 |
| Tokens per sequence | 8,192 | Micro-batch / global batch (sequences) | 4 / 28 |
| Loss tokens per domain | 11,468,800 | Evaluation iterations | 50 |
| GPUs | 7 | Random seed | 1234 |
Table 16: Held-out loss evaluation settings. The same settings and validation sequences are used for all five inference budgets in each domain.
Table 17: Base-model comparison specifications. Parameter counts are in billions and retain source conventions. Active shared weights are not multiplied by recurrence count. Accordion’s active count includes its output head (0.54B without it) and describes an unlooped pass. Training exposure excludes subsequent SFT/RL.
| Task | Gain | 95% CI | Task | Gain | 95% CI |
| MMLU | BoolQ | ||||
| TriviaQA | PIQA | ||||
| Natural Questions | BBH | ||||
| ARC-C | GSM8K | ||||
| ARC-E | Full MATH | ||||
| CommonsenseQA | HumanEval |
Table 18: Paired endpoint gains on the core suite. Accordion-Base differences between and (percentage points), with paired bootstrap 95% confidence intervals reflecting evaluation-sample uncertainty.
| Setting | Value |
| Layers / hidden size / Q / KV / head dim | 24 / 768 / 6 / 2 / 128 |
| Routed / shared / active experts; intermediate size | 128 / 1 / 8; 192 |
| Total / active parameters; vocabulary | 1.61B / 0.14B; 129,280 |
| Loopable window / budgets | 7–18 / |
| Training tokens / context / tokens per step | 80B / 8K / 2.1M |
| Peak LR / final LR / schedule | / 1% of peak / WSD |
Table 19: Configuration for the 1.61B-parameter design studies. Active parameters refer to a single unlooped forward pass and exclude the input embedding and output head. WSD denotes warmup–stable–decay.
| Accordion-Base-32K | Qwen3 | OLMoE | Ouro | ||||||
| Category | 0.6B | 1.7B | 0125 | 1.4B | |||||
| Knowledge and QA | 49.27 | 49.94 | 50.34 | 50.43 | 50.55 | 41.80 | 49.65 | 50.44 | 51.36 |
| Common sense | 65.28 | 66.18 | 66.61 | 66.56 | 66.62 | 57.00 | 65.33 | 66.60 | 67.67 |
| Reasoning | 39.13 | 40.60 | 42.30 | 42.43 | 42.57 | 33.90 | 43.37 | 31.20 | 48.23 |
| Mathematics | 40.25 | 41.83 | 43.40 | 43.85 | 43.66 | 42.41 | 53.18 | 30.53 | 50.30 |
| Code | 40.87 | 43.52 | 44.20 | 44.95 | 44.65 | 31.67 | 46.13 | 17.67 | 45.28 |
Table 20: Category-level performance of Accordion-Base-32K. Categories average the displayed scores in Table 21 , including the Ouro correction. Reference columns follow the main-text order.
Table 21: Base-model comparison across all 37 benchmarks. All models follow the protocol in Appendix B.2 . Accordion-Base-32K uses 3.65T training tokens, including 50B for context extension. Category scores appear in Table 20 .
| Domain | |||||
| Web | 2.3007 | 2.2923 | 2.2883 | 2.2869 | 2.2867 |
| Math | 1.3436 | 1.3347 | 1.3308 | 1.3296 | 1.3295 |
| Code | 0.8681 | 0.8618 | 0.8586 | 0.8576 | 0.8574 |
Table 22: Accordion-Base-32K held-out loss. The 8K evaluation settings match Table 15 ; lower is better.
| Context | [95% CI] | |||||
| 4K | 86.52 | 86.29 | 86.87 | 87.27 | 87.23 | |
| 8K | 78.15 | 79.18 | 79.44 | 79.97 | 79.95 | |
| 16K | 70.10 | 73.94 | 74.38 | 74.15 | 74.03 | |
| 32K | 49.49 | 51.80 | 54.58 | 54.90 | 54.95 |
Table 23: RULER for Accordion-Base-32K. Scores average 13 tasks (%), using 500 examples per task and context length. The last column gives the paired to gain [95% CI] in percentage points. Bold and underline mark the highest and second-highest scores within each row.
| 4K context | |||||
| Task | |||||
| S1 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| S2 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| S3 | 97.20 | 93.20 | 95.00 | 95.40 | 95.40 |
| MK1 | 95.60 | 96.60 | 96.40 | 96.60 | 96.00 |
| MK2 | 99.40 | 99.80 | 100.00 | 100.00 | 100.00 |
Table 24: Complete RULER task scores at 4K and 8K. Scores are percentages over 500 examples per task and budget. S: single-needle; MK: multi-key; MQ: multi-query; MV: multi-value; VT: variable tracking; CWE/FWE: common/frequent-word extraction; QA1/2: SQuAD/HotpotQA. Bold/underline mark the highest/second-highest scores per row, including ties.
| 16K context | |||||
| Task | |||||
| S1 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| S2 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| S3 | 97.80 | 97.80 | 97.40 | 97.60 | 97.80 |
| MK1 | 69.20 | 72.20 | 77.80 | 78.80 | 79.20 |
| MK2 | 86.80 | 91.60 | 93.80 | 92.80 | 93.20 |
Table 25: Complete RULER task scores at 16K and 32K. The protocol, task abbreviations and ranking marks follow Table 24 . Each task and budget uses 500 examples.
Figure 19: Attention readouts change despite similar target selection. Left, cross-pass cosines before gating: normalized input ( ), queries ( ), keys ( ), values ( ), attention-weighted values ( ) and output projection ( ). Right, top-eight target overlap for the same head across passes (Same head), and within-pass controls using another head in the same or a different KV group (Same KV / Other KV), at query positions at least 64.
| Edit / layer | 7 | 8 | 9 | 10 | 11 | 12 |
| Remove attention | 1.86 | 3.11 | 2.71 | 2.34 | 2.45 | 1.98 |
| Remove MoE | 5.32 | 10.23 | 6.73 | 3.03 | 7.80 | 5.62 |
| Remove both writes | 7.94 | 9.07 | 7.64 | 7.28 | 6.86 | 5.66 |
| Fix expert set | 0.57 | 3.23 | 6.08 | 0.08 | 1.93 | 0.21 |
| Fix set and weights | 1.03 | 4.08 | 5.95 | 0.21 | 2.13 | 0.26 |
| Attention, | 0.32 | 0.50 | 0.54 | 0.54 | 0.53 | 0.25 |
Table 26: Complete layerwise intervention effects. NLL in millinats/token. “Both” always refers to second-pass attention and MoE writes, with first-pass execution preserved. The lower block of each panel gives scaling factors.
| Measurement | Base-32K | Instruct | RL |
| Baseline NLL | 1.5674 | 1.6974 | 1.7046 |
| Remove all writes | 13.38 [12.92, 13.87] | 9.66 [8.92, 10.41] | 7.22 [6.44, 7.97] |
| Remove all MoE | 38.23 [37.28, 39.18] | 43.04 [41.76, 44.30] | 47.20 [45.87, 48.53] |
| Remove all attention | 16.94 [16.38, 17.52] | 29.79 [28.82, 30.70] | 29.65 [28.68, 30.56] |
| Fix all expert sets | 12.58 [12.10, 13.06] | 17.98 [17.22, 18.71] | 19.56 [18.78, 20.33] |
Table 27: Second-pass computation remains useful across training stages. Baseline NLL is in nats/token; intervention effects and their 95% document bootstrap intervals are in millinats/token. Each stage uses its own intact baseline on the same raw documents, without chat templates.
| Removed layers | 16–18 | 13–15 | 10–12 | 7–9 | 7, 11, 15 |
| Joint effect | 0.25 | 1.23 | 3.61 | 8.07 | 16.49 |
| 95% interval | [0.18, 0.32] | [1.09, 1.37] | [3.37, 3.84] | [7.72, 8.46] | [15.98, 17.04] |
| Sum of singles | 0.86 | 8.00 | 19.79 | 24.65 | 16.32 |
| Joint / sum | 0.29 | 0.15 | 0.18 | 0.33 | 1.01 |
| Ratio 95% interval | [0.23, 0.36] | [0.14, 0.17] | [0.17, 0.19] | [0.32, 0.34] | [1.00, 1.03] |
Table 28: Joint effects for every tested three-, six- and nine-layer set. Effects and intervals are in millinats/token. Sums use unrounded single-layer estimates. Ratio intervals resample the joint and single-layer effects on the same documents.
| Removed layers | 7, 8 | 8, 9 | 8, 11 | 9, 10 | 9, 11 | 10, 11 | 12, 13 |
| Joint effect | 8.78 | 12.91 | 16.60 | 14.45 | 16.56 | 6.70 | 8.61 |
| 95% interval | [8.43, 9.16] | [12.45, 13.43] | [16.07, 17.18] | [13.97, 14.98] | [16.04, 17.13] | [6.40, 7.01] | [8.28, 8.93] |
| Sum of singles | 17.02 | 16.71 | 15.93 | 14.91 | 14.49 | 14.13 | 9.94 |
| Joint / sum | 0.52 | 0.77 | 1.04 | 0.97 | 1.14 | 0.47 | 0.87 |
| Ratio 95% interval | [0.50, 0.53] | [0.76, 0.79] | [1.03, 1.05] | [0.95, 0.98] | [1.13, 1.16] | [0.46, 0.49] | [0.85, 0.88] |
Table 29: Pair effects depend on layer identity. Seven layer-pair interventions quantify variation in joint effects, with 95% document bootstrap intervals. Joint NLL is in millinats/token.
| Measurement | ||||
| Long / short gain | 1.15 | 1.29 | 1.44 | 1.44 |
| 95% interval | [0.97, 1.43] | [1.10, 1.57] | [1.23, 1.73] | [1.23, 1.73] |
Table 30: Length preference across all nonzero budgets. Long/short gain ratios for Accordion-Base-32K. Long bins end at 512–4,096 tokens; short bins at 32–256 tokens. The ratio compares unweighted means of the four bin-level gains in each group. Intervals are 95% document bootstrap intervals.
Figure 20: Depth gains across input lengths and positions. Accordion-Base-32K, versus . All nine bins are shown; whiskers give 95% document bootstrap intervals. The final bin ends at 8,191 tokens.
| Metric | |||||
| Exact match (%) | 8.50 | 10.00 | 11.75 | 12.00 | 12.00 |
| Answer NLL change | 0.000 | -0.224 | -0.259 | -0.272 | -0.259 |
Table 31: Depth improves bracket completion with long demonstrations. All five budgets on the same 400 prompts with 32 demonstrations. Exact match is teacher-forced, and answer NLL changes are nats per answer relative to . At all tested positive demonstration counts below 32, paired exact-match gain intervals include zero.
| Category | Dataset / retained subset | Single pass (B) | Epochs | Training (B) |
| Nemotron-SFT-Math-v3 | 18.098 | 1 | 18.098 | |
| OpenMathReasoning (CoT) | 23.592 | 1 | 23.592 | |
| Mathematics | OpenR1-Math-220k | 1.028 | 1 | 1.028 |
| Nemotron-SFT-Comp.-Prog.-v2 (Python) | 1.985 | 1 | 1.985 | |
| Nemotron-SFT-Comp.-Prog.-v2 (C++) | 2.014 | 1 | 2.014 | |
| AceReason-1.1-SFT (code) | 9.529 | 1 | 9.529 |
Table 32: SFT training data by source. All 12 packed training components, with input tokens in billions before and after applying the indicated epochs. Category shading identifies mathematics, code, and instruction/other data. Validation is excluded; the 68.5B loss-bearing tokens are a subset of the 71.734B training input tokens. Totals use unrounded counts.
| Tasks | Maximum output tokens |
| GSM8K, IFEval, IFBench, MMLU-Redux | 8,192 |
| MATH500, HumanEval+, MBPP+ | 16,384 |
| AIME 2024, AIME 2025, LiveCodeBench | 30,000 |
Table 33: Output-token limits for the common post-training evaluation. The matched SFT and RL comparison uses the same limits. Public references request these limits, capped by their remaining context space as specified in Appendix D.4 .
| Task | 95% CI | ||||||
| Greedy (one response per problem) | |||||||
| GSM8K | 75.82 | 79.76 | 81.20 | 82.26 | 81.27 | ||
| MATH500 | 69.40 | 76.60 | 77.60 | 78.40 | 82.20 | ||
| HumanEval+ | 65.24 | 63.41 | 70.12 | 68.29 | 65.85 | ||
| MBPP+ | 60.05 | 62.17 | 66.40 | 66.14 | 66.67 | ||
| LiveCodeBench | 18.94 | 23.79 | 27.97 | 26.65 | 26.21 | ||
Table 34: SFT decoding policies across all five budgets. vLLM scores (%), endpoint differences and paired 95% intervals (percentage points). Differences are calculated before rounding. Intervals use NumPy default_rng(0) ; the resample count is unrecorded in the archived evaluation. Sampled scores serve as the SFT baseline in Table 38 ; the greedy comparison tests depth gains across decoding policies. AIME has one 16-response sampled evaluation. Ranks compare the five budgets within each task and decoding-policy row.
Figure 21: SFT budget response under greedy and sampled decoding. Sampling uses , top- and one response per problem. Output limits are held fixed within each task. Complete scores and paired intervals appear in Table 34 .
| Domain | Public dataset / subset | Prompts |
| Instruction following | Nemotron-Cascade-RL-Instruction-Following | 98,092 |
| Nemotron-RL-instruction_following | 45,079 | |
| Nemotron-RL instruction following (structured outputs) | 9,437 | |
| Mathematics | Nemotron-RL-math-OpenMathReasoning | 81,277 |
| OpenR1-Math-220k | 59,703 | |
| AceReason-Math | 46,280 |
Table 35: Public sources for RL. Counts describe the deduplicated and decontaminated source pools before difficulty and code-verifier filtering. The final expert pool sizes are reported in the text. Mathematics counts sum to 207,156, IF to 152,608 and code to 22,016. Dataset names link to their public releases where separately identifiable.
| Expert | Selected step | Validation score (%) | Stopping step | Entropy at stopping | Entropy threshold |
| IF | 130 | 81.11 | 133 | 1.811 | 1.8 |
| Mathematics | 30 | 47.89 | 45 | 1.533 | 1.5 |
| Code | 32 | 70.85 | 35 | 1.527 | 1.5 |
Table 36: Expert selection and entropy-based stopping. Steps count domain-specific rollout/training iterations, each containing up to five per-budget optimizer updates; hardware allocations are reported below. The learning-rate scheduler advances once per training step. Validation scores average all five budgets. Training stops when entropy exceeds the domain-specific threshold. The selected experts are merged to initialize the student, selected after ten distillation training steps.
| Quantity | Value or change |
| IF mean training reward, first 50 recorded steps | |
| Mathematics validation gain over initial evaluation (pp) | |
| Code validation gain over initial evaluation (pp) | |
| MOPD task reward, ten-step run | |
| MOPD token distillation loss, ten-step run | |
| Mathematics validation at step 40 (%) | 48.40 |
Table 37: Expert learning, distillation progress and selection statistics. Expert validation averages the five budgets. The ten-step distillation objective uses teacher supervision; task reward is monitored separately. The lower block reports the paired mathematics checkpoint-selection comparison.
Table 38: Complete matched SFT and RL evaluation. Benchmark cells give Accordion-Instruct / Accordion-RL scores (%). Mean rows average the displayed RL SFT differences equally over tasks within each group, in percentage points. Bold and underlining identify the best and second-best of the ten model–budget scores per benchmark; group changes are not ranked.
Figure 22: Response completion and conditional accuracy on LiveCodeBench. RL and SFT use matched generation settings. Conditional scores characterize each model’s own nontruncated subset, with RL SFT differences of to points across these different subsets. Overall accuracy in Table 38 uses the full paired problem set.
| Task | 95% CI | Task | 95% CI | ||
| GSM8K | HumanEval+ | – | |||
| MATH500 | MBPP+ | ||||
| AIME 2024 | – | LiveCodeBench | |||
| AIME 2025 | – | MMLU-Redux |
Table 39: Accordion-RL gains from additional depth. Differences compare with , in percentage points, and are calculated before rounding. Paired 95% percentile intervals use 2,000 problem-level resamples with Python random.Random(0) ; a dash indicates that no interval is reported here.
| Parameter group | Tensors | IF | Code | Math |
| Attention projections and attention normalization; shared MoE experts; loop gates; layer normalization, including the final norm | 289 | 1 | 0.5 | 0.5 |
| Routed expert projections; routers; embeddings; language-model head | 74 | 1 | 1 | 1 |
| Frozen balancing biases | 48 | Copied from SFT | ||
| Total | 411 | – | – | – |
Table 40: Parameter groups in the selected RL merge. Coefficients multiply expert deltas relative to SFT. The first row defines the 289 shared-component tensors referred to in the merge comparisons. Frozen balancing biases are copied, not merged.
| Variant | Instruction following | Code | Mathematics |
| IF + code | +1.76 [+0.35,+3.11] | +0.47 [−0.77,+1.62] | – |
| IF + math | +1.93 [+0.51,+3.28] | +2.03 [+0.83,+3.21] | −0.52 [−1.34,+0.32] |
| Code + math | −24.67 [−26.44,−22.91] | +1.97 [+0.85,+3.13] | −0.06 [−0.84,+0.69] |
| Unit weights | +1.85 [+0.47,+3.21] | +0.44 [−0.76,+1.66] | −4.92 [−5.95,−3.93] |
| Math 0.5 throughout | +2.69 [+1.31,+4.11] | +1.17 [+0.12,+2.27] | −2.24 [−3.03,−1.44] |
| Shared code/math 0.5 | +2.36 [+0.90,+3.79] | +1.78 [+0.64,+2.98] | −0.93 [−1.75,−0.10] |
Table 41: Merge variants relative to the corresponding domain expert. Entries are equal-task, equal-budget mean gains [paired 95% intervals], in percentage points. Omitted domains in pairwise merges have coefficient zero; unspecified included coefficients are one. “Shared” denotes the 289 tensors in Table 40 . A mathematics-domain interval is unavailable for the IF–code merge.
| Comparison | Mean gain | 95% CI |
| Student versus IF expert, instruction following | ||
| Student versus mathematics expert, mathematics | ||
| Student versus code expert, code | ||
| Student versus grouped merge, mathematics |
Table 42: Domain-level gains after merging and distillation. Differences are percentage points, averaging tasks equally within each budget and then averaging the five budgets. Intervals use paired problem-bootstrap resampling.
| Setting | Value | Setting | Value |
| Training pool (prompts) | 57,600 | Warmup (training steps) | 10 |
| Prompts per rollout batch | 576 | Peak learning rate | |
| Responses per prompt | 8 | Student nodes | 36 |
| Responses per rollout batch | 4,608 | Additional teacher nodes | 4 |
| Max. prompt tokens | 2,048 | GPUs per node | 8 H800 |
| Max. response tokens | 28,672 | Teacher inference instances | 32 |
Table 43: Multi-teacher on-policy distillation settings. The student generates its own responses, and each domain expert evaluates them at the student’s inference budget.
| Calculated model costs at 8K | |||||
| Layer executions | 24 | 27 | 30 | 33 | 36 |
| Rel. train FLOPs | 1.000 | 1.101 | 1.202 | 1.303 | 1.404 |
| KV/token (KiB) | 48 | 54 | 60 | 66 | 72 |
Table 44: Calculated model costs and measured inference cost. The first block is analytical. In the runtime block, each metric cell gives batch 1 / batch 8 , measured on one H800 with 128 output tokens per sequence. Throughput counts tokens across the batch; latency covers the batch. Prompt lengths group all five budgets, with 32,640 prompt tokens per sequence for the 32K workload. Measurement settings appear above.
More in this topic
Efficient Language Model Inference
Efficient language model inference reduces the compute, memory, latency, and energy needed to generate useful outputs.
See all 321 papers