Most LLM architectures have fixed compute demands determined at training time. We propose a novel architecture that combines the Mixture-of-Experts architecture with shallow experts and a novel dynamic-depth system to make the extent of a forward pass dynamically adjustable. Our approach lets the model automatically determine the compute needed for a given task, paired with a system to scale it down after training without retraining when the execution environment demands faster or cheaper inference. To achieve this, we introduce an iterative computation mechanism, sequence-level depth-based routing steered by a halting mechanism within the experts, a depth encoding that lets the model track its progress in the chain, and a regularization mechanism that incentivizes the model to use allocated compute efficiently. We present initial benchmarks and discuss future research directions.
Figures & tables
Figure 1: AlterNet forward pass. Input tokens are embedded and passed through a depth loop with a maximum of D iterations. At each step, a depth-aware router selects k of E shallow experts; a PonderNet halt gate accumulates per-step probabilities against a Gaussian prior.
Figure 2
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
50M (ablations)
350M (scale-up)
Hidden size H
512
2048
Number of experts E
8
10
Top- k
2
2
FFN dimension per expert
2048
2048
Max iterations D
8
12
Attention heads
8
16
Appendix
Table 1: AlterNet model hyperparameters for ablation (50M) and scale-up (350M) experiments.
Hyperparameter
50M ablations
350M Run I / Run II
Effective batch size
64
256
Tokens per step
32,768
131,072
Learning rate
3×10−4
3×10−4 / 3→1×10−4
LR warmup steps
1,000
2,000
Training steps
5,000
10,000 / 30,000
Total tokens seen
≈ 164M
≈ 1.3B / 3.9B
Appendix
Table 2: Training setup for ablation and scale-up versions
Run
Experts
ffn_dim
ppl
lm_loss
iters
savings
cv
A0/G0 (baseline)
8
2048
1342
7.202
3.00 ± 0.01
62.5%
0.31
A1-tk2 (clean)
16
1024
1394
7.240
2.00 ± 0.00
75.0%
0.56
A2
32
512
1437
7.270
3.00 ± 0.03
62.5%
0.85
A3
64
256
1497
7.311
3.00 ± 0.08
62.5%
1.65
Appendix
Table 3: Monotonic degradation as expert count increases and ffn_dim decreases. Experts below ffn_dim ≈ 1024 lack sufficient capacity to specialize.
Run
Depth sig.
Budget sig.
ppl
iters
savings
cv
B1
✓
–
1362
3.00 ± 0.00
62.5%
0.21
B3/G0 (default)
✓
✓
1342
3.00 ± 0.01
62.5%
0.31
Appendix
Table 4: Removing the sinusoidal budget signal sin(1−t) increases ppl by 20 points ( +1.5% ). B0 (no depth signal at all) and B2 (budget-only) remain untested.
Run
Halt type
ppl
iters
savings
C0
Global single head
1354
2.00 ± 0.00
75.0%
C1 (default)
Per-expert (aggregated)
1343
3.00 ± 0.00
62.5%
Appendix
Table 5: Per-expert halt outperforms global halt by 11 ppl. Both lock to a fixed depth.
Run
αKL
ppl
lm_loss
iters
savings
cv
D1
0.001
1347
7.206
3.00 ± 0.00
62.5%
0.18
D2/G0 (default)
0.01
1342
7.202
3.00 ± 0.01
62.5%
0.31
D3
0.1
1351
7.209
2.65 ± 0.46
66.9%
0.25
D4
1.0
1418
7.257
3.37 ± 1.83
57.9%
1.04
Appendix
Table 6: αKL=0.1 achieves the only genuine per-example depth variance (std=0.46) with negligible quality cost ( +9 ppl). αKL=1.0 overwhelms the task loss on very small models.
Table 7: Halt depth locks at 3 regardless of ceiling. Quality peaks at iter=12 and degrades for higher ceilings at 5k/50M scale; savings scale linearly with the ceiling.
Run
αz
clamp
batch
ppl
iters
savings
cv
G0/A0 (baseline)
1e-3
10.0
64
1342
3.00 ± 0.01
62.5%
0.31
G1
1e-2
10.0
64
1342
3.00 ± 0.01
62.5%
0.48
G2
1e-3
off
64
1342
2.00 ± 0.00
75.0%
0.19
G3
1e-3
10.0
128
1334
3.00 ± 0.01
62.5%
0.19
Appendix
Table 8: Neither z-loss weight (1e-3 vs 1e-2) nor the logit clamp affects perplexity at 50M scale. A larger batch (G3) improves perplexity by 8 ppl; removing the logit clamp (G2) improves routing balance.
Run
Configuration
Outcome
Collapse step
III
Geometric prior, no curriculum
Collapsed to depth 1
≈ 300
IV
Geometric prior, slow αKL ramp
Collapsed to depth 1
200–400
V
Curriculum release, geometric prior
Collapsed to depth+1
≈ 900
II
Curriculum release, Gaussian prior, αKL=1.0
Stable through 30k steps
—
Appendix
Table 9: Empirical progression of depth-bootstrap collapse mitigation attempts. All runs use the 350M preset.
Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token compute. However, in memory-constrained inference settings, expert weights must be offloaded to CPU, creating a performance bottleneck from CPU-GPU transfers during decoding. We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts, enabling memory transfers to overlap with computation. Across multiple MoE architectures, we demonstrate that future experts can be reliably predicted by these internal representations. We also demonstrate that executing speculated experts generally maintains downstream task accuracy, thus preserving more effective compute-memory overlap by eliminating the need to re-fetch true router-selected experts. Integrated into an optimized inference engine, our approach achieves up to 14% reduction in time per output token (TPOT) over on-demand loading of experts from CPU memory. For MoEs where speculative execution alone yields suboptimal accuracy, we further examine lightweight estimators that improve expert prediction hit rates, thereby reducing performance degradation. Our code is released in open-source at https://github.com/axonn-ai/yalis/tree/offload_prefetch.
Large language models (LLMs) incur high inference cost due to their depth and parameter scale. Depth pruning can reduce latency by skipping redundant Transformer blocks, but existing methods (i) provide limited control under user-specific compute budgets and (ii) typically fix the routing path, failing to adapt as the context grows during decoding. We propose Buddy, a budget-driven dynamic depth routing framework. Buddy uses a lightweight Decision Module to score intermediate layers conditioned on the input and deterministically executes the top-k layers to satisfy a given budget. To support decode-time adaptation, Buddy reuses the first-layer KV cache as a low-overhead global context source and pools it together with the newest token representation before each routing decision. When no explicit budget is provided, an optional Budget Predictor estimates an input-dependent compute level to balance quality and efficiency. Experiments on Llama-family and Qwen models show that Buddy is competitive with strong static pruning baselines and often improves the accuracy-compute trade-off, while uniquely supporting strict budget control, decode-time rerouting, and multiple budgets within a single trained model.
Yuhua Zhou, Shaoqi Yu, Shichao Weng +4
College of Computer Science and Technology, Zhejiang University · Zhejiang Lab · Ant Group +1
Training large language models (LLMs) is resource-intensive, and adapting them for diverse deployment scenarios with varying computational constraints remains challenging. While elastic architectures enable flexible model deployment and sparsely activated models allow input-adaptive computation, existing approaches treat these dimensions independently. Moreover, models catered towards on-device edge inference need to conform to the memory and compute limitations of the serving devices. In this paper, we introduce a unified framework that combines elastic structures with sparsely gated architectures to create models that adapt simultaneously to both deployment constraints and task requirements. Our approach employs a model backbone that conditions on both the context and target efficiency specifications, enabling fine-grained control over the accuracy-efficiency trade-off at inference time. The model learns to activate task-relevant parameters within elastically-nested sub-networks, allowing a single model to span multiple capacity points while maintaining input-adaptive routing. Through experiments we demonstrate that we can create a model that allows the flexibility to use 1,2,3,4 billion parameters while being more accurate than their dense counter-parts (2-5% on knowledge-intensive benchmarks) and at par with their static versions while delivering similar latency metrics as dense models. Overall, we save on device disk space by sharing the model parameters, allow flexibility of serving based on DRAM and compute available while delivering more accurate results.