Most LLM architectures have fixed compute demands determined at training time. We propose a novel architecture that combines the Mixture-of-Experts architecture with shallow experts and a novel dynamic-depth system to make the extent of a forward pass dynamically adjustable. Our approach lets the model automatically determine the compute needed for a given task, paired with a system to scale it down after training without retraining when the execution environment demands faster or cheaper inference. To achieve this, we introduce an iterative computation mechanism, sequence-level depth-based routing steered by a halting mechanism within the experts, a depth encoding that lets the model track its progress in the chain, and a regularization mechanism that incentivizes the model to use allocated compute efficiently. We present initial benchmarks and discuss future research directions.
Figures & tables
Figure 1: AlterNet forward pass. Input tokens are embedded and passed through a depth loop with a maximum of D iterations. At each step, a depth-aware router selects k of E shallow experts; a PonderNet halt gate accumulates per-step probabilities against a Gaussian prior.
Figure 2
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
50M (ablations)
350M (scale-up)
Hidden size H
512
2048
Number of experts E
8
10
Top- k
2
2
FFN dimension per expert
2048
2048
Max iterations D
8
12
Attention heads
8
16
Appendix
Table 1: AlterNet model hyperparameters for ablation (50M) and scale-up (350M) experiments.
Hyperparameter
50M ablations
350M Run I / Run II
Effective batch size
64
256
Tokens per step
32,768
131,072
Learning rate
3×10−4
3×10−4 / 3→1×10−4
LR warmup steps
1,000
2,000
Training steps
5,000
10,000 / 30,000
Total tokens seen
≈ 164M
≈ 1.3B / 3.9B
Appendix
Table 2: Training setup for ablation and scale-up versions
Run
Experts
ffn_dim
ppl
lm_loss
iters
savings
cv
A0/G0 (baseline)
8
2048
1342
7.202
3.00 ± 0.01
62.5%
0.31
A1-tk2 (clean)
16
1024
1394
7.240
2.00 ± 0.00
75.0%
0.56
A2
32
512
1437
7.270
3.00 ± 0.03
62.5%
0.85
A3
64
256
1497
7.311
3.00 ± 0.08
62.5%
1.65
Appendix
Table 3: Monotonic degradation as expert count increases and ffn_dim decreases. Experts below ffn_dim ≈ 1024 lack sufficient capacity to specialize.
Run
Depth sig.
Budget sig.
ppl
iters
savings
cv
B1
✓
–
1362
3.00 ± 0.00
62.5%
0.21
B3/G0 (default)
✓
✓
1342
3.00 ± 0.01
62.5%
0.31
Appendix
Table 4: Removing the sinusoidal budget signal sin(1−t) increases ppl by 20 points ( +1.5% ). B0 (no depth signal at all) and B2 (budget-only) remain untested.
Run
Halt type
ppl
iters
savings
C0
Global single head
1354
2.00 ± 0.00
75.0%
C1 (default)
Per-expert (aggregated)
1343
3.00 ± 0.00
62.5%
Appendix
Table 5: Per-expert halt outperforms global halt by 11 ppl. Both lock to a fixed depth.
Run
αKL
ppl
lm_loss
iters
savings
cv
D1
0.001
1347
7.206
3.00 ± 0.00
62.5%
0.18
D2/G0 (default)
0.01
1342
7.202
3.00 ± 0.01
62.5%
0.31
D3
0.1
1351
7.209
2.65 ± 0.46
66.9%
0.25
D4
1.0
1418
7.257
3.37 ± 1.83
57.9%
1.04
Appendix
Table 6: αKL=0.1 achieves the only genuine per-example depth variance (std=0.46) with negligible quality cost ( +9 ppl). αKL=1.0 overwhelms the task loss on very small models.
Table 7: Halt depth locks at 3 regardless of ceiling. Quality peaks at iter=12 and degrades for higher ceilings at 5k/50M scale; savings scale linearly with the ceiling.
Run
αz
clamp
batch
ppl
iters
savings
cv
G0/A0 (baseline)
1e-3
10.0
64
1342
3.00 ± 0.01
62.5%
0.31
G1
1e-2
10.0
64
1342
3.00 ± 0.01
62.5%
0.48
G2
1e-3
off
64
1342
2.00 ± 0.00
75.0%
0.19
G3
1e-3
10.0
128
1334
3.00 ± 0.01
62.5%
0.19
Appendix
Table 8: Neither z-loss weight (1e-3 vs 1e-2) nor the logit clamp affects perplexity at 50M scale. A larger batch (G3) improves perplexity by 8 ppl; removing the logit clamp (G2) improves routing balance.
Run
Configuration
Outcome
Collapse step
III
Geometric prior, no curriculum
Collapsed to depth 1
≈ 300
IV
Geometric prior, slow αKL ramp
Collapsed to depth 1
200–400
V
Curriculum release, geometric prior
Collapsed to depth+1
≈ 900
II
Curriculum release, Gaussian prior, αKL=1.0
Stable through 30k steps
—
Appendix
Table 9: Empirical progression of depth-bootstrap collapse mitigation attempts. All runs use the 350M preset.