FARO: Function-Space Adaptive Return-to-Risk Optimization via Cross-Block Equal-Risk-Contribution Budget Allocation
Organizations: IDEA Research · The Hong Kong University of Science and Technology (Guangzhou) · DataArcTech Ltd.
Abstract
Training a large language model is compute-bound, and current recipes divide the work between optimizers by hand: orthogonalized momentum for attention and MLP matrices, Adam-style updates for the rest. This division rests on human priors about which layer type deserves which geometry, and it fixes a single global update budget that every parameter block shares. We introduce FARO, a function-space adaptive optimizer that makes the partition of parameters into blocks the unit of decision. FARO gives each block a geometry by its realized counterfactual return, then freezes the routes and reallocates the frozen per-block budget by cross-block equal risk contribution (ERC), the risk-parity principle from portfolio theory. On a GPT-style model trained on FineWeb, the framework gives two results. First, its FARO-Adaptive arm reaches 3.2780 nats at the full schedule, matching NorMuonAndAdam (3.2794), the best hand-designed recipe: it recovers the human prior without hand-setting per-layer geometry. Second, FARO-ERC improves on that arm: it lowers the clean-data loss at every one of 6 seeds, by 0.0136 nats on average, a modest but robust gain. Its sign test, paired test, and confidence interval all agree, and it exceeds the measured run-to-run noise at no added time or memory. It works by trimming the riskiest blocks rather than expanding the safest, and it holds up under a corrupted stream, still lowering the mean loss and winning the seed majority. The source code is available at https://github.com/wxj630/FARO.
Figures & tables
| clean FineWeb (6 seeds 2 reps) | cost (clean) | scatter (60% noise, 6 seeds) | |||||||||
| Seed | FARO-Ad. | FARO-ERC | win | mem | ms Ad. | ms ERC | FARO-Ad. | FARO-ERC | win | ||
| Per-seed final validation loss and cost | |||||||||||
| seed0 | 3.6019 | 3.5917 | -0.0102 | ✓ | 42,495 | 397.0 | 393.2 | 3.7956 | 3.7690 | -0.0266 | ✓ |
| seed1 | 3.6081 | 3.5850 | -0.0231 | ✓ | 42,495 | 393.7 | 399.3 | 3.7898 | 3.7871 | -0.0027 | ✓ |
| seed2 | 3.5997 | 3.5865 | -0.0131 | ✓ | 42,495 | 392.3 | 398.3 | 3.7699 | 3.7811 | +0.0113 | — |
| seed3 | 3.6001 | 3.5879 | -0.0122 | ✓ | 42,495 | 395.9 | 398.7 | 3.7798 | 3.7742 | -0.0056 | ✓ |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Meaning |
|---|---|
| block index, | |
| number of blocks ( ) | |
| parameters of block | |
| frozen-geometry update direction | |
| base learning rate | |
| frozen reference budget |
| LR | LR | LR | |
|---|---|---|---|
| cap 1.8 | -0.0109 | -0.0219 | -0.0215 |
| cap 2.6 | -0.0118 | -0.0235 | -0.0234 |
| cap | 1.2 | 1.4 | 1.8 | 2.6 | 4.0 |
|---|---|---|---|---|---|
| -0.0083 | -0.0137 | -0.0215 | -0.0234 | -0.0232 |
| Setting | vs. FARO-Adaptive |
|---|---|
| floor (default) | -0.0137 |
| floor | -0.0245 |
| floor | -0.0250 |
| floor | -0.0246 |
| ceiling | -0.0210 |
| ceiling | -0.0205 |
| step | 250 | 350 | 450 | 550 | 650 | 800 |
|---|---|---|---|---|---|---|
| -0.0209 | -0.0298 | -0.0471 | -0.0221 | -0.0191 | -0.0136 |
| Controller | final | wins |
|---|---|---|
| DS-FARO (difficulty-sensed) | -0.0002 | 1/3 |
| continuous fractional-Kelly | -0.0006 | 2/3 |
| Optimizer | seeds | mean std |
|---|---|---|
| Adam | ||
| Muon | ||
| NorMuonAndAdam | ||
| Quantity | Value |
|---|---|
| Blocks participating | 117 |
| Multiplier range | – |
| Fraction shrunk ( ) | 58.7% |
| Fraction expanded ( ) | 40.8% |
| Mean applied multiplier | 0.980 |
| System | tok/s | train | wall | alloc | res |
|---|---|---|---|---|---|
| Adam | 569,045.0 | 595.422 | 635.088 | 33,747 | 40,820 |
| Muon | 544,056.2 | 622.770 | 663.049 | 34,528 | 41,894 |
| NorMuonAndAdam | 607,164.2 | 558.039 | 597.875 | 39,435 | 46,892 |
| FARO-Adaptive | 574,519.6 | 589.748 | 629.567 | 42,489 | 51,654.7 |
| train/wall in s; alloc/res in MiB. | |||||
Deep Learning Optimization
Deep learning optimization develops algorithms and training strategies that help neural networks converge, generalize, and use compute efficiently.
See all 217 papers