cs.LGMay 20, 2026

Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws

Authors: Nandan Kumar Jha, Brandon Reagen

Abstract

Scaling laws have made language-model performance predictable from model size, data, and compute, but they typically treat the optimizer as a fixed training detail. We show that this assumption misses a fundamental axis of representation scaling: how effectively the optimizer converts added FFN width into utilized spectral capacity. Using eigenspectra of feed-forward network representations, measured through soft and hard spectral-ranks, we find that \emph{the same Transformer architecture realizes markedly different spectral scaling laws when trained with different optimizers}. Holding architecture and width schedule fixed, AdamW exhibits weak hard-rank scaling (ββ=0.44) on rare-token (TAIL) representations where learning is known to be hardest, whereas Muon achieves linear scaling (ββ=1.02) in the same regimes, a 2.3×2.3\times increase in the scaling exponent. This difference is not reducible to validation loss: AdamW configurations can match low-rank Dion variants in perplexity, under extended training, while exhibiting sharply different spectral geometry, demonstrating that matched loss does not imply matched representation structure. Hard--soft rank asymmetry further reveals that optimizers differ not only in how much capacity is realized, but also in how that capacity is structured across eigenmodes. To disentangle optimizer effects from architectural ones, we compare against architectural interventions (e.g., attention rank and positional encoding), and find that optimizer-induced spectral shifts often exceed the architectural effects. These results suggest optimization as a first-class axis of representation scaling, motivating optimizer--architecture co-design.

Explore similar work

CardsList
May 28, 2026cs.LG

On the Optimizer Dependence of Neural Scaling Laws

The scaling exponent αα in neural scaling laws L(N)∝N−αL(N) \propto N^{-α} is commonly treated as a fixed constant set by architecture and data. We present evidence that αα depends systematically on the optimizer. In controlled random-feature regression experiments -- the canonical theoretical framework for neural scaling -- we measure αα across five optimizer variants and six spectral conditions. Preconditioned optimizers consistently yield steeper scaling (larger αα), with the αα-shift increasing across most of the tested spectral range, peaking near s=1.5s = 1.5, and remaining large at s=2.0s = 2.0. At s≈1.0s \approx 1.0 (characteristic of natural language), the full natural gradient achieves α≈0.31α\approx 0.31 versus α≈0.12α\approx 0.12 for gradient descent -- a 2.6×2.6\times larger fitted exponent that, within the random-feature model, compounds with each model-size doubling. Whether and how this exponent shift transfers to large-scale LLM training -- where recent evidence suggests the advantage may attenuate with scale -- remains an important open question. Our results imply that scaling-law forecasts should account for optimizer choice, and we provide a spectral diagnostic predicting when advanced optimizers will pay off.
Vansh Ramani, Shourya Vir Jain
Scaling LawsOptimizer DesignLarge Language Model Scale
AskPDF
May 29, 2026cs.LG

Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail

Neural scaling laws describe predictable power-law relationships between model size, dataset size, compute, and performance. While these laws guide the development of modern foundation models, the mechanisms underpinning them remain poorly understood, in part due to the absence of scalable analysis tools. To close this gap, we introduce "spectral position": a scalable measure of which eigenvalues of the empirical neural tangent kernel (eNTK) currently drive loss reduction. Applying this measure to scaling experiments, we find that spectral position decreases throughout training: learning shifts from dominant eigenmodes into the spectral tail. Larger models reach further into the tail than smaller models, revealing a size-dependent capacity we call "spectral reach". This suggests why larger models achieve lower losses: they sustain learning on weak spectral signals inaccessible to smaller models. We further identify feature learning as a key enabler of spectral reach. It adaptively amplifies gradient magnitudes as learning advances, sustaining progress where frozen representations stall. This points to concrete interventions through architecture and optimizer design.
Konstantin Nikolaou, Jonas Scheunemann, Sven Krippendorf +2
Scaling LawsNeural Tangent KernelLarge Language Model ScaleHessian EigenvalueSpectral AlgorithmsCapacity
AskPDF
Jul 14, 2026cs.LG

Reassessing Muon for Matrix Factorization

Muon has recently emerged as a strong optimizer for large-scale deep learning, where it reshapes gradient updates through approximate orthogonalization and has been reported to outperform Adam and AdamW in large language model training. Its empirical success has motivated a growing body of theoretical work that interprets Muon as steepest descent under the spectral norm. Yet it remains unclear which of Muon's advantages stem from its update rule itself and which are artifacts of the scale, architecture, and data of modern deep networks. In this work, we isolate the optimizer from these confounding factors by studying Muon on a simple, well-understood, and spectrally structured problem: low-rank matrix factorization. Through a controlled comparison against carefully tuned adaptive baselines, we find that Muon does not consistently outperform AdamW in this setting and that several previously reported advantages are sensitive to hyperparameter choices. Our results provide a more nuanced picture of when spectrum-aware orthogonalization is beneficial and argue for evaluating modern optimizers on controlled problems in addition to end-to-end benchmarks.
Ali Parviz, Gal Mishne, Alex Cloninger
MuonOptimizer DesignAdamLow-Rank Factorization
AskPDF