cs.LGNov 11, 2026

Learning Rate Transfer and Feature Learning Across Depth for Constrained Spectral Optimizers: Complete Scion

Authors: Eline Pot, Laurent Lefebvre, Mathieu Segond, Antonio Silveti-Falls

Organizations: CVN CentraleSupélec, 91190 Gif-sur-Yvette, France · Framatome, 92400 Courbevoie, France

Abstract

Hyperparameter transfer across model depth is instrumental in training deep models, as it enables learning rates calibrated on shallow networks to be effectively applied to deeper architectures. The recent CompleteP framework establishes a 1/L1/L residual scaling rule that demonstrates this for Adam-family optimizers. We study whether similar transfer extends to constrained optimization methods, specifically Frank-Wolfe and projected non-Euclidean normalized steepest descent, where updates are determined by linear minimization oracles. We derive scaling rules for these methods which confirm that the 1/L1/L residual scaling applies, although the arguments differ from the Adam-family optimizers. Our analysis supports depth-wise transfer and width-wise transfer of the learning rate for the Scion algorithm while ensuring feature learning and avoiding the kernel-regime. In experiments on NLP transformers (nanoGPT on FineWeb), vision transformers (DeiT on ImageNet), and vision MLPs (MLP-Mixer on CIFAR-10), we observe learning rate transfer across depth and feature learning for appropriately chosen residual scaling.

Figures & tables

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Oct 9, 2026

On the Depth Transfer of Neural Optimizers via L-Smoothness

Scaling deep learning models efficiently requires predictable performance transfer across compute regimes. Establishing reliable hyperparameter transfer allows optimal training dynamics and structural choices to be discovered on low-cost proxy models, avoiding catastrophic failures during massive-scale training. Existing depth-transfer analyses are architecture-specific and heuristic: hyperparameter transfer is only justified through toy blocks, and both rely on infinite-depth dynamics, a specific initialization, and feature-learning criteria. Building on LL-smoothness framework of Xu et al. (2026), we understand hyperparameter transfer as how do the Lipschitz and LL-smoothness constants of the loss scale with the depth KK? We show that inverse-depth residual branch scaling α=Θ(1/K)α=Θ(1/K) is sufficient to make both constants depth-independent at fixed width, uniformly over the bounded parameter domain and for arbitrary p→qp\to q geometries. Because the bounds depend on each residual branch only through its Lipschitz and smoothness constants, the argument applies to general residual networks (including branches of block depth ≥2\geq 2 such as transformer and ResNet blocks) without assumptions on initialization or correlations between layer weights. Since the smoothness constant controls the curvature penalty in the descent inequality, our bounds motivate a learning-rate scale without an additional depth-dependent curvature correction for updates bounded in the prescribed norm. This provides a geometric explanation of inverse-depth residual scaling and its application to the operator-norm geometries associated with AdamW, Muon, and MOGA. Learning-rate sweeps on GPT-2 (depths 2--128) and CIFAR-100 ResNets (8--32 residual blocks) confirm that Muon and MOGA exhibit stable learning-rate transfer under these prescriptions, with a common optimal rate across a 64-fold increase in depth.
Feb 24, 2026

Extending μμP: Spectral Conditions for Feature Learning Across Optimizers

Several variations of adaptive first-order and second-order optimization methods have been proposed to accelerate and scale the training of large language models. The performance of these optimization routines is highly sensitive to the choice of hyperparameters (HPs), which are computationally expensive to tune for large-scale models. Maximal update parameterization (μ(μP)) is a set of scaling rules which aims to make the optimal HPs independent of the model size, thereby allowing the HPs tuned on a smaller (computationally cheaper) model to be transferred to train a larger, target model. Despite promising results for SGD and Adam, deriving μμP for other optimizers is challenging because the underlying tensor programming approach is difficult to grasp. Building on recent work that introduced spectral conditions as an alternative to tensor programs, we propose a novel framework to derive μμP for a broader class of optimizers, including AdamW, ADOPT, LAMB, Sophia, Shampoo and Muon. We implement our μμP derivations on multiple benchmark models and demonstrate zero-shot learning rate transfer across increasing model width for the above optimizers. Further, we provide empirical insights into depth-scaling parameterization for these optimizers.
Jun 16, 2026

Loop-μμP: Residual Scaling and Hyperparameter Transfer in Looped Transformers

Looped Transformers have drawn growing attention because looping improves multi-step reasoning and increases the executed depth of a model without adding parameters. They achieve this by reusing a stack of LL layers for NN loops. However, increasing the loop count NN not only destabilizes training but also shifts the best hyperparameters, so they must be searched again for every loop count. Inspired by Depth-μμP (Yang et al., 2024), which multiplies each residual branch by L−1/2L^{-1/2} for depth LL so that a learning rate tuned on a shallow untied network transfers to a deep one, we study how the residual branches and the learning rate should scale with the loop count NN. We find that loops cannot simply be treated as extra depth, because the loops share weights. At initialization, distinct layers add nearly uncorrelated increments to the residual stream, so their sum grows as L\sqrt{L}, whereas a shared layer adds positively correlated increments at every loop, so their sum grows linearly in NN. For a single layer looped NN times, we show in the infinite-width limit that multiplying the residual branch by 1/N1/N and keeping the learning rate independent of NN is the only parameterization with both a stable forward pass and maximal residual-stream updates. We then generalize the analysis to a stack of LL distinct layers looped NN times to obtain a parameterization we call Loop-μμP, in which the loop factor 1/N1/N combines with the depth factor L−1/2L^{-1/2}. Empirically, we find in language modeling that the base learning rate tuned on the smallest model stays best under Loop-μμP for loop counts up to 88 and executed depths up to 384384 layers. These results make Loop-μμP a practical recipe for scaling looped Transformers.