cs.LGNov 11, 2026

Learning Rate Transfer and Feature Learning Across Depth for Constrained Spectral Optimizers: Complete Scion

Authors: Eline Pot, Laurent Lefebvre, Mathieu Segond, Antonio Silveti-Falls

Organizations: CVN CentraleSupélec, 91190 Gif-sur-Yvette, France · Framatome, 92400 Courbevoie, France

Abstract

Hyperparameter transfer across model depth is instrumental in training deep models, as it enables learning rates calibrated on shallow networks to be effectively applied to deeper architectures. The recent CompleteP framework establishes a 1/L1/L residual scaling rule that demonstrates this for Adam-family optimizers. We study whether similar transfer extends to constrained optimization methods, specifically Frank-Wolfe and projected non-Euclidean normalized steepest descent, where updates are determined by linear minimization oracles. We derive scaling rules for these methods which confirm that the 1/L1/L residual scaling applies, although the arguments differ from the Adam-family optimizers. Our analysis supports depth-wise transfer and width-wise transfer of the learning rate for the Scion algorithm while ensuring feature learning and avoiding the kernel-regime. In experiments on NLP transformers (nanoGPT on FineWeb), vision transformers (DeiT on ImageNet), and vision MLPs (MLP-Mixer on CIFAR-10), we observe learning rate transfer across depth and feature learning for appropriately chosen residual scaling.

Figures & tables

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. On the Depth Transfer of Neural Optimizers via L-Smoothness

    Oct 9, 2026Maximilian Wong, Chenyu Xu, Jiajin Li +1Deep Learning OptimizationHyperparameter Transfer

  2. Extending μμP: Spectral Conditions for Feature Learning Across Optimizers

    Feb 24, 2026Akshita Gupta, Marieme Ngom, Sam Foreman +1Hyperparameter TransferAdaptive Gradient Methods

  3. Loop-μμP: Residual Scaling and Hyperparameter Transfer in Looped Transformers

    Jun 16, 2026Shaowen Wang, Bingrui Li, Tansheng Zhu +4Residual LearningHyperparameter Transfer