Learning Rate Transfer and Feature Learning Across Depth for Constrained Spectral Optimizers: Complete Scion
Organizations: CVN CentraleSupélec, 91190 Gif-sur-Yvette, France · Framatome, 92400 Courbevoie, France
Abstract
Hyperparameter transfer across model depth is instrumental in training deep models, as it enables learning rates calibrated on shallow networks to be effectively applied to deeper architectures. The recent CompleteP framework establishes a residual scaling rule that demonstrates this for Adam-family optimizers. We study whether similar transfer extends to constrained optimization methods, specifically Frank-Wolfe and projected non-Euclidean normalized steepest descent, where updates are determined by linear minimization oracles. We derive scaling rules for these methods which confirm that the residual scaling applies, although the arguments differ from the Adam-family optimizers. Our analysis supports depth-wise transfer and width-wise transfer of the learning rate for the Scion algorithm while ensuring feature learning and avoiding the kernel-regime. In experiments on NLP transformers (nanoGPT on FineWeb), vision transformers (DeiT on ImageNet), and vision MLPs (MLP-Mixer on CIFAR-10), we observe learning rate transfer across depth and feature learning for appropriately chosen residual scaling.
Figures & tables
| Embedding | Hidden Layers | Un-embedding | |||
| ( ) | ( ) | ( ) | |||
| Norm | |||||
| LMO | -\rho_{e}{\color[rgb]{0,1,0}\frac{1}{d_{\text{in}}}}\text{sign}(M^{(0)}) | -\rho_{h}{\color[rgb]{0,1,0}\sqrt{\frac{d_{\ell}}{d_{\ell-1}}}}U^{(\ell)}{V^{(\ell)}}^{\top} | -\rho_{u}{\color[rgb]{0,1,0}\frac{1}{d_{L}}}\text{sign}(M^{(L+1)}) | ||
| Scaled LR | {\color[rgb]{0,1,1}L^{\alpha-1}}\gamma_{\text{base}} | ||||
| U NCONSTRAINED S CION Update | -\gamma_{0}\rho_{e}{\color[rgb]{0,1,0}\frac{1}{d_{\text{in}}}}\text{sign}(M^{(0)}) | -\gamma_{\ell}\rho_{h}{\color[rgb]{0,1,0}\sqrt{\frac{d_{\ell}}{d_{\ell-1}}}}U^{(\ell)}{V^{(\ell)}}^{\top} | -\gamma_{L+1}\rho_{u}{\color[rgb]{0,1,0}\frac{1}{d_{L}}}\text{sign}(M^{(L+1)}) | ||
| S CION Update | -\gamma_{0}(\rho_{e}{\color[rgb]{0,1,0}\frac{1}{d_{\text{in}}}}\text{sign}(M^{(0)})+W^{(0)}) | -\gamma_{\ell}(\rho_{h}{\color[rgb]{0,1,0}\sqrt{\frac{d_{\ell}}{d_{\ell-1}}}}U^{(\ell)}{V^{(\ell)}}^{\top}+W^{(\ell)}) | -\gamma_{L+1}(\rho_{u}{\color[rgb]{0,1,0}\frac{1}{d_{L}}}\text{sign}(M^{(L+1)})+W^{(L+1)}) |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Unconstrained Scion and Scion |
| Width | 768 |
| Head dim | 128 |
| Activation function | ScaledReLU 2 |
| Vocabulary size | 50304 |
| Dataset | FineWeb |
| Batch size | 512 |
| Hyperparameter | Scion |
| Head dim | 64 |
| Activation function | GELU |
| Normalization function | RMSNorm |
| Sequence length | 197 |
| Dataset | ImageNet-1k |
| Batch size | 4096 |
| Hyperparameter | Unconstrained Scion , Scion and Projected Scion |
| Depth (in the width-transfer experiments) | 3 |
| Width (in the depth-transfer experiments) | 256 |
| Head dim | 64 |
| Activation function | GELU |
| Vocabulary size | 65 |
| Dataset | Shakespeare-Char |
| Hyperparameter | Unconstrained Scion and Scion |
| Dataset | CIFAR-10 |
| Embedding dimension | 128 |
| Token-mixing MLP hidden width | 64 |
| Channel-mixing MLP hidden width | 512 |
| Activation | GELU |
| Batch size | 2048 |
| Hyperparameter | Unconstrained Scion and Scion |
| Dataset | MNIST |
| Layers per branch | 2 |
| Activation | ReLU |
| Batch size | 128 |
| Iterations | 1 |
| Averaging parameter | 0.9 |
| Hyperparameter | Unconstrained Scion and Scion |
| Dataset | CIFAR-10 |
| Layers per branch | 1 |
| Width | 300 |
| Activation | ReLU |
| Batch size | 1024 |
| Learning rates and |
| Hyperparameter | Scion and Projected Scion |
| Dataset | CIFAR-10 |
| Depth | 3 |
| Activation | ReLU |
| Batch size | 1024 |
| Learning rate | |
| Averaging parameter | 0.9 |
Deep Learning Optimization
Deep learning optimization develops algorithms and training strategies that help neural networks converge, generalize, and use compute efficiently.
See all 216 papers