Layer-Wise Learning Rate Adaptation

Follow this topic Ask about this topic

Layer-wise learning rate adaptation studies how different parts of a neural network should move during training, rather than applying one learning rate everywhere. Approaches include meta-learned or scheduled rates, trust-ratio scaling, and rules based on weight spectra, gradient stability, curvature, or layer divergence; theory also models unequal rates as two-time-scale optimization. Researchers test these methods in large language model pretraining, vision tasks, decentralized federated learning, and continual learning. Results suggest layer-specific rates can help, but the best allocation depends on architecture, task, and training stage, and no single strategy works across all settings.

Latest papers

14 papers1 paper in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Towards Hyperparameter Transfer for Differentially Private Optimization
Oct 16, 2026Tianze Wang, Zhiqi Bu, Linjun ZhangNeural Network Optimization Hyperparameter Transfer
Department of Statistics, Rutgers University · Independent Researcher
The Drift Contract: Spectral Updates for Depth-Robust Local Learning
Sep 16, 2026Fabien PollySpectral Optimization Local Learning
Independent researcher
What to Preserve, Where to Adapt: A Depth-Wise Analysis of Forgetting in Continual Gynecological Image Segmentation
Aug 13, 2026Amal Saqib, Tausifa Jan Saleem, Numan Saeed +1Medical Image Segmentation 3D Medical Image Segmentation
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE
FedA2L: Adaptive layer-wise learning rate adjustment in decentralized federated learning
Aug 10, 2026Van Truong Vo, Khoa Nguyen, Taehong KimNon-IID Federated Learning Federated Learning
School of Information and Communication Engineering, Chungbuk National University, Cheongju, South Korea
LionVote: Per-Layer Learning Rate Adaptation for Lion
Jul 10, 2026Kris AtallahDeep Learning Optimization Layer-Wise Learning Rate Adaptation
New York University, New York, USA · New York University New York, NY, USA
Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam
Jul 4, 2026Ashmitha R, Jörg FrochteGradient Descent Adaptive Gradient Methods
Department of Artificial Intelligence and Data Science, Sri Ramakrishna Engineering College, Anna University, Coimbatore 641022, Tamil Nadu, India · Interdisciplinary Institute for Applied AI and Data Science Ruhr (AKIS), Department of Electrical Engineering and Computer Science, Bochum University of Applied Sciences, Am Hochschulcampus 1, 44801 Bochum, Germany
Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
May 29, 2026Tianyu Pang, Vignesh Kothapalli, Shenyang Deng +3Deep Learning Optimization Neural Network Training Dynamics
Dartmouth College · Stanford University · Virginia Tech
One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs
May 21, 2026Di He, Songjun Tu, Keyu Wang +2Learning Rate Scheduling Efficient Language Model Training
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Peng Cheng Laboratory · University of Chinese Academy of Sciences +5
Training Neural Networks with Optimal Double-Bayesian Learning
May 19, 2026Vy Bui, Hang Yu, Karthik Kantipudi +2Neural Network Optimization Bayesian Decision Theory
Lister Hill National Center for Biomedical Communications, National Library of Medicine National Institutes of Health, Bethesda, MD 20894, USA · Office of Cyber Infrastructure and Computational Biology, National Institute of Allergy and Infectious Diseases National Institutes of Health, Bethesda, MD 20892, USA
Rethinking Neural Network Learning Rates: A Stackelberg Perspective
May 15, 2026Sihan Zeng, Sujay Bhatt, Sumitra GaneshDeep Learning Optimization Neural Network Optimization
JPMorgan AI Research, United States.
Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining
May 11, 2026Jinchang Zhu, Jindong Li, Yuwen Hao +3Language Model Pretraining Self-Attention
The Hong Kong University of Science and Technology (Guangzhou)
OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling
May 8, 2026Yuxuan Lou, Yang YouOrthogonal Optimization Neural Network Optimization
National University of Singapore
Learning Rate Engineering: From Coarse Single Parameter to Layered Evolution
Apr 30, 2026Ming-Hong Yao, Di Wang, Jian Cui +5Deep Learning Optimization Fine-Tuning
School of AI, Jilin International Sciences University, Changchun, China.
A Self-Attentive Meta-Optimizer with Group-Adaptive Learning Rates and Weight Decay
Apr 10, 2026JiangBo Zhao, ZhaoXin LiuDeep Learning Optimization Learning to Optimize

Oct 16, 2026J/K move · Enter open · S save

Towards Hyperparameter Transfer for Differentially Private Optimization

Tianze Wang, Zhiqi Bu, Linjun Zhang

Department of Statistics, Rutgers University · Independent Researcher

Neural Network Optimization Hyperparameter Transfer Layer-Wise Learning Rate Adaptation Differentially Private Stochastic Gradient Descent

Open paper AskPDF arXiv

Scaling laws and hyperparameter tuning remain central challenges in training large-scale models, particularly under differential privacy (DP), where optimization is further complicated by noise injection and per-sample gradient clipping. While Maximal Update Parametrization (

μ

P) allows for zero-shot hyperparameter transfer in standard training, we show that it fails under DP constraints due to the distinct spectral properties of high-dimensional noise versus low-rank gradients. In this work, we introduce DP-

μ

P, a theoretically grounded framework that extends

μ

P to the privacy-preserving setting. By analyzing the interaction between gradient clipping, noise injection, and feature learning, we derive a Signal-to-Noise Ratio (SNR) condition that governs layer-wise learning rate scaling. Combining this with our proposed spectral scaling formulation, we provide specific scaling rules for DP-SGD, DP-Adam, and DP-Muon optimizers. Empirically, we demonstrate that DP-

μ

P enables effective zero-shot transfer of learning rates across model widths on diverse architectures (MLP, ViT, GPT-2), matching or exceeding the performance of individually tuned baselines while significantly reducing computational costs.