Layer-Wise Learning Rate Adaptation

Follow this topic Ask about this topic

Momentum

0 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 11

All topics

What to Preserve, Where to Adapt: A Depth-Wise Analysis of Forgetting in Continual Gynecological Image Segmentation
Aug 13, 2026Amal Saqib, Tausifa Jan Saleem, Numan Saeed +1Medical Image Segmentation 3D Medical Image Segmentation
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE
FedA2L: Adaptive layer-wise learning rate adjustment in decentralized federated learning
Aug 10, 2026Van Truong Vo, Khoa Nguyen, Taehong KimNon-IID Federated Learning Federated Learning
School of Information and Communication Engineering, Chungbuk National University, Cheongju, South Korea
LionVote: Per-Layer Learning Rate Adaptation for Lion
Jul 10, 2026Kris AtallahDeep Learning Optimization Layer-Wise Learning Rate Adaptation
New York University, New York, USA · New York University New York, NY, USA
Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam
Jul 4, 2026Ashmitha R, Jörg FrochteGradient Descent Adaptive Gradient Methods
Department of Artificial Intelligence and Data Science, Sri Ramakrishna Engineering College, Anna University, Coimbatore 641022, Tamil Nadu, India · Interdisciplinary Institute for Applied AI and Data Science Ruhr (AKIS), Department of Electrical Engineering and Computer Science, Bochum University of Applied Sciences, Am Hochschulcampus 1, 44801 Bochum, Germany
Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
May 29, 2026Tianyu Pang, Vignesh Kothapalli, Shenyang Deng +3Deep Learning Optimization Neural Network Training Dynamics
Dartmouth College · Stanford University · Virginia Tech
One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs
May 21, 2026Di He, Songjun Tu, Keyu Wang +2Learning Rate Scheduling Efficient Language Model Training
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Peng Cheng Laboratory · University of Chinese Academy of Sciences +5
Training Neural Networks with Optimal Double-Bayesian Learning
May 19, 2026Vy Bui, Hang Yu, Karthik Kantipudi +2Neural Network Optimization Bayesian Decision Theory
Lister Hill National Center for Biomedical Communications, National Library of Medicine National Institutes of Health, Bethesda, MD 20894, USA · Office of Cyber Infrastructure and Computational Biology, National Institute of Allergy and Infectious Diseases National Institutes of Health, Bethesda, MD 20892, USA
Rethinking Neural Network Learning Rates: A Stackelberg Perspective
May 15, 2026Sihan Zeng, Sujay Bhatt, Sumitra GaneshDeep Learning Optimization Neural Network Optimization
JPMorgan AI Research, United States.
Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining
May 11, 2026Jinchang Zhu, Jindong Li, Yuwen Hao +3Language Model Pretraining Self-Attention
The Hong Kong University of Science and Technology (Guangzhou)
OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling
May 8, 2026Yuxuan Lou, Yang YouNeural Network Optimization Muon Optimizer
National University of Singapore
Learning Rate Engineering: From Coarse Single Parameter to Layered Evolution
Apr 30, 2026Ming-Hong Yao, Di Wang, Jian Cui +5Deep Learning Optimization Fine-Tuning
School of AI, Jilin International Sciences University, Changchun, China.

Aug 13, 2026, cs.CVJ/K move · Enter open · S save

What to Preserve, Where to Adapt: A Depth-Wise Analysis of Forgetting in Continual Gynecological Image Segmentation

Amal Saqib, Tausifa Jan Saleem, Numan Saeed, Mohammad Yaqub

Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE

Medical Image Segmentation 3D Medical Image Segmentation Catastrophic Forgetting Layer-Wise Learning Rate Adaptation

Open paper AskPDF arXiv

The clinical management of gynecological diseases often relies on medical imaging for diagnosis, treatment planning, and follow-up. Segmentation in this setting is challenging because successive tasks may differ in imaging modality, target anatomy, pathology, and annotation structure. Continual learning allows models to adapt to new tasks without simultaneous access to previous datasets. However, when successive tasks differ substantially, learning a new task can degrade performance on earlier ones, a problem known as catastrophic forgetting. Understanding where adaptation disrupts previous knowledge can help guide the design of more targeted continual-learning strategies. We investigate how forgetting changes as different parts of an encoder--decoder network are allowed to adapt. We progressively expand the trainable region of a 3D nnU-Net backbone from the bottleneck toward input- and output-proximal blocks. Under a shared learning rate, adaptation near the bottleneck largely preserves previous-task performance but provides limited current-task learning, whereas broader adaptation improves current-task performance but sharply increases forgetting. This trade-off persists even when the average change in trainable backbone parameters is approximately comparable. Assigning different learning rates to different blocks substantially reduces forgetting when part of the backbone is trainable, although this changes both the size and location of the updates. Forgetting still increases as more blocks are trained and remains severe when the full backbone is updated. These results show that forgetting depends not only on how much the model changes, but also on which parts of the model are allowed to change.

Momentum

Related topics

Latest papers 11