Learning Rates

Follow this topic Ask about this topic

Recent momentum

-8%

11 papers in the last 28 days · 0.3% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

3 new papers

A weekly snapshot of new work published in Learning Rates.

Period ending 2026-09-14

6 new papers

A weekly snapshot of new work published in Learning Rates.

Period ending 2026-09-07

3 new papers

A weekly snapshot of new work published in Learning Rates.

109 papers

Latest in Learning Rates

Open your feed →

Fast Learning Rates for Physics-Informed Kernel Methods
Sep 16, 2026Luc Brogat-Motte, Joachim Bona-Pellissier, Giacomo Meanti +1Physics-Informed Machine Learning Kernel Method
Reaching Every Position Without Searching: Rotating Sparse Wiring on the Hypercube as a Substitute for Attention
Sep 16, 2026Yoshiaki TakashitaPosition Learning Rates
Federated stochastic bilevel optimization with fully first-order gradients
Sep 14, 2026Yihan Zhang, Rohit Dhaipule, Chiu C Tan +2Bilevel Optimization Proximal Gradient Descent
Model-Aware Schedules Improve Generation via Fiberwise Optimal Transport
Sep 12, 2026Luyi Jia, Boyan Zhang, Yilun Liu +1Optimal Transport Learning Rates
Generalization Analysis of Distributed Kernel-based Robust Gradient Descent Algorithms
Sep 10, 2026Jun-Yi Meng, Zheng-Chu Guo, Yuan MaoGeneralization Bounds Kernel Hilbert Spaces
When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay
Sep 8, 2026Hasan Amin, Wei-Kai Chang, Rajiv KhannaLearning Rates Zeroth-Order Optimization
Curriculum Learning as Transport: Understanding Curricula with Wasserstein Geodesics
Sep 8, 2026Changho Shin, David Alvarez-MelisCurriculum Wasserstein Distance
Equivariance Breaks the Learning Rate
Sep 8, 2026Andrei Manolache, Mathias NiepertAdam Equivariant Neural Networks
BrachistoneLR: A Brachistochrone-Inspired Learning-Rate Schedule and a Controlled Benchmark of Scheduling Policies
Sep 8, 2026Md. Sadekur Rahman Roni, Md. Jalal uddin Chowdhury, Moutusi Dash NimiLearning Rates Cosine Annealing
TASTE: Throughput-Aware Batch Size Tuning for On-Device Edge Learning
Sep 7, 2026Avik Bhatnagar, Federico Nicolas Peccia, Oliver BringmannBatch Size Learning Rates
When Does Online Adaptation Pay on the Edge? A Leakage-Free Evaluation of Warmup, Learning-Rate Selection, and Resource Trade-offs for Time-Series Forecasting
Sep 1, 2026Takumi Fujimoto, Hiroaki NishiNon-Stationarity Cold-Start
Stochastic Optimization of Tree Tensor Networks
Sep 1, 2026Marius Willner, Maximilian Scharf, André Uschmajew +2Tensor Networks Neural Network Optimization
Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss
Aug 28, 2026Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan +4Learning Rates Batch Size
Learned, Then Lost: A Measured Single-Example Counterfactual in Pre-training
Aug 19, 2026Zachary Speck, Asa ShepardCounterfactual Learning Counterfactual Reasoning
Sustaining Plasticity via Learnable Wavelet Activations in Continual Learning
Aug 13, 2026Zeyang Zhang, Tieliang Gong, Junyan Lu +1Efficient Continual Learning Plasticity
FedA2L: Adaptive layer-wise learning rate adjustment in decentralized federated learning
Aug 10, 2026Van Truong Vo, Khoa Nguyen, Taehong KimFedavg Learning Rates
Modular TTT: Rethinking Test-Time Training as Composable Modules
Aug 7, 2026Bohao Tang, Zhen Qin, Yuqi Pan +3Sequence Modeling Learning Rates
A Rate Separation for Agnostic Direct Sums
Aug 7, 2026Mihir More, Aritra Das, Debayan GuptaSingular Learning Theory Optimal Sample Complexity
Improving Low-Resolution Face Recognition under Limited Data: How Synthetic Data Generation Can Close the Domain Gap
Aug 6, 2026Luis S. Luevano, Ünsal Öztürk, Hatef Otroshi Shahreza +2Facial Recognition Synthetic-To-Real Domain Gap
Optimal Training-Time Scaling in Gradual Adaptation
Aug 5, 2026Zonghuan Xu, Krishna HarishLearning Rates Training Time
LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
Aug 4, 2026Fengqi Zhu, Shaoxuan Xu, Jingyang Ou +11Mixture-Of-Expert Language Models Large Models
AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization
Aug 3, 2026Alok Kumar Pandey, Umang Chaturvedi, Aatish Rana +1Neural Network Optimization Learning Rates
A Trust-region Framework for Moment Estimation
Jul 26, 2026Oluwasegun A. SomefunTrust Region Stochastic Gradient Descent
Scale Weight Decay and Train Better
Jul 26, 2026Anuj ApteLearning Rates Model Weights
Hyperball May Not Be a Free Lunch
Jul 24, 2026Yihao Xiao, Jialong Sun, Zitian Gao +5Learning Rates Hyperbolic Space
When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers
Jul 22, 2026Tong Zhang, Junhao Hu, Yun Peng +1Recurrence Learning Rates
Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers
Jul 16, 2026Akhilesh GogikarAdam Learning Rates
Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners
Jul 14, 2026Haseeb Shah, Lingwei Zhu, Adam White +1Soft Actor-Critic Learning Rates
A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism
Jul 14, 2026Chengguang Gan, Zhixi Cai, Yunhao Liang +3Group Relative Policy Optimization Reinforcement Learning With Verifiable Reward
Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps
Jul 14, 2026Subham Singh, Ashutosh Mishra, Subha RautLearning Rates Stochastic Gradient Descent
WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training
Jul 12, 2026Jianhao Ma, Yuxin ChenLearning Rates Cosine Annealing
Index SLM Technical Report
Jul 10, 2026Lusheng Zhang, Shien He, Tianxing Yan +5Small Language Models Instruction
LionVote: Per-Layer Learning Rate Adaptation for Lion
Jul 10, 2026Kris AtallahLearning Rates Cross-Layer Interactions
The Silent Freeze: Predicting When Low-Precision Training Stops Learning
Jul 9, 2026Zekai ShangFull-Precision Performance Learning Rates
Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures
Jul 9, 2026Hafsa Mateen, Radu Timofte, Dmitry IgnatovLearning Rates Hyperparameter Optimization
Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing
Jul 8, 2026Tommaso Cerruti, Tim Rieder, George Rowlands +2Quadratic Attention Global Attention Mechanism
Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
Jul 8, 2026Yedi Zhang, Peter E. Latham, Leena Chennuru Vankadara +1Learning Rates Scaling Laws
ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstruction
Jul 6, 2026Xinze Li, Yiyuan Wang, Pengxu Chen +43D Reconstruction Recurrent State
Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam
Jul 4, 2026Ashmitha R, Jörg FrochteAdam Learning Rates
Mixture-of-Gaussians-Guided Schedule Design for Brownian Bridge Diffusion Models
Jul 3, 2026Ron Levi, Michael EladSchrödinger Bridges Image Restoration
The Orthogonalized Read Is a Removable Training Scaffold for Recurrent Memory
Jul 2, 2026Keston Aquino-MichaelsMemorization Recurrent Model
Staleness-Learning Rate Scaling Laws for Asynchronous RLHF
Jul 1, 2026Jingwei Song, Haofeng Xu, Jie Xiao +8Long Rollouts Learning Rates
SGD at the Edge of Stability: Stochastic Stabilization with Large Learning Rates
Jun 29, 2026Konstantinos Emmanouilidis, Lachlan MacDonald, Salma Tarmoun +1Stochastic Gradient Descent Learning Rates
Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics
Jun 29, 2026Mark Rhee, Jamie Simon, Dhruva KarkadaMuon Optimizer Matrix Factorization
Curvature-Weighted Gradient Diversity: A Noise Measure for Geometry-Adaptive SGD Schedules
Jun 29, 2026Muhammad Hamza, Ayush GoelStochastic Gradient Descent Mini-Batch Stochastic Gradient Descent
Smooth Scaling Laws Hide Stepwise Token Learning
Jun 29, 2026Pingjie Wang, Zechen Hu, Peiru Yang +2Scaling Laws Large Models
Anti-Collapse Dynamics and the Emergence of Multi-Time-Scale Learning in Recurrent Neural Networks
Jun 28, 2026Lorenzo LiviRecurrent Neural Networks Learning Rates
On the Nonlinearity of Learning Rate Scaling for LLM Training
Jun 28, 2026Zaiwen Yang, Huaqing Zhang, Jing Xu +1Learning Rates Large Models
Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
Jun 23, 2026Yizhou Liu, Jeff GoreScaling Laws Large Models
Convergence of Gradient Descent for General Neural Network Architectures Beyond the NTK Regime
Jun 22, 2026Yuqing WangNeural Tangent Kernel Gradient Descent
Error Highways: Scaling Predictive Coding to Very Deep Networks
Jun 22, 2026Amirhossein Mohammadi, Alexander G. OrorbiaPredictive Coding Jacobian
Alternate loss functions and regression models that achieve robustness to outliers by modulating the learning rate
Jun 20, 2026Mathew Mithra Noel, Arindam Banerjee, Yug D. Oswal +2Outliers Loss Function
Mitigating Early Training Collapse in CTR Models
Jun 20, 2026Ergun Biçici, Erkan ÇetinyamaçClick-Through Rate Prediction Collapse
Beyond IGO-Flow: Toward Convergence Analysis of IGO in Continuous Spaces
Jun 16, 2026Ryosuke Kimura, Youhei AkimotoInformation-Geometric Optimization Convergence Analysis
Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization
Jun 15, 2026Kaiyue Wen, Xingyu Dang, Kaifeng Lyu +2Large Language Model Pretraining Model Weights
The Weight Norm Sets the Grokking Timescale: A Causal Delay Law
Jun 11, 2026Truong Xuan Khanh, Doan Hoang Viet, Luu Duc Trung +1Grokking Learning Rates
The Hidden Power of Scaling Factor in LoRA Optimization
Jun 11, 2026Zicheng Zhang, Haoran Li, Jiaxing Wang +10Low-Rank Adaptation Learning Rates
Adaptive Learning Rates with Surrogate Probability for Follow-the-Perturbed-Leader
Jun 4, 2026Jongyeong Lee, Junya Honda, Shinji Ito +1Learning Rates Bandits
Unlocking Feature Learning in Gated Delta Networks at Scale
Jun 2, 2026Yifeng Liu, Quanquan GuLarge Models Learning Rates
Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
May 29, 2026Tianyu Pang, Vignesh Kothapalli, Shenyang Deng +3Learning Rates Two-Layer Neural Networks

Sep 16, 2026 · stat.MLJ/K move · Enter open · S save

Fast Learning Rates for Physics-Informed Kernel Methods

Luc Brogat-Motte, Joachim Bona-Pellissier, Giacomo Meanti, Lorenzo Rosasco

In physics-informed machine learning, a target function

u^*

is learned from noisy value observations

y_i=u^*(x_i)+ \varepsilon_i

, together with differential information, given either by noisy observations

d_j=(Du^*)(z_j)+ξ_j

or by a known physical constraint

Du^*=v

. We consider the setting where

D

is a linear differential operator and analyze a physics-informed kernel estimator

\hat u

combining

n

value observations and

m

differential observations. In this context, we ask how much can differential information improve predictions, and how does this improvement depend quantitatively on

n

m

, and

D

. We prove finite-sample bounds, supported by numerical simulations, revealing a two-regime structure for the prediction error. When

m

is limited, the rate depends jointly on

n

and

m

; when

m

exceeds a problem-dependent threshold, the rate saturates and matches the oracle rate obtained when the perfect constraint

D \hat u = Du^*

is imposed. Examples are discussed for Sobolev spaces which are reproducing kernel Hilbert spaces and include partial Laplacian constraints on the torus and gradient observations on bounded domains. These examples illustrate the range of possible learning rate improvements --- from the standard nonparametric

n^{-1/4}

to the parametric rate

n^{-1/2}

. Finally, we derive physically consistent rates in a stronger norm that jointly controls the errors in

\hat u

and

D\hat u

Physics-Informed Machine Learning Kernel Method Learning Rates Gradient Concentration

Open paper Ask about it PDF