Regularization

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

16 new papers

A weekly snapshot of new work published in Regularization.

Period ending 2026-09-14

18 new papers

A weekly snapshot of new work published in Regularization.

Period ending 2026-09-07

11 new papers

A weekly snapshot of new work published in Regularization.

Inside this field

Focused directions

Focused topic

Regularization Techniques

Within Regularization

363 papersExplore

Focused topic

Kullback-Leibler Divergence

Within Regularization

102 papersExplore

Focused topic

Divergence

Within Regularization

58 papersExplore

Focused topic

Bregman Divergences

Within Regularization

53 papersExplore

Focused topic

Entropy-Regularized Equilibrium Selection

Within Regularization

45 papersExplore

Focused topic

Kullback-Leibler Regularization

Within Regularization

27 papersExplore

Focused topic

Quantitative Idea Divergence

Within Regularization

19 papersExplore

638 papers

Latest in Regularization

Browse all topics →

Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalization
May 7, 2026Ruotong Sun, Ermin WeiSpectral Preconditioning Orthogonality
Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networks
May 7, 2026Yi Wei, Xuan Qi, Furao ShenAffine Deep Network
Playing the network backward: A Game Theoretic Attribution Framework
May 7, 2026Jakob Paul Zimmermann, Jim Berend, Georg Loho +2Training Data Attribution Backward Induction
Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes
May 7, 2026Liu Hanqing, Jianjun Cao, Yuanze Li +1Loss Landscape Full-Precision Performance
Safety Anchor: Defending Harmful Fine-tuning via Geometric Bottlenecks
May 7, 2026Guoxin Lu, Letian Sha, Qing Wang +4Harmful Fine-Tuning Large Language Model Safety
HEDP: A Hybrid Energy-Distance Prompt-based Framework for Domain Incremental Learning
May 7, 2026Yu Feng, Zhen Tian, Haoran Luo +8Domain Incremental Learning Domain Adaptation
Jointly Learning Structured Representations and Stabilized Affinity for Human Motion Segmentation
May 7, 2026Xianghan Meng, Zhiyuan Huang, Zhengyu Tong +1Motion-Focused Representations Human Motion Generation
RVPO: Risk-Sensitive Alignment via Variance Regularization
May 7, 2026Ivan Montero, Tomasz Jurczyk, Bhuwan DhingraReinforcement Learning From Human Feedback Regularization Techniques
Convexity in Disguise: A Theoretical Framework for Nonconvex Low-Rank Matrix Estimation
May 6, 2026Chengyu Cui, Gongjun XuLow-Rank Factorization Nonconvex
Estimating Implicit Regularization in Deep Learning
May 6, 2026Joseph H. Rudoler, Kevin Tan, Giles Hooker +1Regularization Techniques Quadratic Slack Penalty
Information Theoretic Adversarial Training of Large Language Models
May 6, 2026Yiwei Zhang, Jeremiah Birrell, Reza Ebrahimi +3Language Model Evasion Attacks Adversarial Transferability
Feature Starvation as Geometric Instability in Sparse Autoencoders
May 6, 2026Faris Chaudhry, Keisuke Yano, Anthea MonodMasked Autoencoders Regularization Techniques
On the Wasserstein Gradient Flow Interpretation of Drifting Models
May 6, 2026Arthur Gretton, Li Kevin Wenliang, Alexandre Galashov +3Wasserstein Gradient Flows Generative Models
Proximal Projection for Doubly Sparse Regularized Models
May 6, 2026Jia Wei He, R. Ayesha Ali, Gerarda DarlingtonRegularization Techniques Graph Representation Learning
Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization
May 6, 2026Xin Yu, Liuchen Liao, Yiwen Zhang +3Unsupervised On-Policy Self-Distillation On-Policy Distillation
Demystifying Manifold Constraints in LLM Pre-training
May 6, 2026Kang An, Jiaxiang Li, Donald Goldfarb +1Riemannian Optimization Large Language Model Pretraining
Beyond Penalization: Diffusion-based Out-of-Distribution Detection and Selective Regularization in Offline Reinforcement Learning
May 6, 2026Qingjun Wang, Hongtu Zhou, Hang Yu +5Offline Reinforcement Learning Out-Of-Distribution Detection
Entropic Riemannian Neural Optimal Transport
May 5, 2026Alessandro Micheli, Silvia Sapora, Anthea Monod +1Optimal Transport Entropy-Regularized Equilibrium Selection
Topology-Constrained Quantized nnUNet for Efficient and Anatomically Accurate 3D Tooth Segmentation
May 5, 2026Paarth Prasad, Ruchika MalhotraTopological Data Analysis Regularization Techniques
Complex Equation Learner: Rational Symbolic Regression with Gradient Descent in Complex Domain
May 5, 2026Sergei Garmaev, Maurice Gauché, Olga FinkSymbolic Regression Regularization Techniques
Vanishing L2 regularization for the softmax Multi Armed Bandit
May 5, 2026Stefana-Lucia Anita, Gabriel TuriniciMulti-Armed Bandits Policy Gradient
Quantile Geometry Regularization for Distributional Reinforcement Learning
May 5, 2026Zhaofan Zhang, Minghao Yang, Rufeng Chen +2Quantile Regression Wasserstein Distance
DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignment
May 5, 2026Hongbo Jin, Rongpeng Zhu, Zhongjing Du +4Multi-Objective Policy Optimization Credit Assignment
Generalized Distributional Alignment Games for Unbiased Answer-Level Fine-Tuning
May 4, 2026Mehryar Mohri, Jon Schneider, Yutao ZhongKullback-Leibler Divergence Training-Inference Mismatch
Binary Rewards and Reinforcement Learning: Fundamental Challenges
May 4, 2026Marc DymetmanReinforcement Learning With Verifiable Reward Kullback-Leibler Divergence
On the Optimal Sample Complexity of Offline Multi-Armed Bandits with KL Regularization
May 4, 2026Kaixuan Ji, Qiwei Di, Heyang Zhao +2Kullback-Leibler Regularization Linear Bandits
DR-SNE: Density-Regularized Stochastic Neighbor Embedding
May 3, 2026Maksim KazanskiiDimensionality Reduction Density Ratio Estimation
Large margin classifier with graph-based adaptive regularization
May 3, 2026Vítor M. Hanriot, Turíbio T. Salis, Luiz C. B. Torres +2Class Imbalance Regularization Techniques
Adversarial Imitation Learning with General Function Approximation: Theoretical Analysis and Practical Algorithms
May 3, 2026Tian Xu, Zhilong Zhang, Zexuan Chen +3Imitation Learning Learning-Augmented Algorithms
Regularized Centered Emphatic Temporal Difference Learning
May 2, 2026Xingguo Chen, Chaohui Wu, Jinguo Ye +5Temporal Difference Regularization Techniques
MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate
May 2, 2026Jianze Wang, Ying Liu, Jinlong Chen +7On-Policy Distillation Token-Level Supervision
Colinearity Decay: Training Quantization-Friendly ViTs with Outlier Decay
May 2, 2026Jin Tong, Guang Liang, Peilin Sun +1Vision Transformer Regularization Techniques
Benchmarking ResNet Backbones in RT-DETR: Impact of Depth and Regularization under environmental conditions
May 1, 2026Pamela Barboza, Víctor Castelli, Belén Pereira +3Rt-Detr Resnet-18
Gradient Regularized Newton Boosting Trees with Global Convergence
May 1, 2026Nikita Zozoulenko, Daniel Falkowski, Thomas Cass +1Gradient Boosted Tree Proximal Gradient Descent
Healthcare AI GYM for Medical Agents
May 1, 2026Minbyul JeongClinical Reasoning Multi-Turn Tool-Use Agent
Divergence is Uncertainty: A Closed-Form Posterior Covariance for Flow Matching
May 1, 2026Jiarui Xing, Song Wang, Jian WangGenerative Models Posterior
Pessimism-Free Offline Learning in General-Sum Games via KL Regularization
Apr 30, 2026Claire Chen, Yuheng ZhangKullback-Leibler Regularization Offline Reinforcement Learning
Adaptive Norm-Based Regularization for Neural Networks
Apr 30, 2026Muhammad Qasim, Farrukh JavedRegularization Techniques Lasso
AIDA-ReID: Adaptive Intermediate Domain Adaptation for Generalizable and Source-Free Person Re-Identification
Apr 30, 2026Sundas Iqbal, Qing Tian, Danish Ali +2Re-Identification Domain Adaptation
Exponential families from a single KL identity
Apr 30, 2026Marc DymetmanKullback-Leibler Divergence Exponents
ITS-Mina: A Harris Hawks Optimization-Based All-MLP Framework with Iterative Refinement and External Attention for Multivariate Time Series Forecasting
Apr 30, 2026Pourya Zamanvaziri, Amirhossein Sadr, Aida Pakniyat +1Multivariate Time Series Forecasting Temporal Convolutional Networks
Better Models, Faster Training: Sigmoid Attention for single-cell Foundation Models
Apr 29, 2026Vijay Sadashivaiah, Georgios Dasoulas, Judith Mueller +1Softmax Attention Wireless Foundation Models
Semi-supervised learning with max-margin graph cuts
Apr 29, 2026Branislav Kveton, Michal Valko, Ali Rahimi +1Semi-Supervised Learning Support Vector Machines
DSIPA: Detecting LLM-Generated Texts via Sentiment-Invariant Patterns Divergence Analysis
Apr 29, 2026Siyuan Li, Aodu Wulianghai, Guangyan Li +5Machine-Generated Text Detection Divergence
Deflation-Free Optimal Scoring
Apr 28, 2026Sharmin Afroz, Brendan AmesSupport Vector Machines Regularization Techniques
SaliencyDecor: Enhancing Neural Network Interpretability through Feature Decorrelation
Apr 28, 2026Ali Karkehabadi, Jamshid Hassanpour, Houman Homayoun +1Saliency Model Interpretability
Adversarial Robustness of NTK Neural Networks
Apr 28, 2026Yuxuan HouNeural Tangent Kernel Neural Network
Alice v1: Distillation-Enhanced Video Generation Surpassing Closed-Source Models
Apr 27, 2026Wang Xiaoyu, Phong Nguyen, Chen ZhaoLarge-Scale Video Generation Models Asymmetric Distillation
Statistical Inference and Quality Measures of KV Cache Quantisations Inspired by TurboQuant
Apr 27, 2026Paolo D'AlbertoKey-Value Cache Quantization Rate-Distortion Theory
Scaling Properties of Continuous Diffusion Spoken Language Models
Apr 27, 2026Jason Ramapuram, Eeshan Gunesh Dhekane, Amitis Shidani +6Diffusion Language Models Scaling Laws
Semantic Segmentation for Histopathology using Learned Regularization based on Global Proportions
Apr 27, 2026Yangping Li, Thomas Pinetz, Michael Hölzel +2Computational Pathology Robust Segmentation
A Divergence-Based Method for Weighting and Averaging Model Predictions
Apr 27, 2026Olav Benjamin VassendBregman Divergences Quantitative Idea Divergence
GEM: Geometric Entropy Mixing for Optimal LLM Data Curation
Apr 27, 2026Yue Min, Ziyun Qiao, Ruining Chen +1Data-Curation Large Language Model Pretraining
Generalising maximum mean discrepancy: kernelised functional Bregman divergences
Apr 27, 2026Russell Tsuchida, Frank NielsenKernel Hilbert Spaces Bregman Divergences
TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents
Apr 27, 2026Jiaqi Wang, Wenhao Zhang, Weijie Shi +2On-Policy Distillation Curriculum
When Does Removing LayerNorm Help? Activation Bounding as a Regime-Dependent Implicit Regularizer
Apr 25, 2026Lucky VermaBatch Normalization Activation Functions
Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distance
Apr 25, 2026Xinzhu Chen, Wei He, Huichuan Fan +7Credit Assignment Group Relative Policy Optimization
Learning Curves and Benign Overfitting of Spectral Algorithms in Large Dimensions
Apr 25, 2026Weihao Lu, Qian Lin, Yingcun Xia +1Kernel Ridge Regression Overfitting
A Unified Fractional Regularization Framework for Sparse Recovery
Apr 25, 2026Yinhao Zhao, Haoyu He, Chuanqi Ma +1Sparse Recovery Regularization Techniques
Turtle shell clustering: A mixture approach to discriminative clustering with applications to flow cytometry and other data
Apr 25, 2026Mackenzie R. Neal, Paul D. McNicholas, Arthur WhiteImage Clustering Single-Cell Rna Sequencing

May 7, 2026 · cs.LGJ/K move · Enter open · S save

Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalization

Ruotong Sun, Ermin Wei

†Department of Industrial Engineering & Management Sciences, Northwestern University · ∗Department of Electrical & Computer Engineering, Northwestern University

Optimizers that exploit the matrix structure of gradients are central to modern LLM pre-training, with two distinct frontiers: explicit Kronecker-factored preconditioning -- most recently KL-Shampoo, which estimates the preconditioner via KL divergence minimization -- and orthogonalization of the gradient momentum, exemplified by Muon and analyzed as steepest descent under the spectral norm. The two routes are typically developed in isolation. We make a structural observation about KL-Shampoo's Kronecker preconditioners: their eigenvalue spectra exhibit a \emph{spike-and-flat} shape -- a few dominant eigenvalues followed by an approximately uniform tail -- across layers and training stages, holding exactly under a rank-

ρ

signal-plus-noise gradient model. We exploit this structure by restricting one of KL-Shampoo's Kronecker factors to a parametric family aligned with the spike-and-flat shape: full spectral structure on a tracked

r

-dimensional subspace, single shared eigenvalue across the remaining

n-r

directions. On these directions, we apply orthogonalization. An identity shows that this orthogonalization recovers the algebraic form of full KL-Shampoo's preconditioner. On four pre-training scales (GPT-2 124M / 350M, LLaMA 134M / 450M), Pro-KLShampoo consistently outperforms KL-Shampoo at every subspace rank we test in validation loss, peak per-GPU memory, and wallclock time to reach each loss level.

Spectral Preconditioning Orthogonality Kullback-Leibler Divergence

Open paper Ask about it PDF