Explicit Regularization

Explicit regularization in machine learning involves adding penalty terms to optimization objectives to constrain model complexity and improve generalization. Recent research focuses on understanding the interplay between explicit and implicit regularization, particularly in deep learning models like convolutional neural networks and transformers, and how this interplay affects performance in various tasks such as image restoration, matrix completion, and continual learning. This research is crucial for developing more robust and efficient machine learning algorithms, addressing challenges like overfitting and catastrophic forgetting, and ultimately improving the reliability and performance of AI systems across diverse applications.

Papers

November 7, 2024

Impact of Label Noise on Learning Complex Features
Rahul Vashisht, P. Krishna Kumar, Harsha Vardhan Govind, Harish G. Ramaswamy
Gradient Descent Global Impact Stochastic Gradient Descent Label Noise Deep Model Pre Training Model Explicit Regularization

November 2, 2024

The Implicit Bias of Gradient Descent on Separable Multiclass Data
Hrithik Ravi, Clayton Scott, Daniel Soudry, Yutong Wang
Gradient Descent Cross Entropy Loss Implicit Bias Explicit Regularization

September 23, 2024

Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling
Lechao Xiao
Large Language Model Machine Learning Strong Generalization Regularization Model Multiplicative Size Scaling Explicit Regularization Human Like Generalization

August 19, 2024

Implicit Gaussian Splatting with Efficient Multi-Level Tri-Plane Representation
Minye Wu, Tinne Tuytelaars
Gaussian Splatting Novel View Synthesis RSD Difference of Gaussian Explicit Regularization Tri Plane

June 17, 2024

How Neural Networks Learn the Support is an Implicit Regularization Effect of SGD
Pierfrancesco Beneventano, Andrea Pinto, Tomaso Poggio
Neural Network Deep Neural Network Stochastic Gradient Descent Customer Service Implicit Regularization Mini Batch Explicit Regularization Implicit Regularization Effect

June 12, 2024

Deep learning from strongly mixing observations: Sparse-penalized regularization and minimax optimality
William Kengne, Modou Wade
Loss Function Evolving Observation Minimax Optimality Explicit Regularization Sparse Regularization Deep Neural Network Estimator

April 28, 2024

IMEX-Reg: Implicit-Explicit Regularization in the Function Space for Continual Learning
Prashant Bhat, Bharath Renjith, Elahe Arani, Bahram Zonooz
Continual LEArning Generalization Performance Consistency Regularization Function Space Explicit Regularization Recency Bias

February 16, 2024

One-Bit Quantization and Sparsification for Multiclass Linear Classification via Regularized Regression
Reza Ghane, Danil Akhtiamov, Babak Hassibi
Gaussian Mixture Model Linear Regression Random Sparsification Linear Classifier Multiclass Classification Classification Error Explicit Regularization One Bit

February 9, 2024

Convergence of Gradient Descent with Small Initialization for Unregularized Matrix Completion
Jianhao Ma, Salar Fattahi
Gradient Descent Early Stage Convergence Matrix Completion Global Convergence Explicit Regularization Vanilla Gradient Descent Small Initialization Positive Semidefinite

February 1, 2024

Plug-and-Play image restoration with Stochastic deNOising REgularization
Marien Renaud, Jean Prost, Arthur Leclaire, Nicolas Papadakis
Inverse Problem Image Restoration Regularization Model Stochastic Way Ill Posed Inverse Problem Explicit Regularization

January 18, 2024

Offline Imitation Learning by Controlling the Effective Planning Horizon
Hee-Jun Ahn, Seong-Woong Shim, Byung-Jun Lee
Expert Trajectory Explicit Regularization Offline Imitation Offline Imitation Learning Sub Optimal

November 29, 2023

Variational Bayes image restoration with compressive autoencoders
Maud Biquard, Marie Chabert, Florence Genin, Christophe Latry, Thomas Oberlin
Variational Inference Variational Autoencoders Implicit Regularization Autoencoder Based Variational Posterior Explicit Regularization Image Restoration Framework

November 10, 2023

Minimum norm interpolation by perceptra: Explicit regularization and implicit bias
Jiyoung Park, Ian Pelakh, Stephan Wojtowytsch
Empirical Risk Minimization Implicit Bias Explicit Regularization Shallow ReLU Decay Regularization Minimum Norm

August 14, 2023

No Regularization is Needed: An Efficient and Effective Model for Incomplete Label Distribution Learning
Xiang Li, Songcan Chen
Implicit Regularization Label Distribution Explicit Regularization Label Distribution Learning Effective Model Incomplete Multi Label Learning

June 12, 2023

Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization
Jiuling Zhang, Zhiming Ding
Sparse Linear Regression Differentiable Architecture Search Supernet Training Explicit Regularization Information Leakage Generalized Gradient

June 1, 2023

Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networks
Dan Zhao
Deep Network Efficient Learning Implicit Bias Implicit Regularization Deep Linear Explicit Regularization Explicit Point Solo

March 29, 2023

Hard Regularization to Prevent Deep Online Clustering Collapse without Data Augmentation
Louis Mahon, Thomas Lukasiewicz
Data Augmentation Explicit Regularization Online Clustering

December 31, 2022

Spatiotemporal implicit neural representation for unsupervised dynamic MRI reconstruction
Jie Feng, Ruimin Feng, Qing Wu, Zhiyong Zhang, Yuyao Zhang, Hongjiang Wei
Implicit Neural Representation Spatio tempOral Explicit Regularization Undersampled Magnetic Resonance Dynamic Magnetic Resonance Imaging

December 20, 2022

Calibrating Deep Neural Networks using Explicit Regularisation and Dynamic Data Pruning
Ramya Hebbalaguppe, Rishabh Patra, Tirtharaj Dash, Gautam Shroff, Lovekesh Vig
Deep Learning Model Calibration Data Pruning Explicit Regularization Confidence Aware

December 14, 2022

ECON: Explicit Clothed humans Optimized via Normal integration
Yuliang Xiu, Jinlong Yang, Xu Cao, Dimitrios Tzionas, Michael J. Black
Real Human Surface Reconstruction 3D Human Body Model Explicit Regularization Explicit Content Normal Distribution Free Form

Explicit Regularization

Papers

Impact of Label Noise on Learning Complex Features

The Implicit Bias of Gradient Descent on Separable Multiclass Data

Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling

Implicit Gaussian Splatting with Efficient Multi-Level Tri-Plane Representation

How Neural Networks Learn the Support is an Implicit Regularization Effect of SGD

Deep learning from strongly mixing observations: Sparse-penalized regularization and minimax optimality

IMEX-Reg: Implicit-Explicit Regularization in the Function Space for Continual Learning

One-Bit Quantization and Sparsification for Multiclass Linear Classification via Regularized Regression

Convergence of Gradient Descent with Small Initialization for Unregularized Matrix Completion

Plug-and-Play image restoration with Stochastic deNOising REgularization

Offline Imitation Learning by Controlling the Effective Planning Horizon

Variational Bayes image restoration with compressive autoencoders

Minimum norm interpolation by perceptra: Explicit regularization and implicit bias

No Regularization is Needed: An Efficient and Effective Model for Incomplete Label Distribution Learning

Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization

Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networks

Hard Regularization to Prevent Deep Online Clustering Collapse without Data Augmentation

Spatiotemporal implicit neural representation for unsupervised dynamic MRI reconstruction

Calibrating Deep Neural Networks using Explicit Regularisation and Dynamic Data Pruning

ECON: Explicit Clothed humans Optimized via Normal integration