Risk Bound

Risk bounds in machine learning quantify the difference between a model's performance on training data and its generalization to unseen data. Current research focuses on tightening these bounds for various algorithms, including stochastic gradient descent (SGD) and its accelerated variants, as well as analyzing their performance under different data assumptions (e.g., heterogeneous, dependent, or heavy-tailed data) and privacy constraints. This work is crucial for understanding and improving the reliability and efficiency of machine learning models, particularly in applications where generalization performance is paramount, such as healthcare and finance. Improved risk bounds inform the design of more robust and efficient algorithms, leading to better predictive models.

Papers

November 5, 2024

Generalization and Risk Bounds for Recurrent Neural Networks
Xuewei Cheng, Ke Huang, Shujie Ma
Strong Generalization Loss Function Recurrent Neural Network Risk Bound

October 29, 2024

Refined Risk Bounds for Unbounded Losses via Transductive Priors
Jian Qian, Alexander Rakhlin, Nikita Zhivotovskiy
Regret Bound Regression Task Linear Regression Risk Bound Unbounded Loss Transductive Linear

October 18, 2024

Provable In-context Learning for Mixture of Linear Regressions using Transformers
Yanhao Jin, Krishnakumar Balasubramanian, Lifeng Lai
Transformer Megatron Decepticons Context Learning Mixture Component Attention Layer Expectation Maximization Linear Regression Risk Bound Mixture Markov Model

October 14, 2024

Tighter Risk Bounds for Mixtures of Experts
Wissam Akretche, Frédéric LeBlanc, Mario Marchand
Mixture Component Expert Knowledge Local Differential Privacy Upper Bound Theoretical Guarantee Gating Mechanism Linear Dependence Risk Bound

October 13, 2024

Stability and Sharper Risk Bounds with Convergence Rate $O(1/n^2)$
Bowei Zhu, Shaojie Li, Yong Liu
Gradient Descent Stochastic Gradient Descent Early Stage Convergence Core Stability Empirical Risk Minimization Risk Bound Nonconvex Nonconcave Problem

October 11, 2024

Towards Sharper Risk Bounds for Minimax Problems
Bowei Zhu, Shaojie Li, Yong Liu
Generalization Bound Minimax Optimization Problem Strongly Convex Risk Bound

July 12, 2024

Private Heterogeneous Federated Learning Without a Trusted Server Revisited: Error-Optimal and Communication-Efficient Algorithms for Convex Losses
Changyu Gao, Andrew Lowy, Xingyu Zhou, Stephen J. Wright
Differential Privacy Communication Efficient Convex Loss Private Federated Learning Cross Silo Federated Learning Risk Bound

April 19, 2024

Risk Bounds for Mixture Density Estimation on Compact Domains via the $h$-Lifted Kullback--Leibler Divergence
Mark Chiu Chong, Hien Duy Nguyen, TrungTin Nguyen
Mixture Model Kullback Leibler Divergence Probability Density Function Risk Bound Density Evolution Likelihood Estimation Continuous Domain

November 23, 2023

Risk Bounds of Accelerated SGD for Overparameterized Linear Regression
Xuheng Li, Yihe Deng, Jingfeng Wu, Dongruo Zhou, Quanquan Gu
Strong Generalization Generalization Performance Faster Convergence Better Generalization Parameterized Model Accelerated Gradient Descent Risk Bound

August 15, 2023

High-Probability Risk Bounds via Sequential Predictors
Dirk van der Hoeven, Nikita Zhivotovskiy, Nicolò Cesa-Bianchi
Regret Bound Online Algorithm Online Learning Algorithm Sequential Prediction Risk Bound

July 31, 2023

Lossless Transformations and Excess Risk Bounds in Statistical Inference
László Györfi, Tamás Linder, Harro Walk
Loss Function Information Theoretic Statistical Inference Excess Risk Risk Bound Lossless Transformation

June 29, 2023

Local Risk Bounds for Statistical Aggregation
Jaouad Mourtada, Tomas Vaškevičius, Nikita Zhivotovskiy
Risk Bound Mean Aggregation Exponential Weight Q Aggregation

March 21, 2023

Uniform Risk Bounds for Learning with Dependent Data Sequences
Fabien Lauer
LeArning Abstract Sample Complexity Sequential Model Risk Bound VC Dimension Complexity Bound Accuracy Constraint

March 2, 2023

Tight Risk Bounds for Gradient Descent on Separable Data
Matan Schliserman, Tomer Koren
Gradient Descent Stochastic Gradient Descent Gradient Method Gradient Free Separable Data Risk Bound Smooth Loss Function

December 19, 2022

The One-Inclusion Graph Algorithm is not Always Optimal
Ishaq Aden-Ali, Yeshwanth Cherapanamjeri, Abhishek Shetty, Nikita Zhivotovskiy
Risk Bound

November 3, 2022

Optimal Algorithms for Stochastic Complementary Composite Minimization
Alexandre d'Aspremont, Cristóbal Guzmán, Clément Lezane
Regularization Technique Optimal Algorithm Regularization Term Risk Bound Convex Composite Optimization Complexity Bound

September 19, 2022

Deep Linear Networks can Benignly Overfit when Shallow Ones Do
Niladri S. Chatterji, Philip M. Long
Gradient Flow Deep Linear Benign Overfitting Risk Bound Shallow Water

September 15, 2022

Private Stochastic Optimization With Large Worst-Case Lipschitz Parameter: Optimal Rates for (Non-Smooth) Convex Losses and Extension to Non-Convex Losses
Andrew Lowy, Meisam Razaviyayn
Convex Loss Private Stochastic Non Convex Loss Function Optimal Rate Convex Loss Function Lipschitz Bound Risk Bound

April 30, 2022

Understanding the Generalization Performance of Spectral Clustering Algorithms
Shaojie Li, Sheng Ouyang, Yong Liu
Generalization Performance Spectral Clustering Continuous Optimization Risk Bound

April 26, 2022

Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD
Konstantinos E. Nikolakakis, Farzin Haddadpour, Amin Karbasi, Dionysios S. Kalogerias
Convex Loss Non Convex Loss Function Risk Bound Full Batch Gradient Descent Norm Based Generalization Bound