Convex Loss Function

Convex loss functions are fundamental in machine learning, enabling efficient optimization algorithms to find model parameters minimizing prediction error. Current research focuses on extending their application to distributed and federated learning settings, addressing challenges like data heterogeneity, adversarial attacks, and stragglers, often employing techniques like mirror descent and gradient coding. This work also explores tighter convex loss functions (e.g., Fitzpatrick losses) and their application in robust optimization frameworks, including distributionally robust optimization and differentially private optimization, to improve model robustness and privacy. The resulting advancements enhance the efficiency, robustness, and privacy of machine learning models across various applications.

Papers

September 15, 2024

Rewind-to-Delete: Certified Machine Unlearning for Nonconvex Functions
Siqiao Mu, Diego Klabjan
Machine Unlearning Harmful Unlearning Non Convex Loss Function Non Convex Objective Convex Loss Function Nonconvex Function

July 19, 2024

A Mirror Descent-Based Algorithm for Corruption-Tolerant Distributed Gradient Descent
Shuche Wang, Vincent Y. F. Tan
Practical Algorithm Gradient Descent Mirror Descent Etruscan Mirror Convex Loss Function Adversarial Corruption

May 23, 2024

Learning with Fitzpatrick Losses
Seta Rakotomandimby, Jean-Philippe Chancelier, Michel de Lara, Mathieu Blondel
LeArning Abstract Loss Function Convex Loss Function Cheeger Inequality

April 1, 2024

Large-Scale Non-convex Stochastic Constrained Distributionally Robust Optimization
Qi Zhang, Yi Zhou, Ashley Prater-Bennette, Lixin Shen, Shaofeng Zou
Robust Optimization Distributionally Robust Optimization Non Convex Loss Function Convex Loss Function

March 20, 2024

Byzantine-resilient Federated Learning With Adaptivity to Data Heterogeneity
Shiyuan Zuo, Xingrun Yan, Rongfei Fan, Han Hu, Hangguan Shan, Tony Q. S. Quek
Adaptive Importance Data Heterogeneity Non Convex Loss Function Byzantine Attack Byzantine Robust Federated Learning Convex Loss Function

March 19, 2024

Distributed Learning based on 1-Bit Gradient Coding in the Presence of Stragglers
Chengxi Li, Mikael Skoglund
Natural Gradient Speech Presence Convergence Guarantee Convex Loss Distributed Learning Convex Loss Function Straggler Mitigation Gradient Coding

February 20, 2024

Fast Rates in Online Convex Optimization by Exploiting the Curvature of Feasible Sets
Taira Tsuchiya, Shinji Ito
Online Convex Optimization Gaussian Curvature Fast Rate Logarithmic Regret Convex Loss Function Online Linear Feasible Set

February 6, 2024

Gradient Coding in Decentralized Learning for Evading Stragglers
Chengxi Li, Mikael Skoglund
Natural Gradient Stochastic Gradient Decentralized Learning Convex Loss Function Straggler Mitigation Gradient Coding

December 26, 2023

An extended asymmetric sigmoid with Perceptron (SIGTRON) for imbalanced linear classification
Hyenkyun Woo
Linear Classifier Convex Loss Convex Loss Function Sigmoid Function Elementary Perceptron

October 17, 2023

Multi-point Feedback of Bandit Convex Optimization with Hard Constraints
Yasunari Hikima
Convex Optimization Proximal Gradient Descent Hard Constraint Convex Loss Function Constraint Violation Bandit Convex Multi Point Feedback

August 17, 2023

July 17, 2023

Universal Online Learning with Gradient Variations: A Multi-layer Online Ensemble Approach
Yu-Hu Yan, Peng Zhao, Zhi-Hua Zhou
Online Convex Optimization Stochastic Convex Optimization Convex Loss Function Universal Learning Small Loss Gradient Difference Online Ensemble

May 22, 2023

Faster Differentially Private Convex Optimization via Second-Order Methods
Arun Ganesh, Mahdi Haghifam, Thomas Steinke, Abhradeep Thakurta
Gradient Descent Convex Optimization Second Order Convex Loss Function Second Order Algorithm Private Convex Optimization

May 14, 2023

An Optimal and Scalable Matrix Mechanism for Noisy Marginals under Convex Loss Functions
Yingtai Xiao, Guanlin He, Danfeng Zhang, Daniel Kifer
Loss Function Marginal Distribution Convex Loss Function

May 3, 2023

Select without Fear: Almost All Mini-Batch Schedules Generalize Optimally
Konstantinos E. Nikolakakis, Amin Karbasi, Dionysis Kalogerias
Generalization Error Convex Loss Mini Batch Convex Loss Function FEAR Speech Mini Batch Gradient Descent

March 31, 2023

Differentially Private Stochastic Convex Optimization in (Non)-Euclidean Space Revisited
Jinyan Su, Changhong Zhao, Di Wang
Convex Function Non Euclidean Private Gradient Convex Loss Function \Ell_p$ Norm

February 17, 2023

Smoothly Giving up: Robustness for Simple Models
Tyler Sypherd, Nathan Stromberg, Richard Nock, Visar Berisha, Lalitha Sankar
Native Robustness Logistic Regression Convex Loss New Hyperparameter Convex Loss Function Simple Model

October 18, 2022

Online Convex Optimization with Unbounded Memory
Raunak Kumar, Sarah Dean, Robert Kleinberg
Regret Bound Online Convex Optimization Convex Loss Function Online Control Algorithm Online Learning Problem

September 15, 2022

Private Stochastic Optimization With Large Worst-Case Lipschitz Parameter: Optimal Rates for (Non-Smooth) Convex Losses and Extension to Non-Convex Losses
Andrew Lowy, Meisam Razaviyayn
Convex Loss Private Stochastic Non Convex Loss Function Optimal Rate Convex Loss Function Lipschitz Bound Risk Bound

Convex Loss Function

Papers

Rewind-to-Delete: Certified Machine Unlearning for Nonconvex Functions

A Mirror Descent-Based Algorithm for Corruption-Tolerant Distributed Gradient Descent

Learning with Fitzpatrick Losses

Large-Scale Non-convex Stochastic Constrained Distributionally Robust Optimization

Byzantine-resilient Federated Learning With Adaptivity to Data Heterogeneity

Distributed Learning based on 1-Bit Gradient Coding in the Presence of Stragglers

Fast Rates in Online Convex Optimization by Exploiting the Curvature of Feasible Sets

Gradient Coding in Decentralized Learning for Evading Stragglers

An extended asymmetric sigmoid with Perceptron (SIGTRON) for imbalanced linear classification

Multi-point Feedback of Bandit Convex Optimization with Hard Constraints

Over-the-Air Computation Aided Federated Learning with the Aggregation of Normalized Gradient

Dual Gauss-Newton Directions for Deep Learning

Universal Online Learning with Gradient Variations: A Multi-layer Online Ensemble Approach

Faster Differentially Private Convex Optimization via Second-Order Methods

An Optimal and Scalable Matrix Mechanism for Noisy Marginals under Convex Loss Functions

Select without Fear: Almost All Mini-Batch Schedules Generalize Optimally

Differentially Private Stochastic Convex Optimization in (Non)-Euclidean Space Revisited

Smoothly Giving up: Robustness for Simple Models

Online Convex Optimization with Unbounded Memory

Private Stochastic Optimization With Large Worst-Case Lipschitz Parameter: Optimal Rates for (Non-Smooth) Convex Losses and Extension to Non-Convex Losses