Non Convex Loss Function

Non-convex loss functions pose significant challenges in machine learning due to the absence of a globally optimal solution, hindering the development of efficient and robust algorithms. Current research focuses on developing novel algorithms for federated learning, Byzantine-resilient training, and differentially private optimization, often employing techniques like gradient normalization, adaptive client selection, and model sparsification to mitigate the difficulties associated with non-convexity. These advancements are crucial for improving the efficiency, privacy, and robustness of machine learning models in distributed and sensitive data settings, impacting various applications from healthcare to large-scale data analysis.

Papers

October 1, 2023

Spectral Neural Networks: Approximation Theory and Optimization Landscape
Chenghui Li, Rishi Sonthalia, Nicolas Garcia Trillos
Training Dynamic Non Convex Loss Function Optimization Landscape Spectral Learning Approximation Theory Spectral Neural Network

September 10, 2023

Generalization error bounds for iterative learning algorithms with bounded updates
Jingwen Fu, Nanning Zheng
Generalization Error Information Theoretic Iterative Training Non Convex Loss Function Information System Update Generalization Theory

July 14, 2023

Improved Convergence Analysis and SNR Control Strategies for Federated Learning in the Presence of Noise
Antesh Upadhyay, Abolfazl Hashemi
Industrial Disturbing Noise Speech Presence Convergence Analysis Signal to Noise Ratio Non Convex Loss Function Noisy Communication

June 22, 2023

Uniform Convergence with Square-Root Lipschitz Loss
Lijia Zhou, Zhen Dai, Frederic Koehler, Nathan Srebro
Loss Function Non Convex Loss Function Uniform Convergence Lipschitz Loss Learning Guarantee

May 11, 2023

An Adaptive Graduated Nonconvexity Loss Function for Robust Nonlinear Least Squares Solutions
Kyungmin Jung, Thomas Hitchcox, James Richard Forbes
Loss Function Least Square Non Convex Loss Function Robust Loss Function Robust Linear

March 22, 2023

Delay-Aware Hierarchical Federated Learning
Frank Po-Chen Lin, Seyyedali Hosseinalipour, Nicolò Michelusi, Christopher Brinton
Federated Learning Stochastic Gradient Descent Non Convex Loss Function Model Convergence

February 7, 2023

Decentralized Inexact Proximal Gradient Method With Network-Independent Stepsizes for Convex Composite Optimization
Luyao Guo, Xinli Shi, Jinde Cao, Zihao Wang
Step Size Non Convex Loss Function Decentralized Algorithm Convex Composite Optimization Composite Optimization Decentralized Proximal

October 14, 2022

Revisiting Optimal Convergence Rate for Smooth and Non-convex Stochastic Decentralized Optimization
Kun Yuan, Xinmeng Huang, Yiming Chen, Xiaohan Zhang, Yingya Zhang, Pan Pan
Sparse Connectivity Non Convex Loss Function Decentralized Optimization Decentralized Algorithm Optimal Convergence Rate Decentralized Nonconvex

September 15, 2022

Private Stochastic Optimization With Large Worst-Case Lipschitz Parameter: Optimal Rates for (Non-Smooth) Convex Losses and Extension to Non-Convex Losses
Andrew Lowy, Meisam Razaviyayn
Convex Loss Private Stochastic Non Convex Loss Function Optimal Rate Convex Loss Function Lipschitz Bound Risk Bound

August 19, 2022

Sudakov-Fernique post-AMP, and a new proof of the local convexity of the TAP free energy
Michael Celentano
Free Energy Non Convex Loss Function Proof Pile 2 Gibbs Distribution Approximate Message Passing Single Tap

July 31, 2022

Online Decentralized Frank-Wolfe: From theoretical bound to applications in smart-building
Angan Mitra, Nguyen Kim Thang, Tuan-Anh Nguyen, Denis Trystram, Paul Youssef
Financial Application Practical Algorithm Online Algorithm Non Convex Loss Function Smart Building

July 15, 2022

Support Vector Machines with the Hard-Margin Loss: Optimal Training via Combinatorial Benders' Cuts
Ítalo Santana, Breno Serrano, Maximilian Schiffer, Thibaut Vidal
Loss Function Non Convex Loss Function Cut and Approximate Margin Loss Soft Margin Optimal Training Bender Decomposition

June 7, 2022

Generalization Error Bounds for Deep Neural Networks Trained by SGD
Mingze Wang, Chao Ma
Deep Neural Network Gradient Descent Stochastic Gradient Descent Langevin Dynamic Non Convex Loss Function Generalization Error Bound Algorithm Dependent Generalization

May 31, 2022

Communication-Efficient Distributionally Robust Decentralized Learning
Matteo Zecchin, Marios Kountouris, David Gesbert
Communication Efficient Decentralized FL Non Convex Loss Function Robust Algorithm Decentralized Gradient

May 27, 2022

April 26, 2022

Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD
Konstantinos E. Nikolakakis, Farzin Haddadpour, Amin Karbasi, Dionysios S. Kalogerias
Convex Loss Non Convex Loss Function Risk Bound Full Batch Gradient Descent Norm Based Generalization Bound

April 20, 2022

Efficient Wireless Federated Learning with Partial Model Aggregation
Zhixiong Chen, Wenqiang Yi, Arumugam Nallanathan, Geoffrey Ye Li
Federated Learning Loss Function Model Aggregation Non Convex Loss Function Device Scheduling

March 24, 2022

Distributionally Robust Optimization via Ball Oracle Acceleration
Yair Carmon, Danielle Hausler
Robust Optimization Distributionally Robust Optimization Convex Loss Non Convex Loss Function Uncertainty Set

March 13, 2022

Private Non-Convex Federated Learning Without a Trusted Server
Andrew Lowy, Ali Ghafelebashi, Meisam Razaviyayn
Differential Privacy Private Federated Learning Non Convex Loss Function Private Optimization

Non Convex Loss Function

Papers

Spectral Neural Networks: Approximation Theory and Optimization Landscape

Generalization error bounds for iterative learning algorithms with bounded updates

Improved Convergence Analysis and SNR Control Strategies for Federated Learning in the Presence of Noise

Uniform Convergence with Square-Root Lipschitz Loss

An Adaptive Graduated Nonconvexity Loss Function for Robust Nonlinear Least Squares Solutions

Delay-Aware Hierarchical Federated Learning

Decentralized Inexact Proximal Gradient Method With Network-Independent Stepsizes for Convex Composite Optimization

Revisiting Optimal Convergence Rate for Smooth and Non-convex Stochastic Decentralized Optimization

Private Stochastic Optimization With Large Worst-Case Lipschitz Parameter: Optimal Rates for (Non-Smooth) Convex Losses and Extension to Non-Convex Losses

Sudakov-Fernique post-AMP, and a new proof of the local convexity of the TAP free energy

Online Decentralized Frank-Wolfe: From theoretical bound to applications in smart-building

Support Vector Machines with the Hard-Margin Loss: Optimal Training via Combinatorial Benders' Cuts

Generalization Error Bounds for Deep Neural Networks Trained by SGD

Communication-Efficient Distributionally Robust Decentralized Learning

Towards Communication-Learning Trade-off for Federated Learning at the Network Edge

AsyncFedED: Asynchronous Federated Learning with Euclidean Distance based Adaptive Weight Aggregation

Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD

Efficient Wireless Federated Learning with Partial Model Aggregation

Distributionally Robust Optimization via Ball Oracle Acceleration

Private Non-Convex Federated Learning Without a Trusted Server