Tailed Loss

Tailed loss, referring to the distribution of errors in machine learning models, is a crucial area of research focusing on improving model robustness and efficiency. Current efforts concentrate on developing methods to mitigate the negative impacts of heavy-tailed loss distributions, such as overfitting to outliers and slow convergence, through techniques like loss-weighted sampling and normalized gradient descent. These advancements aim to enhance the accuracy and efficiency of various machine learning tasks, including time series forecasting and object detection, by better managing the distribution of prediction errors and improving generalization performance. The ultimate goal is to build more reliable and efficient models across diverse scientific and practical applications.

Papers

June 19, 2024

Robust Time Series Forecasting with Non-Heavy-Tailed Gaussian Loss-Weighted Sampler
Jiang You, Arben Cela, René Natowicz, Jacob Ouanounou, Patrick Siarry
Multivariate Time Series Forecasting Weighted Loss Robust Forecasting Tailed Loss

June 5, 2023

On Tail Decay Rate Estimation of Loss Function Distributions
Etrit Haxholli, Marco Lorenzi
Loss Function Decay Function Tail Risk Tailed Loss

May 22, 2023

Fast Convergence in Learning Two-Layer Neural Networks with Separable Data
Hossein Taheri, Christos Thrampoulidis
Gradient Descent Generalization Bound Two Layer Neural Network Linear Classifier Separable Data Exponential Convergence Rate Tailed Loss

April 10, 2023

Head-tail Loss: A simple function for Oriented Object Detection and Anchor-free models
Pau Gallés, Xi Chen
Loss Function Human Prediction Simple Function Bounding Box Thin Tail Oriented Object Detection Anchor Free Tailed Loss

March 13, 2023

General Loss Functions Lead to (Approximate) Interpolation in High Dimensions
Kuo-Wei Lai, Vidya Muthukumar
Loss Function Gradient Descent Average Approximation High Dimension Interpolation Regime Convex Loss General Convex Loss Tailed Loss

October 3, 2022

PAC-Bayes Generalisation Bounds for Heavy-Tailed Losses through Supermartingales
Maxime Haddouche, Benjamin Guedj
PAC Bayesian Martingale Difference PAC Bayesian Generalization Bound Tailed Loss

March 28, 2022

Flexible risk design using bi-directional dispersion
Matthew J. Holland
Gradient Based Test Loss Adaptive Risk Tailed Loss General Tail Behavior

November 15, 2021

Reducing the Long Tail Losses in Scientific Emulations with Active Learning
Yi Heng Lim, Muhammad Firmansyah Kasim
Deep Learning Model Active Learning Model Performance Core Set Tailed Loss Scientific Simulation