Ratio-based Loss Functions

Lena Helgerth; Andreas Christmann

stat.MLMay 7, 2026

Ratio-based Loss Functions

Authors: Lena Helgerth, Andreas Christmann

Organizations: Department of Mathematics, University of Bayreuth, Chair of Stochastics and Machine Learning, 95440 Bayreuth, Germany

Abstract

Algorithms in machine learning and AI do critically depend on at least three key components: (i) the risk function, which is the expectation of the loss function, (ii) the function space, which is often called the hypothesis space, and (iii) the set of probability measures, which are allowed for the specified algorithm. This paper gives a survey of a certain class of loss functions, which we call ratio-based. In supervised learning, margin-based loss functions for classification tasks depending on the product of the output values $y_i$ and the predictions $f(x_i)$ as well as distance-based loss functions depending on the difference of $y_i$ and $f(x_i)$ for regression are common. Distance-based loss functions are in particular useful, if an additive model assumption seems plausible, i.e. the common signal plus noise assumption. However, in the literature, several loss functions proposed for regression purposes have a multiplicative error structure in mind and pay attention to relative errors, i.e. to the ratio of $y_i$ and $f(x_i)$ . In this survey article, we systematically investigate such ratio-based loss functions and propose a few new losses, which may be interesting for future research. We concentrate on investigating general properties of ratio-based loss functions like continuity, Lipschitz-continuity, convexity, and differentiability, because these properties play a central role in most machine learning algorithms. Therefore, we do not focus on some specific machine learning algorithm to derive universal consistency, learning rates, or stability results. Instead, we want to enable future research in this direction.

Explore similar work

Apr 30, 2026cs.LG

Linear-Core Surrogates: Smooth Loss Functions with Linear Rates for Classification and Structured Prediction

The choice of loss function in classification involves a fundamental trade-off: smooth losses (like Cross-Entropy) enable fast optimization rates but yield slow square-root consistency bounds, while piecewise-linear losses (like Hinge) offer fast linear consistency rates but suffer from non-differentiability. We propose Linear-Core (LC) Surrogates, a new family of convex loss functions that resolve this tension by stitching a linear core to a smooth tail. We prove that these surrogates are differentiable everywhere while retaining strict linear

H

-consistency bounds, effectively combining the optimization benefits of smoothness with the statistical efficiency of margin-based losses. In the structured prediction setting, we show that this smoothness unlocks a massive computational and energy advantage: it allows for an unbiased stochastic gradient estimator that bypasses the quadratic complexity

O(|\mathscr{Y}|^2)

of exact inference (e.g., Viterbi). Empirically, our method achieves a 23

\times

speedup over Structured SVMs on large-vocabulary sequence tagging tasks and demonstrates superior robustness to instance-dependent label noise, outperforming Cross-Entropy by 2.6% on corrupted CIFAR-10.

Mehryar Mohri, Yutao Zhong

Loss Function

Ask PDF

Jun 20, 2026cs.LG

Alternate loss functions and regression models that achieve robustness to outliers by modulating the learning rate

Most real-world datasets used for training supervised learning models are contaminated with noisy data and outliers leading to large prediction errors. This paper proposes a new approach for achieving robustness where the learning rate is modulated by a factor that is sensitive to outliers. In this approach a reduction of the learning rate is shown to be achieved by using alternate loss functions that are infinitely differentiable, strictly convex or quasiconvex and more closely approximate the absolute error than Huber and log-cosh losses. A comparison of the performance of regression models trained with different loss functions on a wide variety of benchmarks and datasets is presented to demonstrate the superior performance of the Square Root Loss (SRL) and Smooth Mean Absolute Error (SMAE) losses proposed in this paper. Two new robust linear regression models are presented. Highly vectorized robust parameter update formulae that take advantage of modern GPUs for both stochastic and batch gradient descent are presented.

Mathew Mithra Noel, Arindam Banerjee, Yug D. Oswal +2

Outliers Loss Function Learning Rates Gradient Descent Mean Squared Error

Ask PDF

May 5, 2026cs.LG

Realizable Bayes-Consistency for General Metric Losses

We study strong universal Bayes-consistency in the realizable setting for learning with general metric losses, extending classical characterizations beyond

0

1

classification (Bousquet et al., 2020; Hanneke et al., 2021) and real-valued regression (Attias et al., 2024). Given an instance space

(X,ρ)

, a label space

(Y,\ell)

with possibly unbounded loss, and a hypothesis class

H \subseteq Y^{X}

, we resolve the realizable case of an open problem presented in Tsir Cohen and Kontorovich (2022). Specifically, we find the necessary and sufficient conditions on the hypothesis class

H

under which there exists a distribution-free learning rule whose risk converges almost surely to the best-in-class risk (which is zero) for every realizable data-generating distribution. Our main contribution is this sharp characterization in terms of a combinatorial obstruction: Similarly to Attias et al. (2024), we introduce the notion of an infinite non-decreasing

(γ_k)

-Littlestone tree, where

γ_k \to \infty

. This extends the Littlestone tree structure used in Bousquet et al. (2020) to the metric loss setting.

Dan Tsir Cohen, Steve Hanneke, Aryeh Kontorovich

Empirical Risk Minimization Bayesian Loss Function Distributional Learning Hypothesis

Ask PDF