cs.LGNov 2, 2026

Rethinking Fairness Metrics for Multi-Class and Multi-Group Assessment: Not Everything is Binary

Authors: Matteo Grazioso, Fabio Naccarella, Chiara Gallese, Marco S. Nobile

Organizations: Department of Environmental Sciences, Informatics and Statistics, Ca’ Foscari University of Venice, Venice, Italy · Tilburg Institute for Law, Technology, and Society, Tilburg University, Tilburg, The Netherlands · European Center for Living Technologies (ECLT), Venice, Italy

Abstract

Artificial Intelligence, and Machine Learning in general, is becoming pervasive across all domains and disciplines. By means of this data-driven approach, algorithms can learns complex task just by learning from samples. However, if the data-set is biased, they might lead to the development of unfair computer systems, that is, characterised by a prejudice towards specific individuals or groups, which might get an unmotivated output from the system. This circumstance lead to the development of multiple metrics designed to quantify the level of unfairness of an algorithm. However, a generalised assumption in the literature is that the sensitive features and the output labels are binary. When the data is not binary, it is forcefully binarized using a one-versus-the-rest simplification. In this work, we discuss the limitations of existing metrics and propose novel, mathematically sound, and generalized multi-group and multi-class formulations of the metrics. Ultimately, our results highlight the empirical assessment distortion caused by binarization strategies, which reveals the limitations of binary fairness metrics in the multi class setting.

Explore similar work

CardsList
  1. Fairness in multi-class multi-group classification problems via contextial coherent risk measures

    Aug 31, 2026Darinka Dentcheva, Xiangyu TianAlgorithmic FairnessMulticlass Classification

  2. A Generalised Exponentiated Gradient Approach to Enhance Fairness in Binary and Multi-class Classification Tasks

    Mar 22, 2026Maryam Boubekraoui, Giordano d'Aloisio, Antinisca Di MarcoGradient DescentAlgorithmic Fairness

  3. Demystifying the Optimal Fair Classifier in Multi-Class Classification

    May 30, 2026Li Zhang, Yuyuan Li, XiaoHua Feng +3Algorithmic FairnessGroup Fairness