Fairness Benchmark Datasets

Fairness benchmark datasets are crucial for evaluating and mitigating bias in machine learning models, particularly those used in high-stakes decision-making. Current research focuses on developing standardized fairness metrics, investigating bias across diverse demographic attributes in various model architectures (including vision-language models and tree-based methods), and exploring methods to improve both fairness and model accuracy simultaneously, often using techniques like adversarial learning or multi-objective optimization. These datasets and associated research are vital for advancing the development of fairer and more trustworthy AI systems across numerous sectors, impacting both the scientific understanding of algorithmic bias and the ethical deployment of AI in practice.

Papers

June 25, 2024

Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Prompts
Xuyang Wu, Yuan Wang, Hsin-Tai Wu, Zhiqiang Tao, Yi Fang
Large Vision Language Model Complex Prompt Visual Understanding Socio Demographic Fairness Analysis Visual Bias Fairness Benchmark Datasets

October 6, 2023

Fair Feature Importance Scores for Interpreting Tree-Based Methods and Surrogates
Camille Olivia Little, Debolina Halder Lina, Genevera I. Allen
Inherent Interpretability Feature Importance Tree Based Surrogate Assisted Version Tree Based Ensemble Fairness Benchmark Datasets

August 22, 2023

Standardized Interpretable Fairness Measures for Continuous Risk Scores
Ann-Kristin Becker, Oana Dumitrasc, Klaus Broelemann
Algorithmic Fairness Fairness Measure Fairness Perception Analytic Score Fairness Benchmark Datasets

May 27, 2023

Auditing Fairness by Betting
Ben Chugg, Santiago Cortes-Gomez, Bryan Wilder, Aaditya Ramdas
Procedural Fairness Fairness Audit Fairness Benchmark Datasets Anytime Valid Sequential Inference

October 12, 2022

Fairness via Adversarial Attribute Neighbourhood Robust Learning
Qi Qi, Shervin Ardeshir, Yi Xu, Tianbao Yang
Procedural Fairness Fair Representation Protected Attribute Adversarial Covariate Shift Fairness Benchmark Datasets

September 16, 2022

Survey on Fairness Notions and Related Tensions
Guilherme Alves, Fabien Bernier, Miguel Couceiro, Karima Makhlouf, Catuscia Palamidessi, Sami Zhioua
Timely Survey Fairness Notion Fairness Accuracy Conflict Classification Fairness Measure Fairness Benchmark Datasets

July 11, 2022

How Robust is your Fair Model? Exploring the Robustness of Diverse Fairness Strategies
Edward Small, Wei Shao, Zeliang Zhang, Peihan Liu, Jeffrey Chan, Kacper Sokol, Flora Salim
Native Robustness Fairness Evaluation Fair Model Fairness Method Robustness Measurement Fairness Benchmark Datasets

April 30, 2022

Fair Feature Subset Selection using Multiobjective Genetic Algorithm
Ayaz Ur Rehman, Anas Nadeem, Muhammad Zubair Malik
Machine Learning Feature Selection Genetic Algorithm Feature Subset Fair Feature Selection Fairness Benchmark Datasets