cs.LGNov 6, 2026

A Taxonomy-Driven Synthetic Dataset for Retrieval Effectiveness and Fairness Evaluation in Hiring

Authors: Warren Jouanneau, Morgane Hoffmann, Emma Jouffroy, Marc Palyart

Organizations: Malt, Paris, France

Abstract

While machine learning hiring tools are ubiquitous, the need for privacy regulations and commercial sensitivity makes public benchmarking legally complex. We address this challenge by introducing a taxonomy-driven, fully synthetic HR dataset and benchmark to simultaneously evaluate retrieval effectiveness and algorithmic fairness. Grounded in 660K real freelancer profiles, our multilingual corpus features candidate profiles, project briefs, and counterfactual demographic twins. Auditing state-of-the-art embedding models reveals performance divergence from general MTEB scores, alongside fairness disparities in grammatically gendered languages. This demonstrates that general benchmarks are insufficient for HR, requiring tailored evaluations to identify Pareto-optimal models that balance semantic accuracy and fairness.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. PopResume: Causal Fairness Evaluation of LLM/VLM Resume Screeners with Population-Representative Dataset

    Mar 24, 2026Sumin Yu, Juhyeon Park, Taesup MoonAI-Assisted Decision MakingLLM Auditing

  2. Fairness Beyond Anonymization? Demographic Leakage in German LLM-Generated Resumes

    Aug 28, 2026Charlotte Leininger, Helena Veit, Matthias Aßenmacher +1Algorithmic FairnessDemographic Bias in Language Models

  3. Fast, Cheap, and Fair? Evaluating Decision Models

    Nov 3, 2026Muhammad Uzair Ul Haq, Davide Rigoni, Alessandro Sperduti +1Algorithmic FairnessAlgorithmic Bias