A Taxonomy-Driven Synthetic Dataset for Retrieval Effectiveness and Fairness Evaluation in Hiring
Organizations: Malt, Paris, France
Abstract
While machine learning hiring tools are ubiquitous, the need for privacy regulations and commercial sensitivity makes public benchmarking legally complex. We address this challenge by introducing a taxonomy-driven, fully synthetic HR dataset and benchmark to simultaneously evaluate retrieval effectiveness and algorithmic fairness. Grounded in 660K real freelancer profiles, our multilingual corpus features candidate profiles, project briefs, and counterfactual demographic twins. Auditing state-of-the-art embedding models reveals performance divergence from general MTEB scores, alongside fairness disparities in grammatically gendered languages. This demonstrates that general benchmarks are insufficient for HR, requiring tailored evaluations to identify Pareto-optimal models that balance semantic accuracy and fairness.
Figures & tables
| Metric | Synthetic | Platform | Clusters |
|---|---|---|---|
| Type-Token Ratio | 0.70 | 0.74 | 0.76 |
| Vocabulary Size | 22k | 33k | 24k |
| Tokens (Mean) | 142 | 114 | 95 |
| Paragraphs (Mean) | 2.25 | 3.58 | 2.65 |
| Emojis (Mean) | 0.29 | 0.29 | 0.11 |
| Humans | Synthetic | |||
|---|---|---|---|---|
| Metric | Mean | Range | Mean | Range |
| % fuzzy agreement ( ) | 0.857 | 0.797–0.937 | 0.873 | 0.785–0.962 |
| % relevant agreement ( ) | 0.806 | 0.747–0.899 | 0.848 | 0.747–0.924 |
| Quadratic Weighted | 0.737 | 0.676–0.856 | 0.787 | 0.687–0.860 |
| Binarized ( ) | 0.593 | 0.487–0.756 | 0.660 | 0.484–0.799 |
| Training Setting | NDCG | P | MRR |
|---|---|---|---|
| Zero-Shot Baseline | 0.42 | 0.55 | 0.76 |
| TRTR (Historical Data) | 0.57 | 0.72 | 0.87 |
| TSTR (Synthetic Data) | 0.54 | 0.70 | 0.83 |
| Axis | Scores | Ranks | Top- |
|---|---|---|---|
| Stability | Pearson | Kendall’s | IoU@ |
| Direction | Cohen’s | RBC | OddRatio@ |
| Magnitude | Std MAD | Abs. Shift | Dropout@ |
| Model | Brief (@10) | Short Query (@10) | MTEB | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Name (version) | Params | Test time | NDCG | P | MRR | NDCG | P | MRR | Score |
| Arctic Embed (s) | 33M | 6m | 0.54 | 0.69 | 0.84 | 0.56 | 0.63 | 0.76 | 39.84 |
| E5 (small) | 118M | 6m | 0.46 | 0.61 | 0.81 | 0.57 | 0.64 | 0.76 | 36.25 |
| Arctic Embed (m) | 109M | 17m | 0.59 | 0.74 | 0.85 | 0.62 | 0.70 | 0.78 | 39.33 |
| E5 (base) | 278M | 18m | 0.47 | 0.63 | 0.80 | 0.54 | 0.61 | 0.74 | 37.69 |
| Nomic Embed (v2) | 475M | 25m | 0.60 | 0.77 | 0.88 | 0.59 | 0.66 | 0.75 | - |
| Model | English | French | Spanish | German | Dutch | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Name (version) | IoU @10 | RBC | Rank Shift | IoU @10 | RBC | Rank Shift | IoU @10 | RBC | Rank Shift | IoU @10 | RBC | Rank Shift | IoU @10 | RBC | Rank Shift |
| Arctic Embed (s) | 1.00 | 0.01 | 0.00 | 0.75 | -0.15 | 127 | 1.00 | 0.02 | 100 | 0.79 | -0.21 | 141 | 1.00 | -0.01 | 0.00 |
| E5 (s) | 1.00 | 0.83 | 0.10 | 0.00 | -0.93 | 310 | 0.33 | -0.97 | 369 | 0.00 | -0.97 | 457 | 1.00 | 0.00 | 0.00 |
| Arctic Embed (m) | 1.00 | 0.01 | 0.00 | 0.62 | -0.03 | 162 | 0.73 | 0.25 | 166 | 0.84 | 0.26 | 142 | 1.00 | 0.00 | 0.00 |
| E5 (b) | 1.00 | -0.99 | 0.41 | 0.20 | -0.99 | 525 | 0.41 | -0.98 | 471 | 0.25 | -0.97 | 442 | 1.00 | -0.01 | 0.00 |
| Nomic Embed (v2) | 1.00 | -0.32 | 0.01 | 0.42 | -0.82 | 198 | 0.44 | -0.78 | 205 | 0.74 | -0.70 | 168 | 1.00 | 0.00 | 0.00 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Human–Human | Synthetic–Human | |||||
|---|---|---|---|---|---|---|
| Metric | A–B | A–C | B–C | M–A | M–B | M–C |
| Fuzzy agreement ( ) | 0.937 | 0.797 | 0.835 | 0.873 | 0.962 | 0.785 |
| Relevant agreement ( ) | 0.899 | 0.747 | 0.772 | 0.873 | 0.924 | 0.747 |
| Quadratic Weighted | 0.856 | 0.679 | 0.676 | 0.814 | 0.860 | 0.687 |
| Binarized ( ) | 0.756 | 0.487 | 0.536 | 0.695 | 0.799 | 0.484 |
| Model | English | French | Spanish | German | Dutch | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Name (version) | IoU @10 | RBC | Rank Shift | IoU @10 | RBC | Rank Shift | IoU @10 | RBC | Rank Shift | IoU @10 | RBC | Rank Shift | IoU @10 | RBC | Rank Shift |
| Arctic Embed (s) | 1.00 | -0.01 | 0.00 | 0.78 | -0.04 | 147 | 0.82 | -0.04 | 130 | 0.61 | 0.08 | 176 | 1.00 | 0.01 | 0.00 |
| E5 (s) | 1.00 | -0.88 | 0.23 | 0.52 | -0.46 | 216 | 0.61 | -0.51 | 229 | 0.74 | -0.48 | 227 | 1.00 | 0.01 | 0.00 |
| Arctic Embed (m) | 1.00 | 0.00 | 0.00 | 0.65 | 0.31 | 172 | 0.70 | 0.56 | 221 | 0.64 | 0.34 | 165 | 1.00 | 0.00 | 0.00 |
| E5 (b) | 1.00 | -0.15 | 0.14 | 0.18 | -0.95 | 430 | 0.43 | -0.95 | 390 | 0.44 | -0.80 | 293 | 1.00 | 0.01 | 0.00 |
| Nomic Embed (v2) | 1.00 | -0.04 | 0.02 | 0.55 | -0.63 | 277 | 0.40 | -0.62 | 330 | 0.58 | -0.41 | 272 | 1.00 | 0.00 | 0.00 |
Algorithmic Fairness
Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.
See all 452 papers