Beyond Single-Attribute Auditing: A Configurable Multi-Attribute Benchmark for Intersectional Fairness in LLMs

Zheng Zhang; Yifan Wu; Yuxiang Zhou; Qi Liu; Qingyang Mao; Ning Li; Weibo Gao; Qiyuan Shi

cs.AIOct 26, 2026

Beyond Single-Attribute Auditing: A Configurable Multi-Attribute Benchmark for Intersectional Fairness in LLMs

Authors: Zheng Zhang, Yifan Wu, Yuxiang Zhou, Qi Liu, Qingyang Mao, Ning Li, Weibo Gao, Qiyuan Shi,

Organizations: State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · Nanyang Technological University

Abstract

Large language models (LLMs) are increasingly evaluated for fairness, yet most existing benchmarks audit one sensitive attribute at a time. This single-attribute paradigm can miss intersectional harms, where multiple attributes interact non-additively and produce failures that are invisible under marginal evaluation. We introduce ComFair, a configurable benchmark that converts existing fairness data into 479,066 controlled Dual- and Multi-attribute instances and pairs every augmented instance with its task-equivalent source. Across Recognition, Selection, Continuation, and Conversation, this pairing supports a unified audit of performance or bias-score shifts, how often added attributes introduce new errors or mask existing ones, and model-ranking stability. Experiments on ten LLMs show task- and subgroup-dependent sensitivity: discriminative tasks expose both transition directions even when aggregate changes are modest, generative tasks exhibit larger and directionally asymmetric shifts, and rankings vary with attribute composition. ComFair therefore provides a scalable controlled sensitivity audit---rather than a model of the natural joint distribution of identities---for testing the robustness of fairness-benchmark conclusions beyond single-attribute settings.

Figures & tables

Figure 1: Overview of single-attribute and intersectional fairness auditing.

Benchmark	#Inst.	Task Coverage				Sensitive-Attribute Support
	#Inst.	Cont.	Conv.	Sel.	Rec.	Single	Dual	Multi
BBQ Parrish et al. (2022)	58,492	–	–	✓	–	✓	✓	–
CrowS-Pairs Nangia et al. (2020)	1,508	–	–	✓	–	✓	✓	–
StereoSet Nadeem et al. (2021)	16,995	✓	–	✓	–	✓	–	–
WinoGender Rudinger et al. (2018)	720	–	–	✓	–	✓	–	–
WinoBias Zhao et al. (2018)	3,160	–	–	✓	–	✓	–	–

Table 1: Coverage of representative LLM fairness benchmarks. Our benchmark unifies fairness evaluation across four task types( Continuation, Conversation, Selection, Recognition ) and supports single , dual , multi-way sensitive-attribute assessment.

Figure 2: ComFair’s task-preserving pipeline for constructing intersectional evaluation pairs.

Task	Single	Dual	Multi	Aug. total
Recognition	11,745	89,571	156,479	246,050
Selection	6,272	43,173	75,829	119,002
Continuation	2,368	14,812	24,983	39,795
Conversation	2,378	15,867	58,352	74,219

Table 2: ComFair instances by task and composition depth; Aug. total = Dual + Multi.

Model	Gender			Race
Model	Base	+White ( $\Delta$ )	+Black ( $\Delta$ )	Base	+Male ( $\Delta$ )	+Female ( $\Delta$ )
DS-V3.2-C	31.42	40.69 (+9.27)	35.00 (+3.58)	38.90	39.79 (+0.89)	40.71 (+1.81)
DS-V3.2-R	33.13	36.54 (+3.41)	32.25 (-0.88)	36.53	46.65 (+10.12)	40.71 (+4.18)
Gem-3F	28.99	30.69 (+1.70)	23.41 (-5.58)	40.59	41.67 (+1.08)	43.14 (+2.55)
GPT-5-mini	26.12	34.15 (+8.03)	27.75 (+1.63)	34.22	38.33 (+4.11)	36.86 (+2.64)
Qwen3-8B	33.73	41.23 (+7.50)	37.03 (+3.30)	40.00	45.88 (+5.88)	43.86 (+3.86)

Table 5: Human continuation bias scores for the same paired interventions as Table 4 . Lower is fairer.

Model	+White	+Black	+Male	+Female
DS-V3.2-C	+4.54	-2.69	-3.59	-3.02
DS-V3.2-R	+6.61	-1.22	+13.32	+3.52
Gem-3F	+2.47	-8.19	+7.06	+1.95
GPT-5-mini	+3.38	-1.08	+0.61	-0.28
Qwen3-8B	+4.21	-1.46	+2.67	+3.02
Qwen3-235B	+8.70	-6.52	+0.61	-3.41

Table 6: Representative Continuation bias-score

\Delta

values after re-scoring Table 4 outputs with DeepSeek-V4-Pro.

Model	+White	+Black	+Standing	White $-$ Standing	Black $-$ Standing
DS-V3.2-C	+0.0647	+0.1741	-0.0197	+0.0844	+0.1938
DS-V3.2-R	+0.0344	+0.1184	-0.0059	+0.0403	+0.1243
GPT-5-mini	+0.0159	+0.2012	-0.0362	+0.0521	+0.2374
Qwen3-235B	+0.0109	+0.0155	+0.0218	-0.0109	-0.0063
Qwen3-8B	-0.0023	+0.0006	+0.0007	-0.0030	-0.0001
Mean	+0.0247	+0.1020	-0.0079	+0.0326	+0.1098

Table 7: Selection matched-placebo F1 shifts. The last two columns subtract the non-demographic “standing” shift from each demographic shift.

Model	Gender				Race
	+Black		+White		+Female		+Male
	Agg.	Mask.	Agg.	Mask.	Agg.	Mask.	Agg.	Mask.
Claude-3.5	33.59%	43.66%	43.61%	14.83%	18.97%	29.70%	22.68%	28.00%
Claude-3.7	33.33%	46.60%	51.72%	20.30%	17.13%	28.48%	23.56%	29.19%
DS-V3.2-C	0.00%	20.97%	0.00%	24.62%	0.00%	10.53%	0.00%	5.13%
DS-V3.2-R	31.78%	48.76%	19.84%	32.26%	9.16%	22.69%	13.28%	20.49%

Table 8: Recognition aggravation (Agg.) and masking (Mask.) rates under paired Gender

\leftrightarrow

Race interventions.

Figure 3: Model-ranking stability under intersectional augmentation, measured by Spearman’s rank correlation coefficient (SRCC).

Figure 4: Aggravation case (Correct $\rightarrow$ Wrong).

Figure 5: Masking case (Wrong $\rightarrow$ Correct).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Dataset	Base attribute $\mathbf{A_{\text{base}}}$	Added attribute set $\mathbf{A_{\text{add}}}$	Added values / schema	#Aug / sample	Size (Orig $\rightarrow$ Aug)
Task	Dataset	Base attribute $\mathbf{A_{\text{base}}}$	Added attribute set $\mathbf{A_{\text{add}}}$		Single / Dual / Multi
Recognition	BBQ WinoBias CrowS-Pairs	Age Gender RaceEthnicity PhysicalAppearance Nationality SES DisabilityStatus Religion SexualOrientation	gender; race age; race age; gender age; gender; race age; gender; race age; gender; race age; gender; race age; gender; race age; gender; race	age: young / adult / old gender: female / male race: Asian / White / Black Latinx / Middle_Eastern	1417 / 9160 / 4134 1960 / 20804 / 33633 2468 / 10131 / 6457 624 / 6498 / 14305 1184 / 4832 / 14836 2694 / 25795 / 57442 599 / 1432 / 3020 477 / 6547 / 14991 322 / 4372 / 7661	11745 $\rightarrow$ 246.05k
Selection	BBQ WinoBias CrowS-Pairs	Age Gender RaceEthnicity PhysicalAppearance Nationality SES DisabilityStatus Religion SexualOrientation	gender; race age; race age; gender age; gender; race age; gender; race age; gender; race age; gender; race age; gender; race age; gender; race	age: young / adult / old gender: female / male race: Asian / White / Black Latinx / Middle_Eastern	736 / 4094 / 2924 1134 / 8185 / 13890 1376 / 4453 / 3412 315/ 1792 / 4097 616 / 4859 / 10358 1372 / 11346 / 29349 311 / 5635 / 5635 240 / 1741 / 3872 172 / 1068 / 2292	6272 $\rightarrow$ 119.00k
Continuation	HolisticBias	race gender age religion	age; gender age; race gender; race age; gender; race	age: young / adult / old gender: female / male race: Asian / White / Black Latinx / Middle_Eastern	827 / 4617 / 4230 640 / 4738 / 8107 370 / 1192 / 941 531 / 4265 / 11705	2368 $\rightarrow$ 39.80k
Conversation	HolisticBias	race gender age religion	age; gender age; race gender; race age; gender; race	age: young / adult / old gender: female / male race: Asian / White / Black Latinx / Middle_Eastern	801 / 4680 / 4380 636 / 4808 / 8486 397 / 1569 / 1356 544 / 4810 / 44130	2378 $\rightarrow$ 74.22k

Appendix

Table 15: User-configurable evaluation capabilities across four task types.

Artifact	Source	License / Terms	Use in this paper
BBQ	Parrish et al. (2022)	CC-BY-4.0	Source data for Recognition and Selection; we augment QA-style bias examples with additional sensitive attributes.
CrowS-Pairs	Nangia et al. (2020)	CC-BY-SA-4.0	Source data for Recognition and Selection; we adapt sentence pairs and construct intersectional variants.
WinoBias	Zhao et al. (2018)	MIT License	Source data for discriminative evaluation; we standardize coreference-style examples into our task format.
HolisticBias	Smith et al. (2022)	Dataset: CC-BY-SA-4.0; code: MIT License	Source data for Continuation and Conversation; we use demographic descriptors and seed texts to build generative prompts.
ComFair	This work	Publicly available; underlying artifact licenses continue to apply	Benchmark constructed in this paper; used for paired single-attribute and intersectional fairness evaluation.

Appendix

Table 16: Existing artifacts used in this paper and their license or usage terms.

Panel of Figure 7 — Figure 7: Selection task prompts organized by pipeline stage.

Panel of Figure 8 — Figure 8: Continuation task prompts organized by pipeline stage.

Panel of Figure 9 — Figure 9: Conversation task prompts organized by pipeline stage.

Language Model Bias Evaluation

Language model bias evaluation tests how models’ outputs, decisions, and representations vary across social groups, cultures, languages, and contexts.

See all 493 papers

FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation
Jul 31, 2026Martin LukkBenchmark Auditing LLM Auditing
University of Toronto
Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs
Jan 6, 2026David Hartmann, Lena Pohlmann, Lelia Hanslik +3LLM Auditing Algorithmic Fairness
Weizenbaum Institut Berlin · Technische Universität Berlin · FIZ Karlsruhe +2
Robust Decentralized Fairness Auditing
Oct 7, 2026Sayan Biswas, Jade Garcia Bourrée, Anne-Marie Kermarrec +2Algorithmic Auditing LLM Auditing
EPFL, Switzerland

Jul 31, 2026J/K move · Enter open · S save

FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation

Martin Lukk

University of Toronto

Benchmark Auditing LLM Auditing Fair Resource Allocation Language Model Bias Evaluation Demographic Bias in Language Models

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly involved in the distribution of scarce resources, raising concerns about biased allocations based on characteristics like race and gender. Recent LLM audits have produced inconsistent results, however, finding evidence of both positive and negative discrimination towards women and ethnic minorities, even for the same models. We show that this disagreement can arise from differences in audit format and introduce FairFund-Bench, a benchmark that systematically varies key features of previous audit designs: the evaluation task (rating, ranking, or allocation), comparison context (single or multi-stimulus), and whether the audit is transparent or disguised. The benchmark comprises 600 requests for financial assistance created from human-authored templates (calibrated against 1.3M real GoFundMe campaigns) across three domains, four race and two gender categories, and five causal framings of need derived from welfare deservingness theory. Across 14 models, audit format changes the direction of bias: models advantage minorities when rating claimants individually but penalize some groups when ranking them side by side. Bias magnitude, though small overall, is several times greater in disguised audits than in transparent ones, where, faced with appeals differing only in claimants' names, models overwhelmingly split funds equally. Causal framing effects, by contrast, exceed demographic effects by roughly an order of magnitude and are consistent across models and audit formats, indicating that current LLMs robustly reproduce human deservingness evaluations. The benchmark scores models on four criteria (demographic bias, deservingness alignment, cross-task consistency, and cross-context consistency), is publicly available, and can be readily adapted to other substantive domains.

Abstract

Language Model Bias Evaluation

Explore similar work

FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation

Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs

Robust Decentralized Fairness Auditing