Topic-Level Fairness in Retrieval-Augmented Generation

Jaeyoung Park; Hantian Zhang; Kexin Rong; Steven Euijong Whang

cs.AIOct 26, 2026

Topic-Level Fairness in Retrieval-Augmented Generation

Authors: Jaeyoung Park, Hantian Zhang, Kexin Rong, Steven Euijong Whang

Organizations: KAIST · Meta · Georgia Institute of Technology

Abstract

Group fairness in retrieval-augmented generation (RAG) systems is typically evaluated using metrics aggregated across queries. However, such metrics can conceal topic-level disparities: a system may favor women on questions about human rights and men on questions about actors and filmmakers, appearing nearly neutral overall even though neither topic is fair. We therefore study topic-level fairness, which requires group fairness to hold within each topic rather than only in aggregate. To reduce the topic-level disparity, we adapt the reranker separately for each topic with a Bradley-Terry objective under direct supervision from the generated output: preference pairs built from the generator's own output carry the direction that topic requires. With preference pairs built from RoMQA alone, the adaptation reduces the mean absolute per-topic disparity by 51% on RoMQA and by 40% on TREC Fair Ranking while retaining accuracy. No baseline reaches the reduction, even when modified to treat each topic separately rather than apply one correction to every topic. Neither does our method with one reranker for all topics, which follows the direction that most topics require.

Figures & tables

Figure 1: Overview. The group disparity (

GD

) of a RAG system between genders differs in sign across topics (left; before adaptation, 5 of the 9 topics on RoMQA). Opposing topics cancel in a signed mean :

\operatorname{avg}(GD)

-0.0004

, nearly neutral, while

\operatorname{avg}|GD|

, the mean absolute per-topic disparity, is

0.0175

. Adapting the reranker separately for each topic in the direction that topic requires, on preference pairs built from the generator’s own output (center), moves every shown topic closer to zero and lowers

\operatorname{avg}|GD|

0.0070

(right).

Figure 2: Illustrative example. The method on two topics whose directions of correction are opposite. (1) The retrieve–rerank–generate pipeline, in which the reranker alone is adapted, separately for each topic. (2) Preference pairs (Section 4.2 ): the system favors men on actors and filmmakers before adaptation, so

\delta_{\text{AF}}

is female, and women on human rights, so

\delta_{\text{HR}}

is male. For each retrieved document

d

n=5

replies are sampled from

G(x,d)

and shown beside it, colored by candidate; a document from which no reply is a candidate falls outside

D_{+}

, and of the two that remain, the one with the higher

v_{\delta_{t}}

is preferred. (3) Each reranker

\pi_{t}

is adapted on the preference pairs

\mathcal{P}_{t}

(Section 4.3 ); at the right, each query’s top-

k

documents before and after adaptation are shown.

Figure 3: Difference in recall@5 between the articles of the male and the female candidate, and

GD_{t}

, of the unadapted system on RoMQA. In the shaded rows, the system favors the candidate of higher recall@5. The sign of

GD_{t}

does not follow from recall@5 alone.

	In-dataset (RoMQA)				Cross-dataset (TREC)
System	$\|GD\|$	Accuracy	Male-fav.	Female-fav.	$\|GD\|$	Accuracy	Male-fav.	Female-fav.
Unadapted	0.0169	55.1	–	–	0.0156	89.9	–	–
RRPO	0.0170	56.1	1/4	2/5	0.0156	89.9	3/6	2/3
GB	0.0193	55.4	1/4	1/5	0.0112	92.1	4/6	2/3
RD	0.0162	54.3	1/4	4/5	0.0143	89.9	6/6	0/3
SD	0.0174	54.9	0/4	4/5	0.0166	89.4	2/6	1/3

Table 1: Fairness and utility of the baselines and Ours under the in-dataset and the cross-dataset protocol. In Male-fav. and Female-fav. ,

k/n

means that

k

of the

n

topics on which the unadapted system favors the male or the female candidate move in their direction of correction. Bold marks the lowest

|GD|

and the highest

k

per protocol.

Figure 4:

GD_{t}

of Table 1 under the in-dataset protocol. Ours moves 8 of 9 topics in their direction of correction.

	In-dataset (RoMQA)
System	$\|GD\|$	Accuracy	Male- fav.	Female- fav.
Unadapted	0.0169	55.1	–	–
Ours, one reranker	0.0175	55.5	1/4	5/5
RD per topic	0.0118	54.8	4/4	4/5
SD per topic	0.0170	55.0	0/4	3/5
Ours	0.0083	55.2	3/4	5/5

Table 2: Ours with one reranker shared by all topics, and the systems that treat each topic separately, under the in-dataset protocol. The columns are defined as in Table 1 . Ours, one reranker moves most topics toward the male candidate.

Figure 5:

GD_{t}

of Table 2 under the in-dataset protocol. Ours is the closest to zero overall.

Variant	$\|GD\|$	Accuracy	Male- fav.	Female- fav.
	In-dataset (RoMQA)
Unadapted	0.0169	55.1	–	–
Ours	0.0083	55.2	3/4	5/5
one direction	0.0176	54.8	0/4	5/5
without anchor	0.0142	55.0	0/4	4/5

Table 3: Ablation under the in-dataset protocol. The columns are defined as in Table 1 . One direction corrects every topic in the direction that most topics require; without anchor sets

\lambda_{t}=0

for every topic.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

RoMQA
	Train	Validation	Test
Questions	961	961	961
Male candidates	17,392	17,329	17,254
Female candidates	11,858	11,646	11,429
Queries	148,586	146,230	143,738
TREC Fair Ranking

Appendix

Table 4: Statistics of the two datasets over the 9 topics. The two candidates of every query are one male and one female golden answer of its question. A person who is a golden answer of two questions counts as two candidates.

Topic	Female	Male
Actors and Filmmakers	128	373
Computer science	78	423
Former countries	80	421
Human rights	154	347
Literature	117	384
Medicine	69	432

Appendix

Table 5: Candidates of each gender in each of the 9 topics of TREC Fair Ranking, summed over both splits.

Topic	Questions	Female	Male
Actors and Filmmakers	216	8,847	11,643
Computer science	50	1,879	2,829
Former countries	104	3,001	5,468
Human rights	15	502	778
Literature	209	7,604	11,533
Medicine	26	917	1,467

Appendix

Table 6: Candidates of each gender in each of the 9 topics of RoMQA, summed over the three splits, and the questions of each topic. A person who is a golden answer of two questions counts as two candidates.

Panel of Figure 6 — Figure 6: Difference in recall@5 and $GD_{t}$ of the unadapted system on both datasets. Full version of Figure 3 . Each panel has its own scale for recall@5, and Table 7 lists the values.

	Recall@5
Topic	Male	Female	Male $-$ Female	$GD_{t}$
RoMQA
Actors and Filmmakers	91.2	92.9	$-1.7$	$+0.0211$
Computer science	93.1	95.3	$-2.2$	$-0.0072$
Former countries	94.1	94.8	$-0.7$	$-0.0121$
Human rights	95.7	97.9	$-2.2$	$-0.0356$

Appendix

Table 7: Recall@5 and

GD_{t}

of the unadapted system on each topic, the values that Figure 6 draws.

Panel of Figure 7 — Figure 7: $GD_{t}$ of Table 1 under both protocols. Full version of Figure 4 . Ours moves 8 of 9 topics in their direction of correction under each protocol. Each panel has its own scale, centered on zero.

In-dataset (RoMQA)
Topic	Unadapted	RRPO	GB	RD	SD	Ours
Actors and Filmmakers	$+$ 0.0211	$+$ 0.0213	$+$ 0.0270	$+$ 0.0322	$+$ 0.0219	$+$ 0.0165
Computer science	$-$ 0.0072	$-$ 0.0073	$-$ 0.0108	$+$ 0.0077	$-$ 0.0061	$+$ 0.0043
Former countries	$-$ 0.0121	$-$ 0.0119	$-$ 0.0074	$+$ 0.0047	$-$ 0.0106	$+$ 0.0041
Human rights	$-$ 0.0356	$-$ 0.0413	$-$ 0.0395	$-$ 0.0417	$-$ 0.0395	$+$ 0.0104
Literature	$+$ 0.0024	$+$ 0.0046	$+$ 0.0038	$+$ 0.0122	$+$ 0.0033	$+$ 0.0010

Appendix

Table 8:

GD_{t}

of every system of Table 1 under both protocols. The rows

|GD|

and Accuracy give, for each column, the mean absolute per-topic group disparity and the accuracy that Table 1 reports. Bold marks, in each row, the value closest to zero.

Topic	Both	Male	Female	Neither	Other	Both	Male	Female	Neither	Other	Pairs per query
	Unadapted					Ours
Actors and Filmmakers	51.4	12.5	10.4	21.3	4.4	51.9	12.5	10.9	20.7	4.0	0.848
Computer science	62.9	8.4	9.2	17.5	2.0	62.6	9.0	8.6	17.6	2.1	1.097
Former countries	43.3	10.1	11.3	30.7	4.7	43.7	11.0	10.6	29.9	4.9	0.825
Human rights	62.5	9.5	13.1	10.2	4.7	64.2	10.9	9.9	9.6	5.4	1.065
Literature	54.8	8.5	8.3	25.5	2.9	54.7	8.4	8.3	25.6	3.0	1.055

Appendix

Table 9: Replies of the unadapted system and of Ours under the in-dataset protocol. For each topic: the share (%) of each reply, and the preference pairs built per training query. Both is the accuracy; Male and Female , a reply of that candidate; Neither includes Insufficient information to decide ; Other , a reply that matches no option and no candidate.

Topic	Both	Male	Female	Neither	Other	Both	Male	Female	Neither	Other	Pairs per query
	Unadapted					Ours
Actors and Filmmakers	92.1	5.8	2.0	0.0	0.1	92.5	4.0	3.3	0.0	0.2	0.848
Computer science	99.4	0.2	0.2	0.0	0.2	99.4	0.1	0.2	0.2	0.1	1.071
Former countries	60.7	10.3	9.5	18.2	1.3	59.8	8.4	9.1	20.5	2.2	0.835
Human rights	90.3	2.6	3.3	1.1	2.7	90.3	2.3	3.0	0.9	3.5	1.065
Literature	98.9	0.4	0.7	0.0	0.0	98.9	0.4	0.7	0.0	0.0	1.091

Appendix

Table 10: Replies of the unadapted system and of Ours under the cross-dataset protocol. For each topic: the share (%) of each reply, and the preference pairs built per training query. The columns are defined as in Table 9 .

	bge-reranker-v2-m3		jina-reranker-v2-base-multilingual
Topic	Unadapted	Adapted	Unadapted	Adapted
Actors and Filmmakers	$+0.0211$	$+0.0165$	$+0.0317$	$+0.0308$
Computer science	$-0.0072$	$+0.0043$	$-0.0096$	$-0.0095$
Former countries	$-0.0121$	$+0.0041$	$+0.0030$	$-0.0033$
Human rights	$-0.0356$	$+0.0104$	$-0.0287$	$-0.0174$
Literature	$+0.0024$	$+0.0010$	$+0.0057$	$+0.0041$

Appendix

Table 11:

GD_{t}

under the in-dataset protocol with each reranker, unadapted and adapted. Adapted bge-reranker-v2-m3 is Ours .

Panel of Table 13 — Table 13: Ours with one reranker shared by all topics, and the systems that treat each topic separately, under both protocols. Full version of Table 2 . Ours, one reranker moves most topics toward the male candidate under the in-dataset protocol and toward the female candidate under the cross-dataset protocol.

Variant	$\|GD\|$	Accuracy	Male-fav.	Female-fav.	$\|GD\|$	Accuracy	Male-fav.	Female-fav.
	In-dataset (RoMQA)				Cross-dataset (TREC)
Unadapted	0.0169	55.1	–	–	0.0156	89.9	–	–
Ours	0.0083	55.2	3/4	5/5	0.0094	90.2	6/6	2/3
one direction	0.0176	54.8	0/4	5/5	0.0102	90.3	6/6	0/3
without anchor	0.0142	55.0	0/4	4/5	0.0112	90.8	5/6	1/3

Appendix

Table 16: Ablation under both protocols. Full version of Table 3 .

Group Fairness

Group fairness measures and reduces disparities in how algorithms predict, rank, allocate resources, and make decisions for different groups.

See all 140 papers

Who Benefits from RAG? The Role of Exposure, Utility and Attribution Bias
Mar 25, 2026Mahdi Dehghan, Graham McDonaldGroup Fairness Algorithmic Fairness
University of Glasgow, Glasgow, UK
Does Reasoning Make Search More Fair? Comparing Fairness in Reasoning and Non-Reasoning Rerankers
Mar 11, 2026Saron Samuel, Benjamin Van Durme, Eugene YangAlgorithmic Fairness LLM Reranking
Johns Hopkins University · Human Language Technology Center of Excellence
Retrieval-Augmented Generation Must Move Beyond Factual Grounding to Represent Diverse Opinions
Apr 13, 2026Aditya Agrawal, Alwarappan Nakkiran, Aman Singh Thakur +2Retrieval-Augmented Generation Aleatoric Uncertainty

Mar 25, 2026J/K move · Enter open · S save

Who Benefits from RAG? The Role of Exposure, Utility and Attribution Bias

Mahdi Dehghan, Graham McDonald

University of Glasgow, Glasgow, UK

Group Fairness Algorithmic Fairness Retrieval-Augmented Generation Language Model Bias Evaluation RAG Evaluation

Open paper AskPDF arXiv

Large Language Models (LLMs) enhanced with Retrieval-Augmented Generation (RAG) have achieved substantial improvements in accuracy by grounding their responses in external documents that are relevant to the user's query. However, relatively little work has investigated the impact of RAG in terms of fairness. Particularly, it is not yet known if queries that are associated with certain groups within a fairness category systematically receive higher accuracy, or accuracy improvements in RAG systems compared to LLM-only, a phenomenon we refer to as query group fairness. In this work, we conduct extensive experiments to investigate the impact of three key factors on query group fairness in RAG, namely: Group exposure, i.e., the proportion of documents from each group appearing in the retrieved set, determined by the retriever; Group utility, i.e., the degree to which documents from each group contribute to improving answer accuracy, capturing retriever-generator interactions; and Group attribution, i.e., the extent to which the generator relies on documents from each group when producing responses. We examine group-level average accuracy and accuracy improvements disparities across four fairness categories using three datasets derived from the TREC 2022 Fair Ranking Track for two tasks: article generation and title generation. Our findings show that RAG systems suffer from the query group fairness problem and amplify disparities in terms of average accuracy across queries from different groups, compared to an LLM-only setting. Moreover, group utility, exposure, and attribution can exhibit strong positive or negative correlations with average accuracy or accuracy improvements of queries from that group, highlighting their important role in fair RAG. Our data and code are publicly available from Github.

Abstract

Group Fairness

Explore similar work

Who Benefits from RAG? The Role of Exposure, Utility and Attribution Bias

Does Reasoning Make Search More Fair? Comparing Fairness in Reasoning and Non-Reasoning Rerankers

Retrieval-Augmented Generation Must Move Beyond Factual Grounding to Represent Diverse Opinions