When RAG Fails to Equalize: Geo-bias in Factual Question Answering over Public Companies

Abhinav Havaldar; Enrico Santus

cs.CLAug 26, 2026

When RAG Fails to Equalize: Geo-bias in Factual Question Answering over Public Companies

Authors: Abhinav Havaldar, Enrico Santus

Organizations: Bloomberg

Abstract

Retrieval-augmented generation (RAG) is widely assumed to mitigate factual errors in large language models (LLMs), but it remains unclear whether retrieval uniformly compensates for missing knowledge. We study this question in a controlled factual QA setting over public companies, constructing a benchmark of approximately 2,000 firms across global equity indices. We evaluate six LLMs on four atomic attributes under four conditions: no-context, perfect context, misleading context, and distraction context. We find strong geographic disparities in no-context accuracy, indicating uneven parametric knowledge. While perfect context improves performance, it does not eliminate these gaps: gains are correlated with baseline accuracy, suggesting retrieval effectiveness is coupled to internal representations. Under misleading context, models frequently copy incorrect information. Larger models improve overall performance but do not remove these structural effects. These results challenge the view of RAG as a universal corrective and highlight the interaction between model knowledge, context quality, and entity representation.

Figures & tables

Figure 1: Paired question construction from the same fact.

Panel of Figure 2 — Figure 2: Hypothesis model. Geo/index position and model scale shape baseline parametric knowledge. Prompt settings determine the instantiated question through context regime and direction, thereby moderating how knowledge and evidence translate into answer accuracy.

Variable	Count	Unique	Freq	Top
Dataset Metadata
Company	2165	2135	2	–
Index	2165	15	498	S&P 500 (US)
Industry	1703	643	147	Financial Serv.
Founding Year	1896	212	45	1994
Headquarters	1902	666	160	Tokyo, Japan

Table 1: Summary statistics of the dataset. The top panel reports dataset-level metadata, while the bottom panel summarizes the target attributes used for question construction. Note that companies can be listed multiple times in different indices and therefore there are fewer (2135) unique companies than the total (2165).

Model	Deductive Accuracy	Inductive Accuracy	Odds Ratio (95% CI)	$p$ -value
LLaMA-8B	0.57	0.67	0.66 [0.62, 0.70]	0.00
LLaMA-70B	0.70	0.76	0.75 [0.71, 0.81]	0.00
Claude Sonnet 4	0.78	0.80	0.76 [0.71, 0.82]	0.00
GPT-5 nano	0.82	0.82	0.92 [0.86, 0.99]	0.02
GPT-5 mini	0.89	0.89	1.00 [0.93, 1.08]	1.00
GPT-5 †	0.73	0.78	1.01 [0.92, 1.10]	0.87

Table 2: Directional effect of question framing. Deductive questions are harder than inductive ones for most models. † GPT-5’s lower deductive accuracy relative to GPT-5 mini is an exception to this trend; we did not identify a parsing or evaluation artifact underlying it and report it as observed.

Figure 3: No-context factual accuracy across global equity indices. Grouped bars show the expected probability of a correct answer, with uncertainty intervals, for each model within each index. Horizontal reference lines indicate each model’s global mean.

Panel of Figure 4 — Figure 4: Effect of perfect context across indices. Positive shifts indicate that models benefit from provided evidence, with different magnitude by markets and model families.

Panel of Figure 5 — Figure 5: Effect of misleading context across indices. Decreasing misleading rates indicate the ability of models to use prior information to avoid relying on false information.

Panel of Figure 6 — Figure 6: Effect of distracting context across indices. Decreasing distraction rates indicate the ability of models to use prior information to avoid relying on irrelevant information.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Index	Country	Reg	Wiki Page	#
GSE	Ghana	AF	GSE_Composite_Index	33
N225	Japan	AS	Nikkei_225	458
FTSE	United Kingdom	EU	FTSE_250_Index	250
IBOV	Brazil	LA	List_of_companies_listed_on_B3	68
ASX200	Australia	OC	S&P/ASX_200	146
HSI	Hong Kong	AS	Hang_Seng_Index	81

Appendix

Table 3: Dataset composition and sources (Reg: region).

	Missingness	Uniqueness
Company Name	0.00	96.98
Index Name	0.00	0.81
Summary	0.54	94.74
Website	8.95	90.17
Headquarters	12.19	68.55
Founding Year	13.70	72.75

Appendix

Table 4: Raw Wikipedia-derived fields, with fields used in this study highlighted in gray.

Index	Founding Year	Headquarters	Key People
GSE	38.7	73.3	100.0
N225	56.6	31.0	93.2
FTSE	73.0	29.7	84.9
IBOV	54.5	31.8	87.9
ASX200	65.2	62.6	94.0
HSI	60.5	31.2	79.7

Appendix

Table 5: Perfect-context exclusion rate (%) by attribute and index.

Condition	Quintile coef.	Quintile $p$	Index FE $p$
No-Context	0.036	$<0.001$	$<0.001$
Perfect Context	0.041	$<0.001$	$<0.001$
Distraction Context	0.040	$<0.001$	$<0.001$
Misleading Context	0.030	0.002	$<0.001$

Appendix

Table 6: Company-size (revenue quintile) regression, controlling for index fixed effects.

Model	Parameters	Version	Temperature	Max Output Tokens
LLaMA	8B	3.1	1.0	4,096
LLaMA	70B	3.1	1.0	8,192
GPT-5 nano	$-^{*}$	5.0	$-^{+}$	128,000
GPT-5 mini	$-^{*}$	5.0	$-^{+}$	128,000
GPT-5	$-^{*}$	5.0	$-^{+}$	128,000
Claude Sonnet	$-^{*}$	4.0	1.0	64,000

Appendix

Table 7: Language Model Access Details. ∗ : Closed source models do not disclose details regarding parameters. + : OpenAI does not permit users to specify temperature.

(a) Founding Year

model	Claude Sonnet 4		GPT-5		GPT-5 Mini		GPT-5 Nano		LLaMA-70B		LLaMA-8B
	Accuracy	Center	Accuracy	Center	Accuracy	Center	Accuracy	Center	Accuracy	Center	Accuracy	Center
Quantile
0.1	0.21 ± 0.10	0.22	0.98 ± 0.02	0.93	0.84 ± 0.09	0.79	0.58 ± 0.12	0.54	-	-	0.25 ± 0.09	0.19
0.2	-	-	-	-	-	-	0.55 ± 0.06	0.58	0.62 ± 0.04	0.56	-	-
0.3	-	-	-	-	0.80 ± 0.05	0.83	-	-	0.60 ± 0.07	0.60	0.13 ± 0.07	0.22
0.4	0.52 ± 0.06	0.40	1.00 ± 0.00	0.95	-	-	-	-	0.73 ± 0.04	0.64	0.40 ± 0.08	0.24

Appendix

Table 8: Statistics for Accuracy given Perfect Context for questions about Founding Year.

model

Claude Sonnet 4

GPT-5

GPT-5 Mini

GPT-5 Nano

LLaMA-70B

LLaMA-8B

Accuracy

Center

Accuracy

Center

Accuracy

Center

Accuracy

Center

Accuracy

Center

Accuracy

Center

Quantile

0.1

0.76 ± 0.09

0.77

0.79 ± 0.05

0.80

0.76 ± 0.09

0.77

0.77 ± 0.03

0.78

0.72 ± 0.09

0.73

0.64 ± 0.10

0.65

0.2

0.79 ± 0.07

0.79

0.82 ± 0.05

0.82

0.80 ± 0.03

0.80

0.81 ± 0.03

0.80

0.77 ± 0.03

0.76

0.68 ± 0.03

0.68

0.3

0.82 ± 0.02

0.81

0.83 ± 0.03

0.84

0.81 ± 0.08

0.82

0.83 ± 0.04

0.83

0.77 ± 0.03

0.78

0.71 ± 0.06

0.71

0.4

0.84 ± 0.02

0.83

0.86 ± 0.02

0.86

0.85 ± 0.02

0.84

0.85 ± 0.05

0.85

0.82 ± 0.06

0.81

0.73 ± 0.03

0.73

Appendix

Table 9: Statistics for Accuracy given Perfect Context for questions about Industry.

model	Claude Sonnet 4		GPT-5		GPT-5 Mini		GPT-5 Nano		LLaMA-70B		LLaMA-8B
	Accuracy	Center	Accuracy	Center	Accuracy	Center	Accuracy	Center	Accuracy	Center	Accuracy	Center
Quantile
0.1	0.30 ± 0.09	0.25	0.92 ± 0.08	0.84	0.65 ± 0.08	0.47	-	-	0.50 ± 0.15	0.38	-	-
0.2	-	-	-	-	-	-	0.28 ± 0.09	0.45	0.52 ± 0.10	0.42	0.55 ± 0.16	0.28
0.3	-	-	0.77 ± 0.08	0.87	-	-	-	-	-	-	0.27 ± 0.07	0.32
0.4	0.71 ± 0.11	0.45	-	-	-	-	0.69 ± 0.12	0.53	0.81 ± 0.09	0.51	-	-

Appendix

Table 10: Statistics for Accuracy given Perfect Context for questions about Key People.

model	Claude Sonnet 4		GPT-5		GPT-5 Mini		GPT-5 Nano		LLaMA-70B		LLaMA-8B
	Accuracy	Center	Accuracy	Center	Accuracy	Center	Accuracy	Center	Accuracy	Center	Accuracy	Center
Quantile
0.1	0.62 ± 0.18	0.52	0.88 ± 0.12	0.80	0.75 ± 0.16	0.76	0.74 ± 0.06	0.64	0.40 ± 0.09	0.38	0.30 ± 0.05	0.28
0.3	-	-	-	-	-	-	-	-	-	-	0.36 ± 0.08	0.38
0.4	-	-	-	-	0.90 ± 0.04	0.84	0.72 ± 0.07	0.75	0.65 ± 0.05	0.57	0.42 ± 0.03	0.44
0.5	0.72 ± 0.07	0.72	-	-	0.86 ± 0.07	0.86	-	-	0.65 ± 0.07	0.63	0.49 ± 0.04	0.49

Appendix

Table 11: Statistics for Accuracy given Perfect Context for questions about Headquarters.

model	Claude Sonnet 4		GPT-5		GPT-5 Mini		GPT-5 Nano		LLaMA-70B		LLaMA-8B
	Center	Rate	Center	Rate	Center	Rate	Center	Rate	Center	Rate	Center	Rate
Quantile
0.1	-	-	0.93	0.63 ± 0.04	0.79	0.79 ± 0.08	0.54	0.62 ± 0.12	0.52	0.93 ± 0.07	-	-
0.2	-	-	-	-	-	-	0.58	0.61 ± 0.04	0.56	0.74 ± 0.03	-	-
0.3	-	-	-	-	0.83	0.65 ± 0.03	-	-	0.60	0.73 ± 0.05	-	-
0.4	0.40	0.79 ± 0.04	0.95	0.60 ± 0.05	-	-	0.68	0.71 ± 0.13	0.64	0.67 ± 0.03	0.24	0.52 ± 0.11

Appendix

Table 12: Statistics for Misleading Rate given Misleading Context for questions about Founding Year.

model

Claude Sonnet 4

GPT-5

GPT-5 Mini

GPT-5 Nano

LLaMA-70B

LLaMA-8B

Center

Rate

Center

Rate

Center

Rate

Center

Rate

Center

Rate

Center

Rate

Quantile

0.1

0.77

0.95 ± 0.05

0.80

0.54 ± 0.07

0.77

0.68 ± 0.11

0.78

0.80 ± 0.03

0.73

0.83 ± 0.09

0.65

0.94 ± 0.06

0.2

0.79

0.87 ± 0.06

0.82

0.55 ± 0.07

0.80

0.79 ± 0.03

0.80

0.72 ± 0.03

0.76

0.87 ± 0.03

0.68

0.90 ± 0.03

0.3

0.81

0.90 ± 0.02

0.84

0.67 ± 0.04

0.82

0.73 ± 0.10

0.83

0.80 ± 0.04

0.78

0.91 ± 0.02

0.71

0.83 ± 0.06

0.4

0.83

0.86 ± 0.02

0.86

0.62 ± 0.03

0.84

0.78 ± 0.02

0.85

0.71 ± 0.06

0.81

0.81 ± 0.07

0.73

0.86 ± 0.03

Appendix

Table 13: Statistics for Misleading Rate given Misleading Context for questions about Industry.

model	Claude Sonnet 4		GPT-5		GPT-5 Mini		GPT-5 Nano		LLaMA-70B		LLaMA-8B
	Center	Rate	Center	Rate	Center	Rate	Center	Rate	Center	Rate	Center	Rate
Quantile
0.1	0.25	0.66 ± 0.05	0.58	0.50 ± 0.17	-	-	-	-	0.38	0.45 ± 0.05	0.23	0.47 ± 0.12
0.2	-	-	-	-	0.46	0.47 ± 0.04	0.38	0.32 ± 0.05	0.42	0.38 ± 0.04	0.28	0.30 ± 0.05
0.3	0.38	0.15 ± 0.10	-	-	0.50	0.41 ± 0.04	-	-	0.47	0.42 ± 0.10	0.32	0.37 ± 0.05
0.4	0.45	0.59 ± 0.04	-	-	0.55	0.48 ± 0.09	0.48	0.38 ± 0.04	0.51	0.40 ± 0.04	0.37	0.27 ± 0.05

Appendix

Table 14: Statistics for Misleading Rate given Misleading Context for questions about Key People.

model	Claude Sonnet 4		GPT-5		GPT-5 Mini		GPT-5 Nano		LLaMA-70B		LLaMA-8B
	Center	Rate	Center	Rate	Center	Rate	Center	Rate	Center	Rate	Center	Rate
Quantile
0.1	0.53	0.43 ± 0.14	0.80	0.27 ± 0.10	0.76	0.48 ± 0.11	0.64	0.44 ± 0.06	0.38	0.85 ± 0.10	0.28	0.45 ± 0.08
0.3	-	-	-	-	-	-	-	-	-	-	0.38	0.78 ± 0.09
0.4	-	-	-	-	0.84	0.48 ± 0.05	0.75	0.67 ± 0.06	0.57	0.63 ± 0.05	0.44	0.80 ± 0.04
0.5	0.72	0.62 ± 0.07	-	-	0.86	0.62 ± 0.09	-	-	0.63	0.73 ± 0.06	0.49	0.63 ± 0.04

Appendix

Table 15: Statistics for Misleading Rate given Misleading Context for questions about Headquarters.

model	Claude Sonnet 4		GPT-5		GPT-5 Mini		GPT-5 Nano		LLaMA-70B		LLaMA-8B
	Center	Rate	Center	Rate	Center	Rate	Center	Rate	Center	Rate	Center	Rate
Quantile
0.1	-	-	0.93	0.01 ± 0.01	0.79	0.17 ± 0.08	0.54	0.31 ± 0.12	0.52	0.79 ± 0.11	-	-
0.2	-	-	-	-	-	-	0.58	0.15 ± 0.03	0.56	0.59 ± 0.03	-	-
0.3	-	-	-	-	0.83	0.07 ± 0.02	-	-	0.60	0.62 ± 0.05	-	-
0.4	0.40	0.78 ± 0.04	0.95	0.02 ± 0.02	-	-	0.68	0.14 ± 0.10	0.64	0.47 ± 0.03	0.24	0.39 ± 0.10

Appendix

Table 16: Statistics for Distraction Rate given Irrelevant Context for questions about Founding Year.

model

Claude Sonnet 4

GPT-5

GPT-5 Mini

GPT-5 Nano

LLaMA-70B

LLaMA-8B

Center

Rate

Center

Rate

Center

Rate

Center

Rate

Center

Rate

Center

Rate

Quantile

0.1

0.77

0.32 ± 0.11

0.80

0.00 ± 0.00

0.77

0.00 ± 0.00

0.78

0.01 ± 0.01

0.73

0.17 ± 0.09

0.65

0.44 ± 0.13

0.2

0.79

0.20 ± 0.07

0.82

0.00 ± 0.00

0.80

0.03 ± 0.01

0.80

0.03 ± 0.01

0.76

0.34 ± 0.04

0.68

0.52 ± 0.04

0.3

0.81

0.29 ± 0.02

0.84

0.03 ± 0.01

0.82

0.09 ± 0.06

0.83

0.02 ± 0.02

0.78

0.40 ± 0.04

0.71

0.32 ± 0.07

0.4

0.83

0.19 ± 0.03

0.86

0.01 ± 0.00

0.84

0.02 ± 0.01

0.85

0.02 ± 0.02

0.81

0.16 ± 0.07

0.73

0.41 ± 0.04

Appendix

Table 17: Statistics for Distraction Rate given Irrelevant Context for questions about Industry.

model	Claude Sonnet 4		GPT-5		GPT-5 Mini		GPT-5 Nano		LLaMA-70B		LLaMA-8B
	Center	Rate	Center	Rate	Center	Rate	Center	Rate	Center	Rate	Center	Rate
Quantile
0.1	0.25	0.75 ± 0.05	0.58	0.20 ± 0.13	-	-	-	-	0.38	0.30 ± 0.05	0.23	0.59 ± 0.12
0.2	-	-	-	-	0.46	0.27 ± 0.03	0.38	0.24 ± 0.05	0.42	0.30 ± 0.04	0.28	0.31 ± 0.05
0.3	0.38	0.62 ± 0.14	-	-	0.50	0.28 ± 0.04	-	-	0.47	0.42 ± 0.10	0.32	0.31 ± 0.05
0.4	0.45	0.67 ± 0.04	-	-	0.55	0.18 ± 0.07	0.48	0.27 ± 0.03	0.51	0.38 ± 0.04	0.37	0.48 ± 0.06

Appendix

Table 18: Statistics for Distraction Rate given Irrelevant Context for questions about Key People.

model	Claude Sonnet 4		GPT-5		GPT-5 Mini		GPT-5 Nano		LLaMA-70B		LLaMA-8B
	Center	Rate	Center	Rate	Center	Rate	Center	Rate	Center	Rate	Center	Rate
Quantile
0.1	0.52	0.36 ± 0.13	0.80	0.00 ± 0.00	0.76	0.19 ± 0.09	0.64	0.07 ± 0.03	0.38	0.08 ± 0.08	0.28	0.34 ± 0.07
0.3	-	-	-	-	-	-	-	-	-	-	0.38	0.48 ± 0.11
0.4	-	-	-	-	0.84	0.08 ± 0.03	0.75	0.09 ± 0.04	0.57	0.21 ± 0.04	0.44	0.37 ± 0.05
0.5	0.72	0.69 ± 0.07	-	-	0.86	0.03 ± 0.03	-	-	0.63	0.29 ± 0.06	0.49	0.34 ± 0.04

Appendix

Table 19: Statistics for Distraction Rate given Irrelevant Context for questions about Headquarters.

Jurisdictional Bias in Language Models

See all 9 papers

Faithfulness-QA: A Counterfactual Entity Substitution Dataset for Training Context-Faithful RAG Models
Apr 28, 2026Li Ju, Junzhe Wang, Qi ZhangContextual Faithfulness Retrieval-Augmented Generation
College of Computer Science and Artificial Intelligence, Fudan University
Why Retrieval-Augmented Generation Fails: A Graph Perspective
May 13, 2026Kai Guo, Xinnan Dai, Zhibo Zhang +5Retrieval-Augmented Generation LLM Grounding
Michigan State University · Massachusetts Institute of Technology
Parameters vs. Context: TRACE Fine-Tuning for Robust Retrieval-Augmented Generation
Sep 24, 2026Zhengchen Huang, Yundong Sun, Minrui Song +4RAG Robustness Retrieval-Augmented Generation
School of Information Science & Engineering, LinYi University, China · School of Computer Science and Technology, Harbin Institute of Technology, China

Apr 28, 2026J/K move · Enter open · S save

Faithfulness-QA: A Counterfactual Entity Substitution Dataset for Training Context-Faithful RAG Models

Li Ju, Junzhe Wang, Qi Zhang

College of Computer Science and Artificial Intelligence, Fudan University

Contextual Faithfulness Retrieval-Augmented Generation Knowledge Conflicts in Language Models RAG Evaluation

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) models frequently produce answers grounded in parametric memory rather than the retrieved context, undermining the core promise of retrieval augmentation. A fundamental obstacle to fixing this unfaithfulness is the lack of training data that explicitly requires models to prefer context over internal knowledge. We introduce Faithfulness-QA, a large-scale dataset of 99,094 samples constructed through counterfactual entity substitution. Starting from two established extractive QA benchmarks--SQuAD and TriviaQA--we automatically identify answer-bearing named entities in each context, replace them with type-consistent alternatives drawn from a curated bank of 76,953 entities, and thereby manufacture controlled knowledge conflicts between context and parametric memory. Rigorous quality filtering ensures 100% pass rates across four automated checks on random 200-sample audits. We release the full dataset, the construction pipeline, and a typed entity bank covering eight named entity categories. Faithfulness-QA is designed as a training resource for attention-based faithfulness objectives and as an evaluation benchmark for measuring context-grounding behavior in RAG systems. Data and code are available at https://github.com/qzhangFDU/faithfulness-qa-dataset.

Abstract

Jurisdictional Bias in Language Models

Explore similar work

Faithfulness-QA: A Counterfactual Entity Substitution Dataset for Training Context-Faithful RAG Models

Why Retrieval-Augmented Generation Fails: A Graph Perspective

Parameters vs. Context: TRACE Fine-Tuning for Robust Retrieval-Augmented Generation