AfriEconQA: A Benchmark for Quantitative and Temporal Reasoning over World Bank Economic Reports

Edward Ajayi; Mustapha Alaba; David Stephen; Peace Bakare

cs.CLJan 6, 2026

AfriEconQA: A Benchmark for Quantitative and Temporal Reasoning over World Bank Economic Reports

Authors: Edward Ajayi, Mustapha Alaba, David Stephen, Peace Bakare

Organizations: Carnegie Mellon University Africa Kigali, Rwanda

Abstract

Reliable question answering over long institutional documents requires more than topical retrieval: a system must localize the exact passage that supports a claim and preserve precise numerical and temporal detail when the same indicator recurs across years, countries, and projection horizons in dense, repetitive prose. Existing question-answering benchmarks rarely test this combination of exact grounding and temporal precision at document scale. We introduce AfriEconQA, a benchmark for document-grounded question answering built from 204 World Bank economic reports on African economies, a corpus whose claims are tied to specific fiscal periods, countries, and projection states. AfriEconQA contains 4,138 evidence-linked QA instances across five reasoning categories: Factoid (1,063), List (852), Multiple Choice (889), Synthesis (671), and Comparison (663), targeting quantitative extraction, set recovery, discrimination, causal integration, and cross-period or cross-country reasoning. Each instance carries supporting evidence and source provenance and is constructed through an agentic generation pipeline with evidence-grounding checks and a stratified human-validation subset for gold-label auditing. We evaluate Qwen 3.6 35B, DeepSeek v4-pro, Gemma 4 12B IT, and frontier model GPT-5.5 under zero-shot, oracle, and hybrid retrieval-augmented generation (RAG) conditions on the held-out test split (n = 826). Across all four models, retrieval yields substantial gains, yet the best RAG system reaches only 0.594 F1, with residual errors concentrated in list extraction, synthesis, and temporally scoped comparison. AfriEconQA therefore poses a hard open challenge for exact numerical and temporal grounding over long institutional economic reports.

Figures & tables

Field	Example
Type	Comparison
Question	What is the projected change in monetary poverty in Burkina Faso between 2023, 2024, and 2025?
Answer	A decrease of 2.6 percentage points between 2023 and 2024, and another 3.3 percentage points by 2025
Evidence	Monetary poverty (at the international poverty line) in Burkina Faso is projected to have fallen 2.6 percentage points between 2023 and 2024 and to fall another 3.3 percentage points by 2025
Source	Burkina Faso Poverty and Equity Brief, October 2025

Table 1: Example AfriEconQA instance (id 1995) requiring country-scoped, multi-year, and projection-sensitive grounding.

Question Type	Count	%
Factoid	1,063	25.7
Multiple Choice	889	21.5
List	852	20.6
Synthesis	671	16.2
Comparison	663	16.0
Total	4,138	100.0

Table 2: Question-type distribution in AfriEconQA.

Dataset	Primary source	#QA	#Docs	Evidence	Focus
FinQA	S&P 500 earnings reports	8,281	–	Program + context	Numerical reasoning
ConvFinQA	S&P 500 earnings reports	14,115	–	Conversation + context	Multi-turn numerical QA
TAT-QA	Financial reports (tables+text)	16,552	2,757	Cell / text spans	Hybrid tabular-text reasoning
MultiHiertt	Multi-hierarchical reports	10,440	1,327	Hierarchical tables	Multi-table arithmetic QA
DocFinQA	Full SEC filings	7,437	801	Full document	Long-context financial QA
FinanceBench	Public company filings	10,231	–	Evidence strings	Open-book financial QA

Table 3: Comparison of AfriEconQA with related financial and economic QA datasets. Document counts are reported only when stated by the source paper.

Model	Condition	EM	F1	ROUGE-L	LLM-judge
Qwen	Zero-shot	0.162 [0.140, 0.186]	0.226 [0.204, 0.250]	0.172	0.248
	Oracle	0.386 [0.355, 0.418]	0.492 [0.461, 0.520]	0.273	0.536
	RAG	0.415 [0.379, 0.448]	0.521 [0.488, 0.549]	0.306	0.632
DeepSeek	Zero-shot	0.200 [0.174, 0.228]	0.265 [0.239, 0.292]	0.193	0.326
	Oracle	0.384 [0.350, 0.418]	0.482 [0.450, 0.511]	0.266	0.499
	RAG	0.430 [0.394, 0.461]	0.541 [0.509, 0.570]	0.315	0.641

Table 4: Main results on the AfriEconQA test split (

n=826

). All metrics in

[0,1]

. EM and F1 include 95% bootstrap CIs. LLM-judge denotes semantic evaluation (secondary). Full-benchmark results are in Appendix A .

Metric	BM25	BGE-M3	Hybrid
HR@1	0.268	0.308	0.326
HR@5	0.419	0.488	0.498
MRR@5	0.324	0.380	0.391
Doc@1	0.754	0.797	0.840
Doc@5	0.910	0.947	0.959
Doc MRR@5	0.815	0.859	0.887

Table 5: Retriever-only results on the test split (

n=826

). All metrics are in

[0,1]

. Strict metrics require an evidence-supporting chunk, whereas Doc@* only require the correct source document.

Model	Type	ZS	Oracle	RAG
Qwen	Comparison	0.116	0.379	0.477
	Factoid	0.030	0.572	0.689
	List	0.008	0.187	0.084
	MC	0.704	0.905	0.849
	Synthesis	0.272	0.313	0.415
DeepSeek	Comparison	0.132	0.390	0.498

Table 6: Test-split F1 by question type (3 d.p.,

[0,1]

) under zero-shot (ZS), oracle, and RAG (

n=826

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Condition	EM	F1	ROUGE-L	LLM-judge
Qwen	Zero-shot	0.162 [0.151, 0.173]	0.228 [0.218, 0.239]	0.174	0.411
	Oracle	0.392 [0.377, 0.408]	0.494 [0.480, 0.508]	0.286	0.605
	RAG	0.426 [0.411, 0.440]	0.537 [0.524, 0.550]	0.314	0.708
DeepSeek	Zero-shot	0.190 [0.178, 0.201]	0.259 [0.247, 0.270]	0.192	0.477
	Oracle	0.375 [0.360, 0.389]	0.474 [0.461, 0.488]	0.279	0.572
	RAG	0.439 [0.424, 0.453]	0.551 [0.537, 0.563]	0.319	0.721

Appendix

Table 7: Full-benchmark main results (

n=4{,}138

). All metrics in

[0,1]

. EM and F1 include 95% bootstrap CIs. LLM-judge is the mean Groq-hosted Llama-3.3-70B LLM-as-a-judge score (secondary).

Metric	BM25	BGE-M3	Hybrid
HR@1	0.271	0.307	0.325
HR@5	0.432	0.490	0.496
MRR@5	0.333	0.379	0.392
Doc@1	0.741	0.805	0.825
Doc@5	0.902	0.945	0.954
Doc MRR@5	0.806	0.864	0.879

Appendix

Table 8: Full-benchmark retriever-only results (

n=4{,}138

). All metrics in

[0,1]

Model	Type	ZS	Oracle	RAG
Qwen	Comparison	0.129	0.369	0.486
	Factoid	0.050	0.579	0.705
	List	0.006	0.154	0.084
	MC	0.677	0.926	0.876
	Synthesis	0.276	0.332	0.441
DeepSeek	Comparison	0.161	0.342	0.507

Appendix

Table 9: Full-benchmark F1 by question type (

n=4{,}138

, 3 d.p.,

[0,1]

Setting	Value
Generators
Qwen 3.6 35B	qwen/qwen3.6-35b-a3b (LM Studio)
Gemma 4 12B IT	google/gemma-4-12b (LM Studio)
DeepSeek v4-pro	deepseek-v4-pro (DeepSeek API)
GPT-5.5	gpt-5.5 (LiteLLM)
Decoding (all generators)	temperature 0.2, top-p 0.95, max 320 tokens

Appendix

Table 10: Fixed implementation settings (also in supplementary config.py ).

Case & Type	Audit Comparison (Gold vs. Predictions)
ID 193	Q: What is the inflation rate of Somalia in 2024?
Factoid	Gold: 5.6 percent
	Oracle: Somalia’s inflation rate in 2024 was 5.6 percent.
	RAG (Near-Miss): Somalia’s inflation rate averaged 5.5 percent in 2024.
ID 549	Q: Projected poverty rate in Ghana in 2025 (LMIC line)?
Factoid	Gold: 53.3 percent

Appendix

Table 11: Audited qualitative failure cases from the held-out test split (

n=826

) demonstrating near-miss RAG retrieval collisions and Oracle synthesis omissions.

Label	Majority ( $n=200$ )	Resolved (%)	Labels ( $N=600$ )	Label (%)
Correct	189	95.5%	552	92.0%
Partial	5	2.5%	33	5.5%
Incorrect	3	1.5%	10	1.7%
Unanswerable	1	0.5%	5	0.8%
Ties (no majority)	2	1.0%	–	–
Total	200	100.0%	600	100.0%

Appendix

Table 12: Distribution of human evaluation judgments across all 600 labels and majority-vote outcomes (

n=200

Figure 1: AfriEconQA explorer audit view: question, gold answer, evidence, and validity labels.

Figure 2: Source-PDF view in the explorer, with supporting evidence span highlighted in the World Bank report.

Question Answering

Question answering studies how AI systems interpret questions and produce accurate, evidence-grounded answers across text, dialogue, and multimodal data.

See all 259 papers

Sep 3, 2026

Enhancing Financial Question Answering: A Novel Benchmark Dataset of Banks' financial statements

The comparative analysis of banks' financial statements poses significant challenges for automated question answering systems due to their complexity, substantial length, technical language, and inhomogeneity of both textual and numerical content across different jurisdictions and institutions. We introduce FinRAG-QA, a novel benchmark dataset for financial question answering, which comprises 999 practitioner-curated questions on 10 standardised indicators, grounded in 209 annual and Pillar 3 reports from 24 major European and U.S. banks spanning 2019-2023. Unlike prior financial QA benchmarks, which centre on U.S. filings and single-institution analysis, FinRAG-QA targets cross-institutional retrieval over documents averaging 198k words, longer than any existing financial QA resource. On this benchmark we evaluate a multi-stage RAG pipeline and isolate the contribution of each component. Contextual chunk enrichment combined with a retrieval-optimised embedding model raises NDCG@10 from 0.322 to 0.710; conditional on the ground truth being retrieved, a reasoning-optimised generator raises answer accuracy from 44.6% to 79.0% (+34.4 percentage points), at roughly 20x the generation latency. We further show that cross-encoder reranking degrades retrieval when the first-stage ranking is already strong, and that a single top-ranked chunk outperforms larger contexts at generation time. Experiments were run in late 2024-early 2025 with the models available at that time.

Arianna Miola, Bruno Spaccavento, Lorenzo Silotto +2

Intesa Sanpaolo Innovation Center, Corso Inghilterra 3, 10138, Torino, Italy. · Universit`a degli Studi di Milano-Bicocca, Milan, Italy · Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129, Turin, Italy. +2

Financial QA RAG Evaluation

Ask PDF

Mar 9, 2026

OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning

We introduce OfficeQA Pro, a benchmark for evaluating AI agents on grounded, multi-document reasoning over a large and heterogeneous document corpus. The corpus consists of U.S. Treasury Bulletins spanning nearly 100 years, comprising 89,000 pages and over 26 million numerical values. OfficeQA Pro consists of 133 questions that require precise document parsing, retrieval, and analytical reasoning across both unstructured text and tabular data. Frontier LLMs including Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro Preview achieve less than 5% accuracy on OfficeQA Pro when relying on parametric knowledge, and less than 12% with additional access to the web. When provided directly with the document corpus, frontier agents still struggle on over half of questions, scoring 34.1% on average. We find that providing agents with a structured document representation produced by Databricks' ai_parse_document yields a 16.1% average relative performance gain across agents. We conduct additional ablations to study the effects of model selection, table representation, retrieval strategy, and test-time scaling on performance. Despite these improvements, significant headroom remains before agents can be considered reliable at enterprise-grade grounded reasoning.

Krista Opsahl-Ong, Arnav Singhvi, Jasmine Collins +10

Databricks

Multi-Document QA LLM Agent Evaluation AI Agent Benchmarks Document QA LLM Grounding Table QA

Ask PDF

Apr 4, 2026

Document-Level Numerical Reasoning across Single and Multiple Tables in Financial Reports

Despite the strong language understanding abilities of large language models (LLMs), they still struggle with reliable question answering (QA) over long, structured documents, particularly for numerical reasoning. Financial annual reports exemplify this difficulty: financial statement analysis often hinges on accurate arithmetic, and analysts derive key indicators by integrating evidence scattered across multiple tables and narrative text. However, existing benchmarks focus largely on single-table settings, leaving cross-table document-level numerical reasoning underexplored. To address this gap, we introduce FinLongDocQA, a dataset for both single-table and cross-table financial numerical reasoning in long-context reports. Evaluating both closed-source and open-source LLMs on FinLongDocQA reveals two bottlenecks: (1) annual reports often exceed 129k tokens, exacerbating the context rot problem for locating relevant tables; and (2) even when relevant evidence is located, LLMs remain prone to errors in multi-step numerical reasoning. We propose FinLongDocAgent, a Multi-Agent Multi-Round Retrieval-Augmented Generation (RAG) approach that iteratively retrieves evidence, performs intermediate calculations, and verifies results across rounds. Experiments highlight the importance of iterative retrieval and verification for reliable numerical QA in long financial documents.

Yi-Cheng Wang, Wei-An Wang, Chu-Song Chen

Department of Computer Science and Information Engineering, National Taiwan University · FinTech Center, National Taiwan University

Numerical Reasoning in Language Models Financial QA Mathematical Reasoning Retrieval-Augmented Generation

Ask PDF

Abstract

Question Answering

Explore similar work

Enhancing Financial Question Answering: A Novel Benchmark Dataset of Banks' financial statements

OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning

Document-Level Numerical Reasoning across Single and Multiple Tables in Financial Reports