Geopolitical and strategic intelligence reports constitute a rich source of expert knowledge and have increasing value in the current political climate, but their heterogeneity and analytical complexity pose challenges for retrieval-augmented question answering. We introduce a benchmark for grounded question answering over 415 open-source geopolitical and strategic intelligence reports, comprising 5,525 document-grounded question-answer pairs spanning factual, inferential, comparative, and risk-oriented questions. We propose a document-aware generation pipeline that profiles reports before generating questions and validate dataset quality through expert annotation and an LLM-as-judge protocol. Using the benchmark, we evaluate 15 retrieval systems and six open-weight LLMs for RAG. We find that models perform similarly on document-level retrieval, while at the harder chunk level retrieval quality is not driven by embedding-model scale: bge-m3, a mid-size open encoder matches an order of magnitude larger ones and outperforms commercial embedding APIs. In contrast, answer generation remains challenging even given the gold evidence, particularly for inferential and risk-oriented questions.
Figures & tables
Institutional category
Documents
Regional / local think tanks
129
Specialised domains
0 64
Official & multilateral
0 63
Global think tanks
0 51
Index & data-narrative
0 31
Parliamentary services
0 29
Table 1: Corpus composition by institutional category (122 publishers).
Question type
Count
%
Factual
3,805
68.9
Inferential
789
14.3
Comparative
696
12.6
Risk-oriented
235
0 4.3
Total
5,525
100.0
Table 2: Question-type distribution of the benchmark.
Dimension
Cohen’s κ
exact (3/3)
≥ 2/3
Groundedness
0.42
78%
99%
Faithfulness
0.30
58%
100%
Overall
0.28
54%
98%
Clarity
0.28
60%
98%
Doc-purpose fit
0.12
71%
99%
Atomicity
0.05
60%
99%
Table 3: Inter-annotator agreement on the 100-item overlap block (three annotators). κ is quadratic-weighted, averaged over rater pairs.
Agreement with →
A1
A2
A3
GPT-5.4
80%
63%
75%
Claude Sonnet 5
75%
59%
71%
DeepSeek-V3.2
76%
61%
71%
other annotators
78%
69%
79%
Table 4: Judge validation: pooled exact agreement (all six dimensions) with each human annotator (A1–A3). Judge rows give judge-vs-annotator agreement; the other annotators row gives the mean agreement of the remaining two annotators with that annotator (the per-annotator human ceiling). A2 is the systematically stricter annotator, so all raters (human and model) agree with A2 least.
Table 6: Retrieval leaderboard ( n=4,980 queries, 55,353 chunks), ranked by chunk nDCG@10 against answer-cited gold. Query/passage prefixes follow each family’s recommended format (Appendix G ). ∗ in-construction (BM25 and e5 were the construction hybrid’s components); shown for reference, not ranked as winners.
Figure 1: In-domain retrieval scaling: chunk nDCG@10 (answer-cited gold) vs. encoder size (log scale) by family. Dense encoders largely scale with size (GTE is an exception, with gte-base above gte-large), and the best (bge-m3, Qwen3-Emb-8B) match the BM25 baseline (dashed).
Reader
Size
Faith.
Correct.
Overall [95% CI]
Llama-3.3-70B
70B
2.69
2.60
2.56 [2.48, 2.64]
Mistral-Small-3.2
24B
2.60
2.60
2.55 [2.46, 2.63]
Qwen3-8B
8B
2.46
2.66
2.48 [2.40, 2.56]
Qwen3-32B
32B
2.48
2.67
2.48 [2.40, 2.56]
Llama-3.1-8B
8B
2.48
2.34
2.30 [2.20, 2.39]
Mistral-Nemo
12B
2.57
2.30
2.28 [2.17, 2.38]
Table 7: RAG reader leaderboard (GPT-5.4 judge, 1–3; n=250 ), readers given gold context to isolate generation quality. Faithfulness and overall are validated dimensions; correctness is exploratory.
Question type
n
doc-miss
chunk-miss
recall
Factual
3416
3%
7%
89%
Inferential
719
3%
6%
82%
Comparative
626
2%
6%
88%
Risk-oriented
219
3%
4%
78%
Table 8: BM25 retrieval failures at top-10 by question type, against answer-cited gold (with content-hash dedup credit): fraction of queries retrieving no source-document chunk (doc-miss) / no answer-cited chunk (chunk-miss), and mean per-query recall of the cited set.
Figure 2: RAG answer quality (overall, 1–3) by question type and model size. Both tiers degrade from factual to inferential to risk-oriented questions; the small–mid gap is largest on the harder, reasoning-heavy types.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Persona
Share
Geopolitical Researcher
19.1%
Intelligence Analyst
11.0%
Humanitarian Affairs Analyst
10.5%
Risk Intelligence Analyst
10.4%
Conflict Data Analyst
9.0%
Financial Analyst
8.1%
Appendix
Table 9: Analyst-persona distribution over the external benchmark (16 personas).
Type (persona)
Question
Reference answer
Factual (Military Analyst)
What specific types of decommissioned Soviet-era equipment were returned to service during 2023?
Long-decommissioned equipment returned to service in 2023: tanks T-55 and T-62 [1]; howitzers D-10 and D-20 [1].
Inferential (Geopolitical Researcher)
In the context of the India–China nuclear dyad, what is the implication of the “stability–instability paradox”?
The implication is that nuclear stability at the strategic level decreases the likelihood of nuclear confrontation, which in turn increases the chances of conventional confrontation; it is therefore suggested that India needs to hone its conventional capabilities [1].
Comparative (Intelligence Analyst)
What technical advantage does the Jinan-1 microsatellite have over the Micius satellite?
The Jinan-1 microsatellite weighs one-sixth of the Micius satellite and has 2–3 orders of magnitude better accuracy in quantum communication [1].
Risk-oriented (Geopolitical Researcher)
What specific risk does the document associate with US bilateral deals with the DRC and Ukraine regarding critical minerals?
The deals have been criticised as asymmetric, because potential security assistance is linked to access to resources [1].
Appendix
Table 10: Representative Q&A pairs, one per question type.
Family
Query prefix
Passage
E5 / mE5
query:
passage:
BGE (en)
Represent this sentence for searching relevant passages:
—
BGE-M3
—
—
Qwen3-Emb
Instruct: … Query:
—
GTE, MiniLM
—
—
OpenAI, Mistral
—
—
Appendix
Table 11: Per-family embedding prefixes. The Qwen3 query instruction in full is “Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: ”.
Question answering studies how AI systems interpret questions and produce accurate, evidence-grounded answers across text, dialogue, and multimodal data.