StratRAG: A Multi-Hop Retrieval Evaluation Dataset for Retrieval-Augmented Generation Systems

Aryan Patodiya

cs.IRMar 6, 2026

StratRAG: A Multi-Hop Retrieval Evaluation Dataset for Retrieval-Augmented Generation Systems

Authors: Aryan Patodiya

Organizations: Department of Computer Science, California State University, Fresno, Fresno, CA, USA

Abstract

We introduce StratRAG, an open-source retrieval evaluation dataset for benchmarking Retrieval-Augmented Generation (RAG) systems on multi-hop reasoning tasks under realistic, noisy document-pool conditions. Derived from HotpotQA (distractor setting), StratRAG comprises 2,200 examples across three question types -- bridge, comparison, and yes-no -- each paired with a pool of 15 candidate documents containing exactly 2 gold documents and 13 topically related distractors. We benchmark three retrieval strategies -- BM25, dense retrieval (all-MiniLM-L6-v2), and hybrid fusion -- reporting Recall@k, MRR, and NDCG@5 on the validation set. Hybrid retrieval achieves the best overall performance (Recall@2 = 0.70, MRR = 0.93), yet bridge questions remain substantially harder (Recall@2 = 0.67), motivating future work on reinforcement-learning-based retrieval policies. StratRAG is publicly available at https://huggingface.co/datasets/Aryanp088/StratRAG.

Figures & tables

Statistic	Train	Validation
Total examples	2,000	200
Bridge questions	1,775	171
Comparison questions	112	17
Yes-no questions	113	12
Docs per pool	15	15
Gold docs per example	2	2

Table 1: StratRAG dataset statistics.

Retriever	R@1	R@2	R@5	MRR	NDCG@5
Random	0.0525	0.1425	0.3300	0.3190	0.2336
BM25	0.3950	0.6000	0.8150	0.8732	0.7624
Dense (MiniLM)	0.4175	0.6500	0.8600	0.9035	0.8087
Hybrid	0.4400	0.6975	0.9050	0.9310	0.8543

Table 2: Retrieval benchmark results on StratRAG validation set (

n=200

). Best results in bold .

Type	n	R@2	MRR	NDCG@5
Bridge	171	0.6696	0.9281	0.8418
Yes-no	12	0.8333	0.9167	0.9007
Comparison	17	0.8824	0.9706	0.9473

Table 3: Hybrid retriever performance by question type on the StratRAG validation set.

RAG Evaluation

Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.

See all 337 papers

KohakuRAG: A simple RAG framework with hierarchical document indexing
Mar 8, 2026Shih-Ying Yeh, Yueh-Feng Ku, Ko-Wei Huang +1Retrieval-Augmented Generation Multi-Document QA
National Tsing Hua University · Comfy Org Research · Kohaku-Lab
Automating Multi-Hop RAG Evaluation via TRIAD: From Context Extraction to Validated Dataset Generation
Aug 21, 2026Lorenz Brehme, Adam JatowtSynthetic Benchmark Generation RAG Evaluation
University of Innsbruck Department of Computer Science Innsbruck, Austria
IndexRAG: Index-Time Reasoning for Multi-Hop Retrieval-Augmented Generation
Mar 17, 2026Zhenghua Bao, Yi ShiMulti-Hop QA Retrieval-Augmented Generation
Continuum AI Shanghai, China · Continuum AI San Francisco, CA, USA

Mar 8, 2026J/K move · Enter open · S save

KohakuRAG: A simple RAG framework with hierarchical document indexing

Shih-Ying Yeh, Yueh-Feng Ku, Ko-Wei Huang, Buu-Khang Tu

National Tsing Hua University · Comfy Org Research · Kohaku-Lab

Retrieval-Augmented Generation Multi-Document QA Knowledge-Intensive QA

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) systems that answer questions from document collections face compounding difficulties when high-precision citations are required: flat chunking strategies sacrifice document structure, single-query formulations miss relevant passages through vocabulary mismatch, and single-pass inference produces stochastic answers that vary in both content and citation selection. We present KohakuRAG, a hierarchical RAG framework that preserves document structure through a four-level tree representation (document

\rightarrow

section

\rightarrow

paragraph

\rightarrow

sentence) with bottom-up embedding aggregation, improves retrieval coverage through an LLM-powered query planner with cross-query reranking, and stabilizes answers through ensemble inference with abstention-aware voting. We evaluate on the WattBot 2025 Challenge, a benchmark requiring systems to answer technical questions from 32 documents with

\pm

0.1% numeric tolerance and exact source attribution. KohakuRAG achieves first place on both public and private leaderboards (final score 0.861), as the only team to maintain the top position across both evaluation partitions. Ablation studies reveal that prompt ordering (+80% relative), retry mechanisms (+69%), and ensemble voting with blank filtering (+1.2pp) each contribute substantially, while hierarchical dense retrieval alone matches hybrid sparse-dense approaches (BM25 adds only +3.1pp). We release KohakuRAG as open-source software at https://github.com/KohakuBlueleaf/KohakuRAG.

Abstract

RAG Evaluation

Explore similar work

KohakuRAG: A simple RAG framework with hierarchical document indexing

Automating Multi-Hop RAG Evaluation via TRIAD: From Context Extraction to Validated Dataset Generation

IndexRAG: Index-Time Reasoning for Multi-Hop Retrieval-Augmented Generation