Automating Multi-Hop RAG Evaluation via TRIAD: From Context Extraction to Validated Dataset Generation

Lorenz Brehme; Adam Jatowt

cs.CLAug 21, 2026

Automating Multi-Hop RAG Evaluation via TRIAD: From Context Extraction to Validated Dataset Generation

Authors: Lorenz Brehme, Adam Jatowt

Organizations: University of Innsbruck Department of Computer Science Innsbruck, Austria

Abstract

Recent advances in LLMs and the adoption of RAG systems in industry have created a need for domain-specific question-answer datasets that can assess RAG performance on proprietary data. Existing datasets, such as HotpotQA, challenge current RAG systems on Wikipedia-based knowledge, but they cannot be transferred directly to domain-specific settings. A comprehensive evaluation of RAG system quality requires both multi-hop queries and unanswerable questions. This paper introduces TRIAD, a three-stage automated dataset generation approach. First, it generates question--answer (QA) pairs for the domain-specific knowledge base of a RAG system. Second, a validator checks each QA-pair in a feedback loop. Third, the QA pairs are extended with relevance-labeled context documents for downstream evaluation. We evaluate this approach against the established MuSiQue and HotpotQA datasets. The results show that the generated dataset exhibits similar performance trends across different RAG setups, while human validation indicates that the questions are suitable for evaluating a domain-specific RAG system. The code used to generate the dataset and all validation results are available in our GitHub repository(https://github.com/lorenzbrehme/triad).

Figures & tables

Figure 1: Overview of the TRIAD approach

Question Type	Retrieval Pattern	Description	Example
Comparative	Parallel	Compare two or more entities that can be retrieved independently	Which city is larger, Paris or Berlin?
Intersection	Parallel	Retrieve multiple entities independently, then filter or aggregate them to satisfy constraints	Who acted in both Film A and Film B?
Composition	Sequential	Retrieve information in sequence, where the first retrieval leads to the next	What is the capital of the birthplace of Person C?
Binary	Sequential / Parallel	Questions with a yes/no answer	Is the capital of the birthplace of Person C Town A?
Temporal	Sequential / Parallel	Require temporal reasoning across one or more facts	Who was the U.S. president when the company that developed Android was founded?
Unanswerable	—	Questions that cannot be answered from the provided corpus; the model should abstain	—

Table 1: Classification of multi-hop question types by retrieval pattern.

Dataset	Answer-ability	Correct-ness	Multi-hop	Un-ambiguity
HotpotQA	0.90	0.92	0.91	0.92
MuSiQue	0.93	0.98	0.85	0.92

Table 2: Human validation results for TRIAD QA sets.

Panel of Figure 2 — Figure 2: Embedding-model performance using Gemini-2.5-Flash-Lite (Tabular results Appendix A ).

Metric	Gemini	Gemma	GPT-5
HotpotQA – TRIAD (Generated Dataset)
Correctness	0.431	0.387 $\downarrow$	0.447 $\uparrow$
Faithfulness	0.876	0.874	0.886 $\uparrow$
HotpotQA – Static
Correctness	0.246	0.231 $\downarrow$	0.309 $\uparrow$
Faithfulness	0.627	0.563 $\downarrow$	0.722 $\uparrow$

Table 3: Comparison of LLMs using the BGE(5) retriever. Arrows indicate improvements relative to the baseline. The baseline is shown in bold.

Type	Correct	Faithful	Precision	Recall
Comparison*	0.289	0.870	0.391	0.882
Comparison	0.390	0.895	0.387	0.873
Intersection*	0.383	0.817	0.345	0.782
Intersection	0.583	0.807	0.357	0.809
Composition*	0.318	0.832	0.356	0.806
Composition	0.478	0.823	0.343	0.778

Table 4: Performance by question type. * indicates binary questions.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	BGE(3)	BGE(5)	Sent	E-01	E-02
HotpotQA – Generated
Correctness	0.411 $\downarrow$	0.431	0.225 $\downarrow$	0.443 $\uparrow$	0.421 $\downarrow$
Faithfulness	0.879	0.876	0.586 $\downarrow$	0.882	0.877
Precision	0.582 $\uparrow$	0.364	0.071 $\downarrow$	0.380 $\uparrow$	0.385 $\uparrow$
Recall	0.870 $\downarrow$	0.910	0.176 $\downarrow$	0.950 $\uparrow$	0.940 $\uparrow$
HotpotQA – Static

Appendix

Table 5: Comparison of embedding models using Gemini-2.5-Flash-Lite (cf. Figure 2 ). Arrows indicate improvements relative to the baseline. The baseline is shown in bold.

Dataset	Gemini	Gemma	GPT-5
MuSiQue	99.59%	99.58%	93.78%
HotpotQA	99.58%	99.37%	94.76%

Appendix

Table 6: Comparison of scores for unanswerable questions (cf. Section 5.4 ).

Question	Answer	Type	Binary	Corpus
Did Hijokaidan’s early lineup have more members than the Doberman ensemble?	Yes	Comparison	Yes	HotpotQA
Did the British Interplanetary Society’s study for Project Daedalus last longer than their study for Project Boreas?	Yes	Comparison	Yes	MuSiQue
Which single, Will Young’s ’Love Revolution’ or Tomcraft’s ’Loneliness’, was released earlier in the United Kingdom?	Tomcraft’s ’Loneliness	Comparison	No	HotpotQA
Who was born earlier, Alejandro Llorente or Don Carlos, Duke of Madrid?	Alejandro Llorente	Comparison	No	MuSiQue
Did both the 17/18 and the 16/17 UEFA Champions League knockout phases include a total of 16 teams?	Yes	Intersection	Yes	HotpotQA
Is there an actress mentioned who played both a love interest of Ted Mosby and the first lead female character for the Call of Duty franchise?	No	Intersection	Yes	MuSiQue

Appendix

Table 7: Example multi-hop QA samples from our dataset.

Synthetic Benchmark Generation

Synthetic benchmark generation creates controlled, verifiable evaluation tasks and datasets to test models across diverse settings.

See all 281 papers

StratRAG: A Multi-Hop Retrieval Evaluation Dataset for Retrieval-Augmented Generation Systems
Mar 6, 2026Aryan PatodiyaRAG Evaluation Retrieval-Augmented Reasoning
Department of Computer Science, California State University, Fresno, Fresno, CA, USA
LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation
Mar 6, 2026Koki Itai, Shunichi Hasegawa, Yuta Yamamoto +2RAG Evaluation LLM Evaluation
neoAI Inc., Tokyo, Japan · Tokyo Metropolitan University, Tokyo, Japan · The University of Tokyo, Tokyo, Japan
Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategies
Apr 20, 2026Lorenz Brehme, Thomas Ströhle, Ruth BreuMulti-Hop QA LLM-as-a-Judge
Universität Innsbruck, Technikerstraße 21a, 6020 Innsbruck, Austria

Mar 6, 2026J/K move · Enter open · S save

StratRAG: A Multi-Hop Retrieval Evaluation Dataset for Retrieval-Augmented Generation Systems

Aryan Patodiya

Department of Computer Science, California State University, Fresno, Fresno, CA, USA

RAG Evaluation Retrieval-Augmented Reasoning Information Retrieval Multi-Hop QA Hybrid Retrieval

Open paper AskPDF arXiv

We introduce StratRAG, an open-source retrieval evaluation dataset for benchmarking Retrieval-Augmented Generation (RAG) systems on multi-hop reasoning tasks under realistic, noisy document-pool conditions. Derived from HotpotQA (distractor setting), StratRAG comprises 2,200 examples across three question types -- bridge, comparison, and yes-no -- each paired with a pool of 15 candidate documents containing exactly 2 gold documents and 13 topically related distractors. We benchmark three retrieval strategies -- BM25, dense retrieval (all-MiniLM-L6-v2), and hybrid fusion -- reporting Recall@k, MRR, and NDCG@5 on the validation set. Hybrid retrieval achieves the best overall performance (Recall@2 = 0.70, MRR = 0.93), yet bridge questions remain substantially harder (Recall@2 = 0.67), motivating future work on reinforcement-learning-based retrieval policies. StratRAG is publicly available at https://huggingface.co/datasets/Aryanp088/StratRAG.

Abstract

Synthetic Benchmark Generation

Explore similar work

StratRAG: A Multi-Hop Retrieval Evaluation Dataset for Retrieval-Augmented Generation Systems

LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation

Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategies