MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations

Sara Rosenthal; Yannis Katsis; Vraj Shah; Lihong He; Lucian Popa; Marina Danilevsky

cs.CLFeb 26, 2026

MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations

Authors: Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa, Marina Danilevsky

Organizations: IBM Research, USA

Abstract

We present MTRAG-UN, a benchmark for exploring open challenges in multi-turn retrieval augmented generation, a popular use of large language models. We release a benchmark of 666 tasks containing over 2,800 conversation turns across 6 domains with accompanying corpora. Our experiments show that retrieval and generation models continue to struggle on conversations with UNanswerable, UNderspecified, and NONstandalone questions and UNclear responses. Our benchmark is available at https://github.com/IBM/mt-rag-benchmark

Figures & tables

Figure 1: Portions of three conversations highlighting the challenges in MTRAG-UN. The answerability is shown using the assistant response color: answerable , unanswerable , and underspecified . The multi-turn type is shown using the question circle: follow-up and clarification . The last two examples show non-standalone questions.

Figure 2: Distribution of tasks in MTRAG-UN based on different dimensions.

Corpus	Documents (D)	Passages (P)	Avg P/D
Banking	4,497	33,380	7.4
Telco	4,616	52,350	11.3

Table 1: Statistics of new document corpora in MTRAG-UN.

		Recall		nDCG
		@5	@10	@5	@10
BM25	LT	0.29	0.38	0.27	0.31
	RW	0.36	0.47	0.34	0.39
BGE-base 1.5	LT	0.25	0.32	0.23	0.26
	RW	0.38	0.49	0.35	0.40
Granite R2	LT	0.29	0.38	0.28	0.32

Table 2: Retrieval Performance using Recall and nDCG metrics for Last Turn (LT) and Query Rewrite (RW)

	Subset	LT	RW
Standalone	No (214)	0.39	0.52
Standalone	Yes (254)	0.40	0.46

Table 3: Elser R@5 standalone results

Panel of Figure 3 — Figure 3: Generation results in the Reference ( $\bullet$ ) setting using, $\textrm{RB}_{\textrm{alg}}$ , on three different dimensions.

	RL ${}_{\textrm{F}}$		$\textrm{RB}_{\textrm{llm}}$		$\textrm{RB}_{\textrm{alg}}$
	$\bullet$	$\circ$	$\bullet$	$\circ$	$\bullet$	$\circ$
target	0.85	0.69	0.96	0.92	0.89	0.88
gpt-oss-120b	0.65	0.59	0.76	0.65	0.46	0.37
gpt-oss-20b	0.60	0.55	0.67	0.63	0.43	0.36
deepseek-v3	0.63	0.60	0.61	0.58	0.42	0.37
deepseek-r1	0.47	0.46	0.54	0.52	0.26	0.23

Table 4: Generation by retrieval setting: Reference (

\bullet

) and RAG (

\circ

). The best result is bold and runner-up is underlined .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Distribution of tasks in MTRAG-UN based on conversational turn.

Panel of Figure 5 — Figure 5: Weighted Spearman correlation: automated judge metrics vs human evaluation metrics.

Figure 6: Prompt used for clarification judge.

Pattern	Example
Hedging with answer	There are many astronauts you could be referring to, such as Ellen Ochoa, who was the first Hispanic woman to go to space and has received numerous awards, including the Presidential Medal of Freedom, or Kalpana Chawla, who was the first woman of Indian origin to go to space and tragically died in the Columbia disaster in 2003. Which one are you talking about?
Hedging over list	There are many fast food chains you could be talking about, like In-N-Out Burger, Jack in the Box, Big Boy Restaurants, or Chipotle Mexican Grill. Which one are you referring to?
Open-domain	Which modern smart design segment are you talking about?

Appendix

Table 5: Types of response to underspecified questions.

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 792 papers

MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks
May 20, 2026Junhao Ruan, Abudukeyumu Abudula, Bei Li +8Conversational Search Synthetic Benchmark Generation
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · 2Meituan Inc., 3NiuTrans Research, Shenyang, China · 4Tsinghua University, Beijing, China
Lost in Conversation or Lost in Translation? Diagnosing Multi-Turn Degradation in RAG
Sep 29, 2026Pranav Handa, Ariful AzadRetrieval-Augmented Generation LLM Reliability
Texas A&M University
LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation
Mar 6, 2026Koki Itai, Shunichi Hasegawa, Yuta Yamamoto +2RAG Evaluation LLM Evaluation
neoAI Inc., Tokyo, Japan · Tokyo Metropolitan University, Tokyo, Japan · The University of Tokyo, Tokyo, Japan

May 20, 2026J/K move · Enter open · S save

MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks

Junhao Ruan, Abudukeyumu Abudula, Bei Li, Yongjing Yin+7

School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · 2Meituan Inc., 3NiuTrans Research, Shenyang, China · 4Tsinghua University, Beijing, China

Conversational Search Synthetic Benchmark Generation Multi-Turn Dialogue Evaluation RAG Evaluation IR Evaluation

Open paper AskPDF arXiv

Accurate evaluation of conversational retrieval is pivotal for advancing Retrieval-Augmented Generation (RAG) systems. However, existing conversational retrieval benchmarks suffer from costly, sparse human annotation or rigid, unnatural automated heuristics. To address these challenges, we introduce MTR-Suite, a unified framework for auditing, synthesizing, and benchmarking retrieval. It features: (1) MTR-Eval, an LLM-based auditor quantifying alignment gaps in previous benchmarks; (2) MTR-Pipeline, a multi-agent system using greedy traversal clustering to generate high-fidelity dialogues at 1/400th human cost; and (3) MTR-Bench, a rigorous general-domain benchmark. MTR-Bench mimics production-style challenges (hard topic switching, verbosity), offering superior discriminative power. We make our code and data publicly available to facilitate future research at https://github.com/rangehow/mtr-suite.

Abstract

Retrieval-Augmented Generation

Explore similar work

MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks

Lost in Conversation or Lost in Translation? Diagnosing Multi-Turn Degradation in RAG

LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation