AILS-NTUA at SemEval-2026 Task 8: Evaluating Multi-Turn RAG Conversations

Dimosthenis Athanasiou; Maria Lymperaiou; Giorgos Filandrianos; Athanasios Voulodimos; Giorgos Stamou

cs.CLMar 11, 2026

AILS-NTUA at SemEval-2026 Task 8: Evaluating Multi-Turn RAG Conversations

Authors: Dimosthenis Athanasiou, Maria Lymperaiou, Giorgos Filandrianos, Athanasios Voulodimos, Giorgos Stamou

Organizations: National Technical University of Athens

Abstract

We present the AILS-NTUA system for SemEval-2026 Task 8 (MTRAGEval), addressing all three subtasks of multi-turn retrieval-augmented generation: passage retrieval (A), reference-grounded response generation (B), and end-to-end RAG (C). Our unified architecture is built on two principles: (i) a query-diversity-over-retriever-diversity strategy, where five complementary LLM-based query reformulations are issued to a single corpus-aligned sparse retriever and fused via variance-aware nested Reciprocal Rank Fusion; and (ii) a multistage generation pipeline that decomposes grounded generation into evidence span extraction, dual-candidate drafting, and calibrated multi-judge selection. Our system ranks 1st in Task A (nDCG@5: 0.5776, +20.5% over the strongest baseline) and 2nd in Task B (HM: 0.7698). Empirical analysis shows that query diversity over a well-aligned retriever outperforms heterogeneous retriever ensembling, and that answerability calibration-rather than retrieval coverage-is the primary bottleneck in end-to-end performance.

Figures & tables

Panel of Figure 1 — Figure 1: System architecture: (a) Task A retrieval pipeline, (b) Task B generation pipeline, (c) Task C end-to-end RAG with answerability gate.

Characteristic	ClapNQ	FiQA	Govt	Cloud
Documents	4,293	57,638 †	7,661	8,578
Passages	183,408	61,022	49,607	72,442
Avg. passages/doc	42.7	1.1	6.5	8.4
Domain	Wikipedia	Finance	Government	Tech docs

Table 1: MTRAG corpus statistics. † FiQA contains individual forum posts.

Parameter	ClapNQ	FiQA	Govt	Cloud
$k_{\text{final}}$	20	60	40	20
$w_{\text{Minimal}}$	0.55	0.45	0.65	0.65
$w_{\text{Corpus-Spec}}$	0.40	0.40	0.25	0.30
$w_{\text{WeakCons}}$	0.05	0.15	0.10	0.05

Table 2: Nested RRF parameters per domain.

Task	Metric	Ours	Rank
A	nDCG@5	0.5776	1/38
B	HM	0.7698	2/26
C	HM	0.5409	11/29

Table 3: Official test set results.

Configuration	R@5	$\Delta$
No rewriting	0.483	–
+ Minimal	0.527	$+$ 9.1%
+ Corpus-Specific	0.558	$+$ 15.5%
+ CoT	0.573	$+$ 18.6%
+ HyDE	0.584	$+$ 20.9%
+ Anchor-Keyword	0.591	$+$ 22.4%

Table 4: Task A: ablations on incremental rewriting strategy contribution (dev set).

Configuration	HM
Single generation ( $\tau{=}0.0$ )	0.654
+Span extraction	0.688
+ Dual generation (random select)	0.706
+ Technical Judge	0.725
+ User Satisfaction Judge (60%)	0.738
+ Micro-adjustments	0.743

Table 5: Task B: generation pipeline ablation (dev set).

Method	ANS R	UNANS R	UNANS F1
Always Answer	100.0	0.0	0.0
Single Judge	94.6	18.2	21.5
Pipeline + Verif.	92.3	27.3	23.1
Multi-Judge (ours)	95.8	21.8	24.0
Multi-Judge + Calib.	92.3	27.3	23.1

Table 6: Answerability detection (dev set).

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Corpus	Domain	Docs	Passages	Avg P/D
ClapNQ	Wikipedia	4,293	183,408	42.7
FiQA	Finance forum	57,638	61,022	1.1
Govt	Government	7,661	49,607	6.5
Cloud	Technical docs	8,578	72,442	8.4
Total		78,170	366,479

Appendix

Table 7: Corpus statistics. FiQA “Docs” correspond to individual forum posts (atomic documents).

Domain	Ans	Partial	Unans	Conv
ClapNQ	86%	7%	7%	0%
FiQA	82%	9%	7%	3%
Govt	83%	11%	6%	0%
Cloud	86%	5%	7%	1%
Overall	84%	8%	7%	1%

Appendix

Table 8: Answerability distribution per domain (dev set, 842 turns). Percentages are row-normalized.

Domain	Top-3 Question Types	Coverage
ClapNQ	Factoid, Summarization, Explanation	71%
FiQA	Opinion, Factoid, Explanation	51%
Govt	Factoid, Summarization, Explanation	63%
Cloud	Factoid, Explanation, Summarization	50%

Appendix

Table 9: Top-3 question types per domain (dev set). Coverage indicates the percentage of turns with one of the three most frequent types. FiQA’s lower coverage reflects its more diverse question distribution.

Figure 2: Question-type distribution per domain (dev set, % of domain turns). FiQA’s Opinion-heavy profile and Cloud’s even distribution across types stand in contrast to the Factoid/Summarization-dominated ClapNQ and Govt corpora.

Figure 3: Reference answer length distribution by question type (dev set). Red diamonds indicate means; Summarization answers are significantly longer, motivating type-specific generation targets.

Figure 4: Non-standalone phenomena in dev set non-first turns (

N=732

, multi-label). Pronoun coreference and implicit topic carryover are the most frequent sources of query underspecification, motivating history-aware query rewriting.

Type	User Query	Challenge
Follow-up (coreference)	“How much does it cost?” refers to IBM Cloud Object Storage, 3 turns prior	Non-standalone query; retriever sees no entity without rewriting
Unanswerable (temporal)	“Where do the Cardinals play this week?” corpus contains only 2017 schedule	Topically relevant but temporally mismatched; high hallucination risk
Partial (incomplete)	“Is it worth having a web chat widget?” docs describe features, not subjective value	Evidence supports description but not the evaluative judgment

Appendix

Table 10: Representative MTRAG challenge examples.

Characteristic	Dev	Test
Conversations	110	507
Total turns	842	507
Avg turns/conv	7.7	1.0
Answerable (%)	84.2	56.2
Partial (%)	8.1	9.3
Unanswerable (%)	6.5	19.1

Appendix

Table 11: Development vs. test set comparison. Notable distributional shifts are shown in bold . The test set is substantially harder due to a 3

\times

higher unanswerable rate and single-turn evaluation structure.

Figure 5: Turn position distribution for evaluated turns in the development (blue) and test (orange) sets. The dev set spans positions 1–10+, while all 507 test turns are non-first turns—eliminating the easy first-turn queries that inflate dev-set averages and ensuring every test turn requires contextual understanding.

Figure 6: Answerability distribution: development vs. test set. The test set contains nearly 3

\times

more unanswerable turns and 28 percentage points fewer answerable turns, creating a substantially harder classification setting.

Domain	Ans	Partial	Unans	Conv
ClapNQ	46%	13%	15%	0%
FiQA	66%	9%	16%	0%
Govt	56%	11%	17%	0%
Cloud	62%	4%	27%	0%
Overall	56%	9%	19%	0%

Appendix

Table 12: Answerability distribution per domain (test set, 507 turns). All domains show substantially higher unanswerable rates than in development, with Cloud exhibiting the most severe shift.

Figure 7: Answerability distribution per domain (% of domain turns), development set (left) vs. test set (right). Red borders highlight the Unanswerable column. Cloud exhibits the most severe shift: unanswerable turns rise from 7% (dev) to 27% (test), the highest unanswerable rate across all domains.

Task	Metric	Ours	Rank	Top Score	Top Baseline	Baseline Score
A — Retrieval	nDCG@5	0.5776	1/38	0.5776	ELSER + GPT-OSS-20b rewrite	0.4795
B — Generation	HM	0.7698	2/26	0.7827	GPT-OSS-120b	0.6390
	RB agg / RL F / RB llm	0.633 / 0.897 / 0.832	–	–	–	–
C — RAG	HM	0.5409	11/29	0.5861	Qwen-30B-A3B-Thinking	0.5366
	RB agg / RL F / RB llm	0.400 / 0.729 / 0.598	–	–	–	–

Appendix

Table 13: Official test set results. HM = harmonic mean of RB agg , RL F , RB llm . Top Baseline refers to the strongest non-participant system released by the organizers. Task C rank reflects the answerability bottleneck identified .

Domain	Baseline	Final	$\Delta$	Top Strategy
ClapNQ	0.528	0.690	$+$ 16.2	Minimal
FiQA	0.419	0.527	$+$ 10.8	Corpus-Spec
Govt	0.507	0.629	$+$ 12.2	Minimal
Cloud	0.477	0.541	$+$ 6.4	Minimal
Overall	0.483	0.597	$+$ 12.4	–

Appendix

Table 25: Per-domain R@5 on the dev set: unaugmented ELSER baseline vs. final system.

\Delta

is the absolute R@5 gain. FiQA shows the smallest gain despite benefiting most from Corpus-Specific rewriting, reflecting its structurally harder retrieval problem.

Figure 8: R@5 vs. turn index: ELSER baseline (dashed) vs. final system (solid). Rewriting reduces the later-turn drop from 51% to 39%.

Figure 9: SA vs. Non-SA R@5, baseline vs. final system, per domain.

Strategy	Asst.	Key Behavior
Minimal	3	Coreference only
Corpus-Spec.	0–3	Domain-aware (Tab. 29 )
CoT	3	Reasoning trace
HyDE	3	Hypothetical passage
Anchor-KW	1	Entity + intent extraction

Appendix

Table 28: Rewriting strategy configurations.

Domain	Asst.	Preservation Targets
ClapNQ	3	Entity variants, Wikipedia terminology, temporal phrasing
FiQA	0	Amounts, currencies, tickers, time horizons
Govt	3	Program names, agency acronyms, form IDs, locations
Cloud	3	Error codes, CLI commands/flags, config keys, service names

Appendix

Table 29: Domain-specific preservation rules for Corpus-Specific rewriting. All four variants share the same template; only the preservation targets differ.

Figure 10: Extractiveness shaping function

\phi(r_{4})

. Dashed vertical lines mark the ideal band

[0.28,0.38]

; the grey region

[0.38,0.50]

is acceptable. The asymmetric design penalizes under-extractive (hallucination-prone) responses more strongly than over-extractive ones.

Phrase pattern	Fire rate	Turn type
“I’m / I am not sure”	34%	Partial
“It’s / It is unclear”	27%	Partial / Unans
“I don’t / do not know”	18%	Unanswerable
“I cannot / Unable to answer”	13%	Unanswerable
“I’m uncertain / cannot say”	8%	Mixed

Appendix

Table 31: Forbidden-phrase fire rates (dev set, 842 turns). Hedging dominates on partial-answerability turns, motivating the

-2.0

penalty in Eq. 6 and the force-answer fallback.

Type	Target	$\Delta$	Style hint
Keyword	90w	$0$	Interpret and answer
How-to	95w	$+5$	Step-by-step
Explanation	100w	$+10$	Clear explanation
Comparative	95w	$+5$	Compare systematically
Summarization	105w	$+15$	Comprehensive summary
Factoid	85w	$-5$	Direct answer

Appendix

Table 32: Question-type rules, style hints, and generation length targets. Rules are evaluated top-to-bottom; the first match wins.

Configuration	HM	RL ${}_{\text{F}}$	RB ${}_{\text{alg}}$	Rel. Cost
Uniform GPT-4o	0.748	0.767	0.730	$3.0\times$
Uniform DeepSeek-V3	0.721	0.738	0.705	$0.4\times$
Uniform GPT-4o-mini	0.731	0.748	0.715	$0.3\times$
Routed (ours)	0.743	0.761	0.726	$1.0\times$

Appendix

Table 33: Uniform vs. routed model assignment (dev set). Routing GPT-4o to generation only matches uniform GPT-4o quality at one-third the cost. Relative cost normalized to our routed configuration.

Passages ( $N$ )	HM	RL ${}_{\text{F}}$	RB ${}_{\text{alg}}$
1	0.701	0.719	0.683
2	0.718	0.736	0.701
3	0.733	0.750	0.717
5 (ours)	0.743	0.761	0.726
7	0.738	0.756	0.721
10	0.729	0.747	0.712

Appendix

Table 34: Effect of passage count on generation quality (Task B, dev set). Performance peaks at

N{=}5

; additional passages introduce noise that degrades faithfulness.

Ideal band	HM	$\Delta$
No shaping ( $\phi{=}0$ uniform)	0.724	$-0.019$
$[0.20,0.50]$ (loose)	0.731	$-0.012$
$[0.25,0.45]$	0.738	$-0.005$
$[0.28,0.38]$ (ours)	0.743	–
$[0.30,0.40]$ (tight)	0.737	$-0.006$
$[0.35,0.50]$ (high)	0.733	$-0.010$

Appendix

Table 35: Extractiveness band sensitivity (dev set). Band

[0.28,0.38]

balances faithfulness and naturalness; deviations in either direction degrade HM.

\Delta

is relative to our configuration.

Figure 11:

\text{RL}_{\text{F}}

vs.

\text{RB}_{\text{llm}}

scatter for dev set responses binned by 4-gram extractiveness. Responses in the

[0.28,0.38]

band (blue) cluster near the Pareto frontier; under-extractive (

r_{4}{<}0.28

, red) and over-extractive (

r_{4}{>}0.50

, grey) responses degrade at least one metric.

Judge rate	HM	RB ${}_{\text{llm}}$	Rel. cost
0%	0.725	0.708	$1.0\times$
30%	0.731	0.715	$1.1\times$
60% (ours)	0.738	0.722	$1.2\times$
80%	0.739	0.723	$1.3\times$
100%	0.740	0.724	$1.4\times$

Appendix

Table 36: User-satisfaction judge invocation rate sweep (dev set). HM plateaus after 60%; the marginal gain of

+

0.002 from 60%

\to

100% does not justify the

+

0.2

\times

cost increase.

Figure 12: Representative error cascade in Task C (Cloud domain). A retrieval miss at turn 3 (

\times

) corrupts evidence for turns 4–5, causing generation errors ( ! ) despite correct retrieval at those turns. Turn 6 recovers independently.

Root cause	% of failures	Primary domain
Retrieval error (current turn)	38%	FiQA
Retrieval error (previous turn(s))	34%	Cloud
Answerability misclassification	18%	All
Generation error (correct passages)	10%	ClapNQ

Appendix

Table 37: Root cause analysis of 50 Task C failures (dev set). Cross-turn error propagation (34%) is nearly as frequent as within-turn retrieval errors (38%), highlighting the compounding challenge of multi-turn RAG.

Threshold	HM	ANS R	UNANS F1
0.50	0.573	0.892	0.261
0.60	0.591	0.923	0.251
0.70 (ours)	0.610	0.958	0.240
0.80	0.597	0.971	0.198
0.90	0.578	0.986	0.162

Appendix

Table 38: Confidence threshold sweep for the answerability classifier (Task C dev set). Threshold 0.70 maximises HM by balancing ANS recall and UNANS F1; lower values over-refuse (higher UNANS F1, lower ANS R), while higher values accept weak evidence (higher ANS R, lower UNANS F1).

Configuration	HM	RL ${}_{\text{F}}$	RB ${}_{\text{alg}}$
Always answer (no gate)	0.518	0.798	0.312
Single-judge gate, top-5 passages	0.561	0.821	0.356
Multi-judge gate, top-5 passages	0.586	0.836	0.383
Multi-judge gate, top-3 (ours)	0.610	0.848	0.408
Multi-judge gate, top-1	0.591	0.841	0.389

Appendix

Table 39: Task C ablation (dev set). Reducing retrieved passages from 5 to 3 yields the largest single gain (

+

0.024 HM) by reducing generation noise; the multi-judge gate consistently outperforms single-judge across all metrics. Top-1 underperforms top-3 due to insufficient evidence coverage.

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 792 papers

5ting at SemEval-2026 Task 8: Strong End-to-End Multi-Turn RAG via LLM-Based Reranking and Faithfulness Control
Jun 27, 2026Thien-Qua-T-Nguyen, Chi Hoang, Nguyen Tran +3RAG Citation Faithfulness Retrieval-Augmented Generation
University of Information Technology, Ho Chi Minh City, Vietnam · 2Vietnam National University Ho Chi Minh City, Ho Chi Minh City, Vietnam
Sifei at SemEval-2026 Task 8: Hybrid Retrieval and Query Rewriting for Multi-Turn RAG
Jun 5, 2026Sifei Meng, Dmitry IlvovskyCross-Encoder Reranking Retrieval-Augmented Generation
Faculty of Computer Science HSE University
H-RAG at SemEval-2026 Task 8: Hierarchical Parent-Child Retrieval for Multi-Turn RAG Conversations
May 1, 2026Passant Elchafei, Hossam Emam, Mohamed Alansary +2Retrieval-Augmented Generation Information Retrieval
1Johannes Kepler University Linz, Institute of Computational Perception, Linz, Austria · 3Independent Researcher, Egypt · 2Linz Institute of Technology, Artificial Intelligence Lab, Austria

Jun 27, 2026J/K move · Enter open · S save

5ting at SemEval-2026 Task 8: Strong End-to-End Multi-Turn RAG via LLM-Based Reranking and Faithfulness Control

Thien-Qua-T-Nguyen, Chi Hoang, Nguyen Tran, Tri Le+2

University of Information Technology, Ho Chi Minh City, Vietnam · 2Vietnam National University Ho Chi Minh City, Ho Chi Minh City, Vietnam

RAG Citation Faithfulness Retrieval-Augmented Generation Multi-Turn Dialogue Evaluation RAG Evaluation

Open paper AskPDF arXiv

We introduce 5ting, our system for the SemEval2026 Task 8 (MTRAGEval), which evaluates multi-turn Retrieval Augmented Generation (RAG) systems. Multi turn RAG involves context drift, under specification, and hallucination risk. Our system combines BGE-M3 dense retrieval with FAISS indexing, dual-query merged retrieval, and LLM based reranking, followed by role separated generation constrained to retrieved evidence. The retriever achieved nDCG@5 = 0.4719 in Task A, while the end to end system ranked in Task C with a harmonic score of 0.5597 and RL_F = 0.7692.

Abstract

Retrieval-Augmented Generation

Explore similar work

5ting at SemEval-2026 Task 8: Strong End-to-End Multi-Turn RAG via LLM-Based Reranking and Faithfulness Control

Sifei at SemEval-2026 Task 8: Hybrid Retrieval and Query Rewriting for Multi-Turn RAG

H-RAG at SemEval-2026 Task 8: Hierarchical Parent-Child Retrieval for Multi-Turn RAG Conversations