We present the AILS-NTUA system for SemEval-2026 Task 8 (MTRAGEval), addressing all three subtasks of multi-turn retrieval-augmented generation: passage retrieval (A), reference-grounded response generation (B), and end-to-end RAG (C). Our unified architecture is built on two principles: (i) a query-diversity-over-retriever-diversity strategy, where five complementary LLM-based query reformulations are issued to a single corpus-aligned sparse retriever and fused via variance-aware nested Reciprocal Rank Fusion; and (ii) a multistage generation pipeline that decomposes grounded generation into evidence span extraction, dual-candidate drafting, and calibrated multi-judge selection. Our system ranks 1st in Task A (nDCG@5: 0.5776, +20.5% over the strongest baseline) and 2nd in Task B (HM: 0.7698). Empirical analysis shows that query diversity over a well-aligned retriever outperforms heterogeneous retriever ensembling, and that answerability calibration-rather than retrieval coverage-is the primary bottleneck in end-to-end performance.
Figures & tables
Figure 1: System architecture: (a) Task A retrieval pipeline, (b) Task B generation pipeline, (c) Task C end-to-end RAG with answerability gate.
Characteristic
ClapNQ
FiQA
Govt
Cloud
Documents
4,293
57,638 †
7,661
8,578
Passages
183,408
61,022
49,607
72,442
Avg. passages/doc
42.7
1.1
6.5
8.4
Domain
Wikipedia
Finance
Government
Tech docs
Table 1: MTRAG corpus statistics. † FiQA contains individual forum posts.
Table 7: Corpus statistics. FiQA “Docs” correspond to individual forum posts (atomic documents).
Domain
Ans
Partial
Unans
Conv
ClapNQ
86%
7%
7%
0%
FiQA
82%
9%
7%
3%
Govt
83%
11%
6%
0%
Cloud
86%
5%
7%
1%
Overall
84%
8%
7%
1%
Appendix
Table 8: Answerability distribution per domain (dev set, 842 turns). Percentages are row-normalized.
Domain
Top-3 Question Types
Coverage
ClapNQ
Factoid, Summarization, Explanation
71%
FiQA
Opinion, Factoid, Explanation
51%
Govt
Factoid, Summarization, Explanation
63%
Cloud
Factoid, Explanation, Summarization
50%
Appendix
Table 9: Top-3 question types per domain (dev set). Coverage indicates the percentage of turns with one of the three most frequent types. FiQA’s lower coverage reflects its more diverse question distribution.
Figure 2: Question-type distribution per domain (dev set, % of domain turns). FiQA’s Opinion-heavy profile and Cloud’s even distribution across types stand in contrast to the Factoid/Summarization-dominated ClapNQ and Govt corpora.
Figure 3: Reference answer length distribution by question type (dev set). Red diamonds indicate means; Summarization answers are significantly longer, motivating type-specific generation targets.
Figure 4: Non-standalone phenomena in dev set non-first turns ( N=732 , multi-label). Pronoun coreference and implicit topic carryover are the most frequent sources of query underspecification, motivating history-aware query rewriting.
Type
User Query
Challenge
Follow-up (coreference)
“How much does it cost?” refers to IBM Cloud Object Storage, 3 turns prior
Non-standalone query; retriever sees no entity without rewriting
Unanswerable (temporal)
“Where do the Cardinals play this week?” corpus contains only 2017 schedule
Topically relevant but temporally mismatched; high hallucination risk
Partial (incomplete)
“Is it worth having a web chat widget?” docs describe features, not subjective value
Evidence supports description but not the evaluative judgment
Table 11: Development vs. test set comparison. Notable distributional shifts are shown in bold . The test set is substantially harder due to a 3 × higher unanswerable rate and single-turn evaluation structure.
Figure 5: Turn position distribution for evaluated turns in the development (blue) and test (orange) sets. The dev set spans positions 1–10+, while all 507 test turns are non-first turns—eliminating the easy first-turn queries that inflate dev-set averages and ensuring every test turn requires contextual understanding.
Figure 6: Answerability distribution: development vs. test set. The test set contains nearly 3 × more unanswerable turns and 28 percentage points fewer answerable turns, creating a substantially harder classification setting.
Domain
Ans
Partial
Unans
Conv
ClapNQ
46%
13%
15%
0%
FiQA
66%
9%
16%
0%
Govt
56%
11%
17%
0%
Cloud
62%
4%
27%
0%
Overall
56%
9%
19%
0%
Appendix
Table 12: Answerability distribution per domain (test set, 507 turns). All domains show substantially higher unanswerable rates than in development, with Cloud exhibiting the most severe shift.
Figure 7: Answerability distribution per domain (% of domain turns), development set (left) vs. test set (right). Red borders highlight the Unanswerable column. Cloud exhibits the most severe shift: unanswerable turns rise from 7% (dev) to 27% (test), the highest unanswerable rate across all domains.
Task
Metric
Ours
Rank
Top Score
Top Baseline
Baseline Score
A — Retrieval
nDCG@5
0.5776
1/38
0.5776
ELSER + GPT-OSS-20b rewrite
0.4795
B — Generation
HM
0.7698
2/26
0.7827
GPT-OSS-120b
0.6390
RB agg / RL F / RB llm
0.633 / 0.897 / 0.832
–
–
–
–
C — RAG
HM
0.5409
11/29
0.5861
Qwen-30B-A3B-Thinking
0.5366
RB agg / RL F / RB llm
0.400 / 0.729 / 0.598
–
–
–
–
Appendix
Table 13: Official test set results. HM = harmonic mean of RB agg , RL F , RB llm . Top Baseline refers to the strongest non-participant system released by the organizers. Task C rank reflects the answerability bottleneck identified .
Domain
Baseline
Final
Δ
Top Strategy
ClapNQ
0.528
0.690
+ 16.2
Minimal
FiQA
0.419
0.527
+ 10.8
Corpus-Spec
Govt
0.507
0.629
+ 12.2
Minimal
Cloud
0.477
0.541
+ 6.4
Minimal
Overall
0.483
0.597
+ 12.4
–
Appendix
Table 25: Per-domain R@5 on the dev set: unaugmented ELSER baseline vs. final system. Δ is the absolute R@5 gain. FiQA shows the smallest gain despite benefiting most from Corpus-Specific rewriting, reflecting its structurally harder retrieval problem.
Figure 8: R@5 vs. turn index: ELSER baseline (dashed) vs. final system (solid). Rewriting reduces the later-turn drop from 51% to 39%.
Figure 9: SA vs. Non-SA R@5, baseline vs. final system, per domain.
Strategy
Asst.
Key Behavior
Minimal
3
Coreference only
Corpus-Spec.
0–3
Domain-aware (Tab. 29 )
CoT
3
Reasoning trace
HyDE
3
Hypothetical passage
Anchor-KW
1
Entity + intent extraction
Appendix
Table 28: Rewriting strategy configurations.
Domain
Asst.
Preservation Targets
ClapNQ
3
Entity variants, Wikipedia terminology, temporal phrasing
FiQA
0
Amounts, currencies, tickers, time horizons
Govt
3
Program names, agency acronyms, form IDs, locations
Cloud
3
Error codes, CLI commands/flags, config keys, service names
Appendix
Table 29: Domain-specific preservation rules for Corpus-Specific rewriting. All four variants share the same template; only the preservation targets differ.
Figure 10: Extractiveness shaping function ϕ(r4) . Dashed vertical lines mark the ideal band [0.28,0.38] ; the grey region [0.38,0.50] is acceptable. The asymmetric design penalizes under-extractive (hallucination-prone) responses more strongly than over-extractive ones.
Phrase pattern
Fire rate
Turn type
“I’m / I am not sure”
34%
Partial
“It’s / It is unclear”
27%
Partial / Unans
“I don’t / do not know”
18%
Unanswerable
“I cannot / Unable to answer”
13%
Unanswerable
“I’m uncertain / cannot say”
8%
Mixed
Appendix
Table 31: Forbidden-phrase fire rates (dev set, 842 turns). Hedging dominates on partial-answerability turns, motivating the −2.0 penalty in Eq. 6 and the force-answer fallback.
Type
Target
Δ
Style hint
Keyword
90w
0
Interpret and answer
How-to
95w
+5
Step-by-step
Explanation
100w
+10
Clear explanation
Comparative
95w
+5
Compare systematically
Summarization
105w
+15
Comprehensive summary
Factoid
85w
−5
Direct answer
Appendix
Table 32: Question-type rules, style hints, and generation length targets. Rules are evaluated top-to-bottom; the first match wins.
Configuration
HM
RL F
RB alg
Rel. Cost
Uniform GPT-4o
0.748
0.767
0.730
3.0×
Uniform DeepSeek-V3
0.721
0.738
0.705
0.4×
Uniform GPT-4o-mini
0.731
0.748
0.715
0.3×
Routed (ours)
0.743
0.761
0.726
1.0×
Appendix
Table 33: Uniform vs. routed model assignment (dev set). Routing GPT-4o to generation only matches uniform GPT-4o quality at one-third the cost. Relative cost normalized to our routed configuration.
Passages ( N )
HM
RL F
RB alg
1
0.701
0.719
0.683
2
0.718
0.736
0.701
3
0.733
0.750
0.717
5 (ours)
0.743
0.761
0.726
7
0.738
0.756
0.721
10
0.729
0.747
0.712
Appendix
Table 34: Effect of passage count on generation quality (Task B, dev set). Performance peaks at N=5 ; additional passages introduce noise that degrades faithfulness.
Ideal band
HM
Δ
No shaping ( ϕ=0 uniform)
0.724
−0.019
[0.20,0.50] (loose)
0.731
−0.012
[0.25,0.45]
0.738
−0.005
[0.28,0.38] (ours)
0.743
–
[0.30,0.40] (tight)
0.737
−0.006
[0.35,0.50] (high)
0.733
−0.010
Appendix
Table 35: Extractiveness band sensitivity (dev set). Band [0.28,0.38] balances faithfulness and naturalness; deviations in either direction degrade HM. Δ is relative to our configuration.
Figure 11: RLF vs. RBllm scatter for dev set responses binned by 4-gram extractiveness. Responses in the [0.28,0.38] band (blue) cluster near the Pareto frontier; under-extractive ( r4<0.28 , red) and over-extractive ( r4>0.50 , grey) responses degrade at least one metric.
Judge rate
HM
RB llm
Rel. cost
0%
0.725
0.708
1.0×
30%
0.731
0.715
1.1×
60% (ours)
0.738
0.722
1.2×
80%
0.739
0.723
1.3×
100%
0.740
0.724
1.4×
Appendix
Table 36: User-satisfaction judge invocation rate sweep (dev set). HM plateaus after 60%; the marginal gain of + 0.002 from 60% → 100% does not justify the + 0.2 × cost increase.
Figure 12: Representative error cascade in Task C (Cloud domain). A retrieval miss at turn 3 ( × ) corrupts evidence for turns 4–5, causing generation errors ( ! ) despite correct retrieval at those turns. Turn 6 recovers independently.
Root cause
% of failures
Primary domain
Retrieval error (current turn)
38%
FiQA
Retrieval error (previous turn(s))
34%
Cloud
Answerability misclassification
18%
All
Generation error (correct passages)
10%
ClapNQ
Appendix
Table 37: Root cause analysis of 50 Task C failures (dev set). Cross-turn error propagation (34%) is nearly as frequent as within-turn retrieval errors (38%), highlighting the compounding challenge of multi-turn RAG.
Threshold
HM
ANS R
UNANS F1
0.50
0.573
0.892
0.261
0.60
0.591
0.923
0.251
0.70 (ours)
0.610
0.958
0.240
0.80
0.597
0.971
0.198
0.90
0.578
0.986
0.162
Appendix
Table 38: Confidence threshold sweep for the answerability classifier (Task C dev set). Threshold 0.70 maximises HM by balancing ANS recall and UNANS F1; lower values over-refuse (higher UNANS F1, lower ANS R), while higher values accept weak evidence (higher ANS R, lower UNANS F1).
Configuration
HM
RL F
RB alg
Always answer (no gate)
0.518
0.798
0.312
Single-judge gate, top-5 passages
0.561
0.821
0.356
Multi-judge gate, top-5 passages
0.586
0.836
0.383
Multi-judge gate, top-3 (ours)
0.610
0.848
0.408
Multi-judge gate, top-1
0.591
0.841
0.389
Appendix
Table 39: Task C ablation (dev set). Reducing retrieved passages from 5 to 3 yields the largest single gain ( + 0.024 HM) by reducing generation noise; the multi-judge gate consistently outperforms single-judge across all metrics. Top-1 underperforms top-3 due to insufficient evidence coverage.
We introduce 5ting, our system for the SemEval2026 Task 8 (MTRAGEval), which evaluates multi-turn Retrieval Augmented Generation (RAG) systems. Multi turn RAG involves context drift, under specification, and hallucination risk. Our system combines BGE-M3 dense retrieval with FAISS indexing, dual-query merged retrieval, and LLM based reranking, followed by role separated generation constrained to retrieved evidence. The retriever achieved nDCG@5 = 0.4719 in Task A, while the end to end system ranked in Task C with a harmonic score of 0.5597 and RL_F = 0.7692.
Thien-Qua-T-Nguyen, Chi Hoang, Nguyen Tran +3
University of Information Technology, Ho Chi Minh City, Vietnam · 2Vietnam National University Ho Chi Minh City, Ho Chi Minh City, Vietnam
Multi-turn retrieval-augmented generation (RAG) is challenging due to evolving user intent, conversational noise, and strict context limits. We propose a training-free hybrid retrieval pipeline for SemEval-2026 Task 8 that combines dense and sparse retrieval with controlled query rewriting and cross-encoder reranking. On the official test set of Task A, our system achieves 0.5453 nDCG@5, ranking third among 38 teams and outperforming the strongest baseline score of 0.4795. For Task C, we reuse the documents retrieved for Task A and apply a lightweight generation pipeline guided by the official prompt, achieving 0.5312 as the harmonic mean of relevance and faithfulness and ranking 15th among 29 teams. All retrieval components are open-source, while query rewriting and answer generation rely on LLM APIs.
We present H-RAG, our submission to SemEval-2026 Task 8 (MTRAGEval), addressing both Task A (Retrieval) and Task C (Generation with Retrieved Passages). Task A evaluates standalone retrieval quality, while Task C assesses end-to-end retrieval-augmented generation (RAG) in multi-turn conversational settings, requiring both accurate answer generation and faithful grounding in retrieved evidence. Our approach implements a hierarchical parent-child RAG pipeline that separates fine-grained child-level retrieval from parent-level context reconstruction during generation. Documents are segmented into overlapping sentence-based child chunks, while full documents are preserved as parent units to provide coherent context. Retrieval combines hybrid dense-sparse search, tunable weighting, and embedding-based similarity rescoring over child chunks. Retrieved evidence is aggregated at the parent level and supplied to an instruction-tuned language model for response generation. H-RAG achieves an nDCG@5 score of 0.4271 on Task A and a harmonic mean score of 0.3241 on Task C (RB_agg: 0.2488, RL_F: 0.2703, RB_llm: 0.6508), underscoring the importance of retrieval configuration and parent-level aggregation in multi-turn RAG performance.
1Johannes Kepler University Linz, Institute of Computational Perception, Linz, Austria · 3Independent Researcher, Egypt · 2Linz Institute of Technology, Artificial Intelligence Lab, Austria