We describe the RMIT-ADM+S team's participation in the TREC 2026 Retrieval-Augmented Generation (RAG) track using the ClimbMix corpus. We submitted seven research-agent runs using proprietary OpenAI and open-weight models, exploring different retrieval strategies, writer models, and agent workflows. Pre-submission ablations identified writer selection and adjacent-chunk expansion as the main factors affecting answer quality. In the official NIST evaluation, six of seven runs exceeded the track median across all three rubric/nugget collections. Our best-performing run, brief-hybrid, achieved the highest all-item coverage among our submissions and tied the track maximum (0.892) for Batch-1 vital coverage. Human side-by-side preferences differed across evaluation batches: the open-weight owa-glm5 led Batch 1 despite lower automated coverage, while sol-research-first led Batch 2. These findings highlight differences between automated coverage measures and human preferences, as well as the potential of lower-cost open-weight models for research-agent systems.
Figures & tables
Figure 1. Control flows for three submissions: our leading coverage run ( brief-hybrid ) and two SxS priority runs ( owa-glm5 , sol-research-first ). Purple blocks are expanded in the panel below. Solid arrows show execution order, not prompt inputs; each scout sees only the request. Unboxed notes mark absent stages; k is the typical requested passages per search. Dashed blue arrows are conditional: review feedback resumes the writer loop, which may search again; the revised draft bypasses the once-only review. For staged evidence see Section 3.1 . Three separate vertical pipelines for brief-hybrid, owa-glm5, and sol-research-first. Brief-hybrid uses GPT-5.6-Sol in all roles, with brief and review stages, hybrid RRF retrieval at k=10, and one review pass. Owa-glm5 uses GLM-5 across all stages and adds a blind obligation scout whose items are excluded from the review hook. Sol-research-first uses GPT-5.6-Sol in all roles with a prose coverage plan, blind scout, semantic plus keyword search at k=10, and a final answer from the same 40-round research loop. No blind scout for brief-hybrid and no post-draft review for sol-research-first are unboxed absence notes, not processing stages. Solid arrows show execution order; scouts receive only the request, not the preceding stage's output. Brief, scout, and reviewer stages are separate LLM calls; uncited scans and export are code. Conditional arrows return review feedback to the writer, which may search again, and send revised drafts directly to export without another review. Purple fill links the three research-and-draft blocks to a lower panel expanding their shared staged-evidence loop with adjacent-chunk expansion, commit_context retention, 500k context limit, and code-enforced citation and word limits.
System
Models (main / support)
Retrieval ( k )
Planning / Review
Adj.
sol-default
Sol / —
Sem+Kw (10)
None
No
sol-research-first ⋆
Sol / Sol
Sem+Kw (10)
Plan+Scout
Yes
brief-base
Sol / Sol
Sem+Kw (10)
Brief+Rev
Yes
brief-hybrid
Sol / Sol
Hyb (10)
Brief+Rev
Yes
facets
Luna / —
Sem+Kw (10), Hyb (15)
Facets+coverage check
No
facetrag
GPT-OSS-120B / Qwen3-Next-80B
Sem+Kw+SSR+Bool (10), Hyb (15)
Facets+multi-stage review
—
Table 1. Configurations of the submitted systems. Runs marked ⋆ entered NIST human SxS assessment. Models denote the main and supporting LLMs (orchestrator and analyzer for facetrag ). Sem: semantic; Kw: keyword; Hyb: hybrid RRF; SSR: shortest-substring ranking; Bool: Boolean search; Brief: requirements brief; Scout: blind obligation scout; Rev: post-draft review; Adj.: automatic adjacent-chunk expansion. Retrieval k denotes the most frequently requested number of passages per search call.
System
Srch
Words
Refs
In (K)
Cache (K)
Out (K)
$/topic
sol-default
23.9
882.5
27.9
182.2
202.4
6.1
1.566
sol-research-first ⋆
31.4
936.6
27.7
330.5
369.3
10.5
2.823
brief-base
26.4
830.2
25.8
319.6
388.5
8.0
2.356
brief-hybrid
22.6
834.3
24.7
293.1
322.4
7.5
2.156
facets
11.7
613.0
13.6
138.9
71.4
5.9
0.040
facetrag
33.7
388.9
13.9
411.7
0
49.7
0.106
Table 2. Average resource usage and estimated cost per report across all 119 test topics. Srch: search calls; Refs: cited references; In: uncached input tokens; Cache: cached input tokens; Out: output tokens. Token counts are in thousands (K). Runs marked ⋆ entered NIST human SxS assessment.
System
W
L
T
Win rate
sol-research-first
108
10
1
91.2%
sol-default
107
11
1
90.3%
brief-hybrid
101
18
0
84.9%
brief-base
99
19
1
83.6%
facets
13
105
1
11.3%
facetrag
0
119
0
0.0%
Table 3. Internal both-order RAGDoll arena comparisons against the organizer’s shipped agentic BM25 baseline over all 119 narratives (GPT-5.6-Luna judge, seed 13; not official TREC judgments). Win rate is (W+0.5T)/119 . owa-glm5 was finalized after this sweep.
B1 Human Rubric ( n=17 )
B2 Manual Nuggets ( n=25 )
B2 Auto Nuggets ( n=25 )
Type
System
Vital / All
≥ Med (V/A)
Vital / All
≥ Med (V/A)
Vital / All
≥ Med (V/A)
Proprietary
brief-hybrid
0.892 ⋆ / 0.909
17 / 17
0.691 / 0.723
24 / 25
0.644 / 0.565
25 / 25
sol-research-first
0.892 ⋆ / 0.887
17 / 17
0.684 / 0.702
25 / 24
0.649 / 0.560
25 / 25
facets
0.831 / 0.841
17 / 16
0.627 / 0.642
23 / 23
0.390 / 0.333
25 / 24
brief-base
0.824 / 0.862
17 / 16
0.691 / 0.721
24 / 25
0.648 / 0.555
25 / 25
sol-default
0.790 / 0.861
17 / 17
0.679 / 0.705
25 / 25
0.630 / 0.563
25 / 25
Table 4. Official NIST TREC 2026 RAG coverage ( vital_strict / all_strict ) and numbers of narratives scoring at or above the global track median ( ≥ Med, V/A) across our seven submitted runs. Underlining marks our highest run per metric; ⋆ denotes a tie with the global track maximum before rounding. Two B2 manual narratives ( rag2026-89 , rag2026-114 ) contain no vital nuggets, capping vital_strict at 23/25=0.920 .
Figure 2. Official human side-by-side Bradley–Terry ratings with published 95% confidence intervals and track medians and maxima. Batch 1 ( n=17 ) and Batch 2 ( n=25 ) were fitted separately and used different topic sets and assessment protocols. Two stacked panels showing Bradley–Terry ratings for owa-glm5 and sol-research-first. Owa-glm5 leads in Batch 1, while sol-research-first leads in Batch 2.
Figure 3. Per-topic official all_strict coverage difference ( Δ ) relative to the TREC 2026 RAG track median across Batch 1 ( n=17 , left) and Batch 2 manual nuggets ( n=25 , right), sorted within each panel by brief-hybrid Δ . Shaded bars show the difference between the track maximum and median. Two-panel per-topic plot showing substantial variation in coverage relative to the track median, including the coverage– preference divergence on rag2026-68 and the owa-glm5 failure on rag2026-65.
Measure
owa-glm5
sol-research-first
Words
805.3
951.2
Unique references
16.6
28.9
Flesch–Kincaid Grade Level ↓
15.7
22.2
Topics with shorter report
15/17
2/17
Table 5. Report characteristics on the 17 matched Batch-1 topics (means, except topic counts).
Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.
Deep Research agents are rapidly emerging as primary consumers of modern retrieval systems. Unlike human users who issue and refine queries without documenting their intermediate thought processes, Deep Research agents generate explicit natural language reasoning before each search call, revealing rich intent and contextual information that existing retrievers entirely ignore. To exploit this overlooked signal, we introduce: (1) Reasoning-Aware Retrieval, a retrieval paradigm that jointly embeds the agent's reasoning trace alongside its query; and (2) DR-Synth, a data synthesis method that generates Deep Research retriever training data from standard QA datasets. We demonstrate that both components are independently effective, and their combination yields a trained embedding model, AgentIR-4B, with substantial gains. On the challenging BrowseComp-Plus benchmark, AgentIR-4B achieves 68% accuracy with the open-weight agent Tongyi-DeepResearch, compared to 50% with conventional embedding models twice its size, and 37% with BM25. Code and data are available at: https://texttron.github.io/AgentIR/.
Zijian Chen, Xueguang Ma, Shengyao Zhuang +3
University of Waterloo · University of Queensland · Carnegie Mellon University
Retrieval-Augmented Generation (RAG) systems rely critically on the retriever module to surface relevant context for large language models. Although numerous retrievers have recently been proposed, each built on different ranking principles such as lexical matching, dense embeddings, or graph citations, there remains a lack of systematic understanding of how these mechanisms differ and overlap. Existing benchmarks primarily compare entire RAG pipelines or rely on aggregate ranking metrics (Recall, MRR, nDCG) that assume item-level independence, providing little guidance on redundancy or complementarity between retrievers. This work presents MIGRASCOPE, a Mutual Information based RAG Retriever Analysis Scope. We introduce principled metrics grounded in information theory to quantify retrieval quality, redundancy/synergy, and marginal contribution. Although the framework is designed to be applicable to any setting where a ground-truth answer can construct a pseudo chunk distribution, the present empirical study focuses on multi-hop question answering - a demanding setting where retriever complementarity is especially pronounced. On four corpora and thirteen SOTA retrievers spanning lexical, dense, and graph-based families, we show that (i) the MI-based divergence metric captures retriever quality dimensions not reflected by Recall or MRR, (ii) graph-based retrievers are redundant with each other while BM25 remains informationally complementary, and (iii) in the multi-hop QA setting, a carefully chosen ensemble guided by MI-based attribution can outperform any single retriever. We position these findings as task-specific insights and provide a reusable framework for retriever analysis that practitioners can re-estimate on their own target domains.
Retrieval-Augmented Generation (RAG) has become the dominant paradigm for grounding the outputs of large language models (LLM) in external knowledge. However, practitioners are faced with a vast configuration space comprising chunking, embedding, retrieval, reranking, prompting and generation models, and there is limited guidance on how these components interact. We present RAG-NIMBF, a modular benchmarking framework that attaches to existing RAG pipelines as a non-invasive evaluation layer. It evaluates the configuration space through controlled ablation with end-to-end experiment tracking. The evaluation combines LLM-based answer quality scores, retrieval correctness measures, and runtime indicators. This study validates the framework across multiple generation models on the SQuAD dataset. This process reveals non-obvious interactions and demonstrates how the experiments are tracked. The findings indicate that the retrieval balance of relevance and diversity substantially enhances the relevance and coverage of retrieved context in comparison to a conventional similarity search, with MMR raising nDCG@5 by +4 to +11 percentage points across the swept models. More detailed prompt instructions meaningfully increase how faithfully generated answers stick to the retrieved evidence, especially for capable and reasoning models (+10 to +13pp on Qwen3.6-27B, GPT-4o-mini and Qwen3.5-think, but negligible on the 20B generator). Overall, RAG-NIMBF transforms the configuration space of RAG into reproducible, comparable experiments. By attaching to existing production pipelines in a non-invasive way, it enables practitioners to systematically evaluate and improve deployed systems without disrupting existing workflows. This replaces ad hoc choices with evidence-based decisions.