Retriever-Free Retrieval: Navigating Documents with Decision Models

Chao Wan; Paul Jünger; Beixi Du; Bodong Liu; Navaneetha Krishnan; Kilian Q. Weinberger

cs.AIOct 26, 2026

Retriever-Free Retrieval: Navigating Documents with Decision Models

Authors: Chao Wan, Paul Jünger, Beixi Du, Bodong Liu, Navaneetha Krishnan, Kilian Q. Weinberger

Organizations: Department of Computer Science, Cornell University, New York, USA · ASAPP, New York, USA

Abstract

While language models have become dramatically more capable, retrieval still largely relies on lexical matching or similarity in fixed embedding spaces. This creates a capability asymmetry: what the retriever fails to find, even the strongest generator cannot recover. We propose DecisionRAG, which brings the capabilities of pretrained language models to retrieval using decision models that jointly evaluate candidate branches in a single forward pass, without generating any tokens, and show that judging candidates in a shared context makes their relevance estimates comparable. Like a person browsing a library, DecisionRAG navigates the corpus's inherent structure from broad topics down to individual passages, deciding at each level which branches are worth exploring. Without task-specific training or a first-stage retriever, DecisionRAG outperforms dense retrieval with an 8B reranker by 13.8 R@1 points on MultiDoc2Dial and 17.4 Hit@1 points on Natural Questions. Across four benchmarks, including a corpus of 2.68 million Wikipedia paragraphs, retrieval takes just 0.7-1.8 seconds per query.

Figures & tables

Figure 1: Capability asymmetry: a simple retriever bottlenecks a powerful LLM (a); decision models level the playing field (b).

Figure 2: Retrieval by matching versus retrieval by decisions. (a) BM25 and (b) dense retrieval match query terms or embeddings and can miss the relevant page. (c) DecisionRAG walks the corpus hierarchy: a decision model judges all candidates of a level in one forward pass, keeping the top $k$ documents and then the top $r$ passages; pruned documents are never expanded. Term frequencies, IDF values, BM25 scores, and rankings in panel (a) are illustrative.

Panel of Figure 3 — Figure 3: The two judgment modes of a decision model, each against a shared context that the model reads once.

Figure 4: Analytical model: sequential stages grow with the depth of the hierarchy, not the size of the corpus. Theoretic latency per query ( Equation 4 ,

b=100

), assuming a balanced hierarchy and enough serving capacity to judge every level in parallel: each

100\times

more passages adds one level, i.e., one more sequential decision (

{\approx}0.2

s).

Dataset	Corpus hierarchy	Evaluation set	Gold evidence
MultiDoc2Dial	4 centers / 413 pages / 4,110 passages	4,201 turns	1 passage
ConditionalQA	652 pages / 58,483 paragraphs	285 questions (271 labeled)	1–22 paragraphs
MTRAG (IBM Cloud)	172 products / 8,578 pages / 72,442 passages	205 turns (188 labeled)	1–7 passages
Natural Questions (DAPR)	1,000 clusters / 108,626 pages / 2.68M paragraphs	500 test questions	1–3 paragraphs

Table 1: Overview of the four retrieval benchmarks: MultiDoc2Dial ( Feng et al., 2021 ) , ConditionalQA ( Sun et al., 2022 ) , MTRAG ( Katsis et al., 2025 ) and Natural Questions ( Kwiatkowski et al., 2019 ) in its DAPR version ( Wang et al., 2024 ) .

Conventional retrieval pipelines
	MultiDoc2Dial		ConditionalQA		MTRAG ‡		Natural Questions
Method	R@1	R@10	Hit@1	R@10	R@5	nDCG@5	Hit@1	R@10	nDCG@10
BM25	20.6	56.4	24.7	35.0	18.8	30.9	16.4	42.3	28.4
SPLADE++	24.4	58.8	28.4	38.3	27.8	47.1	36.0	74.6	54.5
Dense	27.9	70.9	47.2	54.0	26.0	46.5	46.2	87.2	66.0
Dense top-100 + BGE	22.8	63.9	53.1	60.6	14.5	25.3	54.8	88.7	71.7

Table 2: Retrieval without task-specific training on four benchmarks (%).

	ConditionalQA	MTRAG	NQ
Retrieval	EM	Judged	EM
None	1.8	6.4	26.9
Dense	43.5	31.4	53.0
Dense top-100 + Qwen3-RR	45.4	39.4	52.4
Dense top-20 + GPT-5	44.3	34.0	54.5
DecisionRAG (Jev)	46.9	44.1	54.2

Table 3: Answer quality of one generator (Qwen3.5-27B) given each system’s top five passages (%): exact match, and on MTRAG the share of replies judged correct (188 labeled turns). Qwen3-RR: Qwen3-Reranker-8B; NQ: Natural Questions.

Task-trained conventional retrieval
	MultiDoc2Dial
Method	R@1	R@10
DPR ( Feng et al., 2021 ) p	49.0	80.0
SFRe2g ( Zhang et al., 2026 ) p	55.0	83.2
Task-trained decision-based retrieval (ours)
DecisionRAG (8B trained)	60.7	85.8

Table 4: Retrieval with task-trained methods (%).

Median latency (s)
	MD2D	CQA	MTRAG	NQ
Dense top-100 + Qwen3-RR	7.5	3.3	14.3	5.8
Dense top-20 + GPT-5	11.8	14.4	12.2	15.0
DecisionRAG (Jev)	0.68	0.82	1.76	1.73
+ dense candidates	0.70	0.85	1.73	1.64
API cost per query (¢)

Table 5: Median latency (seconds) and API cost per query. DecisionRAG rows are live, one query at a time, without reused answers; GPT-5 covers reranking only. Dense top-100 + Qwen3-RR is served locally on one NVIDIA RTX A6000. Qwen3-RR: Qwen3-Reranker-8B; NQ: Natural Questions.

Figure 5: Hit@1 on Natural Questions as the candidate pool grows, with all

N

candidates scored in one request (150 questions).

Figure 6: Hit@1 on Natural Questions when a fixed pool of 200 candidates is split into requests of

m

candidates and ranked together (289 questions; band: 95% interval).

ConditionalQA
	Hit@1	R@10	s	¢
GPT-5, generative	73.8	73.6	28.2	6.1
DecisionRAG	69.7	71.9	0.82	0.29
+ dense candidates	73.4	76.3	0.85	0.29
Natural Questions
GPT-5, generative	59.0	82.5	250.8	57.2

Table 6: The same hierarchy, decided by generation versus decisions. Latency: median seconds per query; cost: cents per query. Natural Questions: the first 100 test questions of Table 2 .

Dataset	Metric	Binary only	+ Multiple- choice
MultiDoc2Dial	R@1	50.1	51.3
ConditionalQA	Hit@1	71.9	76.5

Table 7: Multiple-choice ranking on design data (%; MultiDoc2Dial: 1,832 turns, ConditionalQA: 285 questions). Multiple-choice ranking reorders the top 20 passages, with only these passages in its context ( Section B.6 ).

		Dense top-100 + Jev Rerank	DecisionRAG + dense candidates
MD2D	R@1	49.9	51.2
	R@10	86.6	85.6
CQA	Hit@1	65.7	73.4
	R@10	68.2	76.3
MTRAG	R@5	38.3	38.7
	nDCG@5	36.8	35.7

Table 8: Jev reranking the dense top 100 versus DecisionRAG with dense candidates, both using the same dense index (%). MD2D: MultiDoc2Dial, seeded sample of 1,000 validation turns; CQA: ConditionalQA; MTRAG: official labels; NQ: Natural Questions.

Figure 7: Recall@10 (left axis) and Jev cost per query (right axis) on Natural Questions as the top level keeps more and more clusters. First 100 test questions of Table 2 ; band: 95% interval.

New pages	R@1 (%)	R@10 (%)	API cost (¢/query)	Median time (s)
3	46.9	81.0	0.12	0.53
10	47.4	83.6	0.22	0.68

Table 9: Page budget on MultiDoc2Dial (validation, with multiple-choice ranking).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Conventional retrieval pipelines
Method	R@1	R@5	R@10
BM25 ( Feng et al., 2021 ) †	19.6	41.9	50.8
BM25 (ours)	20.6	46.8	56.4
SPLADE++	24.4	–	58.8
Dense	27.9	60.4	70.9
Dense top-100 + BGE	22.8	52.8	63.9

Appendix

Table 10: MultiDoc2Dial validation (4,201 turns), passage recall by passage ID (%). Dense: Qwen3-Embedding-8B; BGE: bge-reranker-v2-m3. † Published results, not rerun.

Method	Page Hit@1 / 3 / 10	Paragraph Hit@1; R@1 / 5 / 10	nDCG@10 (DAPR)
Conventional retrieval pipelines
BM25 + headings	51.6 / 74.0 / 89.1	24.7 8.9 / 24.6 / 35.0	27.5
SPLADE-v3 (DistilBERT)	62.5 / 80.4 / 94.0	26.9 12.1 / 26.9 / 34.1	28.1
SPLADE++	61.4 / 81.4 / 91.9	28.4 12.1 / 28.3 / 38.3	31.2
Dense	74.7 / 88.4 / 96.1	47.2 21.0 / 43.2 / 54.0	47.6
Dense top-100 + BGE	72.3 / 88.8 / 96.5	53.1 22.1 / 48.9 / 60.6	53.5

Appendix

Table 11: ConditionalQA development set, retrieval from all 652 pages (%). Page metrics use 285 questions and paragraph metrics the 271 questions with evidence. The paragraph column shows Hit@1 above R@1 / R@5 / R@10. nDCG@10 uses DAPR’s relevance judgments ( Wang et al., 2024 ) . Dense and BGE as in Table 10 .

System	R@5	nDCG@5
DecisionRAG + dense candidates	36.6	65.0
DecisionRAG	34.2	62.3
Dense + Qwen3-Reranker, last turn	32.5	56.5
Dense top-20 + GPT-5	31.8	59.4
Dense + Qwen3-Reranker, six turns	28.4	50.9
SPLADE++, last turn	27.8	47.1

Appendix

Table 12: MTRAG, 188 labeled turns, against gold and judged-relevant passages (%); identical texts count once. The remaining nine systems (BM25, dense, SPLADE and BGE variants) trail DecisionRAG by 10 to 20 R@5 points.

Passage context	Notepad	R@1	R@5	R@10
Separate	None	32.7	54.5	63.6
Separate	Own	36.7	61.3	70.4
Shared	Own	44.4	69.7	76.8

Appendix

Table 13: MultiDoc2Dial passage context and page memory on 297 design turns (%).

MultiDoc2Dial
Configuration	Primary	R@10	$\Delta$ primary [95% CI]
3 pages	48.2	82.5	–
3 pages + dense 20	50.1	86.1	+1.9 [+0.9, +3.0]
3 pages + dense 20 + multiple-choice	51.3	86.6	+3.1 [+1.5, +4.7]
ConditionalQA
3 pages	71.2	74.1	–

Appendix

Table 14: Last-stage design comparisons (%). Primary metric: R@1 (MultiDoc2Dial), Hit@1 (ConditionalQA), R@5 (MTRAG-UN).

Dataset	Configuration	Scope	$n$	p50/p95 (s)	API ¢/query
MultiDoc2Dial	DecisionRAG (Jev)	W	48	0.68/0.91	0.22
	+ dense candidates	H	48	0.70/1.16	0.23
	Dense top-20 + GPT-5	R	1,000	11.8/23.3	1.40
ConditionalQA	DecisionRAG (Jev)	W	50	0.82/1.02	0.29
	+ dense candidates	H	50	0.85/1.13	0.29
	Dense top-20 + GPT-5	R	285	14.4/31.4	1.20

Appendix

Table 15: Online timing per query.

n

: timed queries; W: complete retrieval after initialization; H: retrieval and scoring with saved dense candidates; R: GPT-5 listwise reranking of the saved dense top 20, API only.

System	MultiDoc2Dial	ConditionalQA	MTRAG
BM25	0.001	0.001	0.008
SPLADE-v3 (DistilBERT)	–	0.117	0.298
SPLADE++	–	0.090	0.177
Dense	0.066	0.047	0.116
Dense top-100 + Qwen3-Reranker (A6000)	7.51	3.33	14.33

Appendix

Table 16: Median latency of the locally served baselines (seconds per query, RTX 3090 unless noted; BM25 on CPU), including encoding and search. BM25 on ConditionalQA indexes headings with the paragraphs; MTRAG uses the six-turn input.

MultiDoc2Dial validation
Judge / training	Pages	R@1	R@5	R@10
1.7B / general	3	20.5	38.3	44.2
1.7B / general	10	12.8	28.7	36.3
4B / general	3	28.5	52.9	59.8
4B / general	10	21.9	47.2	56.5
8B / general	3	40.4	66.0	71.9

Appendix

Table 17: Open-weight judges (%). General : trained without MultiDoc2Dial or ConditionalQA data; + MD2D + CQA (LoRA) : the DecisionRAG (8B trained) model of Table 4 .

Information Retrieval

Information Retrieval studies how to find, rank, and organize relevant information in large collections of data.

See all 474 papers

Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation
Oct 26, 2026Qiaomu Li, Ying XieLLM Reranking RAG Evaluation
College of Computing and Software Engineering, Kennesaw State University, Marietta, GA, USA
RIT-RAG: Navigating Document Corpora with Retrieval-Induced Trees
Oct 8, 2026Meghanadh Pulivarthi, Swaraj Kumar Biswal, Kushagra Bhushan +3Retrieval-Augmented Generation Structured RAG
IBM · IIT Kharagpur
When to Retrieve During Reasoning: Adaptive Retrieval for Large Reasoning Models
Apr 29, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuRetrieval-Augmented Generation Retrieval-Augmented Reasoning
The University of Hong Kong Hong Kong, China · Brain Investing Limited Hong Kong, China · Stellaris AI Limited Hong Kong, China

Oct 26, 2026J/K move · Enter open · S save

Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation

Qiaomu Li, Ying Xie

College of Computing and Software Engineering, Kennesaw State University, Marietta, GA, USA

LLM Reranking RAG Evaluation Cross-Encoder Reranking LLM Inference Efficiency

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) systems often use a dedicated reranker to choose which retrieved candidates reach the language model. Reranking improves relevance but adds latency, so teams look for faster alternatives. System-1style decision models make direct, structured predictions instead of generating long responses. Could one provide comparable reranking quality with less inference overhead? We first screen 21 open-weight models on 11 benchmarks, then compare a promising decision model with a dedicated reranker on identical candidate lists. The default decision-model setup fails on many long candidate sets, so scoring only the requests it completes gives a misleading ranking. Limiting candidate text length largely restores reliability and makes some configurations competitive. A frozen evaluation on new retrieval data supports the reliability and ranking results, but confirms the latency requirement for only one of the two selected configurations. Across workloads, which model ranks better changes, and a tuned reranker remains faster and uses far less GPU memory. A decision model can perform competitive reranking, but replacing a reranker requires weighing quality, latency and memory together, and teams should tune the reranker they already run first.

Abstract

Information Retrieval

Explore similar work

Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation

RIT-RAG: Navigating Document Corpora with Retrieval-Induced Trees

When to Retrieve During Reasoning: Adaptive Retrieval for Large Reasoning Models