Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation

Qiaomu Li; Ying Xie

cs.LGOct 26, 2026

Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation

Authors: Qiaomu Li, Ying Xie

Organizations: College of Computing and Software Engineering, Kennesaw State University, Marietta, GA, USA

Abstract

Retrieval-augmented generation (RAG) systems often use a dedicated reranker to choose which retrieved candidates reach the language model. Reranking improves relevance but adds latency, so teams look for faster alternatives. System-1style decision models make direct, structured predictions instead of generating long responses. Could one provide comparable reranking quality with less inference overhead? We first screen 21 open-weight models on 11 benchmarks, then compare a promising decision model with a dedicated reranker on identical candidate lists. The default decision-model setup fails on many long candidate sets, so scoring only the requests it completes gives a misleading ranking. Limiting candidate text length largely restores reliability and makes some configurations competitive. A frozen evaluation on new retrieval data supports the reliability and ranking results, but confirms the latency requirement for only one of the two selected configurations. Across workloads, which model ranks better changes, and a tuned reranker remains faster and uses far less GPU memory. A decision model can perform competitive reranking, but replacing a reranker requires weighing quality, latency and memory together, and teams should tune the reranker they already run first.

Figures & tables

Panel of Figure 1 — Fig. 1: Evaluating a reranker replacement in a fixed RAG pipeline. For a given query, both the incumbent dedicated reranker and a proposed decision-model policy receive the same retrieved candidate set. We compare the Top-K candidates they return, including failures and fallback behavior, and measure the serving latency of each policy. The retriever is unchanged, which isolates the reranking step. The downstream LLM is shown for context but is not part of the controlled ranking evaluation. A replacement is justified only when reliability, ranking quality and serving requirements are met together.

Dedicated rerankers (9)
		Reasoning and science				Multi-hop QA				Grounded and domain QA
	Size	BRIGHT	+trunc.	SciFact	QASPER	HotpotQA	MuSiQue	2Wiki	MH-RAG	GaRAGe	Enterpr.	FinBench	CRAG
Model	(B)	nDCG@5			FS@5	FS@5				Rel@5	FS@5	Hit@5	F1
First-stage order, Top-5	–	13.3	–	75.0	14.6	49.8	9.9	24.4	38.9	72.9	55.8	29.4	24.8
Random Top-5	–	5.8	–	–	–	–	–	–	–	72.1	–	–	–
MiniLM-L6 [ 27 ]	0.02	–	–	–	–	59.5	14.3	32.6	–	–	54.4	–	–

TABLE I: How broad is the screen? Quality of 21 open-weight models on 11 benchmarks, from earlier project stages and not one shared protocol (

\times

100)

Benchmark	Content	Queries	Metric	Runs
BRIGHT [ 26 ]	Reasoning, 12 domains	1,384	nDCG@5	13
SciFact [ 33 ]	Scientific claims	277	nDCG@5	8
QASPER [ 34 ]	Research papers	1,502	FS@5	2
HotpotQA [ 35 ]	Multi-hop, Wikipedia	800	FS@5	14
MuSiQue [ 36 ]	Multi-hop, composed	2,167	FS@5	14
2WikiMultihopQA [ 37 ]	Multi-hop, Wikipedia	800	FS@5	14

TABLE II: Which benchmarks does the study use? Sizes, metrics and runs per benchmark

	Done		$\Delta$ nDCG@5 vs Jina
Policy	(%)	nDCG@5	completed only	all requests
BM25 Top-5	100	0.1330	–	$-.0514$
Jina v3.5	100	0.1844	–	–
PPLX Decider, default	44.1	0.0813	$\mathbf{+.0371}$	$\mathbf{-.1031}$
+ BM25 fallback	100	0.1687	–	$-.0157$
Winnow-12B, default	79.6	0.1599	$+.0325$	$-.0245$

TABLE III: Does default one-call reranking survive counting failures? Completed-only versus all-request scores (BRIGHT,

N=1{,}384

)

Fig. 2: Counting failed requests removes the apparent quality gain. Each bar splits a policy’s all-request difference from Jina into its ranking gain on the requests it completed (orange) and the relevance Jina finds on the requests it failed (blue). Diamonds and right-hand values give the net difference; parentheses give completed requests out of 1,384.

Decision model	default	trunc.	default	trunc.	same text	Jina	BM25 Top-5
	Done (%)		nDCG@5		Jina	$\Delta$ of truncated vs
PPLX Decider	44.1	99.7	0.0813	0.1928	0.1765	$\mathbf{+.0163}$	$\mathbf{+.0598}$
Kev-9B	46.8	100	0.0663	0.1213	0.1779	$-.0566$	$-.0116$
Drex-DLM	47.8	100	0.0710	0.1187	0.1770	$-.0582$	$-.0142$
Winnow-12B	79.6	75.5	0.1599	0.1428	0.1844 a	$-.0415$	$+.0099$

TABLE IV: Does truncating candidate text make decision models competitive? Equal-prefix truncation to 8,192 tokens on BRIGHT

Candidate order (row 3: mean over six shuffled orders)
Contrast (nDCG@5)	$\Delta$	$p$	Holm $p$
PPLX: BM25 $-$ shuffled order	$\mathbf{+.0131}$	$<$ .001	0.002
Jina: BM25 $-$ shuffled order	$-.0004$	0.848	–
PPLX truncated $-$ Jina	$-.0051$	0.620	–
Two-stage PPLX (two windows of 10, then 10 to 5)
Two-stage $-$ Jina	$\mathbf{+.0264}$	0.003	0.010

TABLE V: Do multi-stage reranking and its controls hold up? Follow-up experiments on BRIGHT

Budget $L$	Valid	nDCG@5	p95 $\times$	R Q S	nDCG@5	p95 $\times$	R Q S	Valid	nDCG@5	p95 $\times$	R Q S	nDCG@5	p95 $\times$	R Q S
		Single call: PPLX			Single call: Jina				Two-stage: PPLX			Two-stage: Jina
1,024	35.0	0.0458	0.23 †	$\times$ $\times$ $\checkmark$	0.0242	0.03 †	$\times$ $\times$ $\checkmark$	–	–	–	–	–	–	–
2,048	94.9	0.1747	0.44	$\times$ $\times$ $\checkmark$	0.1221	0.04 †	$\times$ $\times$ $\checkmark$	97.1	0.1854	1.4 †	$\times$ $\checkmark$ ?	0.1471	0.13 †	$\times$ $\times$ $\checkmark$
3,072	98.7	0.1839	0.69	$\times$ $\checkmark$ $\checkmark$	0.1500	0.06 †	$\times$ $\times$ $\checkmark$	–	–	–	–	–	–	–
4,096	99.2	0.1875	0.91	$\checkmark$ $\checkmark$ $\checkmark$	0.1637	0.09 †	$\checkmark$ $\times$ $\checkmark$	99.2	0.2013	2.8	$\checkmark$ $\checkmark$ $\times$	0.1761	0.26 †	$\checkmark$ $\times$ $\checkmark$
6,144	99.6	0.1886	1.4	$\checkmark$ $\checkmark$ $\checkmark$	0.1721	0.13	$\checkmark$ $\times$ $\checkmark$	–	–	–	–	–	–	–

TABLE VI: How much input does a decision model need? Input-token budget sweep on BRIGHT (1,384 queries; strict scoring: a failed request scores zero)

Fig. 3: Quality against latency across input budgets (BRIGHT; each point is one budget, from 2,048 to 8,192 tokens). The shaded region meets the quality and latency checks; filled markers also meet the reliability check. Single-call PPLX at 4,096 and 6,144 tokens meets the checks, but only 4,096 is near parity with full-text Jina (6,144 is slower); a tuned Jina is faster; two-stage PPLX is better but slower.

	Valid		$\Delta$ vs	p95	p95	Mem.	Checks
Reranking policy	(%)	nDCG@5	Jina	(ms)	ratio	(GiB)	Rel.	Qual.	Serv.	Takeaway
Baseline and dedicated reranker (Jina v3.5)
BM25 Top-5 (no reranker)	100	0.1330	$-.0514$	–	–	–	$\checkmark$	–	–	Baseline to beat
Jina v3.5, full text (dedicated)	100	0.1844	ref.	1,211	1.0	4.3	$\checkmark$	$\checkmark$	$\checkmark$	Dedicated reranker
Jina v3.5, truncated to 8,192	99.7	0.1765	$-.0079$	245	0.20	1.9	$\checkmark$	$\checkmark$	$\checkmark$	Tuned Jina; passes all
Main case, PPLX Decider: default input, then input budget and multi-stage reranking

TABLE VII: Which policies pass the three replacement checks? Summary of BRIGHT policies against the dedicated reranker (1,384 queries)

EnterpriseRAG-Bench (500 requests, 443 labeled; 74.5% reachable)
	Valid		$\Delta$ vs full-text	p95
Policy	(%)	FS@5	Jina	ratio
First-stage order (hybrid)	100	0.5576	$-.0293$	–
Jina v3.5, full text	100	0.5869	ref.	1.00
Jina v3.5, truncated	100	0.5327	$-.0542$	0.52
Jina v3.5, two-stage	100	0.5688	$-.0181$	1.2

TABLE VIII: Do the replacement results hold across RAG workloads? Which model ranks better depends on the workload; the latency gap does not (controlled transfer check; FS@5; hybrid first-stage lists)

LLM Reranking

Large language model reranking uses LLMs to reorder retrieved documents or candidate outputs by relevance, usefulness, or task-specific criteria.

See all 133 papers

Retriever-Free Retrieval: Navigating Documents with Decision Models
Oct 26, 2026Chao Wan, Paul Jünger, Beixi Du +3Information Retrieval Language Model-Guided Search Heuristics
Department of Computer Science, Cornell University, New York, USA · ASAPP, New York, USA
System One models Are Not Better Rerankers
Oct 23, 2026Michael Dinzinger, Kanishka Ghosh Dastidar, Michael GranitzerLLM Reranking Information Retrieval
University of Passau, Passau, Germany · Interdisciplinary Transformation University Austria (IT:U), Linz, Austria
Revisiting RAG Retrievers: An Information Theoretic Benchmark
Feb 25, 2026Wenqing Zheng, Dmitri Kalaev, Noah Fatsi +5RAG Evaluation IR Evaluation
CapitalOne.

Oct 26, 2026J/K move · Enter open · S save

Retriever-Free Retrieval: Navigating Documents with Decision Models

Chao Wan, Paul Jünger, Beixi Du, Bodong Liu+2

Department of Computer Science, Cornell University, New York, USA · ASAPP, New York, USA

Information Retrieval Language Model-Guided Search Heuristics

Open paper AskPDF arXiv

While language models have become dramatically more capable, retrieval still largely relies on lexical matching or similarity in fixed embedding spaces. This creates a capability asymmetry: what the retriever fails to find, even the strongest generator cannot recover. We propose DecisionRAG, which brings the capabilities of pretrained language models to retrieval using decision models that jointly evaluate candidate branches in a single forward pass, without generating any tokens, and show that judging candidates in a shared context makes their relevance estimates comparable. Like a person browsing a library, DecisionRAG navigates the corpus's inherent structure from broad topics down to individual passages, deciding at each level which branches are worth exploring. Without task-specific training or a first-stage retriever, DecisionRAG outperforms dense retrieval with an 8B reranker by 13.8 R@1 points on MultiDoc2Dial and 17.4 Hit@1 points on Natural Questions. Across four benchmarks, including a corpus of 2.68 million Wikipedia paragraphs, retrieval takes just 0.7-1.8 seconds per query.

Abstract

LLM Reranking

Explore similar work

Retriever-Free Retrieval: Navigating Documents with Decision Models

System One models Are Not Better Rerankers

Revisiting RAG Retrievers: An Information Theoretic Benchmark