Glance: The Generator Is Its Own Reranker

Adilet Metinov; Chingiz Arziev

cs.CLOct 29, 2026

Glance: The Generator Is Its Own Reranker

Authors: Adilet Metinov, Chingiz Arziev

Organizations: Airun Intelligence Labs

Abstract

Retrieval-augmented generation commonly serves separate models for reranking and answering, duplicating model residency and passage processing. We introduce Glance, which equips the generator with a lightweight decision adapter and pointer head to score candidate passages from a shared prefill. Short relevance probes branch from the cached context, producing one score per passage without separately prefilling each query-passage pair. A single resident 9B backbone supports generation, general decisions and reranking, with approximately 0.1 GiB of additional memory per decision skill. After 39 minutes of MS MARCO fine-tuning, Glance achieves nDCG@10 of 0.741 on SciFact and 0.345 on NFCorpus, compared with 0.721 and 0.326 for bge-reranker-v2-m3. On 300 HotpotQA questions, its selected context improves answer F1 by 0.045 over cross-encoder-selected context under the same base generator (paired bootstrap 95% CI [0.014, 0.076]). In a scaling study, reranking quality keeps rising up to 200 candidates (43k tokens) scored from one shared prefill, where it exceeds the cross-encoder by 0.058 nDCG@10, and a probe-only activated adapter lets generation continue from the original passage cache. These initial results demonstrate generator-based reranking with one resident backbone; evaluation currently covers two retrieval datasets, one QA dataset and one backbone family.

Figures & tables

Panel of Figure 1 — Figure 1: One resident backbone, shared passage encoding, passage-specific decisions. Shaded parts run with the decision adapter (LoRA + pointer head). In the main pipeline the selected passages are re-prefilled and answered with the adapter off; in the activated variant the passages are prefilled with the base weights, so generation continues the same cache, which still holds all $k$ candidates.

Name	Initialised from	Training	Adapter active on
zero-shot	–	4.4k general decision records, 20 min, no retrieval data	all tokens
trained	zero-shot	+ 4,000 MS MARCO records, 8–20 passages, 39 min	all tokens
trained-long	trained	+ 450 MS MARCO records, 20–80 passages, 14 min	all tokens
control	zero-shot	+ 1,200 MS MARCO records, 150 steps	all tokens
activated	zero-shot	+ 1,200 MS MARCO records, 150 steps	probe tokens only

Table 1: Adapter configurations (rank-16 LoRA + 256-d pointer head on the frozen 9B backbone).

Method	nDCG@10	ms/query	nDCG@10	ms/query
	SciFact		NFCorpus
BM25	0.662	–	0.306	–
bge-reranker-v2-m3	0.721	31	0.326	60
Decision 9B, separate ( $k$ calls)	0.729	2210	0.323	2485
Glance packed_all (1 pass)	0.715	614	0.341	670
Glance packed_q (1 pass)	0.721	773	0.328	847

Table 2: Reranking BM25 top-20 with the zero-shot adapter (no retrieval training data). One-pass rows run one prefill of the shared state plus one batched continuation over its branches per query; latencies on a shared RTX PRO 6000.

	SciFact		NFCorpus
Method	zero-shot	trained	zero-shot	trained
BM25	0.662		0.306
bge-reranker-v2-m3	0.721		0.326
Glance packed_all (1 pass)	0.715	0.741	0.341	0.345
Glance packed_q (1 pass)	0.721	0.738	0.328	0.338
Glance listwise (1 pass)	0.684	0.692	0.331	0.334

Table 3: nDCG@10 before and after 39 minutes of MS MARCO training (BM25 top-20). Trained packed_all latency on a B200: 420 ms (SciFact), 411 ms (NFCorpus) per query.

$k$	tokens/pass	zero-shot	trained	trained-long	bge-v2-m3	ms/query	peak GiB
10	2.2k	0.651	0.653	–	0.626	252	15.4
20	4.3k	0.662	0.658	0.674	0.624	165	16.7
50	10.8k	0.637	0.674	0.691	0.642	417	25.5
100	21.6k	0.584	0.682	0.697	0.637	1049	56.4
200 †	43.4k	–	–	0.717	0.659	15790	20.5

Table 4: Single-pass reranking with

k

candidates (SciFact, 100 queries, BM25 top-

k

, 100-word passages). Trained : MS MARCO adapter of § 3.6 (8–20 passages per record). Trained-long : warm-started from it and trained 14 more minutes on 450 records with 20–80 passages. Latency and memory: trained-long on a B200 (

k{=}10

: trained adapter, separate run). † Does not fit in one call; run on Kev’s chunked path (one prefill of the state, probe rows in several batches).

Panel of Figure 2 — Figure 2: Quality vs. candidate count (data of Table 4 ). Trained adapters keep improving as more candidates share one prefill ( $k{=}200$ on the chunked path); the zero-shot adapter degrades beyond $k{=}20$ . Peak memory grows from 15 GiB ( $k{=}10$ ) to 56 GiB ( $k{=}100$ ) because of per-branch cache copies.

Context given to the generator	EM	F1	R@2	R@3	rerank ms
(a) all 10 paragraphs	0.533	0.716	–	–	–
(b) bge-reranker-v2-m3 top-3	0.500	0.683	0.863	0.918	29
(c) Glance top-3, adapter off	0.543	0.718	0.762	0.865	378
(b ′ ) bge-reranker-v2-m3 top-3, adapter on	0.547	0.702	0.863	0.918	29
(c ′ ) Glance top-3, adapter on	0.590	0.757	0.762	0.865	378
(d) oracle: 2 gold paragraphs	0.607	0.789	–	–	–

Table 5: End-to-end RAG on HotpotQA distractor (

n{=}300

; packed_all,

k{=}10

). Upper block: zero-shot adapter; lower block: trained adapter. Every row uses the same 9B generator; rows (b ′ ) and (c ′ ) answer with the adapter left on. R@

n

: recall of the gold paragraphs in the top

n

. Latencies: upper block on a shared RTX PRO 6000, lower block on a B200.

Mode	Check against the model loaded alone	Switch into mode (median)
Base	greedy chat text identical on all visits (3 prompts $\times$ 10)	4.2–6.4 ms
General	probabilities bit-identical ( $\max\|\Delta p\|=0$ )	9.4 ms
Rerank	probabilities bit-identical ( $\max\|\Delta p\|=0$ )	8.8 ms

Table 6: One 9B backbone, three modes, switched per request. Memory: 16.85 GiB resident (backbone 14.78, each LoRA 0.08, both heads 0.016, LM head 1.89 GiB), unchanged after the loop (16.86 GiB).

LLM Reranking

Large language model reranking uses LLMs to reorder retrieved documents or candidate outputs by relevance, usefulness, or task-specific criteria.

See all 136 papers

Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation
Oct 26, 2026Qiaomu Li, Ying XieLLM Reranking RAG Evaluation
College of Computing and Software Engineering, Kennesaw State University, Marietta, GA, USA
CAR: Query-Guided Confidence-Aware Reranking for Retrieval-Augmented Generation
May 6, 2026Zhipeng Song, Yizhi Zhou, Xiangyu Kong +6Retrieval-Augmented Generation Learning to Rank
School of Computer Science and Technology, Dalian University of Technology, No.2 Linggong Road, Ganjingzi District, Dalian, 116024, China · School of Information Engineering, Dalian Ocean University, No. 2-52, Heishijiao Street, Shahekou District, Dalian, 116023, China · School of Information Engineering, Liaodong University, No.116 Linjiang Back Street, Zhenan District, Dandong, 118001, China +3
Optimizing RAG Rerankers with LLM Feedback via Reinforcement Learning
Apr 2, 2026Yuhang Wu, Xiangqing Shen, Fanfan Wang +4Retrieval-Augmented Generation Reinforcement Learning
School of Computer Science and Engineering, Nanjing University of Science and Technology, China · School of Intelligence Science and Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China

Oct 26, 2026J/K move · Enter open · S save

Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation

Qiaomu Li, Ying Xie

College of Computing and Software Engineering, Kennesaw State University, Marietta, GA, USA

LLM Reranking RAG Evaluation Cross-Encoder Reranking LLM Inference Efficiency

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) systems often use a dedicated reranker to choose which retrieved candidates reach the language model. Reranking improves relevance but adds latency, so teams look for faster alternatives. System-1style decision models make direct, structured predictions instead of generating long responses. Could one provide comparable reranking quality with less inference overhead? We first screen 21 open-weight models on 11 benchmarks, then compare a promising decision model with a dedicated reranker on identical candidate lists. The default decision-model setup fails on many long candidate sets, so scoring only the requests it completes gives a misleading ranking. Limiting candidate text length largely restores reliability and makes some configurations competitive. A frozen evaluation on new retrieval data supports the reliability and ranking results, but confirms the latency requirement for only one of the two selected configurations. Across workloads, which model ranks better changes, and a tuned reranker remains faster and uses far less GPU memory. A decision model can perform competitive reranking, but replacing a reranker requires weighing quality, latency and memory together, and teams should tune the reranker they already run first.

Abstract

LLM Reranking

Explore similar work

Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation

CAR: Query-Guided Confidence-Aware Reranking for Retrieval-Augmented Generation

Optimizing RAG Rerankers with LLM Feedback via Reinforcement Learning