Glance: The Generator Is Its Own Reranker

Adilet Metinov; Chingiz Arziev

cs.CLOct 29, 2026

Glance: The Generator Is Its Own Reranker

Authors: Adilet Metinov, Chingiz Arziev

Organizations: Airun Intelligence Labs

Abstract

Retrieval-augmented generation commonly serves separate models for reranking and answering, duplicating model residency and passage processing. We introduce Glance, which equips the generator with a lightweight decision adapter and pointer head to score candidate passages from a shared prefill. Short relevance probes branch from the cached context, producing one score per passage without separately prefilling each query-passage pair. A single resident 9B backbone supports generation, general decisions and reranking, with approximately 0.1 GiB of additional memory per decision skill. After 39 minutes of MS MARCO fine-tuning, Glance achieves nDCG@10 of 0.741 on SciFact and 0.345 on NFCorpus, compared with 0.721 and 0.326 for bge-reranker-v2-m3. On 300 HotpotQA questions, its selected context improves answer F1 by 0.045 over cross-encoder-selected context under the same base generator (paired bootstrap 95% CI [0.014, 0.076]). In a scaling study, reranking quality keeps rising up to 200 candidates (43k tokens) scored from one shared prefill, where it exceeds the cross-encoder by 0.058 nDCG@10, and a probe-only activated adapter lets generation continue from the original passage cache. These initial results demonstrate generator-based reranking with one resident backbone; evaluation currently covers two retrieval datasets, one QA dataset and one backbone family.

Figures & tables

Panel of Figure 1 — Figure 1: One resident backbone, shared passage encoding, passage-specific decisions. Shaded parts run with the decision adapter (LoRA + pointer head). In the main pipeline the selected passages are re-prefilled and answered with the adapter off; in the activated variant the passages are prefilled with the base weights, so generation continues the same cache, which still holds all $k$ candidates.

Name	Initialised from	Training	Adapter active on
zero-shot	–	4.4k general decision records, 20 min, no retrieval data	all tokens
trained	zero-shot	+ 4,000 MS MARCO records, 8–20 passages, 39 min	all tokens
trained-long	trained	+ 450 MS MARCO records, 20–80 passages, 14 min	all tokens
control	zero-shot	+ 1,200 MS MARCO records, 150 steps	all tokens
activated	zero-shot	+ 1,200 MS MARCO records, 150 steps	probe tokens only

Table 1: Adapter configurations (rank-16 LoRA + 256-d pointer head on the frozen 9B backbone).

Method	nDCG@10	ms/query	nDCG@10	ms/query
	SciFact		NFCorpus
BM25	0.662	–	0.306	–
bge-reranker-v2-m3	0.721	31	0.326	60
Decision 9B, separate ( $k$ calls)	0.729	2210	0.323	2485
Glance packed_all (1 pass)	0.715	614	0.341	670
Glance packed_q (1 pass)	0.721	773	0.328	847

Table 2: Reranking BM25 top-20 with the zero-shot adapter (no retrieval training data). One-pass rows run one prefill of the shared state plus one batched continuation over its branches per query; latencies on a shared RTX PRO 6000.

	SciFact		NFCorpus
Method	zero-shot	trained	zero-shot	trained
BM25	0.662		0.306
bge-reranker-v2-m3	0.721		0.326
Glance packed_all (1 pass)	0.715	0.741	0.341	0.345
Glance packed_q (1 pass)	0.721	0.738	0.328	0.338
Glance listwise (1 pass)	0.684	0.692	0.331	0.334

Table 3: nDCG@10 before and after 39 minutes of MS MARCO training (BM25 top-20). Trained packed_all latency on a B200: 420 ms (SciFact), 411 ms (NFCorpus) per query.

$k$	tokens/pass	zero-shot	trained	trained-long	bge-v2-m3	ms/query	peak GiB
10	2.2k	0.651	0.653	–	0.626	252	15.4
20	4.3k	0.662	0.658	0.674	0.624	165	16.7
50	10.8k	0.637	0.674	0.691	0.642	417	25.5
100	21.6k	0.584	0.682	0.697	0.637	1049	56.4
200 †	43.4k	–	–	0.717	0.659	15790	20.5

Table 4: Single-pass reranking with

k

candidates (SciFact, 100 queries, BM25 top-

k

, 100-word passages). Trained : MS MARCO adapter of § 3.6 (8–20 passages per record). Trained-long : warm-started from it and trained 14 more minutes on 450 records with 20–80 passages. Latency and memory: trained-long on a B200 (

k{=}10

: trained adapter, separate run). † Does not fit in one call; run on Kev’s chunked path (one prefill of the state, probe rows in several batches).

Panel of Figure 2 — Figure 2: Quality vs. candidate count (data of Table 4 ). Trained adapters keep improving as more candidates share one prefill ( $k{=}200$ on the chunked path); the zero-shot adapter degrades beyond $k{=}20$ . Peak memory grows from 15 GiB ( $k{=}10$ ) to 56 GiB ( $k{=}100$ ) because of per-branch cache copies.

Context given to the generator	EM	F1	R@2	R@3	rerank ms
(a) all 10 paragraphs	0.533	0.716	–	–	–
(b) bge-reranker-v2-m3 top-3	0.500	0.683	0.863	0.918	29
(c) Glance top-3, adapter off	0.543	0.718	0.762	0.865	378
(b ′ ) bge-reranker-v2-m3 top-3, adapter on	0.547	0.702	0.863	0.918	29
(c ′ ) Glance top-3, adapter on	0.590	0.757	0.762	0.865	378
(d) oracle: 2 gold paragraphs	0.607	0.789	–	–	–

Table 5: End-to-end RAG on HotpotQA distractor (

n{=}300

; packed_all,

k{=}10

). Upper block: zero-shot adapter; lower block: trained adapter. Every row uses the same 9B generator; rows (b ′ ) and (c ′ ) answer with the adapter left on. R@

n

: recall of the gold paragraphs in the top

n

. Latencies: upper block on a shared RTX PRO 6000, lower block on a B200.

Mode	Check against the model loaded alone	Switch into mode (median)
Base	greedy chat text identical on all visits (3 prompts $\times$ 10)	4.2–6.4 ms
General	probabilities bit-identical ( $\max\|\Delta p\|=0$ )	9.4 ms
Rerank	probabilities bit-identical ( $\max\|\Delta p\|=0$ )	8.8 ms

Table 6: One 9B backbone, three modes, switched per request. Memory: 16.85 GiB resident (backbone 14.78, each LoRA 0.08, both heads 0.016, LM head 1.89 GiB), unchanged after the loop (16.86 GiB).

LLM Reranking

Large language model reranking uses LLMs to reorder retrieved documents or candidate outputs by relevance, usefulness, or task-specific criteria.

See all 136 papers

Oct 26, 2026

Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation

Retrieval-augmented generation (RAG) systems often use a dedicated reranker to choose which retrieved candidates reach the language model. Reranking improves relevance but adds latency, so teams look for faster alternatives. System-1style decision models make direct, structured predictions instead of generating long responses. Could one provide comparable reranking quality with less inference overhead? We first screen 21 open-weight models on 11 benchmarks, then compare a promising decision model with a dedicated reranker on identical candidate lists. The default decision-model setup fails on many long candidate sets, so scoring only the requests it completes gives a misleading ranking. Limiting candidate text length largely restores reliability and makes some configurations competitive. A frozen evaluation on new retrieval data supports the reliability and ranking results, but confirms the latency requirement for only one of the two selected configurations. Across workloads, which model ranks better changes, and a tuned reranker remains faster and uses far less GPU memory. A decision model can perform competitive reranking, but replacing a reranker requires weighing quality, latency and memory together, and teams should tune the reranker they already run first.

Qiaomu Li, Ying Xie

College of Computing and Software Engineering, Kennesaw State University, Marietta, GA, USA

LLM Reranking RAG Evaluation Cross-Encoder Reranking LLM Inference Efficiency

Ask PDF

May 6, 2026

CAR: Query-Guided Confidence-Aware Reranking for Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) depends on document ranking to provide useful evidence for generation, but conventional reranking methods mainly optimize query-document relevance rather than generation usefulness. A relevant document may still introduce noise, while a lower-ranked document may better reduce the generator's uncertainty. We propose CAR (Confidence-Aware Reranking), a query-guided, training-free, and plug-and-play reranking framework that uses generator confidence change as a document usefulness signal. CAR estimates confidence through the semantic consistency of multiple sampled answers under query-only and query-document conditions. Documents that significantly increase confidence are promoted, those that decrease confidence are demoted, and uncertain cases preserve the baseline order, while a query-level gate avoids unnecessary intervention on already confident queries. Experiments on four BEIR datasets show that CAR consistently improves NDCG@5 across sparse and dense retrievers, LLM-based and supervised rerankers, and four LLM backbones. Notably, CAR improves the YesNo reranker by 25.4 percent on average under Contriever retrieval, and its ranking gains strongly correlate with downstream generation F1 improvements, achieving Spearman rho = 0.964.

Zhipeng Song, Yizhi Zhou, Xiangyu Kong +6

School of Computer Science and Technology, Dalian University of Technology, No.2 Linggong Road, Ganjingzi District, Dalian, 116024, China · School of Information Engineering, Dalian Ocean University, No. 2-52, Heishijiao Street, Shahekou District, Dalian, 116023, China · School of Information Engineering, Liaodong University, No.116 Linjiang Back Street, Zhenan District, Dandong, 118001, China +3

Retrieval-Augmented Generation Learning to Rank LLM Reranking

Ask PDF

Apr 2, 2026

Optimizing RAG Rerankers with LLM Feedback via Reinforcement Learning

Rerankers play a pivotal role in refining retrieval results for Retrieval-Augmented Generation. However, current reranking models are typically optimized on static human annotated relevance labels in isolation, decoupled from the downstream generation process. This isolation leads to a fundamental misalignment: documents identified as topically relevant by information retrieval metrics often fail to provide the actual utility required by the LLM for precise answer generation. To bridge this gap, we introduce ReRanking Preference Optimization (RRPO), a reinforcement learning framework that directly aligns reranking with the LLM's generation quality. By formulating reranking as a sequential decision-making process, RRPO optimizes for context utility using LLM feedback, thereby eliminating the need for expensive human annotations. To ensure training stability, we further introduce a reference-anchored deterministic baseline. Extensive experiments on knowledge-intensive benchmarks demonstrate that RRPO significantly outperforms strong baselines, including the powerful list-wise reranker RankZephyr. Further analysis highlights the versatility of our framework: it generalizes seamlessly to diverse readers (e.g., GPT-4o), integrates orthogonally with query expansion modules like Query2Doc, and remains robust even when trained with noisy supervisors.

Yuhang Wu, Xiangqing Shen, Fanfan Wang +4

School of Computer Science and Engineering, Nanjing University of Science and Technology, China · School of Intelligence Science and Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China

Retrieval-Augmented Generation Reinforcement Learning Learning to Rank LLM Reranking

Ask PDF

Abstract

LLM Reranking

Explore similar work

Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation

CAR: Query-Guided Confidence-Aware Reranking for Retrieval-Augmented Generation

Optimizing RAG Rerankers with LLM Feedback via Reinforcement Learning