Cascaded Retrieval with Cross-Modal Query Refinement for Efficient Visual Document Search

Diana Turmakhan; Aseem Srivastava; Vivek Gupta; Iryna Gurevych; Thamar Solorio

cs.CLOct 14, 2026

Cascaded Retrieval with Cross-Modal Query Refinement for Efficient Visual Document Search

Authors: Diana Turmakhan, Aseem Srivastava, Vivek Gupta, Iryna Gurevych, Thamar Solorio

Organizations: Mohamed bin Zayed University of Artificial Intelligence · Arizona State University · Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science

Abstract

Retrieving information from visually rich documents requires matching queries against pages that contain complex evidence in tables, figures, and layout-dependent structures. Current vision-language retrievers address this but incur query-time costs that grow with corpus size and are impractical to scale. We propose SCOUT, a cascaded two-track retrieval framework that separates retrieval performance from exhaustive search. The visual track first uses a lightweight parent retriever to filter the corpus to a compact candidate pool, then applies a stronger multi-vector reranker only to those candidates. In parallel, a semantic track retrieves over hierarchical page and document summaries generated offline to provide a signal complementary to visual matching. The outputs are fused through cross-modal query refinement, which uses semantic scores to adjust the visual query representation before final ranking. We evaluate SCOUT on 4 major benchmarks against 6 baselines and observe a significant query-time latency improvement up to 80.7x (28.8x on average) relative to the best baseline, while exceeding state-of-the-art retrieval performance by up to 7.8% NDCG across all benchmarks.

Figures & tables

Figure 1: Accurate visual retrieval without exhaustive search. Text-only retrievers ignore visual content, and multi-vector retrievers match every page at high query-time cost. SCOUT applies late interaction only to a compact candidate pool and refines the visual query with offline summaries.

Approach	Vis.	Txt.	Fus.	Cost
Single signal
Text dense (OCR / summ.)	$\times$	✓	–	–
Single-vector visual	page	$\times$	–	–
Multi-vector visual	token	$\times$	–	$O(N)$
Efficiency-oriented
Efficient multi-vector index	token	$\times$	–	$O(N^{\prime})$

Table 1: Taxonomy of prior approaches. Vis. / Txt. : visual/textual evidence used (page: one vector per page; token: token- or patch-level). Fus. : how modalities are combined (rank: rank-level; query: query-level refinement). Cost : number of pages on which the costly stage (late interaction or reranking) runs, where

N

is the corpus size,

N^{\prime}<N

the candidates kept by index pruning, and

K\ll N

the first-stage pool.

Figure 2: Architecture of SCOUT . Left (offline phase): each page in the corpus is dual-encoded as (a) a concatenated doc-level + page-level summary embedding and (b) a single visual embedding. Center (online phase): a single query q goes through two parallel tracks: Track A performs an efficient single-vector retrieval to top-K candidates, then applies expensive multi-vector reranking + max-similarity on the same top-K pages. Track B retrieves top-T summaries through cosine similarity between summary embeddings and query. Right: Their union enters Cross-Modal Query Refinement (CMQR), where semantic scores form a soft target that refines the visual multi-vector query; the refined query gets re-scored for the final ranking of evidence pages.

Model	ViDoSeek		SlideVQA		ViDoRe 3		R-MM-RAG
Model	@5	@10	@5	@10	@5	@10	@5	@10
Qwen3-8B	74.84	77.10	89.01	90.24	48.18	50.63	57.61	60.35
Jina-v4 (t)	75.24	77.09	91.53	92.32	51.30	53.94	52.39	55.41
Linq-Emb.	78.20	80.15	91.12	92.05	46.77	49.44	57.36	60.09
Jina-v4 (v)	85.48	86.46	93.75	94.34	53.01	55.66	66.11	68.87
ColNomic	83.48	85.03	94.06	94.57	55.44	58.02	70.60	72.60

Table 2: Retrieval performance comparison on ViDoSeek, SlideVQA, ViDoRe 3, and REAL-MM-RAG benchmarks.

\Delta(\%)

shows the relative improvement wrt best baseline (underlined).

Variant	Avg. NDCG@10 ( $\Delta\%$ )
SCOUT	61.31 (reference)
w/o Track A	53.73 ( $\downarrow 12.35\%$ )
w/o Track B	57.00 ( $\downarrow 7.01\%$ )
$\hookrightarrow$ w/o Child Reranker	48.49 ( $\downarrow 20.90\%$ )
Replace (Track B): Summary $\rightarrow$ OCR	59.08 ( $\downarrow 3.6\%$ )
Replace (Fusion): CMQR $\rightarrow$ RRF	57.45 ( $\downarrow 6.28\%$ )

Table 3: Ablation Study. Top-down ablation on the major SCOUT modules: Track A (cascaded visual track containing parent retriever and child reranker) and Track B (semantic retriever) in comparison with SCOUT ’s performance on ViDoRe 3 . Additional benchmark ablation in Appendix (c.f. Table 12 ).

Figure 3: (a) Latency-quality trade-off for visual retrieval methods. Each point reports macro-averaged NDCG@10 and per-query latency. (b) Per-query latency breakdown comparing exhaustive ColEmbed-3B-v2 retrieval with our cascaded pipeline averaged over the ViDoRe subsets.

Figure 4: Candidate-depth sweep: Mean NDCG@10 versus latency for different reranker depths

K

Model	ViDoSeek	SlideVQA	ViDoRe 3	R-MM-RAG
ColNomic	88.73 ( $\downarrow$ 3.34)	95.22 ( $\downarrow$ 0.75)	57.41 ( $\downarrow$ 3.64)	72.60 ( $\downarrow$ 17.26)
ColEmbed	89.23 ( $\downarrow$ 3.13)	95.29 ( $\downarrow$ 1.67)	58.14 ( $\downarrow$ 4.64)	74.56 ( $\downarrow$ 17.25)
SCOUT	91.00 ( $\downarrow$ 0.07)	95.52 ( $\downarrow$ 1.58)	59.94 ( $\downarrow$ 2.69)	76.72 ( $\downarrow$ 14.14)

Table 4: Robustness to query perturbation. The table includes NDCG@10 points on the perturbed queries and the shift from the original scores. Values closer to zero indicate greater robustness. Best scores are in bold and second best are underlined.

Figure 5: Case Study. A query and its response with evidence distributed across 3 pages out of 2,255 in Vidore-Energy corpus. Rankings are shown at each stage of SCOUT against ColEmbed-3B-v2 and ColNomic-7B methods. Our full framework retrieves all three gold pages within the top-3 ranks (

\checkmark

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Model	English										French
	CS		Finance		Pharma		HR		Ind.		Finance		Energy		Physics
	@5	@10	@5	@10	@5	@10	@5	@10	@5	@10	@5	@10	@5	@10	@5	@10
Qwen3-8B	70.43	72.45	46.52	48.85	57.41	59.50	46.31	49.37	38.31	40.19	29.08	31.38	53.36	57.07	44.06	46.24
Jina-v4 (Text)	66.61	69.08	54.68	57.25	54.01	55.89	53.34	56.27	41.13	43.17	40.63	44.21	59.25	62.17	40.77	43.44
Linq-Embed	66.39	69.21	47.88	50.85	56.52	59.20	45.79	48.76	31.96	34.12	31.93	33.97	50.81	54.33	42.91	45.11
Jina-v4 (Vis)	69.63	71.91	52.85	56.03	60.32	62.04	53.83	57.00	45.84	47.70	39.70	43.39	58.11	61.10	43.81	46.10

Appendix

Table 5: Fine-grained retrieval performance on multilingual ViDoRe subsets across English and French domains. Results are reported using NDCG@{5,10}. SCOUT achieves the best performance across all domains and shows generalization to multilingual and domain-specific retrieval settings. Consolidated results are in Table 2 .

Model	Finance				Technical
	FinSlides		FinReport		TechReport		TechSlides
	@5	@10	@5	@10	@5	@10	@5	@10
Qwen3-8B	54.76	58.13	54.16	57.03	52.49	55.20	69.04	71.03
Jina-v4 (Text)	49.28	52.67	43.36	46.82	51.47	54.59	65.44	67.55
Linq-Embed	60.72	63.45	49.22	52.50	49.82	53.01	69.70	71.39
Jina-v4 (Vis)	54.47	58.40	57.62	60.77	71.99	74.47	80.36	81.84

Appendix

Table 6: Fine-grained retrieval performance on the REAL-MM-RAG subsets.

Benchmark	Subset	Lang.	#Docs	#Pages
ViDoRe 3	Finance (EN)	en	6	2,942
	Comp. Science	en	2	1,360
	Pharma	en	52	2,313
	HR	en	14	1,110
	Industrial	en	27	5,244
	Physics	fr	42	1,674

Appendix

Table 7: Statistics of the evaluation corpora. Language refers to the documents.

Index Footprint	ViDoRe 3 (19,252 pages)	ViDoSeek (5,385 pages)	SlideVQA (8,000 pages)
Nemotron-Embed-VL-1B	0.08 GB	0.02 GB	0.03 GB
ColEmbed-3B-v2	213.1 GB	59.6 GB	88.6 GB
SCOUT Active	0.63 GB	0.58 GB	0.59 GB
Reduction	337 $\times$	104 $\times$	151 $\times$

Appendix

Table 8: Estimated active retrieval index footprint across corpora.

Parent retriever	Index (GB)	NDCG@10	Latency (ms)
Nemotron-Embed-VL-1B	0.23	67.37	465.3
ColNomic-7B	6.35	67.70	777.7

Appendix

Table 9: Parent retriever ablation across 3 benchmarks: ViDoRe 3, SlideVQA, and ViDoSeek.

Dataset	#Docs	#Pages	Time	Time/page	Cost/page	Total cost
ViDoRe 3	189	19,252	1h 40m	312 ms	$0.0023	$44.81
SlideVQA	400	8,000	40m	300 ms	$0.0022	$17.78
ViDoSeek	292	5,385	25m	279 ms	$0.0024	$12.78
REAL-MM-RAG	162	8,604	45m	314 ms	$0.0025	$21.56
Total	1,043	41,241	3h 30m	–	–	$96.93

Appendix

Table 10: One-time offline cost of generating hierarchical page and document summaries with Gemini 2.5 Flash. Time is effective wall-clock time using 12 parallel API workers. Only newly added pages require summarization, so updates to a corpus scale with the number of changed pages rather than corpus size.

Figure 6: Per-query latency breakdown comparing exhaustive ColEmbed-3B-v2 retrieval with our cascaded pipeline averaged over the (a) SlideVQA and (b) Vidoseek subsets.

Dataset	$\eta$	Steps	$\alpha$	$T$	NDCG@10
ViDoRe (English)
CS	$1\times 10^{-4}$	10	0.5	1.0	78.13
Finance	$5\times 10^{-5}$	50	0.5	1.0	66.86
Pharmaceuticals	$1\times 10^{-3}$	25	0.5	1.0	66.99
HR	$5\times 10^{-5}$	25	0.5	1.0	64.12
Industrial	$1\times 10^{-4}$	10	0.5	1.0	51.09

Appendix

Table 11: Validation-selected CMQR hyperparameters across datasets at the main configuration (

K{=}25

, top-

T

summary feedback

=20

\eta

denotes the learning rate and

T

the softmax temperature.

Figure 7: Validation-set sweep over summary depth: mean NDCG@10 (macro-average over 3 datasets: ViDoRe 3, SlideVQA, and ViDoSeek.) for varying numbers of retrieved summary pages

T

Split	Dataset	Method	NDCG@10
Single-hop	SlideVQA	Parent retrieval	93.87
		+Child reranker	95.44
		SCOUT	95.59
	ViDoRe 3	Parent retrieval	49.37
		+ Child reranker	58.05
		SCOUT	62.41

Appendix

Table 13: Single-hop and multi-hop performance analysis (NDCG@10) across SlideVQA, ViDoRe, and ViDoSeek. ViDoRe values are macro-averaged across its 8 sub-datasets.

Model	CS	Finance	Pharma	HR	Ind.	Fin. (FR)	Energy	Physics	Avg.
ColEmbed-3B-v2	79.76	58.69	65.46	60.25	58.97	52.08	64.73	69.23	63.65
SCOUT + page summaries	82.52	63.09	68.01	63.74	59.85	56.28	67.83	70.49	66.48
$\Delta\%$ vs. ColEmbed	$\uparrow$ 3.5%	$\uparrow$ 7.5%	$\uparrow$ 3.9%	$\uparrow$ 5.8%	$\uparrow$ 1.5%	$\uparrow$ 8.1%	$\uparrow$ 4.8%	$\uparrow$ 1.8%	$\uparrow$ 4.4%

Appendix

Table 14: End-to-end answer quality on the ViDoRe 3 subsets, measured as the average judge score (Correct

=1

, Partially correct

=0.5

, Incorrect

=0

). Generator: Qwen3-VL-8B-Instruct, top-5 retrieved pages; judge: GPT-5.2.

Benchmark	Subset	Method	NDCG@10 ( $\uparrow$ )	Latency (ms) ( $\downarrow$ )
SlideVQA	–	SCOUT	95.01	406.89
		ColNomic-7B	94.57	331.01
		ColEmbed-3B-v2	94.86	32,844.94
ViDoRe	cs	SCOUT	78.13	387.43
		ColNomic-7B	76.93	98.89
		ColEmbed-3B-v2	77.22	7,088.78

Appendix

Table 15: Retrieval performance, latency, and relative speedups grouped by dataset and method.

Panel of Figure 8 — Figure 8: Relative NDCG@10 change over full-corpus ColEmbed-3B-v2 for CMQR ( SCOUT ) and RRF on the same candidate union, by quintile (computed within each of the 12 subsets; 17,162 queries).

Panel of Figure 9 — Figure 9: Prompt used to generate document-level semantic summaries. At indexing time, the JSON fields shown are concatenated and prepended to each page summary before embedding; global_summary is truncated to 300 characters and key_entities to the first 15 entries.

Model alias	Full model name	Size	Vecs/Pg	Dim	Footprint	HF Identifier
		(Params)			(GiB/1M Pgs)
Linq-Embed	Linq-Embed-Mistral	7.1B	1	4096	7.6	Linq-AI-Research/Linq-Embed-Mistral
Qwen3-8B	Qwen3-Embedding-8B	8B	1	4096	7.6	Qwen/Qwen3-Embedding-8B
Jina-v4 (Text)	jina-embeddings-v4	3.8B	1	2048	3.8	jinaai/jina-embeddings-v4
Jina-v4 (Vision)	jina-embeddings-v4	3.8B	$\sim$ 755	128	180.0	jinaai/jina-embeddings-v4
ColNomic-7B	ColNomic-Embed-Multimodal-7B	7B	$\sim$ 755	128	180.0	nomic-ai/colnomic-embed-multimodal-7b

Appendix

Table 16: Specification of the evaluated retrieval models. We list model aliases, full model names, parameter counts, page-level embedding structure, estimated index footprint, and Hugging Face repositories. Footprint is estimated for one million pages using 16-bit precision and excludes metadata, file-system, and indexing overhead.

Panel of Figure 10 — Figure 10: Prompt used to generate page-level semantic summaries for vector-based retrieval. Each page is summarized into a structured text representation comprising a high-level overview, schema-level descriptions of tables/charts/figures, and grouped named entities.

Information Retrieval

Information Retrieval studies how to find, rank, and organize relevant information in large collections of data.

See all 471 papers

LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinement
Jun 22, 2026Tongkun Guan, Haocheng Wang, Wei Shen +1Information Retrieval Visual Document Retrieval
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University
ConceptFormer: Learning Adaptive Latent Concepts for Query-Document Alignment in Visual Document Retrieval
Aug 16, 2026Chunyi Peng, Zhipeng Xu, Yukun Yan +9Visual Document Retrieval Representation Learning
Northeastern University Shenyang, China · Tsinghua University Beijing, China · Peking University Beijing, China
LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrieval
Mar 10, 2026Seonok KimMultimodal IR Visual Document Retrieval
Mazelone

Jun 22, 2026J/K move · Enter open · S save

LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinement

Tongkun Guan, Haocheng Wang, Wei Shen, Xiaokang Yang

MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University

Information Retrieval Visual Document Retrieval

Open paper AskPDF arXiv

Visual document retrieval requires rapidly locating relevant pages from large multi-modal corpora in response to user queries. While recent methods powered by Multi-modal Large Language Models (MLLMs) show competitive accuracy, they suffer from prohibitive computational costs by applying intensive MLLM encoding to every single page. Meanwhile, we observe that user queries are typically keyword-anchored, containing semantically rich words that are expected to appear directly in the visible text of relevant pages, offering an efficient cue for quickly narrowing down candidate pages. Building on this insight, we propose LightSTAR, an efficient framework that decomposes visual document retrieval into: 1) LLM-free Visual Selection, which utilizes content-grounded query encoding to focus on informative words and employs LLM-free visual embeddings to produce a high-recall candidate set; and 2) Vision-adaptive Semantic Refinement, which further performs fine-grained semantic matching exclusively on these top candidates via adaptive region-wise feature fusion to effectively combine textual and layout cues, optimized through a hardness-aware contrastive objective. Experimental results demonstrate that LightSTAR achieves state-of-the-art retrieval accuracy while reducing end-to-end latency by several-fold, offering a highly practical solution to the accuracy-efficiency trade-off in visual document retrieval. Code is available at https://github.com/bokufa/LightSTAR.

Abstract

Information Retrieval

Explore similar work

LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinement

ConceptFormer: Learning Adaptive Latent Concepts for Query-Document Alignment in Visual Document Retrieval

LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrieval