Organizations: Mohamed bin Zayed University of Artificial Intelligence · Arizona State University · Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science
Retrieving information from visually rich documents requires matching queries against pages that contain complex evidence in tables, figures, and layout-dependent structures. Current vision-language retrievers address this but incur query-time costs that grow with corpus size and are impractical to scale. We propose SCOUT, a cascaded two-track retrieval framework that separates retrieval performance from exhaustive search. The visual track first uses a lightweight parent retriever to filter the corpus to a compact candidate pool, then applies a stronger multi-vector reranker only to those candidates. In parallel, a semantic track retrieves over hierarchical page and document summaries generated offline to provide a signal complementary to visual matching. The outputs are fused through cross-modal query refinement, which uses semantic scores to adjust the visual query representation before final ranking. We evaluate SCOUT on 4 major benchmarks against 6 baselines and observe a significant query-time latency improvement up to 80.7x (28.8x on average) relative to the best baseline, while exceeding state-of-the-art retrieval performance by up to 7.8% NDCG across all benchmarks.
Figures & tables
Figure 1: Accurate visual retrieval without exhaustive search. Text-only retrievers ignore visual content, and multi-vector retrievers match every page at high query-time cost. SCOUT applies late interaction only to a compact candidate pool and refines the visual query with offline summaries.
Approach
Vis.
Txt.
Fus.
Cost
Single signal
Text dense (OCR / summ.)
×
✓
–
–
Single-vector visual
page
×
–
–
Multi-vector visual
token
×
–
O(N)
Efficiency-oriented
Efficient multi-vector index
token
×
–
O(N′)
Table 1: Taxonomy of prior approaches. Vis. / Txt. : visual/textual evidence used (page: one vector per page; token: token- or patch-level). Fus. : how modalities are combined (rank: rank-level; query: query-level refinement). Cost : number of pages on which the costly stage (late interaction or reranking) runs, where N is the corpus size, N′<N the candidates kept by index pruning, and K≪N the first-stage pool.
Figure 2: Architecture of SCOUT . Left (offline phase): each page in the corpus is dual-encoded as (a) a concatenated doc-level + page-level summary embedding and (b) a single visual embedding. Center (online phase): a single query q goes through two parallel tracks: Track A performs an efficient single-vector retrieval to top-K candidates, then applies expensive multi-vector reranking + max-similarity on the same top-K pages. Track B retrieves top-T summaries through cosine similarity between summary embeddings and query. Right: Their union enters Cross-Modal Query Refinement (CMQR), where semantic scores form a soft target that refines the visual multi-vector query; the refined query gets re-scored for the final ranking of evidence pages.
Model
ViDoSeek
SlideVQA
ViDoRe 3
R-MM-RAG
@5
@10
@5
@10
@5
@10
@5
@10
Qwen3-8B
74.84
77.10
89.01
90.24
48.18
50.63
57.61
60.35
Jina-v4 (t)
75.24
77.09
91.53
92.32
51.30
53.94
52.39
55.41
Linq-Emb.
78.20
80.15
91.12
92.05
46.77
49.44
57.36
60.09
Jina-v4 (v)
85.48
86.46
93.75
94.34
53.01
55.66
66.11
68.87
ColNomic
83.48
85.03
94.06
94.57
55.44
58.02
70.60
72.60
Table 2: Retrieval performance comparison on ViDoSeek, SlideVQA, ViDoRe 3, and REAL-MM-RAG benchmarks. Δ(%) shows the relative improvement wrt best baseline (underlined).
Variant
Avg. NDCG@10 ( Δ% )
SCOUT
61.31 (reference)
w/o Track A
53.73 ( ↓12.35% )
w/o Track B
57.00 ( ↓7.01% )
↪ w/o Child Reranker
48.49 ( ↓20.90% )
Replace (Track B): Summary → OCR
59.08 ( ↓3.6% )
Replace (Fusion): CMQR → RRF
57.45 ( ↓6.28% )
Table 3: Ablation Study. Top-down ablation on the major SCOUT modules: Track A (cascaded visual track containing parent retriever and child reranker) and Track B (semantic retriever) in comparison with SCOUT ’s performance on ViDoRe 3 . Additional benchmark ablation in Appendix (c.f. Table 12 ).
Figure 3: (a) Latency-quality trade-off for visual retrieval methods. Each point reports macro-averaged NDCG@10 and per-query latency. (b) Per-query latency breakdown comparing exhaustive ColEmbed-3B-v2 retrieval with our cascaded pipeline averaged over the ViDoRe subsets.
Figure 4: Candidate-depth sweep: Mean NDCG@10 versus latency for different reranker depths K .
Model
ViDoSeek
SlideVQA
ViDoRe 3
R-MM-RAG
ColNomic
88.73 ( ↓ 3.34)
95.22 ( ↓ 0.75)
57.41 ( ↓ 3.64)
72.60 ( ↓ 17.26)
ColEmbed
89.23 ( ↓ 3.13)
95.29 ( ↓ 1.67)
58.14 ( ↓ 4.64)
74.56 ( ↓ 17.25)
SCOUT
91.00 ( ↓ 0.07)
95.52 ( ↓ 1.58)
59.94 ( ↓ 2.69)
76.72 ( ↓ 14.14)
Table 4: Robustness to query perturbation. The table includes NDCG@10 points on the perturbed queries and the shift from the original scores. Values closer to zero indicate greater robustness. Best scores are in bold and second best are underlined.
Figure 5: Case Study. A query and its response with evidence distributed across 3 pages out of 2,255 in Vidore-Energy corpus. Rankings are shown at each stage of SCOUT against ColEmbed-3B-v2 and ColNomic-7B methods. Our full framework retrieves all three gold pages within the top-3 ranks ( ✓ ).
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Model
English
French
CS
Finance
Pharma
HR
Ind.
Finance
Energy
Physics
@5
@10
@5
@10
@5
@10
@5
@10
@5
@10
@5
@10
@5
@10
@5
@10
Qwen3-8B
70.43
72.45
46.52
48.85
57.41
59.50
46.31
49.37
38.31
40.19
29.08
31.38
53.36
57.07
44.06
46.24
Jina-v4 (Text)
66.61
69.08
54.68
57.25
54.01
55.89
53.34
56.27
41.13
43.17
40.63
44.21
59.25
62.17
40.77
43.44
Linq-Embed
66.39
69.21
47.88
50.85
56.52
59.20
45.79
48.76
31.96
34.12
31.93
33.97
50.81
54.33
42.91
45.11
Jina-v4 (Vis)
69.63
71.91
52.85
56.03
60.32
62.04
53.83
57.00
45.84
47.70
39.70
43.39
58.11
61.10
43.81
46.10
Appendix
Table 5: Fine-grained retrieval performance on multilingual ViDoRe subsets across English and French domains. Results are reported using NDCG@{5,10}. SCOUT achieves the best performance across all domains and shows generalization to multilingual and domain-specific retrieval settings. Consolidated results are in Table 2 .
Model
Finance
Technical
FinSlides
FinReport
TechReport
TechSlides
@5
@10
@5
@10
@5
@10
@5
@10
Qwen3-8B
54.76
58.13
54.16
57.03
52.49
55.20
69.04
71.03
Jina-v4 (Text)
49.28
52.67
43.36
46.82
51.47
54.59
65.44
67.55
Linq-Embed
60.72
63.45
49.22
52.50
49.82
53.01
69.70
71.39
Jina-v4 (Vis)
54.47
58.40
57.62
60.77
71.99
74.47
80.36
81.84
Appendix
Table 6: Fine-grained retrieval performance on the REAL-MM-RAG subsets.
Benchmark
Subset
Lang.
#Docs
#Pages
ViDoRe 3
Finance (EN)
en
6
2,942
Comp. Science
en
2
1,360
Pharma
en
52
2,313
HR
en
14
1,110
Industrial
en
27
5,244
Physics
fr
42
1,674
Appendix
Table 7: Statistics of the evaluation corpora. Language refers to the documents.
Index Footprint
ViDoRe 3 (19,252 pages)
ViDoSeek (5,385 pages)
SlideVQA (8,000 pages)
Nemotron-Embed-VL-1B
0.08 GB
0.02 GB
0.03 GB
ColEmbed-3B-v2
213.1 GB
59.6 GB
88.6 GB
SCOUT Active
0.63 GB
0.58 GB
0.59 GB
Reduction
337 ×
104 ×
151 ×
Appendix
Table 8: Estimated active retrieval index footprint across corpora.
Parent retriever
Index (GB)
NDCG@10
Latency (ms)
Nemotron-Embed-VL-1B
0.23
67.37
465.3
ColNomic-7B
6.35
67.70
777.7
Appendix
Table 9: Parent retriever ablation across 3 benchmarks: ViDoRe 3, SlideVQA, and ViDoSeek.
Dataset
#Docs
#Pages
Time
Time/page
Cost/page
Total cost
ViDoRe 3
189
19,252
1h 40m
312 ms
$0.0023
$44.81
SlideVQA
400
8,000
40m
300 ms
$0.0022
$17.78
ViDoSeek
292
5,385
25m
279 ms
$0.0024
$12.78
REAL-MM-RAG
162
8,604
45m
314 ms
$0.0025
$21.56
Total
1,043
41,241
3h 30m
–
–
$96.93
Appendix
Table 10: One-time offline cost of generating hierarchical page and document summaries with Gemini 2.5 Flash. Time is effective wall-clock time using 12 parallel API workers. Only newly added pages require summarization, so updates to a corpus scale with the number of changed pages rather than corpus size.
Figure 6: Per-query latency breakdown comparing exhaustive ColEmbed-3B-v2 retrieval with our cascaded pipeline averaged over the (a) SlideVQA and (b) Vidoseek subsets.
Dataset
η
Steps
α
T
NDCG@10
ViDoRe (English)
CS
1×10−4
10
0.5
1.0
78.13
Finance
5×10−5
50
0.5
1.0
66.86
Pharmaceuticals
1×10−3
25
0.5
1.0
66.99
HR
5×10−5
25
0.5
1.0
64.12
Industrial
1×10−4
10
0.5
1.0
51.09
Appendix
Table 11: Validation-selected CMQR hyperparameters across datasets at the main configuration ( K=25 , top- T summary feedback =20 ). η denotes the learning rate and T the softmax temperature.
Figure 7: Validation-set sweep over summary depth: mean NDCG@10 (macro-average over 3 datasets: ViDoRe 3, SlideVQA, and ViDoSeek.) for varying numbers of retrieved summary pages T .
Split
Dataset
Method
NDCG@10
Single-hop
SlideVQA
Parent retrieval
93.87
+Child reranker
95.44
SCOUT
95.59
ViDoRe 3
Parent retrieval
49.37
+ Child reranker
58.05
SCOUT
62.41
Appendix
Table 13: Single-hop and multi-hop performance analysis (NDCG@10) across SlideVQA, ViDoRe, and ViDoSeek. ViDoRe values are macro-averaged across its 8 sub-datasets.
Model
CS
Finance
Pharma
HR
Ind.
Fin. (FR)
Energy
Physics
Avg.
ColEmbed-3B-v2
79.76
58.69
65.46
60.25
58.97
52.08
64.73
69.23
63.65
SCOUT + page summaries
82.52
63.09
68.01
63.74
59.85
56.28
67.83
70.49
66.48
Δ% vs. ColEmbed
↑ 3.5%
↑ 7.5%
↑ 3.9%
↑ 5.8%
↑ 1.5%
↑ 8.1%
↑ 4.8%
↑ 1.8%
↑ 4.4%
Appendix
Table 14: End-to-end answer quality on the ViDoRe 3 subsets, measured as the average judge score (Correct =1 , Partially correct =0.5 , Incorrect =0 ). Generator: Qwen3-VL-8B-Instruct, top-5 retrieved pages; judge: GPT-5.2.
Benchmark
Subset
Method
NDCG@10 ( ↑ )
Latency (ms) ( ↓ )
SlideVQA
–
SCOUT
95.01
406.89
ColNomic-7B
94.57
331.01
ColEmbed-3B-v2
94.86
32,844.94
ViDoRe
cs
SCOUT
78.13
387.43
ColNomic-7B
76.93
98.89
ColEmbed-3B-v2
77.22
7,088.78
Appendix
Table 15: Retrieval performance, latency, and relative speedups grouped by dataset and method.
Figure 8: Relative NDCG@10 change over full-corpus ColEmbed-3B-v2 for CMQR ( SCOUT ) and RRF on the same candidate union, by quintile (computed within each of the 12 subsets; 17,162 queries).
Figure 9: Prompt used to generate document-level semantic summaries. At indexing time, the JSON fields shown are concatenated and prepended to each page summary before embedding; global_summary is truncated to 300 characters and key_entities to the first 15 entries.
Model alias
Full model name
Size
Vecs/Pg
Dim
Footprint
HF Identifier
(Params)
(GiB/1M Pgs)
Linq-Embed
Linq-Embed-Mistral
7.1B
1
4096
7.6
Linq-AI-Research/Linq-Embed-Mistral
Qwen3-8B
Qwen3-Embedding-8B
8B
1
4096
7.6
Qwen/Qwen3-Embedding-8B
Jina-v4 (Text)
jina-embeddings-v4
3.8B
1
2048
3.8
jinaai/jina-embeddings-v4
Jina-v4 (Vision)
jina-embeddings-v4
3.8B
∼ 755
128
180.0
jinaai/jina-embeddings-v4
ColNomic-7B
ColNomic-Embed-Multimodal-7B
7B
∼ 755
128
180.0
nomic-ai/colnomic-embed-multimodal-7b
Appendix
Table 16: Specification of the evaluated retrieval models. We list model aliases, full model names, parameter counts, page-level embedding structure, estimated index footprint, and Hugging Face repositories. Footprint is estimated for one million pages using 16-bit precision and excludes metadata, file-system, and indexing overhead.
Figure 10: Prompt used to generate page-level semantic summaries for vector-based retrieval. Each page is summarized into a structured text representation comprising a high-level overview, schema-level descriptions of tables/charts/figures, and grouped named entities.
Visual document retrieval requires rapidly locating relevant pages from large multi-modal corpora in response to user queries. While recent methods powered by Multi-modal Large Language Models (MLLMs) show competitive accuracy, they suffer from prohibitive computational costs by applying intensive MLLM encoding to every single page. Meanwhile, we observe that user queries are typically keyword-anchored, containing semantically rich words that are expected to appear directly in the visible text of relevant pages, offering an efficient cue for quickly narrowing down candidate pages. Building on this insight, we propose LightSTAR, an efficient framework that decomposes visual document retrieval into: 1) LLM-free Visual Selection, which utilizes content-grounded query encoding to focus on informative words and employs LLM-free visual embeddings to produce a high-recall candidate set; and 2) Vision-adaptive Semantic Refinement, which further performs fine-grained semantic matching exclusively on these top candidates via adaptive region-wise feature fusion to effectively combine textual and layout cues, optimized through a hardness-aware contrastive objective. Experimental results demonstrate that LightSTAR achieves state-of-the-art retrieval accuracy while reducing end-to-end latency by several-fold, offering a highly practical solution to the accuracy-efficiency trade-off in visual document retrieval. Code is available at https://github.com/bokufa/LightSTAR.
Tongkun Guan, Haocheng Wang, Wei Shen +1
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University
Visual document retrieval is a critical component of multimodal retrieval-augmented generation, aiming to identify query-relevant pages from document collections where evidence is distributed across text, layout, charts, and visual structures. Recent efforts toward finer-grained supervision primarily rely on textual descriptions or localized visual regions as evidence proxies. However, such supervision signals may either overlook complex visual structures or provide incomplete and inaccurate representations of the underlying evidence. To address these limitations, we propose ConceptFormer, a latent concept representation learning framework for visual document retrieval. ConceptFormer models query-relevant evidence as continuous, query-conditioned latent concepts that explicitly bridge localized visual evidence and semantic relevance, without requiring either textual intermediate representations or direct reliance on raw visual annotations. During training, ConceptFormer employs a strong vision-language model to dynamically determine the number of latent concept tokens and uses these concepts as an intermediate representation to bridge the semantic gap between queries and documents, thereby guiding the learning of the embedding space. Experiments on diverse visual document retrieval benchmarks demonstrate that ConceptFormer achieves 16.7% and 22.1% relative improvements in average NDCG@10 over the strongest visual retrieval baseline and the strongest OCR-based text retrieval baseline, respectively. Further analysis reveals that latent concepts effectively connect localized visual evidence with semantic relevance, enabling the retriever to capture both fine-grained textual cues and complex document-level visual structures while preserving strong retrieval alignment. Codes and data are available at https://github.com/Neuir/ConceptFormer.
Chunyi Peng, Zhipeng Xu, Yukun Yan +9
Northeastern University Shenyang, China · Tsinghua University Beijing, China · Peking University Beijing, China
Retrieving relevant evidence from visually rich documents such as textbooks, technical reports, and manuals is challenging due to long context, complex layouts, and weak lexical overlap between user questions and supporting pages. We propose LITTA, a query-expansion-centric retrieval framework for evidence page retrieval that improves multimodal document retrieval without retriever retraining. Given a user query, LITTA generates complementary query variants using a large language model and retrieves candidate pages for each variant using a frozen vision retriever with late-interaction scoring. Candidates from expanded queries are then aggregated through reciprocal rank fusion to improve evidence coverage and reduce sensitivity to any single phrasing. This simple test-time strategy significantly improves retrieval robustness while remaining compatible with existing multimodal embedding indices. We evaluate LITTA on visually grounded document retrieval tasks across three domains: computer science, pharmaceuticals, and industrial manuals. Multi-query retrieval consistently improves top-k accuracy, recall, and MRR compared to single-query retrieval, with particularly large gains in domains with high visual and semantic variability. Moreover, the accuracy-efficiency trade-off is directly controllable by the number of query variants, making LITTA practical for deployment under latency constraints. These results demonstrate that query expansion provides a simple yet effective mechanism for improving visually grounded multimodal retrieval.