Visual document retrieval is a critical component of multimodal retrieval-augmented generation, aiming to identify query-relevant pages from document collections where evidence is distributed across text, layout, charts, and visual structures. Recent efforts toward finer-grained supervision primarily rely on textual descriptions or localized visual regions as evidence proxies. However, such supervision signals may either overlook complex visual structures or provide incomplete and inaccurate representations of the underlying evidence. To address these limitations, we propose ConceptFormer, a latent concept representation learning framework for visual document retrieval. ConceptFormer models query-relevant evidence as continuous, query-conditioned latent concepts that explicitly bridge localized visual evidence and semantic relevance, without requiring either textual intermediate representations or direct reliance on raw visual annotations. During training, ConceptFormer employs a strong vision-language model to dynamically determine the number of latent concept tokens and uses these concepts as an intermediate representation to bridge the semantic gap between queries and documents, thereby guiding the learning of the embedding space. Experiments on diverse visual document retrieval benchmarks demonstrate that ConceptFormer achieves 16.7% and 22.1% relative improvements in average NDCG@10 over the strongest visual retrieval baseline and the strongest OCR-based text retrieval baseline, respectively. Further analysis reveals that latent concepts effectively connect localized visual evidence with semantic relevance, enabling the retriever to capture both fine-grained textual cues and complex document-level visual structures while preserving strong retrieval alignment. Codes and data are available at https://github.com/Neuir/ConceptFormer.
Figures & tables
Figure 1. Overview of ConceptFormer. The left panel illustrates a case where the standard InfoNCE method fails to find fine-grained local evidence, whereas ConceptFormer retrieves the relevant page. The right panel shows how ConceptFormer allocates adaptive latent concepts from query-related regions and learns them with retrieval-guided representation learning.
Method
InfoVQA
ChartQA
SlideVQA
TQA
OWID Charts
Wikimedia Maps
Average
R@10
N@10
R@10
N@10
R@10
N@10
R@10
N@10
R@10
N@10
R@10
N@10
R@10
N@10
Text-based retrievers
BM25
81.77
68.91
58.67
48.88
81.43
73.55
24.00
12.73
98.47
88.27
5.93
3.26
58.38
49.27
BGE-1.5
79.87
64.27
82.00
70.83
81.36
70.47
43.40
22.61
99.24
91.92
9.45
5.95
65.89
54.34
E5-Mistral
54.48
40.31
55.33
47.78
62.08
50.42
32.60
17.45
99.24
89.23
6.59
4.08
51.72
41.55
NV-Embed-v2
88.17
73.66
88.67
80.91
88.25
79.40
61.80
34.77
99.24
94.79
15.82
9.92
73.66
62.24
Table 1. Overall performance of ConceptFormer. We report Recall@10 and NDCG@10 as evaluation metrics. † , ‡ , and \lx@sectionsign indicate statistically significant improvements over NV-Embed † , VisRAG-Ret ‡ , and VDocRetriever § , respectively.
Models
InfoVQA
ChartQA
SlideVQA
TQA
OWID Charts
Wikimedia Maps
Average
R@10
N@10
R@10
N@10
R@10
N@10
R@10
N@10
R@10
N@10
R@10
N@10
R@10
N@10
ConceptFormer (Text)
90.27
77.04
100.00
95.61
91.12
81.43
71.50
41.14
99.24
94.82
74.73
59.08
87.81
74.85
ConceptFormer (Vision)
89.22
75.91
98.67
92.78
89.04
79.37
68.60
38.64
99.24
95.37
70.11
54.79
85.81
72.81
Query-Concept Matching
92.37
79.06
99.33
94.87
90.22
80.63
72.00
41.61
99.24
94.67
75.60
60.04
88.13
75.15
ConceptFormer
93.03 †‡
79.23 †‡
98.67
95.79
90.88
82.41 †‡
72.10
41.30
99.24
95.39
76.04 †‡
61.69 †‡
88.33 †‡
75.97 †‡
w/o Mean Pooling
92.56
78.95
99.33
94.78
90.48
81.65
71.80
41.21
99.24
95.29
74.51
60.69
87.99
75.43
Table 2. Ablation study of latent concept representation learning in ConceptFormer. † and ‡ indicate statistically significant improvements over ConceptFormer (Text) † , ConceptFormer w/o KL Loss ‡ , respectively.
λ
I-VQA
C-tQA
S-VQA
TQA
O-Charts
W-Maps
Average
0.0
76.86
95.00
79.01
40.50
96.03
57.91
74.22
0.1
74.33
93.11
79.24
38.54
95.52
58.37
73.18
0.2
79.23
95.79
82.41
41.30
95.39
61.69
75.97
0.3
77.34
93.72
79.72
40.00
96.13
57.79
74.12
0.4
76.80
93.10
79.83
41.31
95.85
57.50
74.06
0.5
78.89
93.72
79.54
41.48
96.23
56.36
74.37
Table 3. Sensitivity to the weight λ of alignment loss Lalign (Eq. 17 ). All models are implemented using Qwen2.5-vl.
Figure 2. 3D retrieval geometry of concept representations. We compare visual and textual concept proxies with ConceptFormer latent concepts using query discriminability, page discriminability, and retrieval alignment.
Figure 3. Effectiveness of adaptive latent concept of ConceptFormer. All models are implemented using Qwen2.5-VL, where P90 denotes the 90th-percentile length.
Figure 4. Case study. Regions with higher color intensity indicate stronger patch-level query relevance.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Field
#Images
#Queries
DocVQA
Industry
12,767
6,382
InfoVQA
Infographic
5,485
9,592
VisualMRC
Webpage
10,229
6,126
OpenWikiTable
Table
1,257
4,261
DUDE
Open
27,955
2,135
MHDocVQA
Open
28,550
9,470
Appendix
Table 4. Training dataset statistics.
Dataset
Field
#Images
#Queries
InfoVQA
Infographic
5,485
1,048
ChartQA
Chart
20,882
150
SlideVQA
Slide
52,380
760
TQA
Table
1,000
1,000
OWID Charts
Chart
131
131
Wikimedia Maps
Map
455
455
Appendix
Table 5. Test dataset statistics.
Benchmark
Dataset Identifier
InfoVQA
NTT-hil-insight/OpenDocVQA ; images from OpenDocVQA-Corpus
ChartQA
NTT-hil-insight/OpenDocVQA ; images from OpenDocVQA-Corpus
SlideVQA
NTT-hil-insight/OpenDocVQA ; images from OpenDocVQA-Corpus
TQA
jinaai/tqa
OWID Charts
jinaai/owid_charts_en
Wikimedia Maps
jinaai/wikimedia-commons-maps
Appendix
Table 6. Sources of the six evaluation benchmarks. All identifiers refer to Hugging Face datasets.
Configuration
Value
Main backbone
Qwen2.5-VL-7B-Instruct
Alternative backbone
Phi3V / DSE initialization
Training epochs
3
Optimizer
AdamW
Learning rate
1×10−4
Per-device batch size
8
Appendix
Table 7. Main training and inference configurations of ConceptFormer.
Category
Method
Checkpoint Identifier
Text retrievers
BM25
No model checkpoint
BGE-1.5
BAAI/bge-large-en-v1.5
E5-Mistral
intfloat/e5-mistral-7b-instruct
NV-Embed-v2
nvidia/NV-Embed-v2
Multimodal retrievers
CLIP
openai/clip-vit-large-patch14-336
E5-V
royokong/e5-v
Appendix
Table 8. Public checkpoints used for baseline evaluation. All identifiers refer to Hugging Face model repositories.
Figure 5. Prompt templates used in Evidence Extraction.
Visual document retrieval requires rapidly locating relevant pages from large multi-modal corpora in response to user queries. While recent methods powered by Multi-modal Large Language Models (MLLMs) show competitive accuracy, they suffer from prohibitive computational costs by applying intensive MLLM encoding to every single page. Meanwhile, we observe that user queries are typically keyword-anchored, containing semantically rich words that are expected to appear directly in the visible text of relevant pages, offering an efficient cue for quickly narrowing down candidate pages. Building on this insight, we propose LightSTAR, an efficient framework that decomposes visual document retrieval into: 1) LLM-free Visual Selection, which utilizes content-grounded query encoding to focus on informative words and employs LLM-free visual embeddings to produce a high-recall candidate set; and 2) Vision-adaptive Semantic Refinement, which further performs fine-grained semantic matching exclusively on these top candidates via adaptive region-wise feature fusion to effectively combine textual and layout cues, optimized through a hardness-aware contrastive objective. Experimental results demonstrate that LightSTAR achieves state-of-the-art retrieval accuracy while reducing end-to-end latency by several-fold, offering a highly practical solution to the accuracy-efficiency trade-off in visual document retrieval. Code is available at https://github.com/bokufa/LightSTAR.
Tongkun Guan, Haocheng Wang, Wei Shen +1
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University
Visual document retrieval has recently become increasingly important in applications such as enterprise search, scientific literature discovery, and retrieval-augmented generation. These applications depend on efficiently identifying query-relevant pages across large collections of visually rich documents. Existing methods commonly adopt late-interaction architectures that encode and index documents offline to enable scalable and low-latency online retrieval. Despite its efficiency, this paradigm requires each document to be encoded into a fixed representation before the query is known. However, the same content in a visual document may induce different interpretations depending on the query intent, which a fixed representation struggles to capture. Yet postponing document encoding until the query arrives would incur prohibitive online retrieval latency. To address this gap, we propose VaRS-Doc, a visual document retrieval framework that diversifies document representations by enabling the model to actively explore variant latent interpretations during document encoding, while preserving efficient late-interaction retrieval in which each query adaptively selects the best-fit representation. We further introduce a two-stage training strategy that encourages the model to capture complementary semantic interpretations and prevents it from falling back to train a single dominant representation. Experiments on visual document retrieval benchmarks show that VaRS-Doc achieves state-of-the-art retrieval performance, offering a practical solution to the mismatch between query-agnostic document encoding and query-specific retrieval needs. Code is available at https://github.com/bokufa/VaRS-Doc.
Haocheng Wang, Tongkun Guan, Wei Shen +1
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University, China
Retrieving information from visually rich documents requires matching queries against pages that contain complex evidence in tables, figures, and layout-dependent structures. Current vision-language retrievers address this but incur query-time costs that grow with corpus size and are impractical to scale. We propose SCOUT, a cascaded two-track retrieval framework that separates retrieval performance from exhaustive search. The visual track first uses a lightweight parent retriever to filter the corpus to a compact candidate pool, then applies a stronger multi-vector reranker only to those candidates. In parallel, a semantic track retrieves over hierarchical page and document summaries generated offline to provide a signal complementary to visual matching. The outputs are fused through cross-modal query refinement, which uses semantic scores to adjust the visual query representation before final ranking. We evaluate SCOUT on 4 major benchmarks against 6 baselines and observe a significant query-time latency improvement up to 80.7x (28.8x on average) relative to the best baseline, while exceeding state-of-the-art retrieval performance by up to 7.8% NDCG across all benchmarks.
Diana Turmakhan, Aseem Srivastava, Vivek Gupta +2
Mohamed bin Zayed University of Artificial Intelligence · Arizona State University · Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science