Visual document retrieval is a critical component of multimodal retrieval-augmented generation, aiming to identify query-relevant pages from document collections where evidence is distributed across text, layout, charts, and visual structures. Recent efforts toward finer-grained supervision primarily rely on textual descriptions or localized visual regions as evidence proxies. However, such supervision signals may either overlook complex visual structures or provide incomplete and inaccurate representations of the underlying evidence. To address these limitations, we propose ConceptFormer, a latent concept representation learning framework for visual document retrieval. ConceptFormer models query-relevant evidence as continuous, query-conditioned latent concepts that explicitly bridge localized visual evidence and semantic relevance, without requiring either textual intermediate representations or direct reliance on raw visual annotations. During training, ConceptFormer employs a strong vision-language model to dynamically determine the number of latent concept tokens and uses these concepts as an intermediate representation to bridge the semantic gap between queries and documents, thereby guiding the learning of the embedding space. Experiments on diverse visual document retrieval benchmarks demonstrate that ConceptFormer achieves 16.7% and 22.1% relative improvements in average NDCG@10 over the strongest visual retrieval baseline and the strongest OCR-based text retrieval baseline, respectively. Further analysis reveals that latent concepts effectively connect localized visual evidence with semantic relevance, enabling the retriever to capture both fine-grained textual cues and complex document-level visual structures while preserving strong retrieval alignment. Codes and data are available at https://github.com/Neuir/ConceptFormer.
Figures & tables
Figure 1. Overview of ConceptFormer. The left panel illustrates a case where the standard InfoNCE method fails to find fine-grained local evidence, whereas ConceptFormer retrieves the relevant page. The right panel shows how ConceptFormer allocates adaptive latent concepts from query-related regions and learns them with retrieval-guided representation learning.
Method
InfoVQA
ChartQA
SlideVQA
TQA
OWID Charts
Wikimedia Maps
Average
R@10
N@10
R@10
N@10
R@10
N@10
R@10
N@10
R@10
N@10
R@10
N@10
R@10
N@10
Text-based retrievers
BM25
81.77
68.91
58.67
48.88
81.43
73.55
24.00
12.73
98.47
88.27
5.93
3.26
58.38
49.27
BGE-1.5
79.87
64.27
82.00
70.83
81.36
70.47
43.40
22.61
99.24
91.92
9.45
5.95
65.89
54.34
E5-Mistral
54.48
40.31
55.33
47.78
62.08
50.42
32.60
17.45
99.24
89.23
6.59
4.08
51.72
41.55
NV-Embed-v2
88.17
73.66
88.67
80.91
88.25
79.40
61.80
34.77
99.24
94.79
15.82
9.92
73.66
62.24
Table 1. Overall performance of ConceptFormer. We report Recall@10 and NDCG@10 as evaluation metrics. † , ‡ , and \lx@sectionsign indicate statistically significant improvements over NV-Embed † , VisRAG-Ret ‡ , and VDocRetriever § , respectively.
Models
InfoVQA
ChartQA
SlideVQA
TQA
OWID Charts
Wikimedia Maps
Average
R@10
N@10
R@10
N@10
R@10
N@10
R@10
N@10
R@10
N@10
R@10
N@10
R@10
N@10
ConceptFormer (Text)
90.27
77.04
100.00
95.61
91.12
81.43
71.50
41.14
99.24
94.82
74.73
59.08
87.81
74.85
ConceptFormer (Vision)
89.22
75.91
98.67
92.78
89.04
79.37
68.60
38.64
99.24
95.37
70.11
54.79
85.81
72.81
Query-Concept Matching
92.37
79.06
99.33
94.87
90.22
80.63
72.00
41.61
99.24
94.67
75.60
60.04
88.13
75.15
ConceptFormer
93.03 †‡
79.23 †‡
98.67
95.79
90.88
82.41 †‡
72.10
41.30
99.24
95.39
76.04 †‡
61.69 †‡
88.33 †‡
75.97 †‡
w/o Mean Pooling
92.56
78.95
99.33
94.78
90.48
81.65
71.80
41.21
99.24
95.29
74.51
60.69
87.99
75.43
Table 2. Ablation study of latent concept representation learning in ConceptFormer. † and ‡ indicate statistically significant improvements over ConceptFormer (Text) † , ConceptFormer w/o KL Loss ‡ , respectively.
λ
I-VQA
C-tQA
S-VQA
TQA
O-Charts
W-Maps
Average
0.0
76.86
95.00
79.01
40.50
96.03
57.91
74.22
0.1
74.33
93.11
79.24
38.54
95.52
58.37
73.18
0.2
79.23
95.79
82.41
41.30
95.39
61.69
75.97
0.3
77.34
93.72
79.72
40.00
96.13
57.79
74.12
0.4
76.80
93.10
79.83
41.31
95.85
57.50
74.06
0.5
78.89
93.72
79.54
41.48
96.23
56.36
74.37
Table 3. Sensitivity to the weight λ of alignment loss Lalign (Eq. 17 ). All models are implemented using Qwen2.5-vl.
Figure 2. 3D retrieval geometry of concept representations. We compare visual and textual concept proxies with ConceptFormer latent concepts using query discriminability, page discriminability, and retrieval alignment.
Figure 3. Effectiveness of adaptive latent concept of ConceptFormer. All models are implemented using Qwen2.5-VL, where P90 denotes the 90th-percentile length.
Figure 4. Case study. Regions with higher color intensity indicate stronger patch-level query relevance.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Field
#Images
#Queries
DocVQA
Industry
12,767
6,382
InfoVQA
Infographic
5,485
9,592
VisualMRC
Webpage
10,229
6,126
OpenWikiTable
Table
1,257
4,261
DUDE
Open
27,955
2,135
MHDocVQA
Open
28,550
9,470
Appendix
Table 4. Training dataset statistics.
Dataset
Field
#Images
#Queries
InfoVQA
Infographic
5,485
1,048
ChartQA
Chart
20,882
150
SlideVQA
Slide
52,380
760
TQA
Table
1,000
1,000
OWID Charts
Chart
131
131
Wikimedia Maps
Map
455
455
Appendix
Table 5. Test dataset statistics.
Benchmark
Dataset Identifier
InfoVQA
NTT-hil-insight/OpenDocVQA ; images from OpenDocVQA-Corpus
ChartQA
NTT-hil-insight/OpenDocVQA ; images from OpenDocVQA-Corpus
SlideVQA
NTT-hil-insight/OpenDocVQA ; images from OpenDocVQA-Corpus
TQA
jinaai/tqa
OWID Charts
jinaai/owid_charts_en
Wikimedia Maps
jinaai/wikimedia-commons-maps
Appendix
Table 6. Sources of the six evaluation benchmarks. All identifiers refer to Hugging Face datasets.
Configuration
Value
Main backbone
Qwen2.5-VL-7B-Instruct
Alternative backbone
Phi3V / DSE initialization
Training epochs
3
Optimizer
AdamW
Learning rate
1×10−4
Per-device batch size
8
Appendix
Table 7. Main training and inference configurations of ConceptFormer.
Category
Method
Checkpoint Identifier
Text retrievers
BM25
No model checkpoint
BGE-1.5
BAAI/bge-large-en-v1.5
E5-Mistral
intfloat/e5-mistral-7b-instruct
NV-Embed-v2
nvidia/NV-Embed-v2
Multimodal retrievers
CLIP
openai/clip-vit-large-patch14-336
E5-V
royokong/e5-v
Appendix
Table 8. Public checkpoints used for baseline evaluation. All identifiers refer to Hugging Face model repositories.
Figure 5. Prompt templates used in Evidence Extraction.
Mohamed bin Zayed University of Artificial Intelligence · Arizona State University · Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science