Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representations
Authors: Yibo Yan, Mingdong Ou, Yi Cao, Xin Zou, Shuliang Liu, Jiahao Huo, Yu Huang, James Kwok, Xuming Hu
Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Alibaba Cloud Computing · The Hong Kong University of Science and Technology
Harnessing the full potential of visually-rich documents requires retrieval systems that understand not just text, but intricate layouts, a core challenge in Visual Document Retrieval (VDR). The prevailing multi-vector architectures, while powerful, face a crucial storage bottleneck that current optimization strategies, such as embedding merging, pruning, or using abstract tokens, fail to resolve without compromising performance or ignoring vital layout cues. To address this, we introduce ColParse, a novel paradigm that leverages a document parsing model to generate a small set of layout-informed sub-image embeddings, which are then fused with a global page-level vector to create a compact and structurally-aware multi-vector representation. Extensive experiments demonstrate that our method reduces storage requirements by over 95% while simultaneously yielding significant performance gains across numerous benchmarks and base models. ColParse thus bridges the critical gap between the fine-grained accuracy of multi-vector retrieval and the practical demands of large-scale deployment, offering a new path towards efficient and interpretable multimodal information systems.
Figures & tables
Figure 1: Comparison of natural image retrieval versus VDR.
Figure 2: The illustration of a multi-vector VDR model and three primary optimization strategies for its efficiency bottleneck.
Figure 3: The simplified illustration of ColParse framework.
Figure 4: The performance comparison (evaluated by nDCG@5) between ColParse and baselines on five VDR benchmarks across ten mainstream single-vector multimodal retrieval models. Refer to Table 2 and Table 3 for detailed result records due to the space limit.
Figure 5: Variant study of ColParse and its variants.
Figure 6: The comparison of the average performance of ColParse across different balancing factors. The dash lines refer to the base results; and the star points refer to the best-performing balancing factors. See Figure 10 for the model-level comparisons.
Figure 7: Comparison of MinerU2.5 against its counterparts. The y-axis represents the overall score of OmniDocBench ( Ouyang et al., 2025 ) , the x-axis shows end-to-end throughput (Pages/Sec), and bubble size indicates the parameter size.
Model
Performance
Storage
Latency
GME-7b
79.50
1.00
0.30
+ ColParse
80.61 ↑1.11
5.90 ↑4.9
0.81 ↑0.51
ColQwen *
80.02 ↑0.52
768.00 ↑767
0.41 ↑0.11
Table 1: Efficiency analysis of ColParse on the best performing model GME-7B and its multi-vector counterpart (w/ original setting). Performance denotes the overall score of MMEB-visdoc, Storage refers to the number of vectors stored per document, and Latency represents the average encoding time per document. * denotes the multi-vector model is trained w/ aligned configuration. Orange arrows denote better and blue ones denote worse.
Figure 8: The illustration of a representative case.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Method
MMLongBench
ViDoRe-V1
Doc
Page
Avg.
Arxiv
DocV
InfoV
Shift
TabF
TatD
S-AI
S-En
S-HC
S-Gov
Avg.
VLM2Vec-V1-2B
25.62
26.23
25.93
17.80
13.98
39.41
9.18
36.32
10.56
16.39
15.96
23.56
24.11
20.73
s2m-add
21.54
15.08
18.31 ↓7.62
35.07
15.61
52.15
6.62
36.51
10.39
26.23
31.22
30.29
33.61
27.77 ↑7.04
s2m-mul
22.07
15.10
18.59 ↓7.34
34.91
16.12
52.61
6.57
36.65
10.34
23.90
30.89
28.88
33.60
27.45 ↑6.72
cl-t-c
16.57
10.59
13.58 ↓12.35
13.97
3.97
21.62
8.73
23.20
11.56
12.94
28.27
19.16
26.72
17.01 ↓3.72
cl-t-m
14.35
8.67
11.51 ↓14.42
16.47
2.98
26.97
16.27
18.48
8.59
14.40
23.03
18.71
13.10
15.90 ↓4.83
Appendix
Table 2: Performance comparison on MMLongBench and ViDoRe-V1 benchmarks. For each model block, we bold the best-performing optimization method in each column (except for the base result). The average scores for optimizations are shown with relative gains ( \color[rgb]{1,0.23,0.13}\uparrow\color[rgb]{0,0,0} / \color[rgb]{0.15,1,0.67}\downarrow\color[rgb]{0,0,0} ) compared to the base model.
Method
ViDoRe-V2
ViDoSeek
VisRAG
Bio-L
Eco-R
ESG-H
ESG-M
Avg.
Doc
Page
Avg.
Arxiv
Chart
InfoV
MP-Doc
Plot
Slide
Avg.
VLM2Vec-V1-2B
6.88
14.15
12.25
20.54
13.46
56.40
67.73
62.07
41.68
58.21
70.79
42.74
23.83
74.07
51.89
s2m-add
10.40
8.59
4.68
3.46
6.78 ↓6.68
40.78
31.23
36.01 ↓26.06
28.94
44.79
59.76
30.86
9.67
59.59
38.94 ↓12.95
s2m-mul
10.00
8.40
5.09
3.51
6.75 ↓6.71
40.63
31.00
35.82 ↓26.25
28.85
44.64
60.48
30.55
9.83
59.52
38.98 ↓12.91
cl-t-c
14.86
14.49
4.14
4.48
9.49 ↓3.97
42.17
31.65
36.91 ↓25.16
13.56
18.52
25.66
13.11
1.41
31.91
17.36 ↓34.53
cl-t-m
13.34
4.60
8.78
5.15
7.97 ↓5.49
36.71
26.80
31.76 ↓30.31
8.29
18.27
37.74
11.85
1.13
32.46
18.29 ↓33.60
Appendix
Table 3: Performance comparison on ViDoRe-V2, ViDoSeek, and VisRAG benchmarks. For each model block, we bold the best-performing optimization method in each column (except for the base result). The average scores for optimizations are shown with relative gains ( \color[rgb]{1,0.23,0.13}\uparrow\color[rgb]{0,0,0} / \color[rgb]{0.15,1,0.67}\downarrow\color[rgb]{0,0,0} ) compared to the base model.
Figure 9: The performance comparison (evaluated by nDCG@5) between ColParse and its variants on five VDR benchmarks across ten mainstream single-vector multimodal retrieval models. Refer to Table 4 and Table 5 for detailed result records due to the space limit.
Method
MMLongBench
ViDoRe-V1
Doc
Page
Avg.
Arxiv
DocV
InfoV
Shift
TabF
TatD
S-AI
S-En
S-HC
S-Gov
Avg.
VLM2Vec-V1-2B
25.62
26.23
25.93
17.80
13.98
39.41
9.18
36.32
10.56
16.39
15.96
23.56
24.11
20.73
single2multi
25.19
19.59
22.39 ↓3.54
38.78
16.38
56.94
12.48
41.15
9.98
38.16
46.42
42.66
40.94
34.39 ↑13.66
s2m-t-c
25.37
18.43
21.90 ↓4.03
34.95
15.04
55.09
15.31
40.07
9.78
32.83
43.07
44.84
37.31
32.83 ↑12.10
s2m-g-i
31.57
26.55
29.06 ↑3.13
48.84
23.86
67.91
39.85
62.15
14.07
55.74
64.41
61.67
60.83
49.93 ↑29.20
ColParse
34.31
29.83
32.07 ↑6.14
47.66
28.12
69.23
47.11
57.05
20.43
62.24
63.77
65.51
62.54
52.37 ↑31.64
Appendix
Table 4: Ablation study on MMLongBench and ViDoRe-V1 benchmarks. For each model block, we bold the best-performing method in each column (except for the base result). The average scores are shown with relative gains ( \color[rgb]{1,0.23,0.13}\uparrow\color[rgb]{0,0,0} / \color[rgb]{0.15,1,0.67}\downarrow\color[rgb]{0,0,0} ) compared to the base model.
Method
ViDoRe-V2
ViDoSeek
VisRAG
Bio-L
Eco-R
ESG-H
ESG-M
Avg.
Doc
Page
Avg.
Arxiv
Chart
InfoV
MP-Doc
Plot
Slide
Avg.
VLM2Vec-V1-2B
6.88
14.15
12.25
20.54
13.46
56.40
67.73
62.07
41.68
58.21
70.79
42.74
23.83
74.07
51.89
single2multi
17.15
18.19
10.24
7.44
13.26 ↓0.20
50.51
41.52
46.02 ↓16.05
30.17
42.91
61.28
37.60
6.80
65.76
40.75 ↓11.14
s2m-t-c
13.46
18.31
11.58
8.85
13.05 ↓0.41
50.22
40.75
45.49 ↓16.58
27.87
41.64
59.37
34.82
6.94
62.76
38.90 ↓12.99
s2m-g-i
28.49
26.29
29.18
39.15
30.78 ↑17.32
73.38
68.14
70.76 ↑8.69
40.39
52.54
70.43
47.82
8.96
76.40
49.42 ↓2.47
ColParse
30.33
29.55
33.21
38.33
32.86 ↑19.40
75.23
70.19
72.71 ↑10.64
38.18
60.09
69.44
48.29
18.83
76.95
51.96 ↑0.07
Appendix
Table 5: Ablation study on ViDoRe-V2, ViDoSeek, and VisRAG benchmarks. For each model block, we bold the best-performing method in each column (except for the base result). The average scores are shown with relative gains ( \color[rgb]{1,0.23,0.13}\uparrow\color[rgb]{0,0,0} / \color[rgb]{0.15,1,0.67}\downarrow\color[rgb]{0,0,0} ) compared to the base model.
Figure 10: The comparison of the model-level performance using ColParse across difference balancing factors. The dash lines refer to the base results; and the star points refer to the best-performing balancing factors.
Model
Para.
Tokens/sec
Pages/sec
MinerU2-VLM
0.9B
3089
2.84
dots.ocr
3.0B
311
0.28
MonkeyOCR-pro-3B
3.7B
520
0.47
MonkeyOCR-pro-1.2B
1.9B
589
0.53
Nanonets-OCR-s
3.7B
605
0.55
MinerU2.5
1.2B
2422
2.25
Appendix
Table 6: Inference efficiency comparison of MinerU2.5. The results for MinerU2.5 and baselines are tested on the A100(80G) machine. The best and runner-up results in each column are bolded and underlined , respectively.
Methods
Para.
Overall ↑
Text Edit ↓
Formula CDM ↑
Table TEDS ↑
Table TEDS-S ↑
Read Order Edit ↓
MinerU2-VLM
0.9B
85.56
0.078
80.95
83.54
87.66
0.086
dots.ocr
3B
88.41
0.048
83.22
86.78
90.62
0.053
MonkeyOCR-pro-3B
3.7B
88.85
0.075
87.25
86.78
90.63
0.128
MonkeyOCR-pro-1.2B
1.9B
86.96
0.084
85.02
84.24
89.02
0.130
Nanonets-OCR-s
3.7B
85.59
0.093
85.90
80.14
85.57
0.108
MinerU2.5
1.2B
90.67
0.047
88.46
88.22
92.38
0.044
Appendix
Table 7: Document parsing performance on OmniDocBench ( Ouyang et al., 2025 ) across multiple tasks. The best and runner-up results are bolded and underlined , respectively.
Benchmark
Dataset
Avg. #Vectors ( k )
MMLongBench
MMLongBench-doc
6.04
MMLongBench-page
6.04
ViDoSeek
ViDoSeek-doc
5.60
ViDoSeek-page
5.77
VisRAG
VisRAG_ArxivQA
1.97
VisRAG_ChartQA
2.98
Appendix
Table 8: Summary of the average number of parsed vectors per document across 24 datasets in five VDR benchmarks. The values represent the number of layout-informed sub-image embeddings ( k ) generated by the document parser (MinerU2.5).
Multi-vector models dominate Visual Document Retrieval (VDR) due to their fine-grained matching capabilities, but their high storage and computational costs present a major barrier to practical deployment. In this paper, we propose ColChunk, a plug-and-play framework that introduces multimodal late chunking to construct efficient, contextualized multi-vectors. Unlike existing pruning or fixed-token approaches, ColChunk employs hierarchical clustering on patch-level embeddings, fused with a 2D position prior to ensure spatial-semantic coherence. This adaptive grouping allows for a content-aware representation that preserves global context while drastically reducing the vector count. Evaluations across 24 VDR datasets demonstrate ColChunk achieves over a 90% reduction in storage requirements while simultaneously delivering a 9-point average improvement in nDCG@5 across representative single-vector models. ColChunk provides a practical solution for balancing retrieval accuracy and efficiency in visual document systems.
Yibo Yan, Mingdong Ou, Yi Cao +5
Hong Kong University of Science and Technology (Guangzhou) · Alibaba Cloud Computing · Hong Kong University of Science and Technology
Visual Document Retrieval (VDR), which aims to retrieve relevant pages within vast corpora of visually-rich documents, is of significance in current multimodal retrieval applications. The state-of-the-art multi-vector paradigm excels in performance but suffers from prohibitive overhead, a problem that current efficiency methods like pruning and merging address imperfectly, creating a difficult trade-off between compression rate and feature fidelity. To overcome this dilemma, we introduce Prune-then-Merge, a novel two-stage framework that synergizes these complementary approaches. Our method first employs an adaptive pruning stage to filter out low-information patches, creating a refined, high-signal set of embeddings. Subsequently, a hierarchical merging stage compresses this pre-filtered set, effectively summarizing semantic content without the noise-induced feature dilution seen in single-stage methods. Extensive experiments on 29 VDR datasets demonstrate that our framework consistently outperforms existing methods, significantly extending the near-lossless compression range and providing robust performance at high compression ratios.
Yibo Yan, Mingdong Ou, Yi Cao +5
Hong Kong University of Science and Technology (Guangzhou) · Alibaba Cloud Computing · Hong Kong University of Science and Technology
Visual Document Retrieval (VDR) models mostly rely on late interaction architectures, in which documents are represented by a set of local patch embeddings and then matched against query tokens. While efficient, this architecture prioritizes local similarity over global layout structure of documents to estimate relevancy between documents and query. In practice, this leads to errors as relevance originates from layout structure of documents with heterogeneous layouts combining figures, tables, and text. We make document layout learnable without changing inference. We propose a multimodal encoder that augments local patch representations with a global layout embedding, trained via textual descriptions encoding document layout information. Across four ViDoRe-v2 datasets, our model improves over the strongest architecturally comparable ColPali/ColQwen baseline by +2.4 nDCG@5 and +2.3 MAP@5, with statistically significant per-dataset gains over ColQwen.
Pascal Tilli, Mohsen Mesgar
University of Stuttgart, Stuttgart, Germany · Bosch Center for Artificial Intelligence, Renningen, Germany