Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representations

Yibo Yan; Mingdong Ou; Yi Cao; Xin Zou; Shuliang Liu; Jiahao Huo; Yu Huang; James Kwok

cs.CLMar 2, 2026

Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representations

Authors: Yibo Yan, Mingdong Ou, Yi Cao, Xin Zou, Shuliang Liu, Jiahao Huo, Yu Huang, James Kwok,

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Alibaba Cloud Computing · The Hong Kong University of Science and Technology

Abstract

Harnessing the full potential of visually-rich documents requires retrieval systems that understand not just text, but intricate layouts, a core challenge in Visual Document Retrieval (VDR). The prevailing multi-vector architectures, while powerful, face a crucial storage bottleneck that current optimization strategies, such as embedding merging, pruning, or using abstract tokens, fail to resolve without compromising performance or ignoring vital layout cues. To address this, we introduce ColParse, a novel paradigm that leverages a document parsing model to generate a small set of layout-informed sub-image embeddings, which are then fused with a global page-level vector to create a compact and structurally-aware multi-vector representation. Extensive experiments demonstrate that our method reduces storage requirements by over 95% while simultaneously yielding significant performance gains across numerous benchmarks and base models. ColParse thus bridges the critical gap between the fine-grained accuracy of multi-vector retrieval and the practical demands of large-scale deployment, offering a new path towards efficient and interpretable multimodal information systems.

Figures & tables

Figure 1: Comparison of natural image retrieval versus VDR.

Figure 2: The illustration of a multi-vector VDR model and three primary optimization strategies for its efficiency bottleneck.

Figure 3: The simplified illustration of ColParse framework.

Figure 4: The performance comparison (evaluated by nDCG@5) between ColParse and baselines on five VDR benchmarks across ten mainstream single-vector multimodal retrieval models. Refer to Table 2 and Table 3 for detailed result records due to the space limit.

Figure 5: Variant study of ColParse and its variants.

Figure 6: The comparison of the average performance of ColParse across different balancing factors. The dash lines refer to the base results; and the star points refer to the best-performing balancing factors. See Figure 10 for the model-level comparisons.

Figure 7: Comparison of MinerU2.5 against its counterparts. The y-axis represents the overall score of OmniDocBench ( Ouyang et al., 2025 ) , the x-axis shows end-to-end throughput (Pages/Sec), and bubble size indicates the parameter size.

Model	Performance	Storage	Latency
GME-7b	79.50	1.00	0.30
+ ColParse	80.61 ↑1.11	5.90 ↑4.9	0.81 ↑0.51
ColQwen *	80.02 ↑0.52	768.00 ↑767	0.41 ↑0.11

Table 1: Efficiency analysis of ColParse on the best performing model GME-7B and its multi-vector counterpart (w/ original setting). Performance denotes the overall score of MMEB-visdoc, Storage refers to the number of vectors stored per document, and Latency represents the average encoding time per document. * denotes the multi-vector model is trained w/ aligned configuration. Orange arrows denote better and blue ones denote worse.

Figure 8: The illustration of a representative case.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Method	MMLongBench			ViDoRe-V1
	Doc	Page	Avg.	Arxiv	DocV	InfoV	Shift	TabF	TatD	S-AI	S-En	S-HC	S-Gov	Avg.
VLM2Vec-V1-2B	25.62	26.23	25.93	17.80	13.98	39.41	9.18	36.32	10.56	16.39	15.96	23.56	24.11	20.73
s2m-add	21.54	15.08	18.31 ↓7.62	35.07	15.61	52.15	6.62	36.51	10.39	26.23	31.22	30.29	33.61	27.77 ↑7.04
s2m-mul	22.07	15.10	18.59 ↓7.34	34.91	16.12	52.61	6.57	36.65	10.34	23.90	30.89	28.88	33.60	27.45 ↑6.72
cl-t-c	16.57	10.59	13.58 ↓12.35	13.97	3.97	21.62	8.73	23.20	11.56	12.94	28.27	19.16	26.72	17.01 ↓3.72
cl-t-m	14.35	8.67	11.51 ↓14.42	16.47	2.98	26.97	16.27	18.48	8.59	14.40	23.03	18.71	13.10	15.90 ↓4.83

Appendix

Table 2: Performance comparison on MMLongBench and ViDoRe-V1 benchmarks. For each model block, we bold the best-performing optimization method in each column (except for the base result). The average scores for optimizations are shown with relative gains (

\color[rgb]{1,0.23,0.13}\uparrow\color[rgb]{0,0,0}

\color[rgb]{0.15,1,0.67}\downarrow\color[rgb]{0,0,0}

) compared to the base model.

Method	ViDoRe-V2					ViDoSeek			VisRAG
	Bio-L	Eco-R	ESG-H	ESG-M	Avg.	Doc	Page	Avg.	Arxiv	Chart	InfoV	MP-Doc	Plot	Slide	Avg.
VLM2Vec-V1-2B	6.88	14.15	12.25	20.54	13.46	56.40	67.73	62.07	41.68	58.21	70.79	42.74	23.83	74.07	51.89
s2m-add	10.40	8.59	4.68	3.46	6.78 ↓6.68	40.78	31.23	36.01 ↓26.06	28.94	44.79	59.76	30.86	9.67	59.59	38.94 ↓12.95
s2m-mul	10.00	8.40	5.09	3.51	6.75 ↓6.71	40.63	31.00	35.82 ↓26.25	28.85	44.64	60.48	30.55	9.83	59.52	38.98 ↓12.91
cl-t-c	14.86	14.49	4.14	4.48	9.49 ↓3.97	42.17	31.65	36.91 ↓25.16	13.56	18.52	25.66	13.11	1.41	31.91	17.36 ↓34.53
cl-t-m	13.34	4.60	8.78	5.15	7.97 ↓5.49	36.71	26.80	31.76 ↓30.31	8.29	18.27	37.74	11.85	1.13	32.46	18.29 ↓33.60

Appendix

Table 3: Performance comparison on ViDoRe-V2, ViDoSeek, and VisRAG benchmarks. For each model block, we bold the best-performing optimization method in each column (except for the base result). The average scores for optimizations are shown with relative gains (

\color[rgb]{1,0.23,0.13}\uparrow\color[rgb]{0,0,0}

\color[rgb]{0.15,1,0.67}\downarrow\color[rgb]{0,0,0}

) compared to the base model.

Figure 9: The performance comparison (evaluated by nDCG@5) between ColParse and its variants on five VDR benchmarks across ten mainstream single-vector multimodal retrieval models. Refer to Table 4 and Table 5 for detailed result records due to the space limit.

Method	MMLongBench			ViDoRe-V1
	Doc	Page	Avg.	Arxiv	DocV	InfoV	Shift	TabF	TatD	S-AI	S-En	S-HC	S-Gov	Avg.
VLM2Vec-V1-2B	25.62	26.23	25.93	17.80	13.98	39.41	9.18	36.32	10.56	16.39	15.96	23.56	24.11	20.73
single2multi	25.19	19.59	22.39 ↓3.54	38.78	16.38	56.94	12.48	41.15	9.98	38.16	46.42	42.66	40.94	34.39 ↑13.66
s2m-t-c	25.37	18.43	21.90 ↓4.03	34.95	15.04	55.09	15.31	40.07	9.78	32.83	43.07	44.84	37.31	32.83 ↑12.10
s2m-g-i	31.57	26.55	29.06 ↑3.13	48.84	23.86	67.91	39.85	62.15	14.07	55.74	64.41	61.67	60.83	49.93 ↑29.20
ColParse	34.31	29.83	32.07 ↑6.14	47.66	28.12	69.23	47.11	57.05	20.43	62.24	63.77	65.51	62.54	52.37 ↑31.64

Appendix

Table 4: Ablation study on MMLongBench and ViDoRe-V1 benchmarks. For each model block, we bold the best-performing method in each column (except for the base result). The average scores are shown with relative gains (

\color[rgb]{1,0.23,0.13}\uparrow\color[rgb]{0,0,0}

\color[rgb]{0.15,1,0.67}\downarrow\color[rgb]{0,0,0}

) compared to the base model.

Method	ViDoRe-V2					ViDoSeek			VisRAG
	Bio-L	Eco-R	ESG-H	ESG-M	Avg.	Doc	Page	Avg.	Arxiv	Chart	InfoV	MP-Doc	Plot	Slide	Avg.
VLM2Vec-V1-2B	6.88	14.15	12.25	20.54	13.46	56.40	67.73	62.07	41.68	58.21	70.79	42.74	23.83	74.07	51.89
single2multi	17.15	18.19	10.24	7.44	13.26 ↓0.20	50.51	41.52	46.02 ↓16.05	30.17	42.91	61.28	37.60	6.80	65.76	40.75 ↓11.14
s2m-t-c	13.46	18.31	11.58	8.85	13.05 ↓0.41	50.22	40.75	45.49 ↓16.58	27.87	41.64	59.37	34.82	6.94	62.76	38.90 ↓12.99
s2m-g-i	28.49	26.29	29.18	39.15	30.78 ↑17.32	73.38	68.14	70.76 ↑8.69	40.39	52.54	70.43	47.82	8.96	76.40	49.42 ↓2.47
ColParse	30.33	29.55	33.21	38.33	32.86 ↑19.40	75.23	70.19	72.71 ↑10.64	38.18	60.09	69.44	48.29	18.83	76.95	51.96 ↑0.07

Appendix

Table 5: Ablation study on ViDoRe-V2, ViDoSeek, and VisRAG benchmarks. For each model block, we bold the best-performing method in each column (except for the base result). The average scores are shown with relative gains (

\color[rgb]{1,0.23,0.13}\uparrow\color[rgb]{0,0,0}

\color[rgb]{0.15,1,0.67}\downarrow\color[rgb]{0,0,0}

) compared to the base model.

Figure 10: The comparison of the model-level performance using ColParse across difference balancing factors. The dash lines refer to the base results; and the star points refer to the best-performing balancing factors.

Model	Para.	Tokens/sec	Pages/sec
MinerU2-VLM	0.9B	3089	2.84
dots.ocr	3.0B	311	0.28
MonkeyOCR-pro-3B	3.7B	520	0.47
MonkeyOCR-pro-1.2B	1.9B	589	0.53
Nanonets-OCR-s	3.7B	605	0.55
MinerU2.5	1.2B	2422	2.25

Appendix

Table 6: Inference efficiency comparison of MinerU2.5. The results for MinerU2.5 and baselines are tested on the A100(80G) machine. The best and runner-up results in each column are bolded and underlined , respectively.

Methods	Para.	Overall ↑	Text Edit ↓	Formula CDM ↑	Table TEDS ↑	Table TEDS-S ↑	Read Order Edit ↓
MinerU2-VLM	0.9B	$85.56$	$0.078$	$80.95$	$83.54$	$87.66$	$0.086$
dots.ocr	3B	$88.41$	$0.048$	$83.22$	$86.78$	$90.62$	$0.053$
MonkeyOCR-pro-3B	3.7B	$88.85$	$0.075$	$87.25$	$86.78$	$90.63$	$0.128$
MonkeyOCR-pro-1.2B	1.9B	$86.96$	$0.084$	$85.02$	$84.24$	$89.02$	$0.130$
Nanonets-OCR-s	3.7B	$85.59$	$0.093$	$85.90$	$80.14$	$85.57$	$0.108$
MinerU2.5	1.2B	$\mathbf{90.67}$	$\mathbf{0.047}$	$\mathbf{88.46}$	$\mathbf{88.22}$	$\mathbf{92.38}$	$\mathbf{0.044}$

Appendix

Table 7: Document parsing performance on OmniDocBench ( Ouyang et al., 2025 ) across multiple tasks. The best and runner-up results are bolded and underlined , respectively.

Benchmark	Dataset	Avg. #Vectors ( $k$ )
MMLongBench	MMLongBench-doc	6.04
MMLongBench	MMLongBench-page	6.04
ViDoSeek	ViDoSeek-doc	5.60
ViDoSeek	ViDoSeek-page	5.77
VisRAG	VisRAG_ArxivQA	1.97
	VisRAG_ChartQA	2.98

Appendix

Table 8: Summary of the average number of parsed vectors per document across 24 datasets in five VDR benchmarks. The values represent the number of layout-informed sub-image embeddings (

k

) generated by the document parser (MinerU2.5).

Visual Document Retrieval

Visual document retrieval finds relevant pages or regions in image-based documents using visual and multimodal representations.

See all 66 papers

Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrieval
Apr 11, 2026Yibo Yan, Mingdong Ou, Yi Cao +5Visual Document Retrieval Multi-Vector Retrieval
Hong Kong University of Science and Technology (Guangzhou) · Alibaba Cloud Computing · Hong Kong University of Science and Technology
Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework
Feb 23, 2026Yibo Yan, Mingdong Ou, Yi Cao +5Visual Document Retrieval Multi-Vector Retrieval
Hong Kong University of Science and Technology (Guangzhou) · Alibaba Cloud Computing · Hong Kong University of Science and Technology
Beyond Bag-of-Patches: Learning Global Layout via Textual Supervision for Late-Interaction Visual Document Retrieval
May 8, 2026Pascal Tilli, Mohsen MesgarMultimodal Embedding Multi-Vector Retrieval
University of Stuttgart, Stuttgart, Germany · Bosch Center for Artificial Intelligence, Renningen, Germany

Apr 11, 2026J/K move · Enter open · S save

Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrieval

Yibo Yan, Mingdong Ou, Yi Cao, Jiahao Huo+4

Hong Kong University of Science and Technology (Guangzhou) · Alibaba Cloud Computing · Hong Kong University of Science and Technology

Visual Document Retrieval Multi-Vector Retrieval Document Chunking Visual Token Compression

Open paper AskPDF arXiv

Multi-vector models dominate Visual Document Retrieval (VDR) due to their fine-grained matching capabilities, but their high storage and computational costs present a major barrier to practical deployment. In this paper, we propose ColChunk, a plug-and-play framework that introduces multimodal late chunking to construct efficient, contextualized multi-vectors. Unlike existing pruning or fixed-token approaches, ColChunk employs hierarchical clustering on patch-level embeddings, fused with a 2D position prior to ensure spatial-semantic coherence. This adaptive grouping allows for a content-aware representation that preserves global context while drastically reducing the vector count. Evaluations across 24 VDR datasets demonstrate ColChunk achieves over a 90% reduction in storage requirements while simultaneously delivering a 9-point average improvement in nDCG@5 across representative single-vector models. ColChunk provides a practical solution for balancing retrieval accuracy and efficiency in visual document systems.

Abstract

Visual Document Retrieval

Explore similar work

Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrieval

Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework

Beyond Bag-of-Patches: Learning Global Layout via Textual Supervision for Late-Interaction Visual Document Retrieval