Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework

Yibo Yan; Mingdong Ou; Yi Cao; Xin Zou; Jiahao Huo; Shuliang Liu; James Kwok; Xuming Hu

cs.CLFeb 23, 2026

Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework

Authors: Yibo Yan, Mingdong Ou, Yi Cao, Xin Zou, Jiahao Huo, Shuliang Liu, James Kwok, Xuming Hu

Organizations: Hong Kong University of Science and Technology (Guangzhou) · Alibaba Cloud Computing · Hong Kong University of Science and Technology

Abstract

Visual Document Retrieval (VDR), which aims to retrieve relevant pages within vast corpora of visually-rich documents, is of significance in current multimodal retrieval applications. The state-of-the-art multi-vector paradigm excels in performance but suffers from prohibitive overhead, a problem that current efficiency methods like pruning and merging address imperfectly, creating a difficult trade-off between compression rate and feature fidelity. To overcome this dilemma, we introduce Prune-then-Merge, a novel two-stage framework that synergizes these complementary approaches. Our method first employs an adaptive pruning stage to filter out low-information patches, creating a refined, high-signal set of embeddings. Subsequently, a hierarchical merging stage compresses this pre-filtered set, effectively summarizing semantic content without the noise-induced feature dilution seen in single-stage methods. Extensive experiments on 29 VDR datasets demonstrate that our framework consistently outperforms existing methods, significantly extending the near-lossless compression range and providing robust performance at high compression ratios.

Figures & tables

Figure 1: Comparison of single-vec vs. multi-vec VDR.

Figure 2: Comparison of pruning-based vs. merging-based efficient VDR paradigms.

Figure 3: Performance comparison (nDCG@5) between Prune-then-Merge and baselines on ViDoRe-V1 ( Faysse et al., 2024 ) across Jina-v4 ( Left ), ColQwen2.5 ( Middle ), and ColNomic ( Right ). solid lines denote adaptive methods, whereas dashed lines denote non-adaptive ones; circular nodes represent pruning methods, whereas square nodes represent merging ones.

Figure 4: Performance comparison (nDCG@5) between Prune-then-Merge and baselines on JinaVDR ( Günther et al., 2025 ) across Jina-v4 ( Left ), ColQwen2.5 ( Middle ), and ColNomic ( Right ). solid lines denote adaptive methods, whereas dashed lines denote non-adaptive ones; circular nodes represent pruning methods, whereas square nodes represent merging ones.

Figure 5: Performance comparison (nDCG@5) between Prune-then-Merge and baselines on REAL-MM-RAG ( Wasserman et al., 2025 ) across Jina-v4 ( Left ), ColQwen2.5 ( Middle ) & ColNomic ( Right ). solid lines denote adaptive methods, whereas dashed lines denote non-adaptive ones; circular nodes represent pruning methods, whereas square nodes represent merging ones.

Figure 6: Variant comparisons of Jina-v4 on ViDoRe-V2.

$\Delta$	ColQwen	ColNomic	JinaV4
nDCG@5	$\downarrow$ 0.27%	$\downarrow$ 0.51%	$\downarrow$ 0.57%
Storage	$\downarrow$ 58.88%	$\downarrow$ 52.16%	$\downarrow$ 52.77%
Latency	$\uparrow$ 56.10%	$\uparrow$ 51.11%	$\uparrow$ 47.06%

Table 1: Relative improvement of Prune-then-Merge wrt. performance , storage , and latency to base models on ViDoRe-V1 (We set adaptation factor as -0.75 and merging factor as 4; orange denotes better and green denotes worse).

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Illustrative examples of ViDoRe-V1 benchmark Faysse et al. (2024) .

Figure 8: Illustrative examples of ViDoRe-V2 benchmark Macé et al. (2025) .

Figure 9: Illustrative examples of JinaVDR benchmark Günther et al. (2025) .

Figure 10: Illustrative examples of Real-MM-RAG benchmark Wasserman et al. (2025) .

Figure 11: Illustrative example of pruned visual documents by different pruning ratios.

Figure 12: Performance comparison (nDCG@5) between Prune-then-Merge and baselines on ViDoRe-V2 ( Macé et al., 2025 ) across Jina-v4 ( Left ), ColQwen2.5 ( Middle ), and ColNomic ( Right ). solid lines denote adaptive methods, whereas dashed lines denote non-adaptive ones; circular nodes represent pruning methods, whereas square nodes represent merging ones.

Figure 13: Performance comparison (nDCG@5) between Prune-then-Merge and baselines on ViDoSeek ( Wang et al., 2025b ) across Jina-v4 ( Left ), ColQwen2.5 ( Middle ), and ColNomic ( Right ). solid lines denote adaptive methods, whereas dashed lines denote non-adaptive ones; circular nodes represent pruning methods, whereas square nodes represent merging ones.

Figure 14: Performance comparison (nDCG@5) between Prune-then-Merge and baselines on MMLongBench-Doc ( Ma et al., 2024b ) across Jina-v4 ( Left ), ColQwen2.5 ( Middle ), and ColNomic ( Right ). solid lines denote adaptive methods, whereas dashed lines denote non-adaptive ones; circular nodes represent pruning methods, whereas square nodes represent merging ones.

Figure 15: Variant comparison of Jina-v4 on ESG dataset of ViDoRe-V2. solid lines denote adaptive methods, whereas dashed lines denote non-adaptive ones.

Figure 16: Variant comparison of Jina-v4 on Biomedical dataset of ViDoRe-V2. solid lines denote adaptive methods, whereas dashed lines denote non-adaptive ones.

Figure 17: Variant comparison of Jina-v4 on Economic dataset of ViDoRe-V2. solid lines denote adaptive methods, whereas dashed lines denote non-adaptive ones.

Figure 18: Variant comparison of Jina-v4 on ESG Human dataset of ViDoRe-V2. solid lines denote adaptive methods, whereas dashed lines denote non-adaptive ones.

		ColQwen2.5	ColNomic	JinaV4	Overall
Performance (nDCG@5)	base	0.8795	0.8999	0.8777	0.8857
	Prune-then-Merge	0.8771	0.8953	0.8727	0.8817
	$\Delta$	0.27%	0.51%	0.57%	0.45%
Storage	base	1.0000	1.0000	1.0000	1.0000
	Prune-then-Merge	0.4112	0.4784	0.4723	0.4500
	$\Delta$	58.88%	52.16%	52.77%	54.60%

Appendix

Table 2: Comparison of our Prune-then-Merge with base models on ViDoRe-V1, in terms of performance , storage , and latency . The table presents absolute values and the relative change (

\Delta

). (We set adaptation factor as -0.75 and merging factor as 4; orange denotes a better outcome and green denotes a worse outcome.)

Type	Methods	Adp.	Para	all		arxqa		docqa		infovqa		tabfquad		tatdqa		shiftproject		synth_ai		synth_energy		synth_gov		synth_health
				ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%
pruning	base		-	0.87	-	0.86	-	0.60	-	0.92	-	0.90	-	0.76	-	0.86	-	0.98	-	0.95	-	0.96	-	0.96	-
pruning	random		ratio=0.1	0.87	0.10	0.86	0.10	0.60	0.10	0.92	0.10	0.91	0.10	0.76	0.10	0.85	0.10	0.98	0.10	0.95	0.10	0.95	0.10	0.97	0.10
pruning	random		ratio=0.3	0.87	0.30	0.86	0.30	0.60	0.30	0.91	0.30	0.91	0.30	0.76	0.30	0.84	0.30	0.98	0.30	0.95	0.30	0.96	0.30	0.97	0.30
pruning	random		ratio=0.5	0.87	0.50	0.86	0.50	0.60	0.50	0.91	0.50	0.91	0.50	0.76	0.50	0.82	0.50	0.98	0.50	0.96	0.50	0.95	0.50	0.97	0.50
pruning	random		ratio=0.7	0.86	0.70	0.85	0.70	0.58	0.70	0.91	0.70	0.91	0.70	0.75	0.70	0.79	0.70	0.97	0.70	0.95	0.70	0.96	0.70	0.97	0.70

Appendix

Table 3: Experimental results of ColQwen on ViDoRe-V1.

Type	Methods	Adp.	Para	all		arxqa		docqa		infovqa		tabfquad		tatdqa		shiftproject		synth_ai		synth_energy		synth_gov		synth_health
				ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%
pruning	base		-	0.89	-	0.87	-	0.60	-	0.93	-	0.94	-	0.82	-	0.91	-	0.98	-	0.97	-	0.97	-	0.98	-
pruning	random		ratio=0.1	0.89	0.10	0.87	0.10	0.60	0.10	0.92	0.10	0.94	0.10	0.82	0.10	0.90	0.10	0.98	0.10	0.96	0.10	0.97	0.10	0.98	0.10
pruning	random		ratio=0.3	0.89	0.30	0.87	0.30	0.60	0.30	0.92	0.30	0.94	0.30	0.81	0.30	0.90	0.30	0.98	0.30	0.96	0.30	0.97	0.30	0.99	0.30
pruning	random		ratio=0.5	0.89	0.50	0.87	0.50	0.59	0.50	0.92	0.50	0.94	0.50	0.81	0.50	0.88	0.50	0.98	0.50	0.97	0.50	0.97	0.50	0.99	0.50
pruning	random		ratio=0.7	0.88	0.70	0.86	0.70	0.58	0.70	0.91	0.70	0.94	0.70	0.80	0.70	0.85	0.70	0.98	0.70	0.96	0.70	0.97	0.70	0.98	0.70

Appendix

Table 4: Experimental results of ColNomic on ViDoRe-V1.

Type	Methods	Adp.	Para	all		arxqa		docqa		infovqa		tabfquad		tatdqa		shiftproject		synth_ai		synth_energy		synth_gov		synth_health
				ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%
pruning	base		-	0.87	-	0.86	-	0.58	-	0.90	-	0.92	-	0.72	-	0.88	-	0.98	-	0.95	-	0.96	-	0.97	-
pruning	random		ratio=0.1	0.87	0.10	0.86	0.10	0.58	0.10	0.90	0.10	0.92	0.10	0.72	0.10	0.88	0.10	0.98	0.10	0.96	0.10	0.96	0.10	0.97	0.10
pruning	random		ratio=0.3	0.87	0.30	0.86	0.30	0.58	0.30	0.89	0.30	0.92	0.30	0.72	0.30	0.88	0.30	0.98	0.30	0.95	0.30	0.96	0.30	0.97	0.30
pruning	random		ratio=0.5	0.87	0.50	0.85	0.50	0.57	0.50	0.89	0.50	0.92	0.50	0.72	0.50	0.86	0.50	0.98	0.50	0.95	0.50	0.96	0.50	0.96	0.50
pruning	random		ratio=0.7	0.85	0.70	0.85	0.70	0.54	0.70	0.87	0.70	0.92	0.70	0.70	0.70	0.84	0.70	0.97	0.70	0.93	0.70	0.97	0.70	0.97	0.70

Appendix

Table 5: Experimental results of Jina-v4 on ViDoRe-V1.

Type	Methods	Adp.	Para	all		esg		biomedical		economics		esg_human
				ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%
pruning	base		-	0.55	-	0.51	-	0.58	-	0.49	-	0.61	-
pruning	random		ratio=0.1	0.53	0.10	0.49	0.10	0.56	0.10	0.49	0.10	0.59	0.10
pruning	random		ratio=0.3	0.50	0.30	0.45	0.30	0.55	0.30	0.49	0.30	0.50	0.30
pruning	random		ratio=0.5	0.46	0.50	0.43	0.50	0.54	0.50	0.45	0.50	0.44	0.50
pruning	random		ratio=0.7	0.45	0.70	0.39	0.70	0.53	0.70	0.47	0.70	0.40	0.70

Appendix

Table 6: Experimental results of ColQwen on ViDoRe-V2.

Type	Methods	Adp.	Para	all		esg		biomedical		economics		esg_human
				ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%
pruning	base		-	0.59	-	0.55	-	0.63	-	0.55	-	0.63	-
pruning	random		ratio=0.1	0.59	0.10	0.54	0.10	0.62	0.10	0.55	0.10	0.65	0.10
pruning	random		ratio=0.3	0.58	0.30	0.52	0.30	0.62	0.30	0.54	0.30	0.65	0.30
pruning	random		ratio=0.5	0.57	0.50	0.50	0.50	0.62	0.50	0.55	0.50	0.61	0.50
pruning	random		ratio=0.7	0.54	0.70	0.45	0.70	0.60	0.70	0.54	0.70	0.57	0.70

Appendix

Table 7: Experimental results of ColNomic on ViDoRe-V2.

Type	Methods	Adp.	Para	all		esg		biomedical		economics		esg_human
				ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%
pruning	base		-	0.56	-	0.50	-	0.60	-	0.54	-	0.62	-
pruning	random		ratio=0.1	0.57	0.10	0.51	0.10	0.59	0.10	0.54	0.10	0.62	0.10
pruning	random		ratio=0.3	0.56	0.30	0.49	0.30	0.59	0.30	0.54	0.30	0.62	0.30
pruning	random		ratio=0.5	0.56	0.50	0.48	0.50	0.60	0.50	0.55	0.50	0.61	0.50
pruning	random		ratio=0.7	0.52	0.70	0.44	0.70	0.58	0.70	0.51	0.70	0.56	0.70

Appendix

Table 8: Experimental results of Jina-v4 on ViDoRe-V2.

Type	Methods	Adp.	Para	all		arabic		japanese		russian		german		spanish		italian		dutch		hindi		chinese
				ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%
pruning	base		-	0.54	-	0.46	-	0.74	-	0.74	-	0.35	-	0.56	-	0.61	-	0.44	-	0.07	-	0.90	-
pruning	random		ratio=0.1	0.54	0.10	0.45	0.10	0.73	0.10	0.75	0.10	0.35	0.10	0.57	0.10	0.61	0.10	0.44	0.10	0.07	0.10	0.90	0.10
pruning	random		ratio=0.3	0.54	0.30	0.46	0.30	0.72	0.30	0.75	0.30	0.34	0.30	0.57	0.30	0.61	0.30	0.44	0.30	0.07	0.30	0.89	0.30
pruning	random		ratio=0.5	0.53	0.50	0.46	0.50	0.73	0.50	0.72	0.50	0.33	0.50	0.55	0.50	0.61	0.50	0.46	0.50	0.07	0.50	0.89	0.50
pruning	random		ratio=0.7	0.52	0.70	0.44	0.70	0.69	0.70	0.74	0.70	0.31	0.70	0.53	0.70	0.59	0.70	0.43	0.70	0.06	0.70	0.88	0.70

Appendix

Table 9: Experimental results of ColQwen on JinaVDR.

Type	Methods	Adp.	Para	all		arabic		japanese		russian		german		spanish		italian		dutch		hindi		chinese
				ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%
pruning	base		-	0.72	-	0.91	-	0.83	-	0.86	-	0.59	-	0.79	-	0.75	-	0.61	-	0.21	-	0.95	-
pruning	random		ratio=0.1	0.72	0.10	0.92	0.10	0.83	0.10	0.87	0.10	0.59	0.10	0.79	0.10	0.74	0.10	0.62	0.10	0.21	0.10	0.95	0.10
pruning	random		ratio=0.3	0.72	0.30	0.91	0.30	0.85	0.30	0.86	0.30	0.59	0.30	0.79	0.30	0.74	0.30	0.63	0.30	0.21	0.30	0.95	0.30
pruning	random		ratio=0.5	0.72	0.50	0.90	0.50	0.86	0.50	0.87	0.50	0.58	0.50	0.77	0.50	0.74	0.50	0.61	0.50	0.21	0.50	0.94	0.50
pruning	random		ratio=0.7	0.71	0.70	0.90	0.70	0.85	0.70	0.87	0.70	0.55	0.70	0.76	0.70	0.73	0.70	0.61	0.70	0.20	0.70	0.96	0.70

Appendix

Table 10: Experimental results of ColNomic on JinaVDR.

Type	Methods	Adp.	Para	all		arabic		japanese		russian		german		spanish		italian		dutch		hindi		chinese
				ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%
pruning	base		-	0.66	-	0.92	-	0.74	-	0.80	-	0.54	-	0.66	-	0.68	-	0.47	-	0.17	-	0.97	-
pruning	random		ratio=0.1	0.66	0.10	0.91	0.10	0.74	0.10	0.80	0.10	0.53	0.10	0.66	0.10	0.68	0.10	0.48	0.10	0.17	0.10	0.97	0.10
pruning	random		ratio=0.3	0.66	0.30	0.91	0.30	0.76	0.30	0.80	0.30	0.54	0.30	0.66	0.30	0.68	0.30	0.48	0.30	0.17	0.30	0.96	0.30
pruning	random		ratio=0.5	0.66	0.50	0.90	0.50	0.75	0.50	0.80	0.50	0.53	0.50	0.65	0.50	0.67	0.50	0.47	0.50	0.16	0.50	0.96	0.50
pruning	random		ratio=0.7	0.64	0.70	0.89	0.70	0.75	0.70	0.79	0.70	0.52	0.70	0.61	0.70	0.66	0.70	0.46	0.70	0.16	0.70	0.96	0.70

Appendix

Table 11: Experimental results of Jina-v4 on JinaVDR.

Type	Methods	Adp.	Para	all		finreport		finslides		techreport		techslides
				ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%
pruning	base		-	0.72	-	0.55	-	0.62	-	0.81	-	0.90	-
pruning	random		ratio=0.1	0.72	0.10	0.55	0.10	0.62	0.10	0.81	0.10	0.90	0.10
pruning	random		ratio=0.3	0.72	0.30	0.55	0.30	0.60	0.30	0.80	0.30	0.91	0.30
pruning	random		ratio=0.5	0.70	0.50	0.54	0.50	0.58	0.50	0.79	0.50	0.90	0.50
pruning	random		ratio=0.7	0.68	0.70	0.50	0.70	0.53	0.70	0.78	0.70	0.90	0.70

Appendix

Table 12: Experimental results of ColQwen on REAL-MM-RAG.

Type	Methods	Adp.	Para	all		finreport		finslides		techreport		techslides
				ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%
pruning	base		-	0.78	-	0.63	-	0.73	-	0.83	-	0.92	-
pruning	random		ratio=0.1	0.84	0.10	0.76	0.10	0.80	0.10	0.88	0.10	0.91	0.10
pruning	random		ratio=0.3	0.83	0.30	0.76	0.30	0.79	0.30	0.87	0.30	0.92	0.30
pruning	random		ratio=0.5	0.81	0.50	0.72	0.50	0.75	0.50	0.86	0.50	0.91	0.50
pruning	random		ratio=0.7	0.77	0.70	0.67	0.70	0.67	0.70	0.82	0.70	0.90	0.70

Appendix

Table 13: Experimental results of ColNomic on REAL-MM-RAG.

Type	Methods	Adp.	Para	overall		finreport		finslides		techreport		techslides
				ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%	ndcg	prun%
pruning	base		-	0.79	-	0.69	-	0.73	-	0.82	-	0.91	-
pruning	random		ratio=0.1	0.79	0.10	0.69	0.10	0.73	0.10	0.82	0.10	0.91	0.10
pruning	random		ratio=0.3	0.79	0.30	0.69	0.30	0.72	0.30	0.82	0.30	0.92	0.30
pruning	random		ratio=0.5	0.78	0.50	0.67	0.50	0.72	0.50	0.81	0.50	0.91	0.50
pruning	random		ratio=0.7	0.74	0.70	0.63	0.70	0.65	0.70	0.79	0.70	0.90	0.70

Appendix

Table 14: Experimental results of Jina-v4 on REAL-MM-RAG.

Type	Methods	Adp.	Para	overall
				ndcg@5	pruning%
pruning	base		-	0.82	-
pruning	random		ratio=0.1	0.82	0.10
pruning	random		ratio=0.3	0.82	0.30
pruning	random		ratio=0.5	0.82	0.50
pruning	random		ratio=0.7	0.82	0.70

Appendix

Table 15: Experimental results of ColQwen on ViDoSeek.

Type	Methods	Adp.	Para	overall
				ndcg@5	pruning%
pruning	base		-	0.84	-
pruning	random		ratio=0.1	0.83	0.10
pruning	random		ratio=0.3	0.83	0.30
pruning	random		ratio=0.5	0.83	0.50
pruning	random		ratio=0.7	0.83	0.70

Appendix

Table 16: Experimental results of ColNomic on ViDoSeek.

Type	Methods	Adp.	Para	overall
				ndcg@5	pruning%
pruning	base		-	0.81	-
pruning	random		ratio=0.1	0.81	0.10
pruning	random		ratio=0.3	0.81	0.30
pruning	random		ratio=0.5	0.81	0.50
pruning	random		ratio=0.7	0.81	0.70

Appendix

Table 17: Experimental results of Jina-v4 on ViDoSeek.

Type	Methods	Adp.	Para	overall
				ndcg@5	pruning%
pruning	base		-	0.54	-
pruning	random		ratio=0.1	0.54	0.10
pruning	random		ratio=0.3	0.53	0.30
pruning	random		ratio=0.5	0.53	0.50
pruning	random		ratio=0.7	0.51	0.70

Appendix

Table 18: Experimental results of ColQwen on MMLongBench-Doc.

Type	Methods	Adp.	Para	overall
				ndcg@5	pruning%
pruning	base		-	0.57	-
pruning	random		ratio=0.1	0.57	0.10
pruning	random		ratio=0.3	0.55	0.30
pruning	random		ratio=0.5	0.54	0.50
pruning	random		ratio=0.7	0.52	0.70

Appendix

Table 19: Experimental results of ColNomic on MMLongBench-Doc.

Type	Methods	Adp.	Para	overall
				ndcg@5	pruning%
pruning	base		-	0.54	-
pruning	random		ratio=0.1	0.54	0.10
pruning	random		ratio=0.3	0.54	0.30
pruning	random		ratio=0.5	0.53	0.50
pruning	random		ratio=0.7	0.52	0.70

Appendix

Table 20: Experimental results of Jina-v4 on MMLongBench-Doc.

Visual Document Retrieval

Visual document retrieval finds relevant pages or regions in image-based documents using visual and multimodal representations.

See all 66 papers

AdaMerge: Tuning-Free Patch Compression for Multi-Vector Visual Document Retrieval
Sep 18, 2026Jianxin You, Kun NiVisual Document Retrieval Visual Token Compression
Université de Montréal Montréal, QC, Canada · Concordia University Montréal, QC, Canada
Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrieval
Apr 11, 2026Yibo Yan, Mingdong Ou, Yi Cao +5Visual Document Retrieval Multi-Vector Retrieval
Hong Kong University of Science and Technology (Guangzhou) · Alibaba Cloud Computing · Hong Kong University of Science and Technology
MAGIC: Marginal-Guided Compression with Optimal Transport for Efficient Visual Document Retrieval
Sep 17, 2026Xu Yuan, Hua Liu, Wenqi Fan +1Visual Document Retrieval Embedding Compression
The Hong Kong Polytechnic University, HK SAR · Southern University of Science and Technology, China

Sep 18, 2026J/K move · Enter open · S save

AdaMerge: Tuning-Free Patch Compression for Multi-Vector Visual Document Retrieval

Jianxin You, Kun Ni

Université de Montréal Montréal, QC, Canada · Concordia University Montréal, QC, Canada

Visual Document Retrieval Visual Token Compression Multi-Vector Retrieval Visual Token Merging

Open paper AskPDF arXiv

Multi-vector visual document retrieval (VDR) models such as ColPali and ColNomic achieve strong accuracy by representing each document with hundreds to thousands of patch-level embeddings, at substantial storage and latency cost. Existing compression methods either prune unimportant patches or merge similar ones into clusters; the recent state-of-the-art merging method Prune-then-Merge (PtM) consistently outperforms pruning-only baselines at high compression, but requires a per-dataset cluster budget m to be tuned by grid search. We observe that the merge-cosine sequence produced by hierarchical clustering exhibits a sharp cliff separating mergeable redundancy from salient signal, and that the location of this cliff is concentrated in a narrow band across more than 11,000 documents from 14 datasets. This suggests the merge boundary can be detected per document rather than tuned per dataset. Building on this observation, we propose AdaMerge, a plug-and-play compression method that (i) detects each document's own cliff via gap analysis on the merge-cosine trajectory, and (ii) builds attention-weighted cluster centroids to preserve salient signal. On the long-document benchmark ViDoRe-V2 (4 datasets, two backbones), AdaMerge significantly outperforms tuned PtM across the operating range (p < 10^-4); on the short-document benchmark ViDoRe-V1 (10 datasets, two backbones), where all merging methods are already near-lossless, AdaMerge matches tuned PtM without any per-dataset tuning. AdaMerge adds only about 10 ms per document and exposes a single global hyperparameter shared across all datasets and backbones.

Abstract

Visual Document Retrieval

Explore similar work

AdaMerge: Tuning-Free Patch Compression for Multi-Vector Visual Document Retrieval

Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrieval

MAGIC: Marginal-Guided Compression with Optimal Transport for Efficient Visual Document Retrieval