MAGIC: Marginal-Guided Compression with Optimal Transport for Efficient Visual Document Retrieval

Xu Yuan; Hua Liu; Wenqi Fan; Qing Li

cs.CVSep 17, 2026

MAGIC: Marginal-Guided Compression with Optimal Transport for Efficient Visual Document Retrieval

Authors: Xu Yuan, Hua Liu, Wenqi Fan, Qing Li

Organizations: The Hong Kong Polytechnic University, HK SAR · Southern University of Science and Technology, China

Abstract

Recent visual document retrieval (VDR) systems such as ColPali use multi-vector page embeddings, in which patch-level vectors enable fine-grained evidence matching but incur substantial index storage and MaxSim scoring overhead. Post-hoc merging offers a practical route to efficient VDR by reducing this cost without retraining the retriever, but its uniform reconstruction objectives are poorly aligned with the sparse, non-uniform patch usage induced by late-interaction retrieval. Under aggressive compression, this misalignment can preserve rarely used patches while concentrating retrieval activity on too few retained representatives. To address this misalignment, we propose Marginal-Guided Compression with Optimal Transport (MAGIC), a training-free post-hoc compressor for efficient retrieval with frozen multi-vector embeddings. MAGIC derives a MaxSim-induced compression surrogate and optimizes it through a two-marginal entropic optimal-transport formulation, where a retrieval-demand source marginal prioritizes high-use patches and a balanced target marginal regularizes retained-facet usage. Across ViDoRe benchmarks, keep ratios, and retrieval backbones, MAGIC consistently outperforms strong post-hoc compressors, with particularly large gains in the aggressive-compression regime; component ablations verify the complementary effects of its two marginals. We release the code at: https://github.com/xandery-geek/MAGIC.

Figures & tables

Figure 1: (a) Patch-level retrieval demand is concentrated: the top 20% of patches account for about 84% of query-token selections. (b) MAGIC reduces retrieval-demand-weighted covering error

C_{\mathrm{err}}

as the facet budget shrinks and maintains a higher retrieval-active facet count

K_{\mathrm{eff}}

than generic merging.

Figure 2: Overview of MAGIC. During offline indexing, frozen patch-level document embeddings are paired with calibration query tokens to estimate MaxSim-induced retrieval demand $\widehat{w}_{j}$ . MAGIC uses this demand as the source marginal and imposes a balanced target marginal in a two-marginal optimal transport problem, producing fixed compressed facets for online retrieval.

Method	ArxivQ	DocQ	InfoQ	TabF	TATQ	Shift	Avg.
Keep-ratio $r=0.1$
1D-Pooling	80.32 / 85.20	52.80 / 58.24	86.96 / 91.34	85.20 / 89.64	72.29 / 83.78	71.43 / 85.00	74.83 / 82.20
2D-Pooling	82.28 / 86.80	53.85 / 60.12	86.80 / 91.09	86.20 / 91.79	67.16 / 79.04	74.47 / 88.00	75.13 / 82.81
K-Means	86.42 / 90.40	58.66 / 64.81	89.31 / 93.73	88.90 / 93.93	77.40 / 87.18	80.07 / 90.00	80.13 / 86.67
DocPruner	74.68 / 81.00	47.40 / 55.61	77.32 / 84.05	85.47 / 92.86	60.95 / 72.27	54.32 / 67.00	66.69 / 75.46
Voronoi Pruning	82.88 / 87.00	53.15 / 60.49	78.08 / 85.36	88.27 / 92.86	66.04 / 76.91	56.03 / 73.00	70.74 / 79.27

Table 1: Comparison with post-hoc baselines on ViDoRe v1 using ColQwen2.5. The metric is nDCG@5 (%) / Recall@5 (%). The upper block reports moderate compression (

r=0.1

); the lower block reports aggressive compression (

r=0.01

Panel of Figure 3 — Figure 3: Budget-sweep retrieval performance (nDCG@5) across main evaluation settings.

Source	Target	Lloyd	0.5	0.25	0.1	0.05	0.025	0.01
✓	✓	✓	83.20	82.71	82.12	80.09	77.33	70.71
✗	✓	✓	82.41	81.67	79.11	76.70	71.60	61.88
✓	✗	✓	82.72	82.52	81.19	79.04	76.83	70.02
✓	✓	✗	82.62	82.36	81.71	79.99	77.01	70.19

Table 2: Marginal decomposition on ViDoRe v1. The metric is nDCG@5 (%). ✓ and ✗ indicate whether each component is used; disabling Lloyd uses the soft barycenter readout.

Figure 6: Query-grounded evidence before and after MAGIC compression.

Method	$\bm{r=0.1}$			$\bm{r=0.01}$
Method	Size (MB/1K)	Latency (ms/query)	Comp. (ms/page)	Size (MB/1K)	Latency (ms/query)	Comp. (ms/page)
ColQwen2.5	380.67	0.98	112.62	380.67	0.98	112.62
DocPruner	43.38	0.18	279.00	9.69	0.08	279.00
Light-ColPali	37.79	0.16	210.41	3.54	0.06	212.72
MAGIC	38.24	0.17	122.79	3.86	0.06	119.91

Table 3: Storage, single-query latency, and offline indexing time on ViDoRe v1 with ColQwen2.5.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Backbones
Model	Public source
ColPali ( Faysse et al., 2025 )	https://huggingface.co/vidore/colpali-v1.3
ColQwen2.5 ( Faysse et al., 2025 )	https://huggingface.co/vidore/colqwen2.5-v0.2
Jina-v4 ( Günther et al., 2025 )	https://huggingface.co/jinaai/jina-embeddings-v4
ColNomic ( Team, 2025 )	https://huggingface.co/nomic-ai/colnomic-embed-multimodal-3b
Post-hoc compression baselines

Appendix

Table 4: Public sources for backbone checkpoints and baseline implementations used in the experiments.

Hyperparameter	Value	Meaning
$L_{\mathrm{cal}}$	1000	calibration query-token count
$\tau$	0.05	selection temperature
$\varepsilon$	0.05	entropic temperature
$T_{\mathrm{out}}$	5	outer iterations
$T_{\mathrm{sk}}$	5	Sinkhorn iterations
$\eta$	0.982	barycenter step size

Appendix

Table 5: Default MAGIC hyperparameters used in the main experiments.

Method	Vectors / page	Biomedical Lectures		Economics Reports		ESG Reports
Method	Vectors / page	EN	Multi. Avg.	EN	Multi. Avg.	EN	Multi. Avg.	Human
ColQwen2.5	759	63.2	60.3	61.0	57.2	60.6	56.2	64.6
AGC	64	60.2	57.7	62.8	56.3	54.5	53.1	54.5
MetaEmbed	64	61.7	58.7	62.3	55.5	62.6	57.4	63.7
MAGIC	64	60.6	57.8	60.3	56.8	57.2	53.2	55.4

Appendix

Table 6: Comparison with learned efficient baselines on ViDoRe v2 using a fixed budget of 64 document embeddings per page. The metric is nDCG@5 (%). “EN” is the English language subset, “Multi. Avg.” denotes the average over multilingual queries, and “Human” denotes the human-labeled ESG subset.

Panel of Figure 7 — Figure 7: Additional budget-sweep retrieval performance on ColNomic and ViDoSeek. Each curve reports nDCG@5 over the keep ratio.

Embedding-size budget = 1536 dimensions
Method	Representation	ArxivQ	DocQ	InfoQ	TabF	TATQ	Shift	Avg.
ColQwen2.5	$763\times 128$ vectors	88.65	62.60	91.25	89.67	81.40	87.87	83.57
VLM2Vec	$1\times 1536$ vector	60.88	32.68	74.30	82.10	31.22	65.16	57.72
VLM2Vec-V2	$1\times 1536$ vector	79.79	44.96	84.41	91.23	48.33	70.87	69.93
UniME-V2	$1\times 1536$ vector	63.79	37.39	73.06	82.31	40.59	53.86	58.50
MAGIC	$12\times 128$ facets	84.36	51.26	85.03	86.76	67.85	74.78	75.01

Appendix

Table 7: Comparison with single-vector baselines under matched document-side embedding-size budgets on ViDoRe v1. The uncompressed ColQwen2.5 retriever is treated as a reference point. MAGIC uses 12 or 16 retained 128-dimensional facets, corresponding to 1536 and 2048 dimensions, respectively. Single-vector baselines use one global page embedding at the corresponding dimensionality. The 2048-dimensional group also includes VisRAG, whose closest available setting uses a 2304-dimensional global embedding. The metric is nDCG@5 (%).

Panel of Figure 8 — Figure 8: Additional MAGIC hyperparameter sensitivity. The curves report nDCG@5 across hyperparameter values at two keep ratios.

Panel of Figure 9 — Figure 9: Additional qualitative example on DocVQA. The query asks for a purchase-order number, and MAGIC preserves the answer-bearing field while operating at a 0.05 keep ratio.

Figure 11: Additional qualitative example on TAT-DQA. MAGIC keeps the definition evidence needed to answer the financial-document query.

Figure 12: Additional qualitative example on ArxivQA. MAGIC preserves the workflow evidence around SCAI-DM.

Visual Document Retrieval

Visual document retrieval finds relevant pages or regions in image-based documents using visual and multimodal representations.

See all 66 papers

AdaMerge: Tuning-Free Patch Compression for Multi-Vector Visual Document Retrieval
Sep 18, 2026Jianxin You, Kun NiVisual Document Retrieval Visual Token Compression
Université de Montréal Montréal, QC, Canada · Concordia University Montréal, QC, Canada
Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework
Feb 23, 2026Yibo Yan, Mingdong Ou, Yi Cao +5Visual Document Retrieval Multi-Vector Retrieval
Hong Kong University of Science and Technology (Guangzhou) · Alibaba Cloud Computing · Hong Kong University of Science and Technology
AnchorFold: A Focus-Then-Fold Framework via Recursive Attention Propagation for Efficient Multi-Vector Visual Document Retrieval
Aug 9, 2026Haoyu Zuo, Yibo Yan, Xin Zou +4Multi-Vector Retrieval Visual Document Retrieval
Hong Kong University of Science and Technology (Guangzhou) · Alibaba Cloud Computing · Hong Kong University of Science and Technology

Sep 18, 2026J/K move · Enter open · S save

AdaMerge: Tuning-Free Patch Compression for Multi-Vector Visual Document Retrieval

Jianxin You, Kun Ni

Université de Montréal Montréal, QC, Canada · Concordia University Montréal, QC, Canada

Visual Document Retrieval Visual Token Compression Multi-Vector Retrieval Visual Token Merging

Open paper AskPDF arXiv

Multi-vector visual document retrieval (VDR) models such as ColPali and ColNomic achieve strong accuracy by representing each document with hundreds to thousands of patch-level embeddings, at substantial storage and latency cost. Existing compression methods either prune unimportant patches or merge similar ones into clusters; the recent state-of-the-art merging method Prune-then-Merge (PtM) consistently outperforms pruning-only baselines at high compression, but requires a per-dataset cluster budget m to be tuned by grid search. We observe that the merge-cosine sequence produced by hierarchical clustering exhibits a sharp cliff separating mergeable redundancy from salient signal, and that the location of this cliff is concentrated in a narrow band across more than 11,000 documents from 14 datasets. This suggests the merge boundary can be detected per document rather than tuned per dataset. Building on this observation, we propose AdaMerge, a plug-and-play compression method that (i) detects each document's own cliff via gap analysis on the merge-cosine trajectory, and (ii) builds attention-weighted cluster centroids to preserve salient signal. On the long-document benchmark ViDoRe-V2 (4 datasets, two backbones), AdaMerge significantly outperforms tuned PtM across the operating range (p < 10^-4); on the short-document benchmark ViDoRe-V1 (10 datasets, two backbones), where all merging methods are already near-lossless, AdaMerge matches tuned PtM without any per-dataset tuning. AdaMerge adds only about 10 ms per document and exposes a single global hyperparameter shared across all datasets and backbones.

Abstract

Visual Document Retrieval

Explore similar work

AdaMerge: Tuning-Free Patch Compression for Multi-Vector Visual Document Retrieval

Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework

AnchorFold: A Focus-Then-Fold Framework via Recursive Attention Propagation for Efficient Multi-Vector Visual Document Retrieval