We study efficient multi-vector retrieval for late interaction in any modality. Late interaction has emerged as a dominant paradigm for information retrieval in text, images, visual documents, and videos, but its computation and storage costs grow linearly with document length, making it costly for image-, video-, and audio-rich corpora. To address this limitation, we explore query-agnostic methods for compressing multi-vector document representations under a constant vector budget. We introduce four approaches for index compression: sequence resizing, memory tokens, hierarchical pooling, and a novel attention-guided clustering (AGC). AGC uses an attention-guided mechanism to identify the most semantically salient regions of a document as cluster centroids and to weight token aggregation. Evaluating these methods on retrieval tasks spanning text (BEIR), visual-document (ViDoRe), and video (MSR-VTT, MultiVENT 2.0), we show that attention-guided clustering consistently outperforms other parameterized compression methods (sequence resizing and memory tokens), provides greater flexibility in index size than non-parametric hierarchical clustering, and achieves competitive or improved performance compared to a full, uncompressed index. The source code is available at: github.com/hanxiangqin/omni-col-press.
Figures & tables
Figure 1. We explore index compression in any modality. We introduce SeqResize , projection-based, MemTok , token-based, H-Pool , heuristic-based, and AGC (Ours), hybrid attention-similarity. AGC better utilizes index tokens while maintaining performance (nDCG@10) at high compression.
Figure 2. Overview of multi-vector index compression techniques. (a) AGC uses universal query tokens to guide attention-based centroid selection and weight the aggregation of clustering. (b) MemTok appends tokens to the document context to act as the final representation. (c) SeqResize down projects a document representation along the sequence dimension. (d) H-Pool iteratively groups similar vectors and replaces them with their mean.
Method
BEIR
ViDoRe
MSR-VTT
MultiVENT 2.0
R@10
nDCG@10
R@1
nDCG@5
R@1
nDCG@10
R@10
nDCG@10
Baseline
37.1
46.2
27.7
60.0
55.7
71.9
– ∗
– ∗
SeqResize
35.8
43.9
23.5
51.7
53.3
69.9
41.1
38.5
96.5%
95.0%
84.8%
86.2%
95.7%
96.9%
N/A
N/A
MemTok
36.3
45.0
25.0
54.4
54.2
69.9
48.7
44.8
97.8%
97.4%
90.3%
90.7%
97.3%
96.9%
N/A
N/A
Table 1. Results of index compression on each retrieval benchmark. Compression budgets: BEIR 32, ViDoRe 64, MSR-VTT 32, MultiVENT 2.0 64. ∗ means baseline was unable to build due to compute. Second row of each method shows percent of baseline.
Meth.
Avg
NF
FQA
SciF
SciD
TC
Tou
Arg
Base
46.2
37.1
43.3
74.4
18.6
78.5
27.5
44.1
Seq
43.9
31.0
36.8
70.3
18.7
75.8
24.8
50.0
95.0
83.6
85.0
94.5
100.5
96.6
90.2
113.4
Mem
45.0
35.2
39.8
71.7
17.6
77.6
27.7
45.1
97.4
94.9
91.9
96.4
94.6
98.9
100.7
102.3
H-P
41.2
33.7
36.2
72.5
18.4
62.0
17.6
48.3
Table 2. nDCG@10 (top) and percent performance relative to uncompressed baseline (bottom) on BEIR datasets. NF: NFCorpus, FQA: FiQA, SciF: SciFact, SciD: SciDocs, TC: TREC-COVID, Tou: Touche, Arg: arguana.
Method
Tok
Avg
Bio
Econ
ESG-R
ESG-H
Base
1297
60.0
61.4
53.9
57.0
67.6
ColPali
-
53.3
56.5
49.9
55.7
51.1
CQO
-
56.5
56.5
53.2
54.2
62.2
MEmbed
64
58.8
58.7
55.5
57.4
63.7
SeqResize
64
51.7
54.7
53.5
45.2
53.5
MemTok
64
54.3
56.8
53.0
46.4
61.4
Table 3. nDCG@5 breakdown by domain on ViDoRe v2 for the multilingual subsets. Bio: Biomedical, Econ: Economics, ESG-R: ESG Reports, ESG-H: ESG Human. ESGs are English. CQO: ColQwenOmni, MEmbed: MetaEmbed.
Tok
Method
R@1
R@10
nDCG@10
1
OmniEmbed-7B ( Ma et al., 2025 )
51.5
83.2
67.1
26
Video-ColBERT ( Reddy et al., 2025 )
51.5
85.5
67.7
1702
ColQwen-Omni 3B
40.8
73.8
56.3
1318
Baseline 3B (Ours)
55.7
88.3
71.9
5
SeqResize
53.4
86.7
69.5
MemTok
52.7
87.3
69.3
Table 4. Retrieval performance on MSR-VTT . We compare compressed methods against uncompressed baselines at budgets of 5, 32, and 128 tokens, alongside SOTA models.
Tok
Appn Tok
R@1
R@10
nDCG@10
5
5
53.9
85.8
69.2
32
53.1
86.7
69.6
32
5
54.2
87.2
70.6
32
56.9
87.0
71.5
128
55.4
87.6
71.2
128
32
55.2
86.8
70.8
Table 5. Retrieval performance metrics of AGC across varying budgets and appending tokens on MSR-VTT. Each combination is used with the same configurations of training. Appn Tok: number of appending tokens.
Method
Train
Test
R@1
R@10
nDCG@10
Baseline
1318
1318
55.7
88.3
71.9
AGC
32
5
53.6
87.4
70.1
32
32
56.9
87.0
71.5
32
128
56.4
87.5
71.7
H-Pool
1318
5
52.6
86.1
68.9
1318
32
54.1
87.3
70.4
Table 6. Generalizability of AGC and H-Pool compression methods on MSR-VTT .
Model Variant
R@1
R@10
nDCG@10
Qwen2.5-VL-3B
56.9
87.0
71.5
Qwen2.5-VL-7B
58.0
89.0
73.0
Qwen3-VL-4B
58.5
88.4
73.0
Table 7. Generalizability of AGC across different model sizes and variants. Results indicate consistent performance scaling with larger and newer model backbones.
Figure 3. Index utilization and inter-position similarity analysis on MSR-VTT. Top row: Per-position matching strength for each method, computed by summing the maximum similarity matches between query tokens and document tokens across all relevant query-document pairs, averaged over query positions. Bottom row: Pairwise cosine similarity between document vectors within each document, averaged across all documents in the index.
Figure 4. Correlation between retrieval performance metrics and distribution evenness measures on MSR-VTT dataset. Dashed lines indicate linear regression fits. All correlations are statistically significant ( p≤0.01 ), with Pearson’s r ranging from 0.959 to 0.996.
CV
Gini
Metric
Pearson r
p-value
Pearson r
p-value
R@1
0.974 ∗∗
0.005
0.959 ∗∗
0.010
nDCG@10
0.978 ∗∗
0.004
0.943 ∗
0.016
MRR
0.996 ∗∗
< 0.001
0.962 ∗∗
0.009
Table 8. Pearson correlation analysis between retrieval metrics and inverse evenness metrics (1/evenness) on MSR-VTT , testing the hyperbolic relationship retrieval ∼ 1/evenness. All variants use a fixed budget of 32. Evenness metrics measured on matching strength of (document position, query position) pairs. CV: Coefficient of Variation, Gini: Gini Coefficient. Significance levels: ∗∗p<0.01 , ∗p<0.05 .
Method
R@1
R@10
nDCG@10
AGC
56.9
87.0
71.5
w/o Attn Weight
55.7
86.5
71.0
w/o Attn Select
54.1
86.8
70.0
w/o Cluster
52.9
87.3
69.8
Table 9. Ablation Study Results on MSR-VTT . We observe a drop in performance as components are removed (individually, not in sequence) from the full AGC model.