Simple Linear Corrections for Compression in Late Interaction

Mihir Agarwal; Sabyasachi Basu; Viraj Agrawal

cs.IROct 9, 2026

Simple Linear Corrections for Compression in Late Interaction

Authors: Mihir Agarwal, Sabyasachi Basu, Viraj Agrawal

Organizations: Microsoft Research India

Abstract

Late-interaction (multi-vector) models represent documents using a set of token-level vectors, providing a middle ground between cross-encoders and dense encoders in terms of quality and cost of embedding. However, compared to the latter, the indices are costly to store and score. Pooling and pruning based compression techniques are deployed widely but deteriorate in quality noticeably with aggressive compression. We investigate whether a lightweight change to the scoring function can recover retrieval quality after document-vector compression. For each combination of encoder, compressor, and vector budget, we learn a global linear correction by distilling the uncompressed model's score distributions over candidate documents. The encoder and compressor remain frozen, and the correction can be applied to each query at search time or once, offline, to the stored vectors. We evaluate the method with ColBERTv2 and Jina v4 on MS MARCO and BEIR, and with ColPali2 on the ViDoRe v3 visual document retrieval benchmark. Experiments cover four training-free compression methods, multiple vector budgets, in-domain adaptation, and transfer from MS MARCO to BEIR. In the in-domain setting, the correction improves over baseline compression in 154 of 160 BEIR settings, while transfer from MS MARCO is more variable across models and datasets. We also find that matrices trained with synthetic queries can improve over baseline compression on ViDoRe v3. These results show that a small learned correction can recover some of the quality lost to compression without retraining the encoder or recompressing the documents.

Figures & tables

Figure 1: Recall@10 relative gain from

A

over baseline compression on MS MARCO : ColBERTv2 (left) and Jina v4 (right). Bars show means. Across plotted settings, Recall@10 sample standard deviations (SDs) range up to 0.20 percentage points. Values below the bars are baseline Recall@10 (%). Uncompressed Recall@10: 68.84% for ColBERTv2 and 49.08% for Jina v4.

Figure 2: Recall@10 relative gains over baseline compression for MS MARCO transfer (

A_{\mathrm{MS}}

) and in-domain correction (

A_{\mathrm{ID}}

) for ColBERTv2 for different compression schemes. Bars show means; across both parts, Recall@10 sample SDs range up to 0.57 percentage points. Values below the bars are baseline Recall@10 (%). Uncompressed references (%): NFCorpus 31.58; HotpotQA 69.12; FiQA 44.26; FEVER 90.94; SciFact 80.46. Jina v4 continues on the next page.

Figure 3

Figure 3: ColPali2 Recall@10 relative gains over baseline compression on ViDoRe v3 , with a 30/70 train/test query-family split. Recall@10 sample SDs range up to 0.10 percentage points. Values below the bars are baseline Recall@10 (%); Full MaxSim Recall@10: 55.12%.

Figure 4: Recall@10 relative gains over baseline Ward for synthetic-query (left bars) and human-query (right bars) training. Left: text, with human queries from the same dataset. Right: ColPali2 on all ViDoRe v3 queries, equally averaging eight subsets; here the human-query matrix is trained on ColPali queries from a different corpus. Human-query bars show means and sample SDs.

Figure 5: ColBERTv2 Recall@10 versus mean single-core reranking latency on MS MARCO , with

C=50

candidates. Baseline Ward and Ward+

A

use

k=4,8,16,32

from left to right; full MaxSim is the uncompressed reference. Ward+

A

points show means, with no error bars. Lines guide the eye.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Training queries	Epoch	$k=4$	$k=8$	$k=16$	$k=32$
Baseline	–	45.525	56.599	64.337	67.536
5,000	1	50.904	59.686	64.967	67.599
	2	51.663	59.667	64.762	67.393
	3	51.841	59.753	65.060	67.396
	4	51.802	59.782	64.891	67.498
	5	51.803	59.860	65.049	67.493

Appendix

Table 1: Number of training queries. Recall@10 (%) of ColBERTv2 with Ward on MS MARCO dev after each epoch; seed 0 and no epoch selection. Baseline is Ward without correction.

Loss	$k=4$	$k=8$	$k=16$	$k=32$
Baseline (no correction)	45.525	56.599	64.337	67.536
KL	51.803	59.860	65.049	67.493
Label	50.026	58.422	64.164	66.940
KL $+\,0.25\beta_{0}\cdot$ label	51.794	59.996	65.094	67.400
KL $+\,0.5\beta_{0}\cdot$ label	51.759	59.633	65.209	67.507
KL $+\,\beta_{0}\cdot$ label	51.461	59.146	65.177	67.635

Appendix

Table 2: Loss function. Recall@10 (%) of ColBERTv2 with Ward on MS MARCO dev, with 5,000 training queries, epoch 5 and seed 0. Every loss also includes

0.01\|A-I\|_{F}^{2}

Correction	Parameters	$k=4$	$k=8$	$k=16$	$k=32$
Baseline (no correction)	0	45.525	56.599	64.337	67.536
$A$	16,384	52.061	60.224	65.025	67.660
MLP $2\times 64$	16,576	54.003	60.893	65.883	67.690
MLP $2\times 128$	33,024	54.026	60.843	65.745	67.575
MLP $2\times 256$	65,920	53.754	60.649	65.591	67.536
MLP $2\times 512$	131,712	53.529	60.616	65.569	67.457

Appendix

Table 3: Non-linear correction. Recall@10 (%) of ColBERTv2 with Ward on MS MARCO dev, with 100,000 training queries, epoch 5 and seed 0. Depth counts the linear layers of

g

. MLPs are unregularized;

A

includes

0.01\|A-I\|_{F}^{2}

. Parameters count only the correction.

Late Interaction Retrieval

Late interaction retrieval matches query and document token vectors after encoding to preserve fine-grained relevance while supporting efficient search.

See all 28 papers

Feb 24, 2026

Multi-Vector Index Compression in Any Modality

We study efficient multi-vector retrieval for late interaction in any modality. Late interaction has emerged as a dominant paradigm for information retrieval in text, images, visual documents, and videos, but its computation and storage costs grow linearly with document length, making it costly for image-, video-, and audio-rich corpora. To address this limitation, we explore query-agnostic methods for compressing multi-vector document representations under a constant vector budget. We introduce four approaches for index compression: sequence resizing, memory tokens, hierarchical pooling, and a novel attention-guided clustering (AGC). AGC uses an attention-guided mechanism to identify the most semantically salient regions of a document as cluster centroids and to weight token aggregation. Evaluating these methods on retrieval tasks spanning text (BEIR), visual-document (ViDoRe), and video (MSR-VTT, MultiVENT 2.0), we show that attention-guided clustering consistently outperforms other parameterized compression methods (sequence resizing and memory tokens), provides greater flexibility in index size than non-parametric hierarchical clustering, and achieves competitive or improved performance compared to a full, uncompressed index. The source code is available at: github.com/hanxiangqin/omni-col-press.

Hanxiang Qin, Alexander Martin, Rohan Jha +3

Johns Hopkins University, Baltimore, MD, USA

Multi-Vector Retrieval Embedding Compression Multimodal IR Clustering

Ask PDF

Aug 30, 2026

Spatial Matryoshka Training for Multi-Granularity Visual Document Retrieval

Multi-modal late-interaction retrievers achieve strong retrieval on visually rich documents by representing each page as per patch embeddings and matching at the token level. However, this approach incurs high storage costs. Existing compression methods typically fix a single compression level at indexing time, limiting flexibility. We present ColSNAP (Spatial Nested Average Pooling)1, a training method that generates a nested hierarchy of compression levels directly from a backbone's patch grid. By spatially pooling patch embeddings into pro- gressively coarser tiers and training all tiers simultaneously, a single model learns to support retrieval at multiple compression levels without architectural changes. Crucially, a single encoding pass yields every tier, enabling the accuracy-storage trade-off to be configured at indexing time to match avail- able storage budgets, rather than being fixed during training. We demonstrate that models trained using ColSNAP maintain near full-resolution retrieval performance under substantial compression and that ColSNAP transfers effectively across multiple late-interaction backbones, and achieves most of its improvements via a lightweight adaptation stage applied to a pre-trained retriever.

Trishan Singha Roy, Arkadeep Acharya, Vishwajeet Kumar +2

IIT Delhi · IBM

Hierarchical Representation Learning Matryoshka Representation Learning Multi-Vector Retrieval Visual Document Retrieval Visual Token Compression

Ask PDF

Oct 19, 2026

FLINT: Fixed-budget Late Interaction

Multi-vector retrieval models such as ColBERT often outperform single-vector models of comparable scale by matching queries and documents at the token level. However, retaining a vector for every token substantially increases storage requirements and slows retrieval compared to single-vector bi-encoders, limiting their scalability. We introduce FLINT (Fixed-budget Late INTeraction), an architecture that uses attention-based pooling to represent each query and document with a fixed number of embedding vectors. Unlike architectures that pool only document representations while retaining token-level queries, FLINT applies pooling symmetrically to both sides, limiting document storage and the number of vector comparisons needed during retrieval. FLINT achieves comparable ranking effectiveness to the closest baselines on BEIR and outperforms them on several benchmarks with longer and more complex queries, including BrowseComp+, CRUMB, and TREC Tip-of-the-Tongue 2025. We demonstrate that FLINT's effectiveness-efficiency benefits extend across encoder architectures, testing it on two separate backbones (BERT and ModernBERT). On long-query corpora, FLINT delivers 1.5-2.8 times faster end-to-end retrieval than ConstBERT, the closest baseline. FLINT also reduces index size by 5.3-9.3 times relative to token-level late-interaction models such as ColBERTv2.

Himank Handa, Negar Arabzadeh, Matei Zaharia

University of Illinois Urbana-Champaign · University of California, Berkeley

Late Interaction Retrieval Multi-Vector Retrieval

Ask PDF

Abstract

Late Interaction Retrieval

Explore similar work

Multi-Vector Index Compression in Any Modality

Spatial Matryoshka Training for Multi-Granularity Visual Document Retrieval

FLINT: Fixed-budget Late Interaction