Working Notes on Late Interaction Dynamics: Analyzing Targeted Behaviors of Late Interaction Models

Antoine Edy; Max Conti; Quentin Macé

cs.IRMar 27, 2026

Working Notes on Late Interaction Dynamics: Analyzing Targeted Behaviors of Late Interaction Models

Authors: Antoine Edy, Max Conti, Quentin Macé

Organizations: Illuin Technology · CentraleSupélec, Paris-Saclay

Abstract

While Late Interaction models exhibit strong retrieval performance, many of their underlying dynamics remain understudied, potentially hiding performance bottlenecks. In this work, we focus on two topics in Late Interaction retrieval: a length bias that arises when using multi-vector scoring, and the similarity distribution beyond the best scores pooled by the MaxSim operator. We analyze these behaviors for state-of-the-art models on the NanoBEIR benchmark. Results show that while the theoretical length bias of causal Late Interaction models holds in practice, bi-directional models can also suffer from it in extreme cases. We also note that no significant similarity trend lies beyond the top-1 document token, validating that the MaxSim operator efficiently exploits the token-level similarity scores.

Figures & tables

Figure 1: Mean length comparison between the retrieved false positive chunks, the relevant ground-truth documents, and the global corpus average. Queries are grouped into quantiles on the x-axis based entirely on the average token length of their corresponding relevant chunks.

Panel of Figure 2 — Figure 2: Expected decrease in retrieval performance (nDCG@10) when a chunk of a specific length is added to the corpus. Chunks are categorized into equal-sized quantile bins by token length on the x-axis. The solid line plots the average nDCG penalty incurred by the presence of a chunk from that bin, evaluated against a random baseline (dashed line) and its 90% confidence interval (shaded area).

Panel of Figure 3 — Figure 3: ColBERT-Zero document token similarities on failed queries. While some datasets exhibit interesting results (e.g., NanoArguAna, right), no clear tendency emerges for positive documents across NanoBEIR (left).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Chunk Length Distribution across the merged NanoBEIR corpus.

Model	Pooling Strategy	Architecture	Parameters
jina-embeddings-v4 Günther et al. (2025)	Multi-vector	Causal	4B
Qwen3-Embedding-4B Zhang et al. (2025)	Single-vector	Causal	4B
GTE-ModernColBERT-v1 Chaffin (2025)	Multi-vector	Bi-directional	0.15B
ColBERT-Zero Chaffin et al. (2026)	Multi-vector	Bi-directional	0.15B

Appendix

Table 1: Summary of the evaluated models, including their pooling strategies, architectures, and sizes.

Panel of Figure 5 — Figure 5: Absolute occurrences of irrelevant chunks ranked above the highest-ranked true positive passage. Bin limits are defined to contain an equal number of chunks. The dashed line plots the no-bias expected baseline, bounded by a 90% variance interval.

Panel of Figure 6 — Figure 6: jina-embeddings-v4 document token similarities on failed queries. The positive document has a larger distance to the top-1 negative, and remains constantly worse across all document tokens.

Late Interaction Retrieval

Late interaction retrieval matches query and document token vectors after encoding to preserve fine-grained relevance while supporting efficient search.

See all 28 papers

Spike Hijacking in Late-Interaction Retrieval
Apr 6, 2026Karthik Suresh, Tushar Vatsa, Tracy King +2Late Interaction Retrieval Multi-Vector Retrieval
Adobe, 345 Park Ave, San Jose, CA 95110, USA
EigenLI: Spectral Approximations to Late Interaction
Sep 7, 2026Archish S, Sabyasachi Basu, Ankit Garg +2Late Interaction Retrieval Multi-Vector Retrieval
Temple · Microsoft Research India
FLINT: Fixed-budget Late Interaction
Oct 19, 2026Himank Handa, Negar Arabzadeh, Matei ZahariaLate Interaction Retrieval Multi-Vector Retrieval
University of Illinois Urbana-Champaign · University of California, Berkeley

Apr 6, 2026J/K move · Enter open · S save

Spike Hijacking in Late-Interaction Retrieval

Karthik Suresh, Tushar Vatsa, Tracy King, Asim Kadav+1

Adobe, 345 Park Ave, San Jose, CA 95110, USA

Late Interaction Retrieval Multi-Vector Retrieval Retrieval Robustness

Open paper AskPDF arXiv

Late-interaction retrieval models rely on hard maximum similarity (MaxSim) to aggregate token-level similarities. Although effective, this winner-take-all pooling rule may structurally bias training dynamics. We provide a mechanistic study of gradient routing and robustness in MaxSim-based retrieval. In a controlled synthetic environment with in-batch contrastive training, we demonstrate that MaxSim induces significantly higher patch-level gradient concentration than smoother alternatives such as Top-k pooling and softmax aggregation. While sparse routing can improve early discrimination, it also increases sensitivity to document length: as the number of document patches grows, MaxSim degrades more sharply than mild smoothing variants. We corroborate these findings on a real-world multi-vector retrieval benchmark, where controlled document-length sweeps reveal similar brittleness under hard max pooling. Together, our results isolate pooling-induced gradient concentration as a structural property of late-interaction retrieval and highlight a sparsity-robustness tradeoff. These findings motivate principled alternatives to hard max pooling in multi-vector retrieval systems.

Abstract

Late Interaction Retrieval

Explore similar work

Spike Hijacking in Late-Interaction Retrieval

EigenLI: Spectral Approximations to Late Interaction

FLINT: Fixed-budget Late Interaction