Recent works have highlighted the expressive limitations of embedding based retrieval models through both theoretical analyses and challenging benchmarks such as LIMIT. While multi-vector embeddings consistently outperform single-vector embeddings, the precise representational gap between them remains poorly understood. In this work, following Jayaram's work, we provide the first explicit family of query and document sets, together with their relevance matrices, for which single-vector embeddings that rank all relevant documents above irrelevant ones require exponential size, whereas polynomial-size multi-vector embeddings suffice. Our result establishes an exponential separation between the expressive power of single-vector and multi-vector embeddings for the task of ranking of documents as opposed to approximating numerical scores as in the work of Jayaram. Motivated by our theoretical construction, we introduce ANDOR, a new retrieval benchmark that naturally instantiates these hard examples. We show that state-of-the-art single-vector embedding models perform poorly on ANDOR in the zero-shot setting and exhibit only marginal improvements after fine-tuning, highlighting the inherent difficulty of the benchmark compared to prior work. In contrast, multi-vector models consistently outperform their single-vector counterparts and improve substantially with fine-tuning, closely aligning with our theoretical predictions.
Figures & tables
Figure 1: Recall (left) and nDCG (right) for the single-vector (SV; blue) and multi-vector (MV; red) representations of Jina Embeddings v4 after joint fine-tuning on ANDOR at a mean training width of 5.5 , evaluated at test width 3.5 .
Figure 2: Example query with mandatory constraints across nine categories (AND-of-OR semantics) on the left. On the right, we provide one matching document (positive), one near-miss (level-1 hard negative), and one broad mismatch (easy negative).
Model
Type
Dim.
Open
FT
GTE ModernColBERT v1 [ 3 ]
Multi-vector
128/token
✓
✓
Jina Embeddings v4 [ 6 ]
Multi-vector
128/token
✓
✓
Jina Embeddings v4 [ 6 ]
Single-vector
2048
✓
✓
Qwen3 Embedding 0.6B [ 27 ]
Single-vector
1024
✓
✓
Snowflake Arctic Embed L v2 [ 26 ]
Single-vector
1024
✓
✓
Cohere Embed v4 [ 4 ]
Single-vector
1536
✗
✗
Table 1: Embedding models evaluated on ANDOR.
Figure 3: Retrieval performance across nine test widths at three recall cutoffs, evaluated in the zero-shot setting (top) and after fine-tuning at a training width of 9.5 (bottom). Multi-vector models consistently outperform single-vector models across all test widths, although the performance of both approaches decreases as the test width increases.
Zero-shot baselines
Baseline
Recall@2
Recall@10
Recall@100
Cohere v4
80.6%
93.8%
87.0%
Qwen3
352.2%
346.8%
223.8%
Snowflake
432.3%
396.9%
321.9%
OAI-3-large
1662.1%
1204.5%
1025.0%
Mean gain
631.8%
510.5%
414.4%
Table 2: Relative gains of GTE ModernColBERT over zero-shot and fine-tuned single-vector baselines.
Baseline
Recall@2
Recall@10
Recall@100
SV fine-tuned
104.8%
83.8%
61.5%
SV zero-shot
1613.8%
1263.3%
767.5%
Table 3: Relative gains of jointly fine-tuned Jina Embeddings v4 MV over zero-shot and fine-tuned SV.
Figure 4: Retrieval performance across nine test widths at three recall cutoffs for Jina Embeddings v4, evaluated after joint fine-tuning at a mean training width of 5.5 .
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Count
Range
μ
r
Queries
Categories, test
{7,…,9}
8.0
1
Categories, training
{5,…,7}
6.0
1
Accept-set size, test
{2,…,6}
3.5
0.77
Accept-set size, training
{4,…,8}
5.5
0.77
Documents
Appendix
Table 4: Range and target mean for every sampled count, with the resulting ratio r . The ratio is 1 , and the distribution therefore uniform, wherever the target mean sits at the midpoint of the range. Accept-set sizes are the sole exception and are listed at the narrowest width.
Statistic
Value
Documents
50,000
Categories per document
11–15
Mean categories per document
13.000
Values per present category
2–4
Mean values per present category
2.978
Total values per document
24–55
Appendix
Table 5: Realized corpus statistics. The category-count distribution is exactly balanced by construction, with 10,000 documents at each count from 11 to 15.
Construction role
Documents
Designated test positives
2,000
Injected hard negatives
38,780
level 1 (one category missed)
26,780
level 2 (two categories missed)
8,000
level 3 (three categories missed)
4,000
Random distractors
9,220
Appendix
Table 6: Corpus composition by construction role. A level- k hard negative violates exactly k queried categories at every width. Roles describe construction only; serialized records carry no role field, since relevance is query-dependent.
Suite
Accepted values per category
Realized mean
Total accepted per query
Mean total
Positives
3.5
2–6
3.506
16–41
28.166
2
4.5
3–7
4.506
23–50
36.199
2
5.5
4–8
5.506
30–59
44.232
2
6.5
5–9
6.506
37–68
52.265
2
7.5
6–10
7.506
44–77
60.298
2
8.5
7–11
8.506
51–86
68.331
2
Appendix
Table 7: Test suites. All nine share one corpus, the same 1,000 query skeletons, and the same category counts: 305 queries with 7 categories, 357 with 8, and 338 with 9, for a mean of 8.033. Every test query has exactly two relevant documents at every width, so difficulty is not confounded by a change in relevance cardinality.
Suite
Accepted values per category
Realized mean
Total accepted per query
Mean total
Positives
Mean positives
5.5
4–8
5.491
20–49
32.945
11–1,114
208.104
6.5
5–9
6.491
25–56
38.945
36–1,571
378.955
7.5
6–10
7.491
30–63
44.945
95–2,028
605.574
8.5
7–11
8.491
35–70
50.945
180–2,584
880.398
9.5
8–12
9.491
40–77
56.945
294–3,076
1,185.989
Appendix
Table 8: Training suites used for fine-tuning. All share the same 800 query skeletons, with exactly 267 queries at 5 categories, 266 at 6, and 267 at 7. Relevance is computed exactly against the frozen corpus and no skeleton is filtered by density, so the growth in positives is the effect of widening the accept sets.
Table 13
Figure 5: Recall across test mean widths for fine-tuned GTE ModernColBERT v1, Qwen3 Embedding 0.6B, and Snowflake Arctic Embed L v2 at training mean widths 5.5 , 6.5 , 7.5 , and 8.5 . The corresponding results at training width 9.5 appear in Figure 3 .
Paradigm
Model
Scoring
P1 Single-vector
Qwen3 Embedding 0.6B
Cosine
P2 Single-vector
Snowflake Arctic L v2
Cosine
P3 Late interaction
GTE ModernColBERT v1
MaxSim
P4 Joint SV–MV
Jina Embeddings v4
Both
P5 Separate SV, MV
Jina Embeddings v4
Both
Appendix
Table 10: The five fine-tuning paradigms. Each is run once per training width, giving five runs per paradigm. P5 trains two independent models per width.
Run
P
τ
LR
WD
Clip
P1 Qwen3
2
0.05
3
0.01
64
P2 Snowflake
2
0.05
3
0.01
64
P3 ColBERT
1
1/320
10
0
1
P4 Jina joint
2
0.05
3
0.01
—
P5 Jina SV
2
0.05
3
0.01
—
P5 Jina MV
1
1/320
10
0.01
—
Appendix
Table 11: Objective and optimization settings. P is the number of supervised positives in Equation 31 ; all runs score D=32 candidates. Learning rates are given in units of 10−7 . The temperature 1/320 is the reciprocal of the query-token budget and reproduces unnormalized summed MaxSim logits.
Run
Precision
Batch
Accum.
Ckpt. grad.
P1 Qwen3
bf16
8
4
Yes
P2 Snowflake
bf16
8
4
Yes
P3 ColBERT
fp16
1
32
No
P4 Jina joint
bf16
1
32/G
No
P5 Jina SV
bf16
1
32/G
No
P5 Jina MV
bf16
1
32/G
No
Appendix
Table 12: Systems settings. The global effective batch size, computed as the number of devices times the per-device batch times the accumulation count, is 32 in every run; G is the number of visible devices, which is one, two, or four for the Jina runs.
Figure 6: nDCG@2, nDCG@10, and nDCG@100 across test mean widths for fine-tuned GTE ModernColBERT v1, Qwen3 Embedding 0.6B, and Snowflake Arctic Embed L v2 at all five training mean widths from 5.5 to 9.5 .
Figure 7: MRR across test mean widths for fine-tuned GTE ModernColBERT v1, Qwen3 Embedding 0.6B, and Snowflake Arctic Embed L v2 at all five training mean widths from 5.5 to 9.5 .
Figure 8: Recall@100 as a function of training mean width for fine-tuned GTE ModernColBERT v1, Qwen3 Embedding 0.6B, and Snowflake Arctic Embed L v2. The nine panels cover every test mean width from 3.5 to 11.5 .
Figure 9: Recall across test mean widths for the jointly trained Jina Embeddings v4 single-vector and multi-vector heads at training mean widths 6.5 , 7.5 , 8.5 , and 9.5 . Solid lines are fine-tuned results and dotted lines are the corresponding zero-shot baselines. Results at training width 5.5 appear in Figure 4 .
Figure 10: nDCG@2, nDCG@10, and nDCG@100 across test mean widths for the jointly trained Jina Embeddings v4 single-vector and multi-vector heads at all five training mean widths from 5.5 to 9.5 . Solid lines are fine-tuned results and dotted lines are the corresponding zero-shot baselines.
Figure 11: MRR across test mean widths for the jointly trained Jina Embeddings v4 single-vector and multi-vector heads at all five training mean widths from 5.5 to 9.5 . Solid lines are fine-tuned results and dotted lines are the corresponding zero-shot baselines.
Figure 12: Recall across test mean widths for Jina Embeddings v4 single-vector and multi-vector models trained separately at all five training mean widths. Solid lines are fine-tuned results and dotted lines are the corresponding zero-shot baselines.
Figure 13: nDCG@2, nDCG@10, and nDCG@100 across test mean widths for Jina Embeddings v4 single-vector and multi-vector models trained separately at all five training mean widths. Solid lines are fine-tuned results and dotted lines are the corresponding zero-shot baselines.
Figure 14: MRR across test mean widths for Jina Embeddings v4 single-vector and multi-vector models trained separately at all five training mean widths. Solid lines are fine-tuned results and dotted lines are the corresponding zero-shot baselines.
Multimodal retrieval relies heavily on single-vector retrievers, which compress rich, sequential token sequences into one single global representation. While efficient, they discard fine-grained, local evidence critical for dense retrieval tasks. Multi-vector approaches were introduced as a solution, but they strictly require training and many ignore the necessity of a globally summarizing representation. To address this, we introduce SMART, a framework that unlocks the latent multi-vector capabilities of standard single-vector models. We first observe that gradients from the pooled contrastive objective reach the preceding hidden states, and then show empirically that these states are already organized for cosine-based matching. By applying direct late-interaction over these frozen hidden states during inference, SMART acts as a plug-and-play upgrade that consistently improves performance across diverse modalities, improving even the state-of-the-art models further on MMEB-V2. We also show that lightweight post-training saves time and compute while further improving Visual Document retrieval, allowing a single-vector model to perform competitively with SoTA multi-vector counterparts. SMART targets dense retrieval: its gains concentrate on fine-grained and visual-document tasks, it is on par on global tasks such as classification, and it trades additional per-token index storage for these gains. Ultimately, SMART offers both a highly efficient inference enhancement and a powerful finetuning technique for multimodal retrieval. We open source our code and weights at https://github.com/HanSolo9682/SMART.
Multi-vector retrieval (MVR) models, exemplified by ColBERT, have established new benchmarks in retrieval accuracy by preserving fine-grained token-level interactions. However, this granularity imposes prohibitive storage and retrieval efficiency bottlenecks: to manage the immense memory footprint and computational overhead of billion-scale token vectors, state-of-the-art systems are forced to rely on aggressive dimension reduction and complex clustering (e.g., K-means). This compromise introduces two critical limitations: excessive indexing latency of clustering large-scale corpora and semantic information loss inherent to compression. In this paper, we propose Single-stage Sparse Retrieval (SSR}, a paradigm shift that replaces expensive clustering with efficient sparse coding. Instead of compressing features into low-dimensional dense vectors, we utilize Sparse Autoencoder (SAE) to project token embeddings into a high-dimensional but highly sparse representation. This transformation enables us to bypass vector clustering entirely and leverage inverted indexing for precise, high-throughput retrieval. Extensive experiments on the BEIR benchmark demonstrate that SSR achieves a "trifecta" of improvements: it reduces indexing time by 15x compared to ColBERTv2, halves retrieval latency, and simultaneously improves retrieval performance over leading baselines.
Lixuan Guo, Yifei Wang, Tiansheng Wen +3
Stony Brook University · Xidian University · Amazon AGI SF Lab +4
We propose Latent Terms, a method revealing that models trained for dense retrieval, whether single- or multi-vector, learn representations that can trivially be decomposed into retrieval-ready sparse features. When trained on frozen retrievers, Sparse Autoencoders without any retrieval-specific adjustments extract a latent vocabulary with approximately Zipfian collection statistics, directly suitable for classical sparse retrieval scoring via BM25. This approach enables sparse retrieval while requiring no learned expansion objective or sparse retrieval supervision whatsoever, and can be readily applied to any dense retriever. Latent Terms is able to match or outperform single-vector scoring methods from its own base model as well as comparable SPLADE variants. In addition, it substantially outperforms its base model on LIMIT, a task specifically designed to highlight the failures of single-vector retrieval. Overall, our results highlight that neural retrievers contain more expressive and indexable structure than their default scoring functions expose, but that other methods can nonetheless be leveraged.
Benjamin Clavié, Sean Lee, Aamir Shakir +1
Mixedbread AI · National Institute of Informatics (NII) · University of Tsukuba