Multi-vector representations have emerged as an effective paradigm for multimodal retrieval, representing each sample with multiple complementary embeddings to capture fine-grained cross-modal information. However, existing approaches typically employ a fixed representation capacity, assigning the same number of vectors to all samples regardless of their individual retrieval demands. Such a fixed-capacity formulation overlooks the fact that different samples may require different amounts of representation capacity for effective retrieval. In this work, we introduce Sample-Adaptive Multi-Vector Representation (SAMVR), a new problem setting for multimodal retrieval that studies how multi-vector representation capacity can be allocated at the sample level. Under SAMVR, each sample is represented by a content-adaptive embedding set (CAES), whose capacity is determined according to the sample-specific retrieval utility of additional representation vectors. To instantiate SAMVR, we propose AdaptiveEmbed, a unified framework for learning sample-adaptive multi-vector representations. AdaptiveEmbed learns structured multi-vector representations through Multi-Group Contrastive Learning (MGCL) with the symmetric set-to-set similarity (SetSim), and further employs Utility Policy Optimization (UPO) to determine sample-specific representation capacity via Marginal Utility Allocation (MUA). Experiments across multimodal retrieval benchmarks involving image, text, video, and audio show that sample-adaptive capacity allocation achieves overall better retrieval performance than fixed-capacity multi-vector representations, validating the effectiveness of SAMVR for multimodal retrieval. These results establish SAMVR as a viable formulation for adaptive capacity allocation in multi-vector multimodal retrieval.
Figures & tables
Figure 1: Motivation for sample-adaptive multi-vector representation in multimodal retrieval. (a) Multimodal samples exhibit diverse content characteristics. (b) Samples grouped by oracle-optimal k show different capacity–performance trends, motivating sample-adaptive allocation. (c) Oracle-preferred capacities are distributed across the available range, indicating substantial sample-level variation in representation demand. Results are visualized on COCO.
Figure 2: Comparison between MaxSim and SetSim for bidirectional multimodal retrieval. (a) MaxSim defines a directed comparison, leading the two retrieval directions to use role-specific query/database configurations. (b) SetSim performs symmetric one-to-one matching between embedding sets, allowing each modality to use a single embedding set for both retrieval directions.
Figure 3: Overview of AdaptiveEmbed. The framework first learns a structured multi-vector embedding set through Multi-Group Contrastive Learning (MGCL), and then constructs a sample-specific content-adaptive embedding set (CAES) using a Token Selection Transformer (TST) trained with Utility Policy Optimization (UPO) and a Marginal Utility Allocation (MUA) strategy.
COCO
Flickr30K
ADE20K
OpenImages
Method
Avg. Tokens
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
Single-Vector Retrieval
CLIP ViT-L/14
1
9.38
12.78
15.58
21.42
7.16
11.80
3.85
7.71
SigLIP
1
23.04
23.16
32.55
37.20
15.76
19.87
9.19
15.65
VLM2Vec-V2
1
35.15
36.96
47.12
45.43
24.36
27.00
15.19
18.61
Qwen3-VL-Embedding-2B
1
41.62
41.11
56.25
57.05
28.41
28.63
14.63
16.36
Table 1: Full-gallery bidirectional image–text retrieval results on four datasets, measured by mAP (%). T2I and I2T denote text-to-image and image-to-text retrieval, respectively. Avg. Tokens is computed on COCO.
ActivityNet
DiDeMo
Clotho
MACS
Method
Avg. Tokens
T2V
V2T
T2V
V2T
T2A
A2T
T2A
A2T
Single-Vector Retrieval
VLM2Vec-V2
1
33.59
33.34
20.97
18.90
–
–
–
–
Qwen3-VL-Embedding-2B
1
60.32
48.46
47.12
41.42
–
–
–
–
Qwen3-VL-Embedding-8B
1
66.26
57.48
54.76
48.35
–
–
–
–
LAION-CLAP
1
–
–
–
–
16.21
20.42
10.21
9.60
Table 2: Generalization of sample-adaptive multi-vector representations to video–text and audio–text retrieval, measured by mAP (%). T2V, V2T, T2A, and A2T denote text-to-video, video-to-text, text-to-audio, and audio-to-text retrieval, respectively. Avg. Tokens is computed on ActivityNet.
COCO
Flickr30K
ActivityNet
Clotho
Method
Avg. Tokens
T2I
I2T
T2I
I2T
T2V
V2T
T2A
A2T
AdaptiveEmbed (Ours)
2.1
63.45
61.14
78.27
75.72
61.62
56.48
32.56
31.75
w/o MGCL
2.1
62.44
60.03
77.85
75.57
61.35
56.21
30.16
29.94
w/o bank-feedback
2.5
62.89
59.58
77.85
75.19
61.41
56.25
31.43
30.62
Table 3: Ablation study of AdaptiveEmbed. We evaluate the contribution of Multi-Group Contrastive Learning (MGCL) and bank feedback on bidirectional multimodal retrieval. Results are measured by mAP (%). Avg. Tokens is computed on COCO.
Figure 4: Analysis of adaptive representation on Flickr30K under zero-shot transfer. (a) Under the same MGCL-trained representation, fixed- K retrieval does not improve monotonically with capacity, whereas AdaptiveEmbed surpasses every fixed- K operating point at a lower average capacity; the oracle marks the upper bound of sample-wise allocation. (b) Multi-group representation learning outperforms single-group structures under markedly different token allocations. (c) Under MaxSim, the fraction of query tokens matched to distinct database tokens drops from 100.0% to 80.8% as capacity grows, whereas SetSim guarantees one-to-one coverage by construction.
Figure 5: Efficiency and parameter sensitivity analysis of AdaptiveEmbed.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Oracle content-adaptive embedding sets on COCO. Each sample shows its narrative and, for both retrieval directions, the oracle-selected tokens, the resulting SetSim score, and the capacity k .
Modality
Dataset
Train
Queries
Gallery
Query split
Text construction
Image–Text
COCO
118,287
5,000
123,287
val
Full localized narrative
Flickr30K
29,783
1,000
30,783
val
Full localized narrative
ADE20K
20,210
2,000
22,210
val
Full localized narrative
OpenImages
504,413
41,620
546,033
val
Full localized narrative
Video–Text
ActivityNet
10,009
4,917
14,926
val1
Temporally ordered event captions
DiDeMo
8,395
1,004
9,399
test
Joined moment descriptions
Appendix
Table 4: Dataset statistics and unified full-gallery retrieval protocols. The Train column denotes the gallery prefix used for representation or policy training when applicable. Each query has exactly one positive item in the full gallery.
Method
kˉ
R@1
R@5
R@10
nDCG
mAP
MeanR ↓
MetaEmbed (2/4)
3
51.79
72.54
79.16
69.13
61.40
30.58
MetaEmbed (4/8)
6
51.52
72.10
79.30
68.96
61.17
31.34
MetaEmbed (8/16)
12
50.99
71.86
79.05
68.65
60.76
30.18
MetaEmbed (16/64)
40
50.92
71.88
78.78
68.59
60.69
30.78
AdaptiveEmbed (Ours)
2.1
52.86
73.32
79.55
69.85
62.30
31.26
Appendix
Table 5: Additional retrieval metrics on COCO under the unified full-gallery protocol ( 5,000 queries, 123,287 gallery items), averaged over both directions. AdaptiveEmbed leads every Recall and nDCG metric at the lowest average capacity, indicating that the conclusions of Table 1 are not an artifact of the mAP metric.
Method
kˉ
R@1
R@5
R@10
nDCG
mAP
MeanR ↓
MetaEmbed (2/4)
3
35.83
58.40
67.75
56.81
46.56
46.09
MetaEmbed (4/8)
6
37.28
59.78
68.60
57.88
47.83
44.26
MetaEmbed (8/16)
12
37.33
59.88
69.25
58.03
47.99
44.49
MetaEmbed (16/64)
40
37.28
59.53
68.93
57.84
47.81
48.08
AdaptiveEmbed (Ours)
2.0
38.08
61.33
69.78
58.66
48.74
46.04
Appendix
Table 6: Additional retrieval metrics on ADE20K under strict zero-shot transfer ( 2,000 queries, 22,210 gallery items), averaged over both directions. The representation, policy, threshold, and retrieval bank are all taken from COCO without any ADE20K tuning. AdaptiveEmbed outperforms every MetaEmbed configuration on all Recall, nDCG, and mAP metrics at the lowest average capacity.
Figure 7: Retrieval utility versus representation cost under adaptive and fixed-capacity allocation, all using the same MGCL-trained representation.
Figure 8: Ablation of single-group and multi-group representation structures. Multi-group contrastive learning consistently improves retrieval performance across different datasets, demonstrating that separating representation capacity into multiple semantic groups provides a more effective basis for adaptive token activation.
Figure 9: Decision complexity analysis of the hierarchical adaptive policy. The bars show the average number of independent responses required at each decision stage, while the curves indicate the remaining response distinctiveness after each stage. Early decisions provide coarse capacity separation, allowing later stages to operate on a reduced candidate space.
Figure 10: Oracle and AdaptiveEmbed selection distributions on COCO, ActivityNet, and MACS. Each configuration is represented as a binary mask over the eight candidate tokens indicating which are activated; the valid group prefixes yield 20 canonical masks in total. The oracle exhaustively evaluates all 20 masks per sample, whereas AdaptiveEmbed approximates the optimal allocation through its hierarchical policy.
Multimodal retrieval relies heavily on single-vector retrievers, which compress rich, sequential token sequences into one single global representation. While efficient, they discard fine-grained, local evidence critical for dense retrieval tasks. Multi-vector approaches were introduced as a solution, but they strictly require training and many ignore the necessity of a globally summarizing representation. To address this, we introduce SMART, a framework that unlocks the latent multi-vector capabilities of standard single-vector models. We first observe that gradients from the pooled contrastive objective reach the preceding hidden states, and then show empirically that these states are already organized for cosine-based matching. By applying direct late-interaction over these frozen hidden states during inference, SMART acts as a plug-and-play upgrade that consistently improves performance across diverse modalities, improving even the state-of-the-art models further on MMEB-V2. We also show that lightweight post-training saves time and compute while further improving Visual Document retrieval, allowing a single-vector model to perform competitively with SoTA multi-vector counterparts. SMART targets dense retrieval: its gains concentrate on fine-grained and visual-document tasks, it is on par on global tasks such as classification, and it trades additional per-token index storage for these gains. Ultimately, SMART offers both a highly efficient inference enhancement and a powerful finetuning technique for multimodal retrieval. We open source our code and weights at https://github.com/HanSolo9682/SMART.
Universal multimodal retrieval requires compact embeddings that preserve task-relevant semantic information across diverse modalities. Prior works have incorporated latent reasoning into multimodal embedding learning to refine this information before embedding extraction. However, most existing approaches remain confined to deterministic latent paths, without exploring alternative trajectories to discover better embeddings. Thus, we propose VaME (Variational Multimodal Embeddings), a framework that models latent reasoning as a learnable distribution over trajectories. Specifically, we first introduce Variational Latent Reasoning (VLR) to enable autoregressive exploration in latent space, guided by answer reconstruction through a lightweight decoder. Meanwhile, we augment the original embedding-token readout with a latent-fused embedding to facilitate exploration during subsequent reinforcement learning. Finally, we optimize latent reasoning over stochastic variational trajectories through reinforcement learning, using Semantic Decoding Reward (SDR) to favor semantically meaningful trajectories with interpretable decoded outcomes. On the 78-task MMEB-V2 benchmark, spanning image, video, and visual-document retrieval, VaME outperforms most explicit CoT-based models and all latent-reasoning baselines. VaME also demonstrates robust performance on reasoning-intensive benchmarks such as MRMR, with substantial gains after reinforcement learning. Importantly, VaME achieves these gains with at least a 4.25x inference speedup over the deterministic latent autoregressive baselines. The code will be made publicly available.
Peixi Wu, Mingzhou Jiang, Feipeng Ma +11
University of Science and Technology of China · Tsinghua University · Kuaishou +1
Different machine learning models can represent the same underlying concept in different ways. This variability is particularly valuable for in-the-wild multimodal retrieval, where the objective is to identify the corresponding representation in one modality given another modality as input. This challenge can be effectively framed as a representation alignment problem. For example, given a sentence encoded by a language model, retrieve the most semantically aligned image based on representations produced by an image encoder, or vice versa. To gain insights into the performance impact of different metrics, embedding spaces, and representation alignment for retrieval tasks, we first empirically investigate the geometric relationships between visual and textual embeddings derived from both vision-language models and combined unimodal models. We then align these representations using four standard similarity metrics as well as two learned ones, implemented via neural networks of different architectures with varying losses across multiple benchmarks. Our experimental findings indicate that cosine similarity consistently outperforms all the investigated metrics in representation alignment tasks, and that Wasserstein distance provides a complementary perspective on cross-modal distributional differences. We also observe that our proposed custom contrastive loss is advantageous over the MSE loss for aligning image and text representations, for both multilayer perceptrons and transformer-based models. Taken together, our findings offer novel insights and practical considerations for researchers working in multimodal information retrieval, particularly in real-world, cross-modal applications. Our code is publicly available.
Fan Xu, Luis A. Leiva
aDepartment of Computer Science, University of Luxembourg, 6, avenue de la Fonte, Esch-sur-Alzette, L-4364, Luxembourg