Multi-vector representations have emerged as an effective paradigm for multimodal retrieval, representing each sample with multiple complementary embeddings to capture fine-grained cross-modal information. However, existing approaches typically employ a fixed representation capacity, assigning the same number of vectors to all samples regardless of their individual retrieval demands. Such a fixed-capacity formulation overlooks the fact that different samples may require different amounts of representation capacity for effective retrieval. In this work, we introduce Sample-Adaptive Multi-Vector Representation (SAMVR), a new problem setting for multimodal retrieval that studies how multi-vector representation capacity can be allocated at the sample level. Under SAMVR, each sample is represented by a content-adaptive embedding set (CAES), whose capacity is determined according to the sample-specific retrieval utility of additional representation vectors. To instantiate SAMVR, we propose AdaptiveEmbed, a unified framework for learning sample-adaptive multi-vector representations. AdaptiveEmbed learns structured multi-vector representations through Multi-Group Contrastive Learning (MGCL) with the symmetric set-to-set similarity (SetSim), and further employs Utility Policy Optimization (UPO) to determine sample-specific representation capacity via Marginal Utility Allocation (MUA). Experiments across multimodal retrieval benchmarks involving image, text, video, and audio show that sample-adaptive capacity allocation achieves overall better retrieval performance than fixed-capacity multi-vector representations, validating the effectiveness of SAMVR for multimodal retrieval. These results establish SAMVR as a viable formulation for adaptive capacity allocation in multi-vector multimodal retrieval.
Figures & tables
Figure 1: Motivation for sample-adaptive multi-vector representation in multimodal retrieval. (a) Multimodal samples exhibit diverse content characteristics. (b) Samples grouped by oracle-optimal k show different capacity–performance trends, motivating sample-adaptive allocation. (c) Oracle-preferred capacities are distributed across the available range, indicating substantial sample-level variation in representation demand. Results are visualized on COCO.
Figure 2: Comparison between MaxSim and SetSim for bidirectional multimodal retrieval. (a) MaxSim defines a directed comparison, leading the two retrieval directions to use role-specific query/database configurations. (b) SetSim performs symmetric one-to-one matching between embedding sets, allowing each modality to use a single embedding set for both retrieval directions.
Figure 3: Overview of AdaptiveEmbed. The framework first learns a structured multi-vector embedding set through Multi-Group Contrastive Learning (MGCL), and then constructs a sample-specific content-adaptive embedding set (CAES) using a Token Selection Transformer (TST) trained with Utility Policy Optimization (UPO) and a Marginal Utility Allocation (MUA) strategy.
COCO
Flickr30K
ADE20K
OpenImages
Method
Avg. Tokens
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
Single-Vector Retrieval
CLIP ViT-L/14
1
9.38
12.78
15.58
21.42
7.16
11.80
3.85
7.71
SigLIP
1
23.04
23.16
32.55
37.20
15.76
19.87
9.19
15.65
VLM2Vec-V2
1
35.15
36.96
47.12
45.43
24.36
27.00
15.19
18.61
Qwen3-VL-Embedding-2B
1
41.62
41.11
56.25
57.05
28.41
28.63
14.63
16.36
Table 1: Full-gallery bidirectional image–text retrieval results on four datasets, measured by mAP (%). T2I and I2T denote text-to-image and image-to-text retrieval, respectively. Avg. Tokens is computed on COCO.
ActivityNet
DiDeMo
Clotho
MACS
Method
Avg. Tokens
T2V
V2T
T2V
V2T
T2A
A2T
T2A
A2T
Single-Vector Retrieval
VLM2Vec-V2
1
33.59
33.34
20.97
18.90
–
–
–
–
Qwen3-VL-Embedding-2B
1
60.32
48.46
47.12
41.42
–
–
–
–
Qwen3-VL-Embedding-8B
1
66.26
57.48
54.76
48.35
–
–
–
–
LAION-CLAP
1
–
–
–
–
16.21
20.42
10.21
9.60
Table 2: Generalization of sample-adaptive multi-vector representations to video–text and audio–text retrieval, measured by mAP (%). T2V, V2T, T2A, and A2T denote text-to-video, video-to-text, text-to-audio, and audio-to-text retrieval, respectively. Avg. Tokens is computed on ActivityNet.
COCO
Flickr30K
ActivityNet
Clotho
Method
Avg. Tokens
T2I
I2T
T2I
I2T
T2V
V2T
T2A
A2T
AdaptiveEmbed (Ours)
2.1
63.45
61.14
78.27
75.72
61.62
56.48
32.56
31.75
w/o MGCL
2.1
62.44
60.03
77.85
75.57
61.35
56.21
30.16
29.94
w/o bank-feedback
2.5
62.89
59.58
77.85
75.19
61.41
56.25
31.43
30.62
Table 3: Ablation study of AdaptiveEmbed. We evaluate the contribution of Multi-Group Contrastive Learning (MGCL) and bank feedback on bidirectional multimodal retrieval. Results are measured by mAP (%). Avg. Tokens is computed on COCO.
Figure 4: Analysis of adaptive representation on Flickr30K under zero-shot transfer. (a) Under the same MGCL-trained representation, fixed- K retrieval does not improve monotonically with capacity, whereas AdaptiveEmbed surpasses every fixed- K operating point at a lower average capacity; the oracle marks the upper bound of sample-wise allocation. (b) Multi-group representation learning outperforms single-group structures under markedly different token allocations. (c) Under MaxSim, the fraction of query tokens matched to distinct database tokens drops from 100.0% to 80.8% as capacity grows, whereas SetSim guarantees one-to-one coverage by construction.
Figure 5: Efficiency and parameter sensitivity analysis of AdaptiveEmbed.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Oracle content-adaptive embedding sets on COCO. Each sample shows its narrative and, for both retrieval directions, the oracle-selected tokens, the resulting SetSim score, and the capacity k .
Modality
Dataset
Train
Queries
Gallery
Query split
Text construction
Image–Text
COCO
118,287
5,000
123,287
val
Full localized narrative
Flickr30K
29,783
1,000
30,783
val
Full localized narrative
ADE20K
20,210
2,000
22,210
val
Full localized narrative
OpenImages
504,413
41,620
546,033
val
Full localized narrative
Video–Text
ActivityNet
10,009
4,917
14,926
val1
Temporally ordered event captions
DiDeMo
8,395
1,004
9,399
test
Joined moment descriptions
Appendix
Table 4: Dataset statistics and unified full-gallery retrieval protocols. The Train column denotes the gallery prefix used for representation or policy training when applicable. Each query has exactly one positive item in the full gallery.
Method
kˉ
R@1
R@5
R@10
nDCG
mAP
MeanR ↓
MetaEmbed (2/4)
3
51.79
72.54
79.16
69.13
61.40
30.58
MetaEmbed (4/8)
6
51.52
72.10
79.30
68.96
61.17
31.34
MetaEmbed (8/16)
12
50.99
71.86
79.05
68.65
60.76
30.18
MetaEmbed (16/64)
40
50.92
71.88
78.78
68.59
60.69
30.78
AdaptiveEmbed (Ours)
2.1
52.86
73.32
79.55
69.85
62.30
31.26
Appendix
Table 5: Additional retrieval metrics on COCO under the unified full-gallery protocol ( 5,000 queries, 123,287 gallery items), averaged over both directions. AdaptiveEmbed leads every Recall and nDCG metric at the lowest average capacity, indicating that the conclusions of Table 1 are not an artifact of the mAP metric.
Method
kˉ
R@1
R@5
R@10
nDCG
mAP
MeanR ↓
MetaEmbed (2/4)
3
35.83
58.40
67.75
56.81
46.56
46.09
MetaEmbed (4/8)
6
37.28
59.78
68.60
57.88
47.83
44.26
MetaEmbed (8/16)
12
37.33
59.88
69.25
58.03
47.99
44.49
MetaEmbed (16/64)
40
37.28
59.53
68.93
57.84
47.81
48.08
AdaptiveEmbed (Ours)
2.0
38.08
61.33
69.78
58.66
48.74
46.04
Appendix
Table 6: Additional retrieval metrics on ADE20K under strict zero-shot transfer ( 2,000 queries, 22,210 gallery items), averaged over both directions. The representation, policy, threshold, and retrieval bank are all taken from COCO without any ADE20K tuning. AdaptiveEmbed outperforms every MetaEmbed configuration on all Recall, nDCG, and mAP metrics at the lowest average capacity.
Figure 7: Retrieval utility versus representation cost under adaptive and fixed-capacity allocation, all using the same MGCL-trained representation.
Figure 8: Ablation of single-group and multi-group representation structures. Multi-group contrastive learning consistently improves retrieval performance across different datasets, demonstrating that separating representation capacity into multiple semantic groups provides a more effective basis for adaptive token activation.
Figure 9: Decision complexity analysis of the hierarchical adaptive policy. The bars show the average number of independent responses required at each decision stage, while the curves indicate the remaining response distinctiveness after each stage. Early decisions provide coarse capacity separation, allowing later stages to operate on a reduced candidate space.
Figure 10: Oracle and AdaptiveEmbed selection distributions on COCO, ActivityNet, and MACS. Each configuration is represented as a binary mask over the eight candidate tokens indicating which are activated; the valid group prefixes yield 20 canonical masks in total. The oracle exhaustively evaluates all 20 masks per sample, whereas AdaptiveEmbed approximates the optimal allocation through its hierarchical policy.