Organizations: AI Center, Samsung Electronics, Republic of Korea · Dept. of CSE, POSTECH, Republic of Korea · Dept. of AI, Ajou University, Republic of Korea · Graduate School of AI, POSTECH, Republic of Korea
At the heart of composed visual data retrieval is the fusion of a reference visual input and a textual modification into a single query. While current state-of-the-art methods utilize multimodal large language models for this fusion, their complexity introduces prohibitive querytime latency, limiting their scalability. We instead revisit the efficacy of simple linear interpolation within an embedding space, and introduce SRAIN, the first framework that dynamically predicts query-specific interpolation weights. The key challenge lies in the fact that the quality of an interpolation weight should be measured by the interpolated embedding's discriminability from negatives as well as its proximity to true targets; this makes collecting and predicting optimal weights intractable. We overcome this bottleneck through two key innovations: batch-wise rank-aware weight estimation during training, and a compact memory bank that synthesizes hard negatives during inference. SRAIN achieves the best in composed video retrieval and matches the current state of the art in composed image retrieval, all while substantially reducing querytime latency compared to MLLM-based alternatives.
Figures & tables
Figure 1: Comparison of SRAIN with MLLM-based methods. (a) MLLM-based methods process reference and modification text together through MLLMs to generate a composed query, introducing substantial computational overhead. (b) SRAIN predicts a sample-specific interpolation weight α and fuses reference and modification embeddings through linear interpolation, enabling efficient retrieval. The right plot shows that SRAIN achieved comparable performance (mAP@5 on CIRCO) while being × 24 faster in throughput compared to MLLM-based methods. Meanwhile, BASIC [ 26 ] , a training-free method without MLLM, underperformed in throughput as well as retrieval accuracy since it requires numerous text encoder forward passes.
Figure 2: Rank-aware optimal weight estimation. Rather than determining the weight solely based on the similarity to the positive target (a), SRAIN selects the weight based on rank, where the positive consistently outranks nearby hard negatives (b).
Figure 3: Overall architecture of SRAIN. The reference video, modification text, and target video are encoded using the pretrained encoders. For each batch, optimal interpolation weights are estimated in a rank-aware manner. The model is trained in two stages: In Stage\penalty1 , the encoders are fine-tuned to align composite query embeddings with target embeddings through hard-negative contrastive learning. In Stage\penalty2 , the encoders are frozen and a lightweight weight predictor is trained to regress αi∗ . A memory bank is updated via exponential moving average (EMA) during training to collect representative target prototypes. At inference, the memory bank supplies top- n prototypes as auxiliary tokens, enabling consistent weight prediction.
Methods
External
Modality
Fusion
Backbone
Evaluation Metric
Knowledge
R@1
R@5
R@10
R@50
Text-only ( α=1 )
✗
Text
-
BLIP-2
23.32
46.21
56.22
78.36
Visual-only ( α=0 )
✗
Visual
-
BLIP-2
36.03
64.24
74.77
92.64
Average ( α=0.5 )
✗
Visual+Text
Avg
BLIP-2
58.69
83.88
90.49
98.21
CoVR [ 43 ]
✗
Visual+Text
MLP
BLIP
50.59
74.65
83.57
95.46
CoVR-2 [ 42 ]
✗
Visual+Text
MLP
BLIP-2
51.88
79.38
86.46
97.42
Table 1: Composed video retrieval results on the WebVid-CoVR-Test, where all compared methods are fine-tuned on the WebVid-CoVR training set. Compared methods differ in fusion strategy: Avg indicates element-wise averaging of reference and modification embeddings, MLP denotes a multi-layer perceptron-based fusion module, CA employs cross-attention, and LI refers to linear interpolation. External Knowledge denotes the use of external models for additional video descriptions generation.
Methods
ViT
External
Pretraining
Dress
Shirt
Toptee
Average
Knowledge
Data
R@10
R@50
R@10
R@50
R@10
R@50
R@10
R@50
Zero-shot
CIReVL [ 15 ]
G
GPT-4 [ 1 ]
✗
27.07
49.53
33.71
51.42
35.80
56.14
32.19
52.36
OSrCIR [ 38 ]
G
GPT-4o [ 1 ]
✗
33.02
54.78
38.65
54.71
41.04
61.83
37.57
57.11
Pic2Word [ 28 ]
L
✗
CC3M
20.00
40.20
26.20
43.60
27.90
47.40
24.70
43.70
SEARLE-XL [ 3 ]
L
✗
CC3M
26.89
45.58
20.48
43.13
29.32
49.97
25.56
46.23
Table 2: Composed image retrieval results on the FashionIQ validation set. CC3M [ 32 ] , LAION2B (L2B) [ 29 ] , LAION2M (L2M), and LLaVA-Align (L-A) [ 23 ] contain 3M, 2.3B, 2M, and 585k image-text pairs, respectively, where LAION2M is a subset of LAION400M [ 30 ] . MagicLens36M [ 53 ] comprises 36M (query image, instruction, target image) triplets. SDP [ 51 ] provides 2.47M text prompts, while ST18M [ 10 ] and LaSCo [ 18 ] comprise 18.8M and 389k triplets for training, respectively.
Methods
ViT
External
mAP@ K
Knowledge
K =5
K =10
K =25
K =50
TFCIR [ 34 ]
G
✓
26.52
28.25
31.23
31.99
CIReVL [ 15 ]
G
✓
26.77
27.59
29.96
31.03
OSrCIR [ 38 ]
G
✓
30.47
31.14
35.03
36.59
SEARLE-XL [ 3 ]
L
✗
11.68
12.73
14.33
15.12
CompoDiff [ 10 ]
G
✗
15.33
17.71
19.45
21.01
Table 3: Zero-shot composed image retrieval results on CIRCO.
Methods
ViT
R@ K
K =1
K =5
K =10
CLIP4CIR [ 4 ]
L
33.59
65.35
77.35
CompoDiff [ 10 ]
G
32.39
57.61
77.25
CoVR [ 43 ]
L
49.69
78.60
86.77
CoVR-2 [ 42 ]
G
50.87
80.80
88.84
SRAIN (Ours)
G
50.95
81.21
89.60
Table 4: Composed image retrieval results on CIRR.
Figure 4: Top-4 retrieval results on the WebVid-CoVR test set. The ground-truth target video is highlighted in green.
Methods
R@1
R@5
R@10
R@50
(a) Training Strategy
Average (BLIP-2 frozen)
45.66
71.71
81.30
94.80
Stage 2 only (BLIP-2 frozen)
48.08
73.40
83.22
95.74
End-to-end training
57.36
82.71
89.20
97.77
Stage 1 only
60.50
83.90
91.09
98.40
(b) Effect of Ci in Eq. 5
Table 5: Ablation studies on key components of our method.
CIRCO
FIQ-D
FIQ-S
FIQ-T
FIQ-A
Methods
mAP@5
R@10
R@10
R@10
R@10
w/o Memory
23.79
21.49
30.98
29.63
27.37
Random Vector
23.72
21.17
30.96
29.57
27.23
SRAIN (Ours)
26.16
23.22
33.14
31.91
29.42
Table 6: Ablation studies on the memory bank for zero-shot CoIR.
Composed image retrieval, multi-turn composed image retrieval, and composed video retrieval all share a common paradigm: composing the reference visual with modification text to retrieve the desired target. Despite this shared structure, the three tasks have been studied in isolation, with no prior work proposing a unified framework, let alone a zero-shot solution. In this paper, we propose UniCVR, the first unified zero-shot composed visual retrieval framework that jointly addresses all three tasks without any task-specific human-annotated data. UniCVR strategically combines two complementary strengths: Multimodal Large Language Models (MLLMs) for compositional query understanding and Vision-Language Pre-trained (VLP) models for structured visual retrieval. Concretely, UniCVR operates in two stages. In Stage I, we train the MLLM as a compositional query embedder via contrastive learning on a curated multi-source dataset of approximately 3.5M samples, bridging the heterogeneous embedding spaces between the MLLM and the frozen VLP gallery encoder. A cluster-based hard negative sampling strategy is proposed to strengthen contrastive supervision. In Stage II, we introduce an MLLM-guided dual-level reranking mechanism that applies adaptive budgeted subset scoring to a small number of top-ranked candidates, and then exploits the resulting relevance signals through a dual-level re-scoring scheme, producing more accurate final rankings with minimal computational overhead. Extensive experiments across five benchmarks covering all three tasks demonstrate that UniCVR achieves cutting-edge performance, validating its effectiveness and generalizability. Our data and code will be released upon acceptance.
Haokun Wen, Xuemeng Song, Haoyu Zhang +3
Harbin Institute of Technology (Shenzhen) Shenzhen, China · Southern University of Science and Technology Shenzhen, China · Pengcheng Laboratory Shenzhen, China +1
The CoVR-R challenge evaluates composed video retrieval, where a system must retrieve a target video from a large gallery given a reference video and a textual edit instruction. This setting is not a standard video-text retrieval problem: the query is defined by both the visual evidence in the source video and the transformation implied by the edit. A strong embedding model can provide scalable candidate recall, but it may under-express target-side consequences such as state changes, action replacement, object preservation, or temporal consistency. A pairwise multimodal reranker can verify such details more directly, but exhaustive reranking over the full gallery is computationally infeasible. We present R3, a zero-shot composed video retrieval pipeline built around Reasoning-guided Recalling and Reranking. The core idea is to turn the source-edit query into a reasoning-grounded retrieval program rather than treating the edit text as a short caption. First, the model generates a reasoning trace that describes the expected target video after applying the edit. Then the trace is encoded together with the source video as a reasoning-augmented query, and its retrieval score is fused with the base composed query through an agreement-gated residual rule. At last, a re-ranker verifies the recalled candidates with direct source-candidate comparison. Experiments have demonstrated the effectiveness of our method in addressing this challenge. Codes are available on https://github.com/Lee-zixu/R-3.
Zixu Li, Yupeng Hu, Zhiheng Fu +3
1Shandong University · 2Harbin Institute of Technology (Shenzhen)
Composed Video Retrieval (CVR) is designed to retrieve a target video that matches a reference video modified by a modification text. While existing methods explore cross-modal correspondences, they often assume modified objects appear directly in videos. However, modification texts frequently describe concepts not explicitly presented but implicitly expressed through semantically related visual cues (e.g., "cake" implying "birthday party"). Current approaches typically rely on aligning explicit feature representations within the concrete space, neglecting critical latent associations. To address this, we propose an adaptIve scheMa-ImAGery enhanced composItional NEtwork (IMAGINE). Unlike standard explicit matching, IMAGINE materializes implicit semantics (termed schema imagery) via dynamic multimodal prototypes. These prototypes capture shared latent concepts to adaptively modulate visual features, effectively injecting implicit guidance into the retrieval process. By bridging the gap between explicit visual contents and implicit retrieval intentions, IMAGINE achieves state-of-the-art performance in both CVR and Composed Image Retrieval (CIR) across three widely used benchmarks.
Jiale Huang, Zixu Li, Zhiwei Chen +3
Shandong University · Jinan, Shandong, China · Qilu University of Technology +1