Learning Sample-wise Rank-aware Interpolation Weights for Composed Visual Data Retrieval

Boseung Jeong; Taegyu Park; Donghyeon Kwon; Hyunsouk Cho; Suha Kwak

cs.CVAug 23, 2026

Learning Sample-wise Rank-aware Interpolation Weights for Composed Visual Data Retrieval

Authors: Boseung Jeong, Taegyu Park, Donghyeon Kwon, Hyunsouk Cho, Suha Kwak

Organizations: AI Center, Samsung Electronics, Republic of Korea · Dept. of CSE, POSTECH, Republic of Korea · Dept. of AI, Ajou University, Republic of Korea · Graduate School of AI, POSTECH, Republic of Korea

Abstract

At the heart of composed visual data retrieval is the fusion of a reference visual input and a textual modification into a single query. While current state-of-the-art methods utilize multimodal large language models for this fusion, their complexity introduces prohibitive querytime latency, limiting their scalability. We instead revisit the efficacy of simple linear interpolation within an embedding space, and introduce SRAIN, the first framework that dynamically predicts query-specific interpolation weights. The key challenge lies in the fact that the quality of an interpolation weight should be measured by the interpolated embedding's discriminability from negatives as well as its proximity to true targets; this makes collecting and predicting optimal weights intractable. We overcome this bottleneck through two key innovations: batch-wise rank-aware weight estimation during training, and a compact memory bank that synthesizes hard negatives during inference. SRAIN achieves the best in composed video retrieval and matches the current state of the art in composed image retrieval, all while substantially reducing querytime latency compared to MLLM-based alternatives.

Figures & tables

Figure 1: Comparison of SRAIN with MLLM-based methods. (a) MLLM-based methods process reference and modification text together through MLLMs to generate a composed query, introducing substantial computational overhead. (b) SRAIN predicts a sample-specific interpolation weight $\alpha$ and fuses reference and modification embeddings through linear interpolation, enabling efficient retrieval. The right plot shows that SRAIN achieved comparable performance (mAP@5 on CIRCO) while being $\times$ 24 faster in throughput compared to MLLM-based methods. Meanwhile, BASIC [ 26 ] , a training-free method without MLLM, underperformed in throughput as well as retrieval accuracy since it requires numerous text encoder forward passes.

Figure 2: Rank-aware optimal weight estimation. Rather than determining the weight solely based on the similarity to the positive target (a), SRAIN selects the weight based on rank, where the positive consistently outranks nearby hard negatives (b).

Figure 3: Overall architecture of SRAIN. The reference video, modification text, and target video are encoded using the pretrained encoders. For each batch, optimal interpolation weights are estimated in a rank-aware manner. The model is trained in two stages: In $Stage\penalty\ 1$ , the encoders are fine-tuned to align composite query embeddings with target embeddings through hard-negative contrastive learning. In $Stage\penalty\ 2$ , the encoders are frozen and a lightweight weight predictor is trained to regress $\alpha_{i}^{*}$ . A memory bank is updated via exponential moving average (EMA) during training to collect representative target prototypes. At inference, the memory bank supplies top- $n$ prototypes as auxiliary tokens, enabling consistent weight prediction.

Methods	External	Modality	Fusion	Backbone	Evaluation Metric
	Knowledge				R@1	R@5	R@10	R@50
Text-only ( $\alpha=1$ )	✗	Text	-	BLIP-2	23.32	46.21	56.22	78.36
Visual-only ( $\alpha=0$ )	✗	Visual	-	BLIP-2	36.03	64.24	74.77	92.64
Average ( $\alpha=0.5$ )	✗	Visual+Text	Avg	BLIP-2	58.69	83.88	90.49	98.21
CoVR [ 43 ]	✗	Visual+Text	MLP	BLIP	50.59	74.65	83.57	95.46
CoVR-2 [ 42 ]	✗	Visual+Text	MLP	BLIP-2	51.88	79.38	86.46	97.42

Table 1: Composed video retrieval results on the WebVid-CoVR-Test, where all compared methods are fine-tuned on the WebVid-CoVR training set. Compared methods differ in fusion strategy: Avg indicates element-wise averaging of reference and modification embeddings, MLP denotes a multi-layer perceptron-based fusion module, CA employs cross-attention, and LI refers to linear interpolation. External Knowledge denotes the use of external models for additional video descriptions generation.

Methods	ViT	External	Pretraining	Dress		Shirt		Toptee		Average
		Knowledge	Data	R@10	R@50	R@10	R@50	R@10	R@50	R@10	R@50
Zero-shot
CIReVL [ 15 ]	G	GPT-4 [ 1 ]	✗	27.07	49.53	33.71	51.42	35.80	56.14	32.19	52.36
OSrCIR [ 38 ]	G	GPT-4o [ 1 ]	✗	33.02	54.78	38.65	54.71	41.04	61.83	37.57	57.11
Pic2Word [ 28 ]	L	✗	CC3M	20.00	40.20	26.20	43.60	27.90	47.40	24.70	43.70
SEARLE-XL [ 3 ]	L	✗	CC3M	26.89	45.58	20.48	43.13	29.32	49.97	25.56	46.23

Table 2: Composed image retrieval results on the FashionIQ validation set. CC3M [ 32 ] , LAION2B (L2B) [ 29 ] , LAION2M (L2M), and LLaVA-Align (L-A) [ 23 ] contain 3M, 2.3B, 2M, and 585k image-text pairs, respectively, where LAION2M is a subset of LAION400M [ 30 ] . MagicLens36M [ 53 ] comprises 36M (query image, instruction, target image) triplets. SDP [ 51 ] provides 2.47M text prompts, while ST18M [ 10 ] and LaSCo [ 18 ] comprise 18.8M and 389k triplets for training, respectively.

Methods	ViT	External	mAP@ $K$
		Knowledge	$K$ =5	$K$ =10	$K$ =25	$K$ =50
TFCIR [ 34 ]	G	✓	26.52	28.25	31.23	31.99
CIReVL [ 15 ]	G	✓	26.77	27.59	29.96	31.03
OSrCIR [ 38 ]	G	✓	30.47	31.14	35.03	36.59
SEARLE-XL [ 3 ]	L	✗	11.68	12.73	14.33	15.12
CompoDiff [ 10 ]	G	✗	15.33	17.71	19.45	21.01

Table 3: Zero-shot composed image retrieval results on CIRCO.

Methods	ViT	R@ $K$
		$K$ =1	$K$ =5	$K$ =10
CLIP4CIR [ 4 ]	L	33.59	65.35	77.35
CompoDiff [ 10 ]	G	32.39	57.61	77.25
CoVR [ 43 ]	L	49.69	78.60	86.77
CoVR-2 [ 42 ]	G	50.87	80.80	88.84
SRAIN (Ours)	G	50.95	81.21	89.60

Table 4: Composed image retrieval results on CIRR.

Figure 4: Top-4 retrieval results on the WebVid-CoVR test set. The ground-truth target video is highlighted in green.

Methods	R@1	R@5	R@10	R@50
(a) Training Strategy
Average (BLIP-2 frozen)	45.66	71.71	81.30	94.80
Stage 2 only (BLIP-2 frozen)	48.08	73.40	83.22	95.74
End-to-end training	57.36	82.71	89.20	97.77
Stage 1 only	60.50	83.90	91.09	98.40
(b) Effect of $C_{i}$ in Eq. 5

Table 5: Ablation studies on key components of our method.

	CIRCO	FIQ-D	FIQ-S	FIQ-T	FIQ-A
Methods	mAP@5	R@10	R@10	R@10	R@10
w/o Memory	23.79	21.49	30.98	29.63	27.37
Random Vector	23.72	21.17	30.96	29.57	27.23
SRAIN (Ours)	26.16	23.22	33.14	31.91	29.42

Table 6: Ablation studies on the memory bank for zero-shot CoIR.

Methods	R@1	R@5	R@10
CoVR	55.95	81.22	89.05
Ours (BLIP)	56.91	81.53	88.90

Table 7: Performance on WebVid-CoVR.

Explore similar work

UniCVR: From Alignment to Reranking for Unified Zero-Shot Composed Visual Retrieval
Apr 22, 2026Haokun Wen, Xuemeng Song, Haoyu Zhang +3Cross-Modal Learning Learning to Rank
Harbin Institute of Technology (Shenzhen) Shenzhen, China · Southern University of Science and Technology Shenzhen, China · Pengcheng Laboratory Shenzhen, China +1
R^3: Composed Video Retrieval via Reasoning-Guided Recalling and Re-ranking
May 31, 2026Zixu Li, Yupeng Hu, Zhiheng Fu +3Cross-Modal Retrieval Composed Video Retrieval
1Shandong University · 2Harbin Institute of Technology (Shenzhen)
IMAGINE: Adaptive Schema-Imagery Enhanced Composition for Composed Video Retrieval
Jun 6, 2026Jiale Huang, Zixu Li, Zhiwei Chen +3Cross-Modal Alignment Cross-Modal Retrieval
Shandong University · Jinan, Shandong, China · Qilu University of Technology +1

Apr 22, 2026, cs.CVJ/K move · Enter open · S save

UniCVR: From Alignment to Reranking for Unified Zero-Shot Composed Visual Retrieval

Haokun Wen, Xuemeng Song, Haoyu Zhang, Xiangyu Zhao+2

Harbin Institute of Technology (Shenzhen) Shenzhen, China · Southern University of Science and Technology Shenzhen, China · Pengcheng Laboratory Shenzhen, China+1

Cross-Modal Learning Learning to Rank Cross-Modal Retrieval Vision-Language Retrieval Image Retrieval Composed Video Retrieval+1

Open paper AskPDF arXiv

Composed image retrieval, multi-turn composed image retrieval, and composed video retrieval all share a common paradigm: composing the reference visual with modification text to retrieve the desired target. Despite this shared structure, the three tasks have been studied in isolation, with no prior work proposing a unified framework, let alone a zero-shot solution. In this paper, we propose UniCVR, the first unified zero-shot composed visual retrieval framework that jointly addresses all three tasks without any task-specific human-annotated data. UniCVR strategically combines two complementary strengths: Multimodal Large Language Models (MLLMs) for compositional query understanding and Vision-Language Pre-trained (VLP) models for structured visual retrieval. Concretely, UniCVR operates in two stages. In Stage I, we train the MLLM as a compositional query embedder via contrastive learning on a curated multi-source dataset of approximately 3.5M samples, bridging the heterogeneous embedding spaces between the MLLM and the frozen VLP gallery encoder. A cluster-based hard negative sampling strategy is proposed to strengthen contrastive supervision. In Stage II, we introduce an MLLM-guided dual-level reranking mechanism that applies adaptive budgeted subset scoring to a small number of top-ranked candidates, and then exploits the resulting relevance signals through a dual-level re-scoring scheme, producing more accurate final rankings with minimal computational overhead. Extensive experiments across five benchmarks covering all three tasks demonstrate that UniCVR achieves cutting-edge performance, validating its effectiveness and generalizability. Our data and code will be released upon acceptance.

Abstract

Explore similar work

UniCVR: From Alignment to Reranking for Unified Zero-Shot Composed Visual Retrieval

R^3: Composed Video Retrieval via Reasoning-Guided Recalling and Re-ranking

IMAGINE: Adaptive Schema-Imagery Enhanced Composition for Composed Video Retrieval