Composed Image Retrieval (CIR) task aims to retrieve target images based on reference images and modification texts. Current CIR methods primarily rely on fine-tuning vision-language pre-trained models. However, we find that these approaches commonly suffer from severe overfitting, posing challenges for CIR with limited triplet data. To better understand this issue, we present a systematic study of overfitting in VLP-based CIR, revealing a significant and previously overlooked generalization gap across different models and datasets. Motivated by these findings, we introduce WRF4CIR, a Weight-Regularized Fine-tuning network for CIR. Specifically, during the fine-tuning process, we apply adversarial perturbations to the model weights for regularization, where these perturbations are generated in the opposite direction of gradient descent. Intuitively, WRF4CIR increases the difficulty of fitting the training data, which helps mitigate overfitting in CIR under limited triplet supervision. Extensive experiments on benchmark datasets demonstrate that WRF4CIR significantly narrows the generalization gap and achieves substantial improvements over existing methods.
Figures & tables
Figure 1. Visualization of learning curve and generalization gap across (a) different baseline methods on FashionIQ (LF: late-fusion; EF: early-fusion; TV: sentence-level text inversion); and (b) different datasets, including FashionIQ, CIRR and Shoes. These results suggest that overfitting is a prevalent phenomenon in VLP-based CIR models. Visualization of learning curve and generalization gap across (a) different baseline methods on FashionIQ (LF: late-fusion; EF: early-fusion; TV: sentence-level text inversion); and (b) different datasets, including FashionIQ, CIRR and Shoes. Most methods achieve over 90% average recall on the FashionIQ training set upon convergence, while their average recall on the validation set plateaus at around 50%. Similar phenomena are observed across different datasets, as shown in (b). These results suggest that overfitting is a prevalent phenomenon in VLP-based CIR models.
Dataset
SPRC
+CO
+TA
+AM
+SPN
CIRR
83.02
82.69
82.65
82.70
83.56
FashionIQ
65.15
65.36
65.46
65.41
66.41
Table 1. Average Recall of the baseline SPRC under various data augmentation methods. Most of them show limited effectiveness except for SPN, which constructs additional triplets for CIR. (CO: CutOut; TA: TrivialAugment; AM: AugMix; SPN: SPN4CIR.)
Figure 2. Generalization gap and recall on FashionIQ under varying training data sizes. Limited training data contributes to overfitting during fine-tuning. Generalization gap and recall on FashionIQ under varying training data sizes. It is observed that as the size of the training data decreases, the validation recall drops and the generalization gap continues to widen. These experimental results indicate that the limited amount of training data contributes to the overfitting phenomenon during the fine-tuning of pre-trained models.
Figure 3. Illustration of our WRF4CIR framework. The reference image and modification text are first fused through the Q-Former, then aligned with the target image by optimizing a contrastive loss. Building on this process, adversarial perturbations are generated and applied to the model weights for regularization. Finally, the gradients computed from the perturbed model are used to update the parameters of the normal model. Illustration of our WRF4CIR framework. The reference image and modification text are first fused through the Q-Former, then aligned with the target image by optimizing a contrastive loss. Building on this process, adversarial perturbations are generated and applied to the model weights for regularization. Finally, the gradients computed from the perturbed model are used to update the parameters of the normal model.
Method
Backbone
Dress
Shirt
TopTee
Average
R10
R50
R10
R50
R10
R50
R10
R50
Rmean
CLIP4CIR ( Baldrati et al., 2023 )
CLIP ResNet-50
37.67
63.16
39.87
60.84
44.88
68.59
40.80
64.20
52.50
BLIP4CIR+Bi ( Liu et al., 2024 )
BLIP ViT-B
42.09
67.33
41.76
64.28
46.61
70.32
43.49
67.31
55.40
CASE ( Levy et al., 2024 )
BLIP ViT-B
47.44
69.36
48.48
70.23
50.18
72.24
48.79
70.68
59.74
CaLa ( Jiang et al., 2024 )
BLIP-2 ViT-L
42.38
66.08
46.76
68.16
50.93
73.42
46.69
69.22
57.95
Re-ranking ( Liu et al., 2023 )
BLIP ViT-B
48.14
71.43
50.15
71.25
55.23
76.80
51.17
73.13
62.15
Table 2. Results on FashionIQ val set. The best and second-best results are marked in Red and Blue , respectively. The symbol † indicates methods that leverage additional data.
Method
Backbone
Recall@K
Recall subset @K
Avg.
K=1
K=5
K=10
K=50
K=1
K=2
K=3
CLIP4CIR ( Baldrati et al., 2023 )
CLIP ResNet-50
40.91
74.53
84.77
97.35
70.22
87.80
94.46
72.38
BLIP4CIR+Bi ( Liu et al., 2024 )
BLIP ViT-B
40.15
73.08
83.88
96.27
72.10
88.27
95.93
72.59
CASE ( Levy et al., 2024 )
BLIP ViT-B
48.00
79.11
87.25
97.57
75.88
90.58
96.00
77.50
CaLa ( Jiang et al., 2024 )
BLIP-2 ViT-L
49.11
81.21
89.59
98.00
76.27
91.04
96.46
78.74
TG-CIR ( Wen et al., 2023 )
CLIP ViT-B
45.25
78.29
87.16
97.30
72.84
89.25
95.13
75.57
Table 3. Results on CIRR test set. The best and second-best results are marked in Red and Blue , respectively. The symbol † indicates methods that leverage additional data.
Figure 4. Case study between the baseline (w/o WRF) and our WRF4CIR on CIRR ((a) and (b)) and FashionIQ (c). The green boxes indicate the ground-truth targets. Case study between the baseline (w/o WRF) and our WRF4CIR on CIRR ((a) and (b)) and FashionIQ (c). The green boxes indicate the ground-truth targets. These results demonstrate that the effectiveness of our method in understanding reference image and text qualitatively, enabling more accurate target image retrieval.
Figure 5. The ablation studies of WRF4CIR. (a) Effect of perturbation strength γ on test performance; (b), (c), and (d) Visualization of the learning curve, generalization gap, and weight loss landscape under different perturbation strength γ on FashionIQ. The ablation studies of WRF4CIR. (a) Effect of perturbation strength γ on test performance; (b), (c), and (d) Visualization of the learning curve, generalization gap, and weight loss landscape under different perturbation strength γ on FashionIQ. These results collectively demonstrate that WRF4CIR effectively enhances generalization by flattening the loss landscape without significantly impairing the training process.
Method
PD
PS
FashionIQ
CIRR
R10
R50
Rmean
R1
R5
R10
Baseline
-
-
54.57
75.02
64.80
55.36
85.16
91.89
WRF4CIRRWP
-
✓
56.02
76.34
66.18
56.78
85.69
91.86
WRF4CIR
✓
✓
57.35
77.33
67.34
57.85
86.82
92.70
Table 4. Ablation study towards the weight perturbation strategy of WRF4CIR.
Method
Backbone
R10
R50
Rmean
CLIP4CIR ( Baldrati et al., 2023 )
ViT-B
42.87
66.77
54.82
CLIP4CIR+Ours
ViT-B
47.28
69.26
58.27
SPRC ( Bai et al., 2024 )
ViT-G
54.92
74.97
64.85
SPRC+Ours
ViT-G
56.67
77.12
66.90
Table 5. Effectiveness of our method on existing CIR approaches (Average only). The best results are shown in bold.
Figure 6. Evaluation of WRF4CIR under different CIR mechanisms, training data sizes, LoRA ranks and regularization strategies. Evaluation of WRF4CIR under different CIR mechanisms, training data sizes, \LoRA ranks and regularization strategies. These experiments demonstrate that our method achieves substantial improvements across multiple experimental settings.
Method
Ratio
R10
R50
Rmean
Time
Baseline
—
54.57
75.02
64.80
329s/epoch
WRF4CIRRWP
0
56.02
76.34
66.18
336s/epoch
WRF4CIR
0.25
56.53
76.67
66.60
388s/epoch
0.5
56.99
76.67
66.83
436s/epoch
0.75
57.07
76.95
67.01
490s/epoch
1.0
57.35
77.33
67.34
573s/epoch
Table 6. Ablation study of performance and training time under different adversarial ratios.
Composed image retrieval (CIR) aims to retrieve images that satisfy a user-specified modification while preserving relevant visual content from a reference image. Collecting target images for this purpose is costly, motivating zero-shot CIR methods that instead use target captions as supervision. However, target captions may omit source details that should be preserved. We therefore propose, Preserve-and-Compose Training, which complements target-caption supervision with visual evidence from the source image. PACT learns from image--text--text (ITT) triplets without target images or gallery updates, aligning composed queries with target captions while preserving source evidence through visual supervision. We further introduce Chord scoring, which combines target similarity with source-relative directional agreement in the frozen image space. Results across four ZS-CIR benchmarks show that combining target-caption supervision with source-image evidence leads to strong retrieval performance across datasets, backbone scales, and external galleries. The code is available on https://github.com/sehyunkwon/PACT.
Composed Image Retrieval (CIR) has attracted significant attention due to its flexible multimodal query method, yet its development is severely constrained by the Noisy Triplet Correspondence (NTC) problem. Most existing robust learning methods rely on the "small loss hypothesis", but the unique semantic ambiguity in NTC, such as "partial matching", invalidates this assumption, leading to unreliable noise identification. This entraps the model in a self dependent vicious cycle where the learner is intertwined with the arbiter, ultimately causing catastrophic "representation pollution". To address this critical challenge, we propose a novel "Expert-Proxy-Diversion" decoupling paradigm, named Air-Know (ArbIteR calibrated Knowledge iNternalizing rObust netWork). Air-Know incorporates three core modules: (1) External Prior Arbitration (EPA), which utilizes Multimodal Large Language Models (MLLMs) as an offline expert to construct a high precision anchor dataset; (2) Expert Knowledge Internalization (EKI), which efficiently guides a lightweight proxy "arbiter" to internalize the expert's discriminative logic; (3) Dual Stream Reconciliation (DSR), which leverages the EKI's matching confidence to divert the training data, achieving a clean alignment stream and a representation feedback reconciliation stream. Extensive experiments on multiple CIR benchmark datasets demonstrate that Air-Know significantly outperforms existing SOTA methods under the NTC setting, while also showing strong competitiveness in traditional CIR.
Zero-Shot Composed Image Retrieval (ZS-CIR) aims to retrieve target images given a multimodal query (comprising a reference image and a modification text), without training on annotated triplets. Existing methods typically convert the multimodal query into a single modality-either as an edited caption for Text-to-Image retrieval (T2I) or as an edited image for Image-to-Image retrieval (I2I). However, each paradigm has inherent limitations: T2I often loses fine-grained visual details, while I2I struggles with complex semantic modifications. To effectively leverage their complementary strengths under diverse query intents, we propose WISER, a training-free framework that unifies T2I and I2I via a "retrieve-verify-refine" pipeline, explicitly modeling intent awareness and uncertainty awareness. Specifically, WISER first performs Wider Search by generating both edited captions and images for parallel retrieval to broaden the candidate pool. Then, it conducts Adaptive Fusion with a verifier to assess retrieval confidence, triggering refinement for uncertain retrievals, and dynamically fusing the dual-path for reliable ones. For uncertain retrievals, WISER generates refinement suggestions through structured self-reflection to guide the next retrieval round toward Deeper Thinking. Extensive experiments demonstrate that WISER significantly outperforms previous methods across multiple benchmarks, achieving relative improvements of 45% on CIRCO (mAP@5) and 57% on CIRR (Recall@1) over existing training-free methods. Notably, it even surpasses many training-dependent methods, highlighting its superiority and generalization under diverse scenarios. Code will be released at https://github.com/Physicsmile/WISER.
Tianyue Wang, Leigang Qu, Tianyu Yang +4
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · National University of Singapore +3