When a text description is extended with an additional detail, image-text similarity should drop if that detail is wrong. We show that CLIP-style dual encoders often violate this intuition: appending a plausible but incorrect object or relation to an otherwise correct description can increase the similarity score. We call such cases half-truths. On COCO, CLIP prefers the correct shorter description only 40.6% of the time, and performance drops to 32.9% when the added detail is a relation. We trace this vulnerability to weak supervision on caption parts: contrastive training aligns full sentences but does not explicitly enforce that individual entities and relations are grounded. We propose CS-CLIP (Component-Supervised CLIP), which decomposes captions into entity and relation units, constructs a minimally edited foil for each unit, and fine-tunes the model to score the correct unit above its foil while preserving standard dual-encoder inference. CS-CLIP raises half-truth accuracy to 69.3% and improves average performance on established compositional benchmarks by 5.7 points, suggesting that reducing half-truth errors aligns with broader gains in compositional understanding. Code is publicly available at: https://github.com/kargibora/CS-CLIP
Figures & tables
Figure 1: The half-truth vulnerability. Starting from a short, caption-supported description (the anchor), we form a half-truth by adding one realistic but incorrect detail: either a wrong entity description or a wrong relation between entities. CLIP ( Radford et al., 2021 ) and NegCLIP ( Yuksekgonul et al., 2023 ) assign higher similarity to the half-truth, while CS-CLIP (ours) correctly penalizes the incorrect addition.
Figure 2: Half-truth construction. (i) Parse captions into units and generate foils via minimal edits. (ii) Sample an anchor A and append one foil to form half-truth A− . (iii) CLIP-style models can assign higher similarity to A− than A , motivating unit-level supervision.
Figure 3: Half-Truth Accuracy ( AccHT ), entity vs. relation additions. Higher is better; dashed line indicates random chance. Sentence-level negatives improve the easier case, but relation additions remain difficult to reject.
Figure 4: CS-CLIP training pipeline. (i) Unit extraction & foil generation. Given a caption Tj , a text-only LLM extracts entity units and relation units, then generates a matched foil for each unit via a minimal, realistic edit (e.g., “brown horse” → “white horse”). (ii) Unit sampling & encoding. We sample one unit–foil pair (Uj,U~j) per image and encode the image Ij with fϕ and the unit texts with gϕ . (iii) Unit-level supervision. The unit loss Lunit pulls the image embedding ij toward the correct unit uj and pushes it away from the matched foil u~j and other in-batch units. Sentence-level contrastive training is applied in parallel (not shown).
Entity
Relation
Overall
Model
AccHT
Δ
AccHT
Δ
AccHT
Δ
Pre-training variants (zero-shot)
CLIP ( Radford et al., 2021 )
52.9
-0.002
32.9
-0.017
40.6
-0.011
SigLIP ( Zhai et al., 2023 )
56.9
-0.000
38.8
-0.018
45.7
-0.011
SigLIP-2 ( Tschannen et al., 2025 )
69.9
+0.013
45.2
-0.005
54.6
+0.002
Fine-tuning variants (COCO)
Table 1: Half-truth results on COCO. We report Half-Truth Accuracy ( AccHT , %) and mean similarity gap Δ=s(I,A)−s(I,A−) (higher indicates better penalization of incorrect additions) for entity additions, relation additions, and overall. Green parentheses show absolute AccHT improvement of CS-CLIP over zero-shot CLIP. Full results are in Appendix C .
Figure 5: Compositional benchmark accuracy vs. Half-Truth Accuracy ( AccHT ). Each point represents a model. The x-axis shows average Image-to-Text accuracy across compositional benchmarks; the y-axis shows Half-Truth Accuracy AccHT (Section 3 ). CS-CLIP improves both metrics.
Figure 6: Compositional benchmark improvements relative to CLIP (I2T). Top: Per-benchmark change in Image-to-Text compositional accuracy relative to zero-shot CLIP across controlled foil benchmarks. Bottom: Average improvement across benchmarks. CS-CLIP shows consistent positive gains across most benchmarks, rather than improvements driven by a small subset.
Figure 7: T2I and Group Accuracy on paired compositional benchmarks. We report average Text-to-Image (T2I) accuracy and Group Accuracy (Grp) across the paired compositional datasets.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Model
Qwen3-8B-AWQ
Quantization
AWQ (4-bit)
Inference framework
vLLM
Batch size
256
Output format
Structured JSON
Appendix
Table 5: Model and inference configuration for the text-only pipeline.
Figure 8: Half-truth condition showcase. Starting from a caption-supported anchor, we add exactly one unit (entity or relation) to form either a truthful refinement or a half-truth. The figure enumerates half-truth variants by corruption type: entity additions (wrong object/attribute/random distractor) and relation additions (predicate change, role swap, argument corruption).
Figure 9: Additional half-truth examples with model scores. Multiple anchors and their half-truth refinements with similarity scores from CLIP, NegCLIP, and CS-CLIP. Some baselines assign comparable or higher similarity to incorrect refinements, while CS-CLIP more consistently lowers similarity for half-truths.
Entity additions
Relation additions
Model
+Obj
+Attr
+Rand
Rel:Attr
Rel:Obj
Ant
Swap
Subj
Pre-training variants (zero-shot)
CLIP ( Radford et al., 2021 )
41.3
27.1
72.6
25.5
37.9
21.0
24.2
59.4
SigLIP ( Zhai et al., 2023 )
46.8
34.8
73.8
38.1
48.4
26.2
21.6
69.9
SigLIP2 ( Tschannen et al., 2025 )
60.8
41.8
87.9
43.9
63.3
31.3
24.5
72.6
TripletCLIP ( Patel et al., 2024 )
51.4
37.1
75.8
35.1
45.8
25.3
27.8
57.3
Appendix
Table 6: Per-condition Half-Truth Accuracy (COCO, higher is better). We report Acc (%) for entity additions ( +Obj : wrong object, +Attr : wrong attribute, +Rand : random distractor) and relation additions ( Rel:Attr / Rel:Obj : corrupt an attribute/head noun inside a relation argument, Ant : change the predicate, Swap : swap subject/object roles, Subj : corrupt one relation argument).
Entity additions
Relation additions
Model
+Obj
+Attr
+Rand
Rel:Attr
Rel:Obj
Ant
Swap
Subj
Pre-training variants (large-scale / zero-shot)
CLIP
83.5
71.7
90.6
72.2
82.7
68.0
69.4
88.8
SigLIP
87.2
77.6
92.7
77.8
86.8
60.7
44.3
92.9
SigLIP2
89.2
78.6
92.9
79.0
88.6
62.6
46.7
93.8
TripletCLIP
74.7
65.8
83.5
67.2
74.5
54.4
57.8
80.0
Appendix
Table 7: Truthful completion vs. matched foil ( A+>A− ) by corruption type (COCO, higher is better). Win rate (%) for Pr[s(I,A+)>s(I,A−)] , where A+ appends a caption-consistent entity/relation and A− replaces only the added content with a matched foil.
Table 16: Unit construction hyperparameters. We vary the number of units per caption N , the fraction of relation-type units, and the fraction of swap-style foils. Gray : baseline configuration.
Setting
Compositionality
Half-Truth
Downstream
I2T
T2I
Grp
All
Ent
Rel
ZS Avg
Retr I2T
Retr T2I
Learning rate
LR =10−6
56.6
39.9
27.2
61.6
71.7
55.4
61.7
70.8
55.7
LR =2×10−6
57.3
39.0
27.2
63.8
72.2
58.6
61.3
71.4
56.6
LR =5×10−6 (Ours)
57.8
39.8
27.8
69.3
75.4
65.5
59.9
71.7
56.8
LR =10−5
57.6
40.2
28.1
71.8
76.7
68.8
58.0
70.0
56.3
Appendix
Table 17: Optimization hyperparameters. We vary learning rate, weight decay, and batch size. Gray : baseline configuration.
CLIP models learn transferable multi-modal features via image-text contrastive learning on internet-scale data. They are widely used in zero-shot classification, multi-modal retrieval, text-to-image diffusion, and as image encoders in large vision-language models. However, CLIP's pretraining is dominated by images paired with short captions, biasing the model toward encoding simple descriptions of salient objects and leading to coarse alignment on complex scenes and dense descriptions. While recent work mitigates this by fine-tuning on small-scale long-caption datasets, we identify an important common bias: both human- and LLM-generated long captions typically begin with a one-sentence summary followed by a detailed description. We show that this acts as a shortcut during training, concentrating attention on the opening sentence and early tokens and weakening alignment over the rest of the caption. To resolve this, we introduce DeBias-CLIP, which removes the summary sentence during training and applies sentence sub-sampling and text token padding to distribute supervision across all token positions. DeBias-CLIP achieves state-of-the-art long-text retrieval, improves short-text retrieval, and is less sensitive to sentence order permutations. It is a drop-in replacement for Long-CLIP with no additional trainable parameters.
Marc-Antoine Lavoie, Anas Mahmoud, Aldo Zaimi +2
University of Toronto Robotics Institute · Mila - Quebec AI Institute
Vision-language retrieval with CLIP-style dual encoders achieves strong cross-modal performance, yet practical accuracy often hinges on localized semantic distinctions where top-ranked near misses differ from the true match by a single critical detail. Hard-sample mining can select confusable candidates but cannot construct corrected counterparts; synthetic augmentation can generate novel samples but, without conditioning on actual model failures, targets irrelevant dimensions of hardness. We observe that a top-ranked false positive is a counterfactual scaffold---sharing most of the query's semantics while differing in a localized failure-causing residual. Minimally correcting this residual yields a hard positive of the ground truth in the same modality; the corrected and unedited versions form a hard negative pair that straddles the decision boundary, producing complementary pull--push supervision. We introduce RePair, guided by three principles---Validity, Minimality, and Locality---which mines false positives bidirectionally, applies LLM-guided counterfactual editing, and trains with a local hard-pair contrastive objective. On Flickr30K and COCO30K, RePair outperforms controlled augmentation baselines with only 107K synthetic samples---26%--75% fewer than comparable methods---confirming failure-conditioned repair is more data-efficient than error-agnostic augmentation.
Siyi Liu, Xiaorong Zhu, Enjun Du +6
The Hong Kong University of Science and Technology (Guangzhou) · Tencent Yuanbao · The University of Hong Kong +1
CLIP-style contrastive pretraining typically curates web-scale image-text pairs using sample-level filtering signals, often based on pair-level alignment. We show that this signal saturates: once coarse mismatches are removed, stricter global filtering no longer tracks the compositional supervision provided by the retained captions. The reason is structural - a global score conflates whether a pair is broadly plausible with whether the individual object, attribute, and relation phrases inside the caption materially support the image-text match. The latter is what compositional generalization demands, yet pair-level filters are blind to it. We address this with Counterfactual Phrase Intervention (CPI), a phrase-level curation framework that converts controlled nonce-token substitutions into image-conditioned phrase-sensitivity scores. CPI uses global alignment only for coarse mismatch removal, then ranks the surviving pool by whether caption phrases measurably affect the image-text score under controlled substitution. We frame CPI as a first-order phrase-sensitivity signal rather than a grounding or identification result, and evaluate it at CC3M scale. Ranking by this signal yields a 50%-data subset that improves VL-CheckList-VG Relation by +1.91 over the full-data baseline and +1.00 over alignment-only filtering at matched budget, while improving SugarCrepe overall and preserving general transfer. CPI is loss-orthogonal: applied unchanged to NegCLIP, it further improves VL-CheckList-VG Relation by +3.84, with additional CE-CLIP gains in the main text.