Half-Truths Break Similarity-Based Retrieval

Bora Kargi; Arnas Uselis; Seong Joon Oh

cs.CVFeb 27, 2026

Half-Truths Break Similarity-Based Retrieval

Authors: Bora Kargi, Arnas Uselis, Seong Joon Oh

Organizations: University of Tübingen, Tübingen AI Center · Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA)

Abstract

When a text description is extended with an additional detail, image-text similarity should drop if that detail is wrong. We show that CLIP-style dual encoders often violate this intuition: appending a plausible but incorrect object or relation to an otherwise correct description can increase the similarity score. We call such cases half-truths. On COCO, CLIP prefers the correct shorter description only 40.6% of the time, and performance drops to 32.9% when the added detail is a relation. We trace this vulnerability to weak supervision on caption parts: contrastive training aligns full sentences but does not explicitly enforce that individual entities and relations are grounded. We propose CS-CLIP (Component-Supervised CLIP), which decomposes captions into entity and relation units, constructs a minimally edited foil for each unit, and fine-tunes the model to score the correct unit above its foil while preserving standard dual-encoder inference. CS-CLIP raises half-truth accuracy to 69.3% and improves average performance on established compositional benchmarks by 5.7 points, suggesting that reducing half-truth errors aligns with broader gains in compositional understanding. Code is publicly available at: https://github.com/kargibora/CS-CLIP

Figures & tables

Figure 1: The half-truth vulnerability. Starting from a short, caption-supported description (the anchor), we form a half-truth by adding one realistic but incorrect detail: either a wrong entity description or a wrong relation between entities. CLIP ( Radford et al., 2021 ) and NegCLIP ( Yuksekgonul et al., 2023 ) assign higher similarity to the half-truth, while CS-CLIP (ours) correctly penalizes the incorrect addition.

Figure 2: Half-truth construction. (i) Parse captions into units and generate foils via minimal edits. (ii) Sample an anchor $A$ and append one foil to form half-truth $A^{-}$ . (iii) CLIP-style models can assign higher similarity to $A^{-}$ than $A$ , motivating unit-level supervision.

Figure 3: Half-Truth Accuracy (

\text{Acc}_{\mathrm{HT}}

), entity vs. relation additions. Higher is better; dashed line indicates random chance. Sentence-level negatives improve the easier case, but relation additions remain difficult to reject.

Figure 4: CS-CLIP training pipeline. (i) Unit extraction & foil generation. Given a caption $T_{j}$ , a text-only LLM extracts entity units and relation units, then generates a matched foil for each unit via a minimal, realistic edit (e.g., “brown horse” → “white horse”). (ii) Unit sampling & encoding. We sample one unit–foil pair $(U_{j},\tilde{U}_{j})$ per image and encode the image $I_{j}$ with $f_{\phi}$ and the unit texts with $g_{\phi}$ . (iii) Unit-level supervision. The unit loss $\mathcal{L}_{\mathrm{unit}}$ pulls the image embedding $i_{j}$ toward the correct unit $u_{j}$ and pushes it away from the matched foil $\tilde{u}_{j}$ and other in-batch units. Sentence-level contrastive training is applied in parallel (not shown).

	Entity		Relation		Overall
Model	$\mathrm{Acc}_{\mathrm{HT}}$	$\Delta$	$\mathrm{Acc}_{\mathrm{HT}}$	$\Delta$	$\mathrm{Acc}_{\mathrm{HT}}$	$\Delta$
Pre-training variants (zero-shot)
CLIP ( Radford et al., 2021 )	52.9	-0.002	32.9	-0.017	40.6	-0.011
SigLIP ( Zhai et al., 2023 )	56.9	-0.000	38.8	-0.018	45.7	-0.011
SigLIP-2 ( Tschannen et al., 2025 )	69.9	+0.013	45.2	-0.005	54.6	+0.002
Fine-tuning variants (COCO)

Table 1: Half-truth results on COCO. We report Half-Truth Accuracy (

\mathrm{Acc}_{\mathrm{HT}}

, %) and mean similarity gap

\Delta=s(I,A)-s(I,A^{-})

(higher indicates better penalization of incorrect additions) for entity additions, relation additions, and overall. Green parentheses show absolute

\mathrm{Acc}_{\mathrm{HT}}

improvement of CS-CLIP over zero-shot CLIP. Full results are in Appendix C .

Figure 5: Compositional benchmark accuracy vs. Half-Truth Accuracy (

\mathrm{Acc}_{\mathrm{HT}}

). Each point represents a model. The x-axis shows average Image-to-Text accuracy across compositional benchmarks; the y-axis shows Half-Truth Accuracy

\mathrm{Acc}_{\mathrm{HT}}

(Section 3 ). CS-CLIP improves both metrics.

Figure 6: Compositional benchmark improvements relative to CLIP (I2T). Top: Per-benchmark change in Image-to-Text compositional accuracy relative to zero-shot CLIP across controlled foil benchmarks. Bottom: Average improvement across benchmarks. CS-CLIP shows consistent positive gains across most benchmarks, rather than improvements driven by a small subset.

Figure 7: T2I and Group Accuracy on paired compositional benchmarks. We report average Text-to-Image (T2I) accuracy and Group Accuracy (Grp) across the paired compositional datasets.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Model	Qwen3-8B-AWQ
Quantization	AWQ (4-bit)
Inference framework	vLLM
Batch size	256
Output format	Structured JSON

Appendix

Table 5: Model and inference configuration for the text-only pipeline.

Panel of Figure 8 — Figure 8: Half-truth condition showcase. Starting from a caption-supported anchor, we add exactly one unit (entity or relation) to form either a truthful refinement or a half-truth. The figure enumerates half-truth variants by corruption type: entity additions (wrong object/attribute/random distractor) and relation additions (predicate change, role swap, argument corruption).

Figure 9: Additional half-truth examples with model scores. Multiple anchors and their half-truth refinements with similarity scores from CLIP, NegCLIP, and CS-CLIP. Some baselines assign comparable or higher similarity to incorrect refinements, while CS-CLIP more consistently lowers similarity for half-truths.

	Entity additions			Relation additions
Model	+Obj	+Attr	+Rand	Rel:Attr	Rel:Obj	Ant	Swap	Subj
Pre-training variants (zero-shot)
CLIP ( Radford et al., 2021 )	41.3	27.1	72.6	25.5	37.9	21.0	24.2	59.4
SigLIP ( Zhai et al., 2023 )	46.8	34.8	73.8	38.1	48.4	26.2	21.6	69.9
SigLIP2 ( Tschannen et al., 2025 )	60.8	41.8	87.9	43.9	63.3	31.3	24.5	72.6
TripletCLIP ( Patel et al., 2024 )	51.4	37.1	75.8	35.1	45.8	25.3	27.8	57.3

Appendix

Table 6: Per-condition Half-Truth Accuracy (COCO, higher is better). We report Acc (%) for entity additions ( +Obj : wrong object, +Attr : wrong attribute, +Rand : random distractor) and relation additions ( Rel:Attr / Rel:Obj : corrupt an attribute/head noun inside a relation argument, Ant : change the predicate, Swap : swap subject/object roles, Subj : corrupt one relation argument).

	Entity additions			Relation additions
Model	+Obj	+Attr	+Rand	Rel:Attr	Rel:Obj	Ant	Swap	Subj
Pre-training variants (large-scale / zero-shot)
CLIP	83.5	71.7	90.6	72.2	82.7	68.0	69.4	88.8
SigLIP	87.2	77.6	92.7	77.8	86.8	60.7	44.3	92.9
SigLIP2	89.2	78.6	92.9	79.0	88.6	62.6	46.7	93.8
TripletCLIP	74.7	65.8	83.5	67.2	74.5	54.4	57.8	80.0

Appendix

Table 7: Truthful completion vs. matched foil (

A^{+}>A^{-}

) by corruption type (COCO, higher is better). Win rate (%) for

\Pr[s(I,A^{+})>s(I,A^{-})]

, where

A^{+}

appends a caption-consistent entity/relation and

A^{-}

replaces only the added content with a matched foil.

Benchmark	Cite	Text	Img	Primary phenomenon
ARO	( Yuksekgonul et al., 2023 )	✓		Binding, relations, word order
BLA	( Chen et al., 2023 )	✓		Syntax and word order
COCO-CF	( Le et al., 2023 )	✓	✓	Counterfactual edits
COLA	( Ray et al., 2023 )	✓	✓	Multi-object binding
ColorFoil	( Samin et al., 2025 )	✓		Color attributes
ColorSwap	( Burapacheep et al., 2024 )	✓		Attribute binding

Appendix

Table 8: Compositional benchmark suite. Text: foil captions provided; Img: foil images provided.

Category	Sub-capabilities
Entity Content	Object Recognition, Attribute Recognition, Counting, Existence
Relational Structure	Predicate Sensitivity, Role Sensitivity
Binding	Attribute Binding
Linguistic	Syntax, Coreference, Negation

Appendix

Table 9: Capability taxonomy summary.

Benchmark	Subset	Category	Sub-capability
Entity Content
SugarCrepe	add_att	Entity Content	Attribute Recognition
SugarCrepe	replace_att	Entity Content	Attribute Recognition
SugarCrepe	add_obj	Entity Content	Object Recognition
SugarCrepe	replace_obj	Entity Content	Object Recognition
SugarCrepe++	replace_attribute	Entity Content	Attribute Recognition

Appendix

Table 10: Benchmark subset to capability mapping.

Figure 10: Per-subset compositional I2T accuracy (Part 1/2). Green: improvement over CLIP; red: degradation.

Figure 11: Per-subset compositional I2T accuracy (Part 2/2). Continuation of Figure 10 .

Setting	Compositionality			Half-Truth			Downstream
	I2T	T2I	Grp	All	Ent	Rel	ZS Avg	Retr I2T	Retr T2I
Number of units per caption $N$
$N{=}1$	57.4	39.1	27.4	69.8	75.9	66.0	59.5	70.7	56.3
$N{=}2$ (Ours)	57.8	39.8	27.8	69.3	75.4	65.5	59.9	71.7	56.8
$N{=}3$	57.3	38.7	27.4	69.0	75.7	64.9	59.6	70.6	56.4
$N{=}4$	57.5	39.0	27.2	68.9	74.6	65.3	58.9	70.2	56.5

Appendix

Table 16: Unit construction hyperparameters. We vary the number of units per caption

N

, the fraction of relation-type units, and the fraction of swap-style foils. Gray : baseline configuration.

Setting	Compositionality			Half-Truth			Downstream
	I2T	T2I	Grp	All	Ent	Rel	ZS Avg	Retr I2T	Retr T2I
Learning rate
LR $=10^{-6}$	56.6	39.9	27.2	61.6	71.7	55.4	61.7	70.8	55.7
LR $=2{\times}10^{-6}$	57.3	39.0	27.2	63.8	72.2	58.6	61.3	71.4	56.6
LR $=5{\times}10^{-6}$ (Ours)	57.8	39.8	27.8	69.3	75.4	65.5	59.9	71.7	56.8
LR $=10^{-5}$	57.6	40.2	28.1	71.8	76.7	68.8	58.0	70.0	56.3

Appendix

Table 17: Optimization hyperparameters. We vary learning rate, weight decay, and batch size. Gray : baseline configuration.

Contrastive Learning for VLMs

See all 18 papers

CLIP Is Shortsighted: Paying Attention Beyond the First Sentence
Feb 25, 2026Marc-Antoine Lavoie, Anas Mahmoud, Aldo Zaimi +2Vision-Language Alignment Vision-Language Pretraining
University of Toronto Robotics Institute · Mila - Quebec AI Institute
RePair: Turning Retrieval Failures into Counterfactual Hard Pairs
Aug 30, 2026Siyi Liu, Xiaorong Zhu, Enjun Du +6Cross-Modal Retrieval Counterfactual Data Augmentation
The Hong Kong University of Science and Technology (Guangzhou) · Tencent Yuanbao · The University of Hong Kong +1
What Does the Caption Really Say? Counterfactual Phrase Intervention for Compositional Data Selection in Vision-Language Pretraining
May 21, 2026Hyejin Go, Semi Lee, Hyesong ChoiTraining Data Selection Vision-Language Pretraining
Soongsil University Seoul, Republic of Korea

Feb 25, 2026J/K move · Enter open · S save

CLIP Is Shortsighted: Paying Attention Beyond the First Sentence

Marc-Antoine Lavoie, Anas Mahmoud, Aldo Zaimi, Arsene Fansi Tchango+1

University of Toronto Robotics Institute · Mila - Quebec AI Institute

Vision-Language Alignment Vision-Language Pretraining Vision-Language Retrieval Image-Text Retrieval Long-Context VLMs

Open paper AskPDF arXiv

CLIP models learn transferable multi-modal features via image-text contrastive learning on internet-scale data. They are widely used in zero-shot classification, multi-modal retrieval, text-to-image diffusion, and as image encoders in large vision-language models. However, CLIP's pretraining is dominated by images paired with short captions, biasing the model toward encoding simple descriptions of salient objects and leading to coarse alignment on complex scenes and dense descriptions. While recent work mitigates this by fine-tuning on small-scale long-caption datasets, we identify an important common bias: both human- and LLM-generated long captions typically begin with a one-sentence summary followed by a detailed description. We show that this acts as a shortcut during training, concentrating attention on the opening sentence and early tokens and weakening alignment over the rest of the caption. To resolve this, we introduce DeBias-CLIP, which removes the summary sentence during training and applies sentence sub-sampling and text token padding to distribute supervision across all token positions. DeBias-CLIP achieves state-of-the-art long-text retrieval, improves short-text retrieval, and is less sensitive to sentence order permutations. It is a drop-in replacement for Long-CLIP with no additional trainable parameters.

Abstract

Contrastive Learning for VLMs

Explore similar work

CLIP Is Shortsighted: Paying Attention Beyond the First Sentence

RePair: Turning Retrieval Failures into Counterfactual Hard Pairs

What Does the Caption Really Say? Counterfactual Phrase Intervention for Compositional Data Selection in Vision-Language Pretraining