No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Models

Hai X. Pham; David T. Hoffmann; Ricardo Guerrero; Brais Martinez

cs.CVMar 26, 2026

No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Models

Authors: Hai X. Pham, David T. Hoffmann, Ricardo Guerrero, Brais Martinez

Organizations: Samsung AI Center Cambridge, UK

Abstract

Contrastive vision-language (V&L) models remain a popular choice for various applications. However, several limitations have emerged, most notably the limited ability of V&L models to learn compositional representations. Prior methods often addressed this limitation by generating custom training data to obtain hard negative samples. Hard negatives have been shown to improve performance on compositionality tasks, but are often specific to a single benchmark, do not generalize, and can cause substantial degradation of basic V&L capabilities such as zero-shot or retrieval performance, rendering them impractical. In this work we follow a different approach. We identify two root causes that limit compositionality performance of V&Ls: 1) Long training captions do not require a compositional representation; and 2) The final global pooling in the text and image encoders lead to a complete loss of the necessary information to learn binding in the first place. As a remedy, we propose two simple solutions: 1) We obtain short concept centric caption parts using standard NLP software and align those with the image; and 2) We introduce a parameter-free cross-modal attention-pooling to obtain concept centric visual embeddings from the image encoder. With these two changes and simple auxiliary contrastive losses, we obtain SOTA performance on standard compositionality benchmarks, while maintaining or improving strong zero-shot and retrieval capabilities. This is achieved without increasing inference cost. We release the code for this work at https://github.com/saic-fi/concept_centric_clip.

Figures & tables

Panel of Figure 1 — Figure 1: Method overview . (a) SigLIP uses a learnable query token in combination with an attention layer to pool the visual tokens into a single token. Aligning only global representations hampers the learning of a compositional representation. (b) Similar to SigLIP, our method aligns the global representations $v$ and $t$ . To simplify learning of a compositional representation, our method extends SigLIP by first, pooling the text encoder output tokens into concept embeddings $\{c_{k}\}$ , which are used to attention-pool concept-specific information from the visual tokens $\bar{V}$ , resulting in $\hat{v}(c)$ . $\hat{v}(c_{k})$ and the corresponding $c_{k}$ are aligned using $L_{xac}$ . Furthermore, the global visual representation $v$ is aligned with all $c_{k}$ via ${\mathcal{L}}_{\textbf{npc}}$ , a multi-positive variant of SigLIP loss. Similarly, the global image and text representations $v$ and $t$ are aligned via $\mathcal{L}_{contrastive}$ .

Model	Training data	Train from scratch
Composition-aware
CE-CLIP [ 35 ]	MSCOCO
NegCLIP [ 33 ]	MSCOCO
CLIC [ 24 ]	LAION-1.5B
DAC [ 4 ]	CC3M
SLVC [ 5 ]

Table 1: Training data of baselines. Summary of baseline methods and their corresponding training datasets, excluding the original CLIP and SigLIP models. The last column indicates whether the provided checkpoints were trained from scratch (✓).

Models	SugarCrepe			SugarCrepe++				ImNet1K	Flickr30k	MSCOCO	DOCCI	IIW	Average
	Add	Replace	Swap	Replace		Swap
	Add	Replace	Swap	I2T	TOT	I2T	TOT
SigLIP ViT-B/16	86.5	84.1	65.8	73.8	62.8	48.0	34.7	76.1	95.2	78.9	58.9	75.3	70.0
CLIP (OpenAI) ViT-B/32	73.0	80.0	62.7	69.5	60.5	45.7	27.4	63.3	89.0	65.4	47.1	69.5	62.8
CLIP (OpenAI) ViT-B/16	72.7	80.4	62.5	70.1	60.2	43.8	23.9	68.4	90.9	67.6	50.4	71.3	63.5
Fine-grained models

Table 2: Performance comparison of ViT-B-based models . C 2 LIP achieves consistently high performance on SugarCrepe and SugarCrepe++ compared to the composition-aware models, despite only relying on regular captions during training. Additionally, our model is able to maintain and improve the retrieval and zero-shot capabilities of the original SigLIP, achieving the best overall score.

Model	SugarCrepe		SugarCrepe++				Average
Model	Replace	Swap	Replace-I2T	Replace-TOT	Swap-I2T	Swap-TOT	Average
SigLIP ViT-B/16	86.7	71.5	75.5	64.2	56.3	46.4	66.8
CE-CLIP	88.8	7 7.0	52.0	64.2	32.3	36.3	58.4
NegCLIP	85.9	75.4	67.1	70.7	55.0	5 0.3	6 7.4
CLIC	86.6	74.0	7 5.9	52.4	6 2.2	31.2	63.7
DAC-SAM	85.9	75.1	44.3	56.0	33.5	25.4	53.3

Table 3: Attribute-binding performance comparison. We compare the accuracy of C 2 LIP with composition-aware baselines on attribute replacement and swapping experiments of SugarCrepe and SugarCrepe++. Our model consistently outperforms the competing baselines on most tasks and attains the best overall score. Moreover, previous methods rely on hard negatives to induce a compositional representation, which often harms generalization to natural data and leads to a considerable performance drop on SugarCrepe++. In contrast, C 2 LIP maintains high performance on all metrics, further highlighting the strength of our proposed approach.

Models	SugarCrepe			SugarCrepe++				ImNet1K	Flickr30k	MSCOCO	DOCCI	IIW	Average
	Add	Replace	Swap	Replace		Swap
	Add	Replace	Swap	I2T	TOT	I2T	TOT
Multi-task SOTA
BLIP-B	92.2	84.1	73.6	76.7	86.0	52.1	41.3	49.2	97.2	79.1	51.6	72.4	71.3
FLAVA	79.6	84.9	73.8	74.3	75.0	56.6	51.0	56.9	91.8	73.9	48.3	72.9	69.9
SigLIP2-ViT-B/16	89.0	86.0	71.9	77.7	69.7	57.9	42.0	78.5	96.5	82.8	62.2	78.1	74.4

Table 4: Comparison with SOTA foundation models. We compare performance of C 2 LIP with larger contrastive models (ViT-L), and SOTA models trained with multimodal/multi-task objectives (BLIP, FLAVA, SigLIP-2). Despite the substantial disadvantages in model size and training, C 2 LIP has the best performance on “Add” and “Replace” tasks of SugarCrepe, and maintains close performance on other benchmarks. On average our model is only

1.7

percentage points below the best model, CLIPS, which has 5x more parameters.

Panel of Figure 2 — Figure 2: Change in attention when using C 2 LIP compared to SigLIP. We visualize the difference in attention between C 2 LIP and SigLIP to the visual tokens given a caption and an image. Higher attention for C 2 LIP is shown as green . Lower attention for C 2 LIP indicated with violet . White means no change. (a) Black regions that are not a sweater get reduced attention, the sweater gets more or unchanged attention. (b) The background, the cups and beer cans get attended less after training with our method, while attention to the green tablecloth increases or stays the same.

Training Configurations	SugarCrepe								Flickr8K
	Replace			Swap		Add		Average
	Object	Attribute	Relation	Object	Attribute	Object	Attribute	Average
Original SigLIP ViT-B/16	95.3	86.7	70.3	60.0	71.5	89.1	83.8	79.5	94.9
+ fine-tuned on CC3M	95.8	87.4	73.5	65.3	74.2	88.9	87.0	81.7	95.2
+ npc	96.7 (+0.9)	89.7 (+2.3)	78.5 (+5.0)	66.5 (+1.2)	78.5 (+4.3)	93.0 (+4.1)	93.9 (+6.9)	85.3 (+3.6)	96.3 (+1.1)
+ xac ( full )	96.7 (+0.9)	89.2 (+1.8)	78.9 (+5.4)	67.3 (+2.0)	78.8 (+4.6)	93.5 (+4.6)	94.9 (+7.9)	85.6 (+3.9)	96.4 (+1.2)

Table 5: Training objective ablation. Improvements in (green) with respect to the variant fine-tuned on CC3M. npc : noun-phrase loss, xac : cross-attended concept-aware loss. Both components lead to significant gains.

Model	SugarCrepe				SugarCrepe++
Model	Replace	Swap	Add	Average	Replace	Swap	Average
SigLIP-ViT-B-16	84.1	65.8	86.5	79.5	73.8	48.0	63.5
C 2 LIP	89.1	75.2	93.3	86.3	79.7	55.2	69.9
LLaVA-SigLIP	86.9	77.0	85.0	83.5	76.5	55.4	68.1
LLaVA-C 2 LIP	86.8	78.5	85.0	83.9	77.6	57.8	69.7

Table 6: Performance of VLM using C 2 LIP vision encoder. We follow the LLaVa [ 20 ] recipe to train VLMs with frozen SigLIP and C 2 LIP image encoders, resulting in LLaVA-SigLIP and LLaVA-C 2 LIP, respectively. LLaVA-C 2 LIP shows improvements on compositionality, demonstrating that the vision encoder trained with our proposed concept-centric approach also benefits the VLM that utilizes it.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Training data	Train from scratch
Composition-aware
CE-CLIP [ 35 ]	MSCOCO
NegCLIP [ 33 ]	MSCOCO
CLIC [ 24 ]	LAION-1.5B
DAC [ 4 ]	CC3M
SLVC [ 5 ]

Appendix

Table A1 : Baseline methods. Summary of baseline methods and their corresponding training datasets. The last column indicates whether the provided checkpoints were trained from scratch (✓).

$\lambda_{hnc}$	$\lambda_{xac}$	SugarCrepe Average Accuracy
0.5	0.5	84.6
0.5	0.1	85.2
0.5	0.01	85.2
1	0.5	85.1
1	0.01	85.6

Appendix

Table A2 : Ablation of trade-off hyperparameters in the objective function. We experimented with different values of

\lambda_{hnc}

\lambda_{xac}

showing our proposed method incurs little sensitivity to these hyperparameters.

Models	Replace				Swap			Add			Average
Models	Obj	Attr	Rel	Avg	Obj	Attr	Avg	Obj	Attr	Avg	Average
SigLIP ViT-B/16	95.3	86.7	70.3	84.1	60.0	71.5	65.8	89.1	83.8	86.5	79.5
CLIP (OpenAI) ViT-B/32	90.9	80.0	69.2	80.0	61.2	64.1	62.7	77.2	68.8	73.0	73.1
CLIP (OpenAI) ViT-B/16	93.5	81.1	66.7	80.4	60.0	65.0	62.5	78.5	66.9	72.7	73.1
FG-CLIP	9 5.9	87.1	72.2	85.1	66.5	73.3	69.9	87.6	81.8	84.7	80.6
FineCLIP	95.6	85.2	75.0	85.3	63.3	70.3	66.8	90.0	80.8	85.4	80.0

Appendix

Table A3 : SugarCrepe compositionality benchmark – all subtasks . The results in this table are summarized in Tab. 2 of the main paper. We compare the accuracy of C 2 LIP with baseline methods on different tasks. On average, C 2 LIP is the second best method, only 0.8 percentage points below DAC-LLM, while being trained on much less data, without custom compositional captions.

Models	Replace - I2T				Replace - TOT				Swap - I2T			Swap - TOT			Average
Models	Obj	Attr	Rel	Avg	Obj	Attr	Rel	Avg	Obj	Attr	Avg	Obj	Attr	Avg	Average
SigLIP ViT-B/16	91.2	75.5	54.8	73.8	79.2	64.2	45.0	62.8	39.6	56.3	48.0	22.9	46.4	34.7	57.5
CLIP (OpenAI) ViT-B/32	86.7	65.6	56.3	69.5	83.7	59.3	38.6	60.5	46.1	45.2	45.7	19.1	35.6	27.4	53.6
CLIP (OpenAI) ViT-B/16	89.6	67.6	53.2	70.1	84.4	57.2	39.0	60.2	39.2	48.4	43.8	16.3	31.4	23.9	52.6
FG-CLIP	9 2.6	7 6.8	58.0	75.8	90.6	67.8	44.2	67.5	47.8	55.3	51.5	29.4	47.0	38.2	6 0.9
FineCLIP	90.8	70.7	57.0	72.8	91.3	67.9	47.0	68.7	39.6	48.2	43.9	20.8	32.7	26.8	56.6

Appendix

Table A4 : SugarCrepe++ compositionality benchmark – all subtasks . The results in this table are summarized in Tab. 2 of the main paper. We compare the accuracy of C 2 LIP with baseline methods on different compositionality tasks. SugarCrepe++ addresses the limitation of the SugarCrepe benchmark, where the positive and negative captions are “hackable” by using custom training data created using similar rules. Here we observe significant performance drops from the baseline methods, while our model, which maintains generalization capabilities, achieves the best result overall.

Models	MSCOCO		Flickr30k		DOCCI		IIW		Average
Models	t2i	i2t	t2i	i2t	t2i	i2t	t2i	i2t	Average
SigLIP ViT-B/16	72.4	85.4	92.3	98.0	35.8	8 2.0	53.0	97.5	77.1
CLIP (OpenAI) ViT-B/32	56.0	74.9	83.4	94.6	27.1	67.1	44.7	94.3	67.8
CLIP (OpenAI) ViT-B/16	58.4	76.7	85.6	96.2	29.3	71.5	46.7	95.9	70.0
FG-CLIP	71.4	85.5	9 3.0	9 8.6	33.9	79.4	52.5	98.7	76.6
FineCLIP	7 3.7	84.2	90.2	96.7	27.9	61.9	44.6	89.5	71.1

Appendix

Table A5 : Zero-shot retrieval benchmarks – all subtasks . The results in this table are summarized in Tab. 2 of the main paper. We record Recall@5 metrics of all models on text-to=image (t2i) and image-to-text (i2t) tasks, across two standard retrieval benchmarks: MSCOCO and Flickr30K, and two fine-grained retrieval benchmarks: DOCCI and Image-in-words (IIW). It can be observed that C 2 LIP can maintain or improve the performance of the original base model across all tasks, with the best average score.

Model	ImageNet1K	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	Average
SigLIP ViT-B/16	76.1	91.6	92.3	72.2	69.9	90.9	4 3.8	6 4.7	9 4.1	88.0	8 6.0	7 9.0
CLIP (OpenAI) ViT-B/32	63.3	83.9	89.8	64.3	63.2	59.7	19.6	44.0	87.5	83.8	66.5	66.0
CLIP (OpenAI) ViT-B/16	68.4	88.8	90.8	67.0	65.5	64.7	24.5	45.2	89.2	84.0	71.5	69.0
FG-CLIP	69.0	85.2	93.9	76.4	71.2	84.2	24.4	57.2	90.5	86.2	70.1	73.5
FineCLIP	55.8	60.1	9 4.3	69.0	55.9	6.3	10.7	41.6	57.9	85.4	41.4	52.6
DreamLIP-3m	31.6	23.6	75.7	43.7	41.3	3.5	1.6	18.8	28.9	70.3	18.5	32.5

Appendix

Table A6 : Zero-shot classification benchmarks . We evaluate classification accuracies of C 2 LIP and the baselines on 11 standard zero-shot classification benchmarks. We observe significantly better performance of our model compared the composition-aware baselines in general, particularly on the challenging Cars and Aircraft benchmarks where the classes are actually sub-classes of the same object category.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 722 papers

Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality
Jun 11, 2026Wei Li, Zhen Huang, Xinmei TianCross-Modal Learning Vision-Language Models
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China · Independent Researcher
CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning
Sep 8, 2026SeongJun Jeong, Minjoon Jung, Woo Suk Choi +2VLM Robustness Contrastive Learning
Seoul National University
Contrastive vision-language learning with paraphrasing and negation
Nov 20, 2025Kwun Ho Ngan, Saman Sadeghi Afgeh, Joe Townsend +1VLM Robustness VLM Reasoning
Fujitsu Research of Europe · United Kingdom · City St George’s, University of London

Jun 11, 2026J/K move · Enter open · S save

Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality

Wei Li, Zhen Huang, Xinmei Tian

MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China · Independent Researcher

Cross-Modal Learning Vision-Language Models Vision-Language Alignment Vision-Language Pretraining Compositional Reasoning in VLMs

Open paper AskPDF arXiv

Contrastively trained vision-language models like CLIP, have made remarkable progress in learning joint image-text representations, but still face challenges in compositional understanding. They often exhibit a "bag-of-words" behavior--struggling to capture the object relations, attribute-object bindings, and word order dependencies. This limitation arises not only from the reliance on global, single-vector representations for optimization, but also from the insufficient exploitation and modeling of the rich compositional information inherently present in paired image text data. In this work, we propose MACCO (MAsked Compositional Concept MOdeling), a framework that masks compositional concepts in one modality and reconstructs them conditioned on the full contextual information from the other, enabling the model to capture and align cross-modal compositional structures more effectively. To facilitate this process, we introduce two auxiliary objectives that jointly align and regularize masked features both inter-modally and intra-modally. Extensive experiments on five compositional benchmarks, along with in-depth analyses, demonstrate that our approach not only significantly enhances compositionality in VLMs but also improves their ability to capture syntactic structure and linguistic information. Additionally, the improved compositionality also benefits text-to-image generation and multimodal large language model. Code is available at https://github.com/hiker-lw/MACCO.

Abstract

Vision-Language Models

Explore similar work

Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality

CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning

Contrastive vision-language learning with paraphrasing and negation