Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understanding

Eun Woo Im; Dhruv Madhwal; Vivek Gupta

cs.LGApr 14, 2026

Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understanding

Authors: Eun Woo Im, Dhruv Madhwal, Vivek Gupta

Organizations: Arizona State University

Abstract

Vision-Language Models demonstrate remarkable capabilities but often struggle with compositional reasoning, exhibiting vulnerabilities regarding word order and attribute binding. This limitation arises from a scarcity of informative samples needed to differentiate subtle semantic variations during contrastive pretraining. Although hard negative mining offers a promising remedy, existing methods lack explicit mechanisms to dictate which linguistic elements undergo modification. Instead of engineering generative architectures, this study establishes lexical concreteness as a fundamental determinant of negative sample efficacy. Modifying highly concrete terms generates more pronounced structural and visual discrepancies, providing a substantially stronger learning signal. Leveraging this principle, ConcretePlant is proposed to systematically isolate and manipulate perceptually grounded concepts. Analyses of the InfoNCE further reveals a severe gradient imbalance, where easily distinguishable pairs disproportionately overwhelm the optimization process and restrict the bandwidth available for nuanced learning. To resolve this degradation, the Cement loss is formulated utilizing a margin-based approach. By correlating psycholinguistic scores with sample difficulty, this objective dynamically calibrates the penalization applied to individual training pairs. Comprehensive evaluations substantiate these theoretical claims. The integrated framework, designated as Slipform, achieves state-of-the-art accuracy across diverse compositional evaluation benchmarks, general cross-modal retrieval, single and multi label linear probing.

Figures & tables

Figure 1: Examples of our main hypothesis. Perturbing abstract keywords yields minor visual changes, whereas concrete words produce larger structural changes.

Figure 2: Overview of ConcretePlant. The pipeline selects a concrete target word from the caption, generates a hard negative caption with Qwen3-32B, and edits the image with SDXL-Turbo to produce a hard negative pair.

Figure 3: Gradient magnitude w.r.t.

s_{i,i}

s_{i,i^{\prime}}

\sum_{j}s_{i,j}

. The dotted lines indicate the average gradient values across the training steps. Severe gradient imbalance cause catastrophic forgetting but can be controlled by margin. Note that

|\frac{\partial L}{\partial s_{i,i}}s_{i,i}|=|\frac{\partial L}{\partial s_{i,i^{\prime}}}s_{i,i^{\prime}}|+|\frac{\partial L}{\partial s_{i,j}}\sum_{j\notin\{i,i^{\prime}\}}^{2N}s_{i,j}|

Figure 4: Estimated similarity scores w.r.t.

c_{i}

. Curves are fitted with LOESS [ 12 ] .

Panel of Figure 5 — Figure 5: Analysis of dataset distributions and metric correlations regarding concreteness scores, dataset difficulties and intra-modal similarities.

Metrics	$\mathcal{D}_{wo}$	$\mathcal{D}_{lc}$	$\mathcal{D}_{hc}$
Mean $c_{i}\uparrow$	4.29	3.81	4.77
Median $c_{i}\uparrow$	4.63	3.89	4.85
I2T acc. $\uparrow$	0.8712	0.8357	0.9225
T2I acc. $\uparrow$	0.4433	0.4020	0.5087
BERTScore	0.8992	0.8981	0.8907
CLIP-L	0.9197	0.9279	0.9029

Table 1: Statistics summary of datasets by sampling bias.

Figure 6: Gradient imbalance gets worse as batch size grows, but reducing the batch size degrades performance.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Qualitative examples of high- and low-concreteness perturbations.

Compositional Reasoning in VLMs

See all 23 papers

No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Models
Mar 26, 2026Hai X. Pham, David T. Hoffmann, Ricardo Guerrero +1Vision-Language Models Compositional Reasoning in VLMs
Samsung AI Center Cambridge, UK
Mastering Negation: Boosting Grounding Models via Grouped Opposition-Based Learning
Mar 13, 2026Zesheng Yang, Xi Jiang, Bingzhang Hu +4Vision-Language Grounding
Department of Computer Science, Southern University of Science and Technology, Shenzhen, China · Vision Intelligence Center, Hefei CAS Dihuge Automation Co., LTD, Hefei, China · School of Information Science and Technology, Harbin Institute of Technology (Shenzhen), China +2
CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning
Sep 8, 2026SeongJun Jeong, Minjoon Jung, Woo Suk Choi +2VLM Robustness Contrastive Learning
Seoul National University

Mar 26, 2026J/K move · Enter open · S save

No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Models

Hai X. Pham, David T. Hoffmann, Ricardo Guerrero, Brais Martinez

Samsung AI Center Cambridge, UK

Vision-Language Models Compositional Reasoning in VLMs Contrastive Learning for VLMs Multimodal Contrastive Learning Vision-Language Alignment

Open paper AskPDF arXiv

Contrastive vision-language (V&L) models remain a popular choice for various applications. However, several limitations have emerged, most notably the limited ability of V&L models to learn compositional representations. Prior methods often addressed this limitation by generating custom training data to obtain hard negative samples. Hard negatives have been shown to improve performance on compositionality tasks, but are often specific to a single benchmark, do not generalize, and can cause substantial degradation of basic V&L capabilities such as zero-shot or retrieval performance, rendering them impractical. In this work we follow a different approach. We identify two root causes that limit compositionality performance of V&Ls: 1) Long training captions do not require a compositional representation; and 2) The final global pooling in the text and image encoders lead to a complete loss of the necessary information to learn binding in the first place. As a remedy, we propose two simple solutions: 1) We obtain short concept centric caption parts using standard NLP software and align those with the image; and 2) We introduce a parameter-free cross-modal attention-pooling to obtain concept centric visual embeddings from the image encoder. With these two changes and simple auxiliary contrastive losses, we obtain SOTA performance on standard compositionality benchmarks, while maintaining or improving strong zero-shot and retrieval capabilities. This is achieved without increasing inference cost. We release the code for this work at https://github.com/saic-fi/concept_centric_clip.

Abstract

Compositional Reasoning in VLMs

Explore similar work

No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Models

Mastering Negation: Boosting Grounding Models via Grouped Opposition-Based Learning

CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning