cs.LGApr 14, 2026

Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understanding

Authors: Eun Woo Im, Dhruv Madhwal, Vivek Gupta

Organizations: Arizona State University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-Language Models demonstrate remarkable capabilities but often struggle with compositional reasoning, exhibiting vulnerabilities regarding word order and attribute binding. This limitation arises from a scarcity of informative samples needed to differentiate subtle semantic variations during contrastive pretraining. Although hard negative mining offers a promising remedy, existing methods lack explicit mechanisms to dictate which linguistic elements undergo modification. Instead of engineering generative architectures, this study establishes lexical concreteness as a fundamental determinant of negative sample efficacy. Modifying highly concrete terms generates more pronounced structural and visual discrepancies, providing a substantially stronger learning signal. Leveraging this principle, ConcretePlant is proposed to systematically isolate and manipulate perceptually grounded concepts. Analyses of the InfoNCE further reveals a severe gradient imbalance, where easily distinguishable pairs disproportionately overwhelm the optimization process and restrict the bandwidth available for nuanced learning. To resolve this degradation, the Cement loss is formulated utilizing a margin-based approach. By correlating psycholinguistic scores with sample difficulty, this objective dynamically calibrates the penalization applied to individual training pairs. Comprehensive evaluations substantiate these theoretical claims. The integrated framework, designated as Slipform, achieves state-of-the-art accuracy across diverse compositional evaluation benchmarks, general cross-modal retrieval, single and multi label linear probing.

Figures & tables

Figure 1: Examples of our main hypothesis. Perturbing abstract keywords yields minor visual changes, whereas concrete words produce larger structural changes.
Figure 2: Overview of ConcretePlant. The pipeline selects a concrete target word from the caption, generates a hard negative caption with Qwen3-32B, and edits the image with SDXL-Turbo to produce a hard negative pair.
Figure 3: Gradient magnitude w.r.t. si,is_{i,i} , si,i′s_{i,i^{\prime}} , ∑jsi,j\sum_{j}s_{i,j} . The dotted lines indicate the average gradient values across the training steps. Severe gradient imbalance cause catastrophic forgetting but can be controlled by margin. Note that ∣∂L∂si,isi,i∣=∣∂L∂si,i′si,i′∣+∣∂L∂si,j∑j∉{i,i′}2Nsi,j∣|\frac{\partial L}{\partial s_{i,i}}s_{i,i}|=|\frac{\partial L}{\partial s_{i,i^{\prime}}}s_{i,i^{\prime}}|+|\frac{\partial L}{\partial s_{i,j}}\sum_{j\notin\{i,i^{\prime}\}}^{2N}s_{i,j}| .
Figure 4: Estimated similarity scores w.r.t. cic_{i} . Curves are fitted with LOESS [ 12 ] .
Figure 5: Analysis of dataset distributions and metric correlations regarding concreteness scores, dataset difficulties and intra-modal similarities.
MetricsDwo\mathcal{D}_{wo}Dlc\mathcal{D}_{lc}Dhc\mathcal{D}_{hc}
Mean ci↑c_{i}\uparrow4.293.814.77
Median ci↑c_{i}\uparrow4.633.894.85
I2T acc. ↑\uparrow0.87120.83570.9225
T2I acc. ↑\uparrow0.44330.40200.5087
BERTScore0.89920.89810.8907
CLIP-L0.91970.92790.9029
Table 1: Statistics summary of datasets by sampling bias.
Figure 6: Gradient imbalance gets worse as batch size grows, but reducing the batch size degrades performance.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Qualitative examples of high- and low-concreteness perturbations.
More in this topic

Compositional Reasoning in VLMs

See all 23 papers

Explore similar work

CardsList
Mar 26, 2026

No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Models

Contrastive vision-language (V&L) models remain a popular choice for various applications. However, several limitations have emerged, most notably the limited ability of V&L models to learn compositional representations. Prior methods often addressed this limitation by generating custom training data to obtain hard negative samples. Hard negatives have been shown to improve performance on compositionality tasks, but are often specific to a single benchmark, do not generalize, and can cause substantial degradation of basic V&L capabilities such as zero-shot or retrieval performance, rendering them impractical. In this work we follow a different approach. We identify two root causes that limit compositionality performance of V&Ls: 1) Long training captions do not require a compositional representation; and 2) The final global pooling in the text and image encoders lead to a complete loss of the necessary information to learn binding in the first place. As a remedy, we propose two simple solutions: 1) We obtain short concept centric caption parts using standard NLP software and align those with the image; and 2) We introduce a parameter-free cross-modal attention-pooling to obtain concept centric visual embeddings from the image encoder. With these two changes and simple auxiliary contrastive losses, we obtain SOTA performance on standard compositionality benchmarks, while maintaining or improving strong zero-shot and retrieval capabilities. This is achieved without increasing inference cost. We release the code for this work at https://github.com/saic-fi/concept_centric_clip.

Hai X. Pham, David T. Hoffmann, Ricardo Guerrero +1

Samsung AI Center Cambridge, UK

Vision-Language ModelsCompositional Reasoning in VLMsContrastive Learning for VLMsMultimodal Contrastive LearningVision-Language Alignment
AskPDF
Mar 13, 2026

Mastering Negation: Boosting Grounding Models via Grouped Opposition-Based Learning

Current vision-language detection and grounding models predominantly focus on prompts with positive semantics and often struggle to accurately interpret and ground complex expressions containing negative semantics. A key reason for this limitation is the lack of high-quality training data that explicitly captures discriminative negative samples and negation-aware language descriptions. To address this challenge, we introduce D-Negation, a new dataset that provides objects annotated with both positive and negative semantic descriptions. Building upon the observation that negation reasoning frequently appears in natural language, we further propose a grouped opposition-based learning framework that learns negation-aware representations from limited samples. Specifically, our method organizes opposing semantic descriptions from D-Negation into structured groups and formulates two complementary loss functions that encourage the model to reason about negation and semantic qualifiers. We integrate the proposed dataset and learning strategy into a state-of-the-art language-based grounding model. By fine-tuning fewer than 10 percent of the model parameters, our approach achieves improvements of up to 4.4 mAP and 5.7 mAP on positive and negative semantic evaluations, respectively. These results demonstrate that explicitly modeling negation semantics can substantially enhance the robustness and localization accuracy of vision-language grounding models.

Zesheng Yang, Xi Jiang, Bingzhang Hu +4

Department of Computer Science, Southern University of Science and Technology, Shenzhen, China · Vision Intelligence Center, Hefei CAS Dihuge Automation Co., LTD, Hefei, China · School of Information Science and Technology, Harbin Institute of Technology (Shenzhen), China +2

Vision-Language Grounding
AskPDF
Sep 8, 2026

CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning

Vision-language models (VLMs) demonstrate strong performance across compositional reasoning benchmarks, which require reasoning over semantic perturbations of objects, attributes, relations, and their interactions. However, our controlled analysis reveals that existing compositionality-aware VLMs exhibit element-specific biases, often underperforming vanilla CLIP on certain compositional elements. To address this, we propose Compositional Scene Graph-guided CLIP (CS-CLIP), which uses scene graphs to identify compositional elements and construct structured negatives via selective masking. We further retain negatives that are most contradictory to the original caption, forcing the model to rely on compositional structure rather than surface cues. CS-CLIP achieves state-of-the-art compositional reasoning with robust performance across compositional elements. It also preserves general vision-language capabilities such as cross-modal retrieval and downstream visual reasoning, while requiring fewer training samples than prior methods.

SeongJun Jeong, Minjoon Jung, Woo Suk Choi +2

Seoul National University

VLM RobustnessContrastive LearningCompositional Reasoning in VLMsVLM Reasoning
AskPDF