Not Just What's There: Enabling CLIP to Comprehend Negated Visual Descriptions Without Fine-tuning

Junhao Xiao; Zhiyu Wu; Hao Lin; Yi Chen; Yahui Liu; Xiaoran Zhao; Zixu Wang; Zejiang He

cs.CVFeb 24, 2026

Not Just What's There: Enabling CLIP to Comprehend Negated Visual Descriptions Without Fine-tuning

Authors: Junhao Xiao, Zhiyu Wu, Hao Lin, Yi Chen, Yahui Liu, Xiaoran Zhao, Zixu Wang, Zejiang He

Organizations: Central China Normal University · Fudan University · Huazhong University of Science and Technology · Kuaishou Technology ·

Abstract

Vision-Language Models (VLMs) like CLIP struggle to understand negation, often embedding affirmatives and negatives similarly (e.g., matching "no dog" with dog images). Existing methods refine negation understanding via fine-tuning CLIP's text encoder, risking overfitting. In this work, we propose CLIPGlasses, a plug-and-play framework that enhances CLIP's ability to comprehend negated visual descriptions. CLIPGlasses adopts a dual-stage design: a Lens module disentangles negated semantics from text embeddings, and a Frame module predicts context-aware repulsion strength, which is integrated into a modified similarity computation to penalize alignment with negated semantics, thereby reducing false positive matches. Experiments show that CLIP equipped with CLIPGlasses achieves competitive in-domain performance and outperforms state-of-the-art methods in cross-domain generalization. Its superiority is especially evident under low-resource conditions, indicating stronger robustness across domains.

Figures & tables

Figure 1: CLIPGlasses enhances CLIP’s capacity for negation understanding by introducing a dynamic repulsion mechanism that suppresses image-text similarity for negated concepts, thus enabling inverse matching while preserving alignment with affirmed content.

Figure 2: t-SNE ( Cai and Ma 2022 ) visualization of CLIP text features for multiple positive-negative sentence pairs ( e.g. , “ there is a woman ” vs. “ there is not a woman ”). Circles and squares denote positive and negative forms, colors distinguish different pairs. Feature clusters across pairs are well-separated, showing CLIP’s strong instance-level discrimination. However, positive and negative features within individual pair remain closely positioned, indicating that while CLIP has limited negation modeling capabilities, there exists clear potential for semantic disentanglement.

Figure 3: CLIPGlasses enhances CLIP’s capability to model negative semantics by introducing two modules: Lens and Frame . Lens disentangles negated concepts ( e.g. , “ dog ” in “no dog ”) from the text embedding $T_{\text{clip}}$ . Frame dynamically predicts a repulsion strength $\lambda$ based on cross-modal context. The final similarity score is computed as $S=S_{\text{I2T}}-\lambda\cdot S_{\text{I2T}}^{\text{neg}}$ , aligning images with affirmed content while repelling from negated concepts when negation is present in the text.

Method	Training Set	Training Size	Testing
		(Images/Captions)	Testing Set	Accuracy	Testing Set	Accuracy
NegCLIP	COCO-caption	82K/31K	CC-Neg-val	62.63%	Neg-COCO-MCQ	10.20%
CoN-CLIP	CC-Neg	188K/376K	CC-Neg-val	99.70%	Neg-COCO-MCQ	25.70%
CLIPGlasses	CC-Neg	188K/376K	CC-Neg-val	96.56% -3.14%	Neg-COCO-MCQ	34.51% +8.81%
CoN-CLIP	Neg-COCO-R	5K/22K	CC-Neg-val	65.91%	Neg-COCO-MCQ	30.61%
CLIPGlasses	Neg-COCO-R	5K/22K	CC-Neg-val	93.36% +27.45%	Neg-COCO-MCQ	35.90% +5.29%

Table 1: Performance comparison highlighting the trade-off between in-domain fitting and cross-domain generalization.

Method	Training Set	ImageNet	caltech101
Vanilla CLIP	-	53.87	90.96
CoN-CLIP	CC-Neg	50.98	88.91
CLIPGlasses	Neg-COCO-R	53.51	90.54
CLIPGlasses	CC-Neg	53.28	90.97

Table 2: Zero-shot performance on standard non-negation benchmarks for models trained on negation datasets.

————————————-Lens Module———————————
Ablation Setting	Acc	$\Delta$ Acc	FAR	$\Delta$ FAR
w/o Syntax Stream	94.09%	-2.47%	4.4927	-3.3326
w/o Semantic Stream	94.93%	-1.63%	6.0925	-1.7328
w/o Residual Gating	68.93%	-27.63%	0.7875	-7.0378
————————————-Frame Module——————————–
w/o Cross-modal	93.79%	-2.77%	7.3051	-0.5202

Table 3: Ablation on CC-Neg evaluating the Lens and Frame modules. “Acc” and “FAR” refer to Accuracy and false alignment rate respectively,

\Delta

denotes the performance drop relative to the full model.

Figure 4: Distribution of predicted repulsion weight

\lambda

under varying negation strengths. Stronger negations ( e.g. , “ no ”) yield higher

\lambda

, confirming the model’s ability to adaptively modulate semantic repulsion.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 846 papers

Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understanding
Mar 31, 2026Jingqi XuVLM Adaptation VLM Reasoning
University of Southern California, Los Angeles, CA, USA
When Negation Is a Geometry Problem in Vision-Language Models
Mar 20, 2026Fawaz Sammani, Tzoulio Chamiti, Paul Gavrikov +1VLM Reasoning VLM Evaluation
ETRO Department, Vrije Universiteit Brussel, Belgium · imec, Kapeldreef 75, B-3001 Leuven, Belgium · Independent Researcher
What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features
Jul 25, 2026Chen-Yi Lu, Yueh-Shao Chen, Somali ChaterjiVision-Language Models VLM Adaptation
Purdue University, West Lafayette, IN 47907, USA

Mar 31, 2026J/K move · Enter open · S save

Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understanding

Jingqi Xu

University of Southern California, Los Angeles, CA, USA

VLM Adaptation VLM Reasoning Contrastive Learning

Open paper AskPDF arXiv

Vision-Language Models (VLMs) have demonstrated strong capabilities across a wide range of multimodal tasks. However, recent studies have shown that VLMs, such as CLIP, perform poorly in understanding negation expressions, which are common in natural language. In this work, we propose Omni-NegCLIP, a fine-tuned CLIP model that improves CLIP's understanding of two types of negation, namely presence-based negation and absence-based negation, which correspond to negated expressions of objects that are actually present in an image and those that may plausibly exist in an image but are in fact absent, respectively, by modifying CLIP's original InfoNCE contrastive loss. Specifically, we design a presence-based contrastive objective that pulls image embeddings closer to their original caption embeddings while pushing them away from the corresponding presence-based negated caption embeddings, and an absence-based contrastive objective that aligns image embeddings with both original and absence-based negated caption embeddings while maintaining a semantic distinction between the two text embeddings. Based on our observation that the front transformer layers of CLIP text encoder have stronger learning ability for negated text than the later layers, we fine-tune the front transformer layers of the CLIP text encoder at each training step using the combined contrastive objective. Experimental results show that, compared with pretrained CLIP, Omni-NegCLIP improves performance on presence-based negation and absence-based negation tasks by up to 52.65% and 12.50%, respectively, without sacrificing general capability in image-text retrieval and even improving it by up to 19.62%. Compared with prior works, Omni-NegCLIP demonstrates a more comprehensive ability to understand multiple types of negation tasks.

Abstract

VLM Reasoning

Explore similar work

Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understanding

When Negation Is a Geometry Problem in Vision-Language Models

What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features