Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understanding

Jingqi Xu

cs.CVMar 31, 2026

Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understanding

Authors: Jingqi Xu

Organizations: University of Southern California, Los Angeles, CA, USA

Abstract

Vision-Language Models (VLMs) have demonstrated strong capabilities across a wide range of multimodal tasks. However, recent studies have shown that VLMs, such as CLIP, perform poorly in understanding negation expressions, which are common in natural language. In this work, we propose Omni-NegCLIP, a fine-tuned CLIP model that improves CLIP's understanding of two types of negation, namely presence-based negation and absence-based negation, which correspond to negated expressions of objects that are actually present in an image and those that may plausibly exist in an image but are in fact absent, respectively, by modifying CLIP's original InfoNCE contrastive loss. Specifically, we design a presence-based contrastive objective that pulls image embeddings closer to their original caption embeddings while pushing them away from the corresponding presence-based negated caption embeddings, and an absence-based contrastive objective that aligns image embeddings with both original and absence-based negated caption embeddings while maintaining a semantic distinction between the two text embeddings. Based on our observation that the front transformer layers of CLIP text encoder have stronger learning ability for negated text than the later layers, we fine-tune the front transformer layers of the CLIP text encoder at each training step using the combined contrastive objective. Experimental results show that, compared with pretrained CLIP, Omni-NegCLIP improves performance on presence-based negation and absence-based negation tasks by up to 52.65% and 12.50%, respectively, without sacrificing general capability in image-text retrieval and even improving it by up to 19.62%. Compared with prior works, Omni-NegCLIP demonstrates a more comprehensive ability to understand multiple types of negation tasks.

Figures & tables

Figure 1: (a) An example of presence-based negation from CC-Neg [ 29 ] . (b) An example of absence-based negation from NegRefCOCOg [ 22 ] .

Figure 2: (a) Illustration of our designed presence-based contrastive objective. (b) Illustration of our designed absence-based contrastive objective. (c) Illustration of the fine-tuning pipeline of Omni-NegCLIP.

Figure 3: (a) presence-based negation learning capability across transformer layers in the CLIP text encoder on CC-Neg [ 29 ] . (b) absence-based negation learning capability across transformer layers in the CLIP text encoder on NegRefCOCOg [ 22 ] . Front layers show higher accuracy than later layers, indicating stronger negation learning capability.

Model	Arch.	CC-Neg [ 29 ]	NegRefCOCOg [ 22 ]	Avg.	COCO [ 19 ]
CLIP	ViT-B/32	65.70	57.73	61.72	40.88
CoN-CLIP [ 29 ]		99.70	55.23	77.47	41.72
NegationCLIP [ 22 ]		66.20	63.86	65.03	52.12
Omni-NegCLIP		99.60	64.55	82.08	48.90
CLIP	ViT-B/16	65.22	58.18	61.70	42.90
CoN-CLIP [ 29 ]		99.70	55.45	77.58	45.38

Table 1: Comparison of the performance of different models on negation tasks and the general task across different CLIP architectures. Avg. denotes the average accuracy on CC-Neg [ 29 ] and NegRefCOCOg [ 22 ] . Bold numbers indicate the best performance within each architecture group.

Choice	CC-Neg [ 29 ]	NegRefCOCOg [ 22 ]
Influence of Loss Terms
$(\{\ell_{k}\}_{k=1}^{6},\;m=0.9)$
Omni-NegCLIP(w/o $\mathcal{L}_{p3}$ )	99.44	63.41
Omni-NegCLIP(w/o $\mathcal{L}_{a1}$ )	99.63	54.77
Omni-NegCLIP(w/o $\mathcal{L}_{a3}$ )	99.66	61.82
Omni-NegCLIP(all loss terms)	99.60	64.55

Table 2: Ablation study on loss terms, the selection of transformer layers in the text encoder during fine-tuning, and the margin value

m

in the caption contrastive loss

\mathcal{L}_{a3}

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 538 papers

When Negation Is a Geometry Problem in Vision-Language Models
Mar 20, 2026Fawaz Sammani, Tzoulio Chamiti, Paul Gavrikov +1VLM Reasoning VLM Evaluation
ETRO Department, Vrije Universiteit Brussel, Belgium · imec, Kapeldreef 75, B-3001 Leuven, Belgium · Independent Researcher
Not Just What's There: Enabling CLIP to Comprehend Negated Visual Descriptions Without Fine-tuning
Feb 24, 2026Junhao Xiao, Zhiyu Wu, Hao Lin +5VLM Reasoning VLM Adaptation
Central China Normal University · Fudan University · Huazhong University of Science and Technology +2
What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features
Jul 25, 2026Chen-Yi Lu, Yueh-Shao Chen, Somali ChaterjiVision-Language Models VLM Adaptation
Purdue University, West Lafayette, IN 47907, USA

Mar 20, 2026J/K move · Enter open · S save

When Negation Is a Geometry Problem in Vision-Language Models

Fawaz Sammani, Tzoulio Chamiti, Paul Gavrikov, Nikos Deligiannis

ETRO Department, Vrije Universiteit Brussel, Belgium · imec, Kapeldreef 75, B-3001 Leuven, Belgium · Independent Researcher

VLM Reasoning VLM Evaluation VLM Robustness Representation Engineering

Open paper AskPDF arXiv

Joint Vision-Language Embedding models such as CLIP typically fail at understanding negation in text queries, for example, failing to distinguish "no" in the query: "a plain blue shirt with no logos". Prior work has largely addressed this limitation through data-centric approaches, fine-tuning CLIP on large-scale synthetic negation datasets. However, these efforts are commonly evaluated using retrieval-based metrics that cannot reliably reflect whether negation is actually understood. In this paper, we identify two key limitations of such evaluation metrics and investigate an alternative evaluation framework based on Multimodal LLMs-as-a-judge, which typically excel at understanding simple yes/no questions about image content, providing a fair evaluation of negation understanding in CLIP models. We then ask whether there already exists a direction in the CLIP embedding space associated with negation. We find evidence that such a direction exists, and show that it can be manipulated through test-time intervention via representation engineering to steer CLIP toward negation-aware behavior without any fine-tuning. Finally, we test negation understanding on non-common image-text samples to evaluate generalization under distribution shifts. Code is at https://github.com/fawazsammani/negation-steering

Abstract

VLM Adaptation

Explore similar work

When Negation Is a Geometry Problem in Vision-Language Models

Not Just What's There: Enabling CLIP to Comprehend Negated Visual Descriptions Without Fine-tuning

What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features