cs.CVMar 31, 2026

Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understanding

Authors: Jingqi Xu

Organizations: University of Southern California, Los Angeles, CA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-Language Models (VLMs) have demonstrated strong capabilities across a wide range of multimodal tasks. However, recent studies have shown that VLMs, such as CLIP, perform poorly in understanding negation expressions, which are common in natural language. In this work, we propose Omni-NegCLIP, a fine-tuned CLIP model that improves CLIP's understanding of two types of negation, namely presence-based negation and absence-based negation, which correspond to negated expressions of objects that are actually present in an image and those that may plausibly exist in an image but are in fact absent, respectively, by modifying CLIP's original InfoNCE contrastive loss. Specifically, we design a presence-based contrastive objective that pulls image embeddings closer to their original caption embeddings while pushing them away from the corresponding presence-based negated caption embeddings, and an absence-based contrastive objective that aligns image embeddings with both original and absence-based negated caption embeddings while maintaining a semantic distinction between the two text embeddings. Based on our observation that the front transformer layers of CLIP text encoder have stronger learning ability for negated text than the later layers, we fine-tune the front transformer layers of the CLIP text encoder at each training step using the combined contrastive objective. Experimental results show that, compared with pretrained CLIP, Omni-NegCLIP improves performance on presence-based negation and absence-based negation tasks by up to 52.65% and 12.50%, respectively, without sacrificing general capability in image-text retrieval and even improving it by up to 19.62%. Compared with prior works, Omni-NegCLIP demonstrates a more comprehensive ability to understand multiple types of negation tasks.

Figures & tables

Figure 1: (a) An example of presence-based negation from CC-Neg [ 29 ] . (b) An example of absence-based negation from NegRefCOCOg [ 22 ] .
Figure 2: (a) Illustration of our designed presence-based contrastive objective. (b) Illustration of our designed absence-based contrastive objective. (c) Illustration of the fine-tuning pipeline of Omni-NegCLIP.
Figure 3: (a) presence-based negation learning capability across transformer layers in the CLIP text encoder on CC-Neg [ 29 ] . (b) absence-based negation learning capability across transformer layers in the CLIP text encoder on NegRefCOCOg [ 22 ] . Front layers show higher accuracy than later layers, indicating stronger negation learning capability.
ModelArch.CC-Neg [ 29 ]NegRefCOCOg [ 22 ]Avg.COCO [ 19 ]
CLIPViT-B/3265.7057.7361.7240.88
CoN-CLIP [ 29 ]99.7055.2377.4741.72
NegationCLIP [ 22 ]66.2063.8665.0352.12
Omni-NegCLIP99.6064.5582.0848.90
CLIPViT-B/1665.2258.1861.7042.90
CoN-CLIP [ 29 ]99.7055.4577.5845.38
Table 1: Comparison of the performance of different models on negation tasks and the general task across different CLIP architectures. Avg. denotes the average accuracy on CC-Neg [ 29 ] and NegRefCOCOg [ 22 ] . Bold numbers indicate the best performance within each architecture group.
ChoiceCC-Neg [ 29 ]NegRefCOCOg [ 22 ]
Influence of Loss Terms
({ℓk}k=16,  m=0.9)(\{\ell_{k}\}_{k=1}^{6},\;m=0.9)
Omni-NegCLIP(w/o Lp3\mathcal{L}_{p3} )99.4463.41
Omni-NegCLIP(w/o La1\mathcal{L}_{a1} )99.6354.77
Omni-NegCLIP(w/o La3\mathcal{L}_{a3} )99.6661.82
Omni-NegCLIP(all loss terms)99.6064.55
Table 2: Ablation study on loss terms, the selection of transformer layers in the text encoder during fine-tuning, and the margin value mm in the caption contrastive loss La3\mathcal{L}_{a3} .
More in this topic

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 538 papers

Explore similar work

CardsList
Mar 20, 2026

When Negation Is a Geometry Problem in Vision-Language Models

Joint Vision-Language Embedding models such as CLIP typically fail at understanding negation in text queries, for example, failing to distinguish "no" in the query: "a plain blue shirt with no logos". Prior work has largely addressed this limitation through data-centric approaches, fine-tuning CLIP on large-scale synthetic negation datasets. However, these efforts are commonly evaluated using retrieval-based metrics that cannot reliably reflect whether negation is actually understood. In this paper, we identify two key limitations of such evaluation metrics and investigate an alternative evaluation framework based on Multimodal LLMs-as-a-judge, which typically excel at understanding simple yes/no questions about image content, providing a fair evaluation of negation understanding in CLIP models. We then ask whether there already exists a direction in the CLIP embedding space associated with negation. We find evidence that such a direction exists, and show that it can be manipulated through test-time intervention via representation engineering to steer CLIP toward negation-aware behavior without any fine-tuning. Finally, we test negation understanding on non-common image-text samples to evaluate generalization under distribution shifts. Code is at https://github.com/fawazsammani/negation-steering

Fawaz Sammani, Tzoulio Chamiti, Paul Gavrikov +1

ETRO Department, Vrije Universiteit Brussel, Belgium · imec, Kapeldreef 75, B-3001 Leuven, Belgium · Independent Researcher

VLM ReasoningVLM EvaluationVLM RobustnessRepresentation Engineering
AskPDF
Feb 24, 2026

Not Just What's There: Enabling CLIP to Comprehend Negated Visual Descriptions Without Fine-tuning

Vision-Language Models (VLMs) like CLIP struggle to understand negation, often embedding affirmatives and negatives similarly (e.g., matching "no dog" with dog images). Existing methods refine negation understanding via fine-tuning CLIP's text encoder, risking overfitting. In this work, we propose CLIPGlasses, a plug-and-play framework that enhances CLIP's ability to comprehend negated visual descriptions. CLIPGlasses adopts a dual-stage design: a Lens module disentangles negated semantics from text embeddings, and a Frame module predicts context-aware repulsion strength, which is integrated into a modified similarity computation to penalize alignment with negated semantics, thereby reducing false positive matches. Experiments show that CLIP equipped with CLIPGlasses achieves competitive in-domain performance and outperforms state-of-the-art methods in cross-domain generalization. Its superiority is especially evident under low-resource conditions, indicating stronger robustness across domains.

Junhao Xiao, Zhiyu Wu, Hao Lin +5

Central China Normal University · Fudan University · Huazhong University of Science and Technology +2

VLM ReasoningVLM AdaptationVision-Language Alignment
AskPDF
Jul 25, 2026

What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features

Contrastive vision-language models such as CLIP map semantically opposite phrases (e.g., "a dog" vs. "not a dog") to nearly identical embeddings, rendering them insensitive to negation. We attribute this failure to a phenomenon we call Representational Collapse: by tracking compositional divergence and visual alignment across the CLIP text encoder, we show that middle layers build compositional syntax, but the final layers collapse this structure as visual alignment rises, producing a syntax-blind final representation. To recover the lost negation signal without altering pretrained weights, we propose PeakPatch, a lightweight post-hoc correction system that intercepts the encoder at its compositional peak. An Embedding Correction Network (ECN) uses cross-attention to extract a negation-specific signal from the peak layer, anchored to a stable baseline, and predicts a deviation vector that re-injects the lost syntax into the final-layer embedding space. A complementary Score Correction Network (SCN) predicts bounded scalar score offsets for discriminative tasks. Both modules are trained jointly end-to-end while all CLIP parameters remain frozen, adding only 5.2M parameters (3.5% of the backbone) and preserving the standard cosine similarity interface. On NegBench, PeakPatch achieves 74.3% on COCO MCQ (+35.1 over CLIP, +17.8 over the best encoder fine-tuning method) and 65.5% on VOC MCQ, while outperforming all fine-tuning baselines on fully out-of-distribution negation retrieval despite training only 3.5% of the parameters. The corrected embeddings also transfer to text-to-image generation (+18.4 negation score) and generalize across ViT-B/32, ViT-L/14, and SigLIP backbones. Project URL: https://stevencylu.github.io/PeakPatch/.

Chen-Yi Lu, Yueh-Shao Chen, Somali Chaterji

Purdue University, West Lafayette, IN 47907, USA

Vision-Language ModelsVLM AdaptationVLM ReasoningVision-Language Alignment
AskPDF