Dual-encoder vision-language models systematically fail at negation, scoring ''a cat without a dog'' nearly identically to ''a cat with a dog.'' We show that this failure is inherent to the single-embedding framework. Under contrastive training, single-embedding methods can at best ignore the negated concept (dilution) but never penalize its presence (rejection). We propose Know-Thy-Not (KTN), a training-free method that introduces an image-side rejection dimension to penalize the negated concept directly in the visual representation. We also introduce COCO-Confusion, the first benchmark requiring genuine rejection to succeed. Across 9 backbones, KTN improves COCO-Confusion mAP by up to +25.6 over vanilla VLMs and +9.6 over the strongest existing method.
Figures & tables
Figure 1: Top-5 retrieval on our COCO-Confusion benchmark (leftmost is rank 1). A vanilla VLM ignores the negation, retrieving primarily confusing images that contain a traffic light . The embedding correction approach [ 1 ] removes the negated direction from the text embedding; while this filters out some confusing images (ranks 4–5), it still places traffic-light scenes at the top. In contrast, our proposed Know-Thy-Not (KTN) introduces an explicit image-grounding step, successfully retrieving correct images. Green borders indicate images containing only the affirmed concept ( 2 ), while red borders indicate confusing images containing the additional negated concept ( 3 ).
Figure 2: Given the query “A photo of a cat without a dog ”, a vanilla VLM ignores the negation and activates on both concepts (left). Embedding correction approach [ 1 ] removes the dog direction from the text embedding but never verifies the visual features, which amounts to passively diluting the concept (center). Faithful negation requires explicitly locating the dog in the image and penalizing its presence (right), which forms the crux of our method Know-Thy-Not (KTN).
Figure 3: Dilution cannot separate correct from confusing images. Top-5 similarity scores for “a car is present but no traffic light” on COCO-Confusion, SigLIP2 SO400M. Raw SigLIP2 (left): all top-5 are confusing (red) with uniform scores ( ≈0.15 ). Embedding Correction (right): scores drop overall ( ≈0.12 ), but red (confusing) and green (correct) bars remain interleaved, failing to separate correct from confusing.
MCQ
Retrieval
COCO
VOC2007
MSRvtt
COCO
MSRvtt
Conf-2C
Conf-3C
Model
R@5
R-Neg@5
R@5
R-Neg@5
mAP
mAP
CLIP [ 19 ]
24.4
27.2
30.1
65.3
57.9
55.7
48.8
63.2
60.5
NegCLIP [ 28 ]
28.5
30.4
28.4
69.5
65.2
57.0
51.6
64.4
61.4
CLIP* (NegBench) [ 2 ]
54.4
54.9
45.3
54.9
52.3
48.8
43.1
68.8
65.0
NegationCLIP [ 18 ]
35.9
44.2
37.4
62.8
62.0
52.5
49.7
72.9
68.4
Table 1: Negation performance on CLIP ViT-B/32 [ 19 ] . We compare against training-based negation methods [ 28 , 18 ] and training-free embedding correction method (EC) [ 1 ] across MCQ, Retrieval, and our COCO-Confusion benchmark. (*) indicates models fine-tuned on CC12M-NegFull [ 2 ] . Best in bold ; second-best underlined .
MCQ
Retrieval
COCO
VOC2007
MSRvtt
COCO
MSRvtt
Conf-2C
Conf-3C
Model
Backbone
R@5
R-Neg@5
R@5
R-Neg@5
mAP
mAP
Vanilla
CLIP [ 19 ] ViT-L/14
25.4
25.3
30.9
70.8
64.1
60.4
55.0
62.1
59.4
EC [ 1 ]
48.4
45.3
41.9
70.8
66.6
60.4
55.9
73.8
67.4
KTN (Ours)
79.6
92.4
69.1
70.8
67.8
60.5
56.5
82.4
74.8
Vanilla
CLIP [ 19 ] ViT-H/14
30.2
31.9
29.9
73.2
66.8
62.6
55.0
62.0
59.3
Table 2: Negation performance across multiple VLM backbones [ 29 , 19 , 26 , 5 ] . Each backbone shows three rows: Vanilla VLM, embedding correction (EC) [ 1 ] , and our KTN. Bold indicates best-per-metric across the three methods. KTN consistently outperforms both vanilla and embedding correction on MCQ comprehension and Confusion retrieval, while preserving standard retrieval R@5.
Table 4: KTN hyperparameters. All values fixed across backbones and tasks. N denotes the candidate-pool size for the task ( N=4 for MCQ, N=∣G∣ for retrieval).
2-object
3-object
Total pairs (post-verification)
950
1,740
Unique affirmed concepts
75
62
Unique negated concepts
76
62
COCO categories used
76/80
62/80
Appendix
Table 5: Concept coverage in COCO-Confusion.
2-object
3-object
Total pairs
950
1,740
Appendix
Table 6: COCO-Confusion final statistics.
Figure 4: 2-object COCO-Confusion example. Query: “a cat is present but no dog.” Top row: correct images, containing a cat but no dog. Bottom row: confusing images, containing both a cat and a dog. All ten images contain a cat—only the negated concept differs across rows. A method that merely dilutes the “dog” direction in the text embedding scores all ten images similarly. Only a method that genuinely rejects c− separates correct from confusing.
Figure 5: 3-object COCO-Confusion example. Query: “an apple and banana are present but no bowl.” The query requires affirming both apple and banana while rejecting bowl. All shown images contain an apple and a banana. Only the bottom row also contains a bowl.
Dual-encoder vision-language models (VLMs) expose a similarity interface that enables zero-shot retrieval but fails compositional constraints: queries like "umbrella and no person" retrieve images containing both, even when concept detection is reliable. We trace this to an interface-level Bag-of-Concepts effect, where similarity scores approximate mean pooling of concept evidence regardless of operators. Although operator-dependent signals exist in text embeddings, they are too weak or misaligned to affect rankings. Fine-tuning does not reliably resolve this failure because the dominant bottleneck is how similarity aggregates evidence rather than what encoders represent. We propose factored inference, which separates evidence extraction from constraint execution, and introduce LCSE (Logic-Constrained Score Editing), a training-free method that executes constraints externally using concept scores from frozen encoders. We also introduce FACTOR-Bench, where LCSE achieves 85.5% accuracy versus 73.2% for the best fine-tuned baseline, 90.7% when applied to SigLIP 2, and improves NegBench COCO MCQ accuracy from 27.2% to 65.2% while preserving retrieval performance.
Joint Vision-Language Embedding models such as CLIP typically fail at understanding negation in text queries, for example, failing to distinguish "no" in the query: "a plain blue shirt with no logos". Prior work has largely addressed this limitation through data-centric approaches, fine-tuning CLIP on large-scale synthetic negation datasets. However, these efforts are commonly evaluated using retrieval-based metrics that cannot reliably reflect whether negation is actually understood. In this paper, we identify two key limitations of such evaluation metrics and investigate an alternative evaluation framework based on Multimodal LLMs-as-a-judge, which typically excel at understanding simple yes/no questions about image content, providing a fair evaluation of negation understanding in CLIP models. We then ask whether there already exists a direction in the CLIP embedding space associated with negation. We find evidence that such a direction exists, and show that it can be manipulated through test-time intervention via representation engineering to steer CLIP toward negation-aware behavior without any fine-tuning. Finally, we test negation understanding on non-common image-text samples to evaluate generalization under distribution shifts. Code is at https://github.com/fawazsammani/negation-steering
Contrastive vision-language models such as CLIP map semantically opposite phrases (e.g., "a dog" vs. "not a dog") to nearly identical embeddings, rendering them insensitive to negation. We attribute this failure to a phenomenon we call Representational Collapse: by tracking compositional divergence and visual alignment across the CLIP text encoder, we show that middle layers build compositional syntax, but the final layers collapse this structure as visual alignment rises, producing a syntax-blind final representation. To recover the lost negation signal without altering pretrained weights, we propose PeakPatch, a lightweight post-hoc correction system that intercepts the encoder at its compositional peak. An Embedding Correction Network (ECN) uses cross-attention to extract a negation-specific signal from the peak layer, anchored to a stable baseline, and predicts a deviation vector that re-injects the lost syntax into the final-layer embedding space. A complementary Score Correction Network (SCN) predicts bounded scalar score offsets for discriminative tasks. Both modules are trained jointly end-to-end while all CLIP parameters remain frozen, adding only 5.2M parameters (3.5% of the backbone) and preserving the standard cosine similarity interface. On NegBench, PeakPatch achieves 74.3% on COCO MCQ (+35.1 over CLIP, +17.8 over the best encoder fine-tuning method) and 65.5% on VOC MCQ, while outperforming all fine-tuning baselines on fully out-of-distribution negation retrieval despite training only 3.5% of the parameters. The corrected embeddings also transfer to text-to-image generation (+18.4 negation score) and generalize across ViT-B/32, ViT-L/14, and SigLIP backbones. Project URL: https://stevencylu.github.io/PeakPatch/.