cs.CVOct 12, 2026
SaveKnow-Thy-Not: Training-Free Targeted Negation in Dual-Encoder Vision-Language Models
Organizations: Sungkyunkwan University, South Korea · University of Oslo, Norway
Abstract
Dual-encoder vision-language models systematically fail at negation, scoring ''a cat without a dog'' nearly identically to ''a cat with a dog.'' We show that this failure is inherent to the single-embedding framework. Under contrastive training, single-embedding methods can at best ignore the negated concept (dilution) but never penalize its presence (rejection). We propose Know-Thy-Not (KTN), a training-free method that introduces an image-side rejection dimension to penalize the negated concept directly in the visual representation. We also introduce COCO-Confusion, the first benchmark requiring genuine rejection to succeed. Across 9 backbones, KTN improves COCO-Confusion mAP by up to +25.6 over vanilla VLMs and +9.6 over the strongest existing method.
Figures & tables
Figure 1: Top-5 retrieval on our COCO-Confusion benchmark (leftmost is rank 1). A vanilla VLM ignores the negation, retrieving primarily confusing images that contain a traffic light . The embedding correction approach [ 1 ] removes the negated direction from the text embedding; while this filters out some confusing images (ranks 4–5), it still places traffic-light scenes at the top. In contrast, our proposed Know-Thy-Not (KTN) introduces an explicit image-grounding step, successfully retrieving correct images. Green borders indicate images containing only the affirmed concept ( 2 ), while red borders indicate confusing images containing the additional negated concept ( 3 ).
Figure 2: Given the query “A photo of a cat without a dog ”, a vanilla VLM ignores the negation and activates on both concepts (left). Embedding correction approach [ 1 ] removes the dog direction from the text embedding but never verifies the visual features, which amounts to passively diluting the concept (center). Faithful negation requires explicitly locating the dog in the image and penalizing its presence (right), which forms the crux of our method Know-Thy-Not (KTN).
Figure 3: Dilution cannot separate correct from confusing images. Top-5 similarity scores for “a car is present but no traffic light” on COCO-Confusion, SigLIP2 SO400M. Raw SigLIP2 (left): all top-5 are confusing (red) with uniform scores ( ). Embedding Correction (right): scores drop overall ( ), but red (confusing) and green (correct) bars remain interleaved, failing to separate correct from confusing.
| MCQ | Retrieval | ||||||||
| COCO | VOC2007 | MSRvtt | COCO | MSRvtt | Conf-2C | Conf-3C | |||
| Model | R@5 | R-Neg@5 | R@5 | R-Neg@5 | mAP | mAP | |||
| CLIP [ 19 ] | 24.4 | 27.2 | 30.1 | 65.3 | 57.9 | 55.7 | 48.8 | 63.2 | 60.5 |
| NegCLIP [ 28 ] | 28.5 | 30.4 | 28.4 | 69.5 | 65.2 | 57.0 | 51.6 | 64.4 | 61.4 |
| CLIP* (NegBench) [ 2 ] | 54.4 | 54.9 | 45.3 | 54.9 | 52.3 | 48.8 | 43.1 | 68.8 | 65.0 |
| NegationCLIP [ 18 ] | 35.9 | 44.2 | 37.4 | 62.8 | 62.0 | 52.5 | 49.7 | 72.9 | 68.4 |
Table 1: Negation performance on CLIP ViT-B/32 [ 19 ] . We compare against training-based negation methods [ 28 , 18 ] and training-free embedding correction method (EC) [ 1 ] across MCQ, Retrieval, and our COCO-Confusion benchmark. (*) indicates models fine-tuned on CC12M-NegFull [ 2 ] . Best in bold ; second-best underlined .
| MCQ | Retrieval | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| COCO | VOC2007 | MSRvtt | COCO | MSRvtt | Conf-2C | Conf-3C | ||||
| Model | Backbone | R@5 | R-Neg@5 | R@5 | R-Neg@5 | mAP | mAP | |||
| Vanilla | CLIP [ 19 ] ViT-L/14 | 25.4 | 25.3 | 30.9 | 70.8 | 64.1 | 60.4 | 55.0 | 62.1 | 59.4 |
| EC [ 1 ] | 48.4 | 45.3 | 41.9 | 70.8 | 66.6 | 60.4 | 55.9 | 73.8 | 67.4 | |
| KTN (Ours) | 79.6 | 92.4 | 69.1 | 70.8 | 67.8 | 60.5 | 56.5 | 82.4 | 74.8 | |
| Vanilla | CLIP [ 19 ] ViT-H/14 | 30.2 | 31.9 | 29.9 | 73.2 | 66.8 | 62.6 | 55.0 | 62.0 | 59.3 |
Table 2: Negation performance across multiple VLM backbones [ 29 , 19 , 26 , 5 ] . Each backbone shows three rows: Vanilla VLM, embedding correction (EC) [ 1 ] , and our KTN. Bold indicates best-per-metric across the three methods. KTN consistently outperforms both vanilla and embedding correction on MCQ comprehension and Confusion retrieval, while preserving standard retrieval R@5.
| Method | COCO MCQ | R-Neg@5 | Conf-2C |
|---|---|---|---|
| SigLIP2 | 23.9 | 71.5 | 61.5 |
| + Proj (Dim 1) | 55.1 | 74.6 | 78.6 |
| + Ours (Dim 1+2) | 81.0 | 75.9 | 87.1 |
Table 3: Ablation study (SigLIP2 SO400M). Dimension 1 alone (Proj) achieves dilution. Dimension 2 provides rejection.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Value | Description |
|---|---|---|
| 50 | Sigmoid sharpness in | |
| Rejection weight, scaled by candidate-pool size |
Table 4: KTN hyperparameters. All values fixed across backbones and tasks. denotes the candidate-pool size for the task ( for MCQ, for retrieval).
| 2-object | 3-object | |
| Total pairs (post-verification) | 950 | 1,740 |
| Unique affirmed concepts | 75 | 62 |
| Unique negated concepts | 76 | 62 |
| COCO categories used | 76/80 | 62/80 |
Table 5: Concept coverage in COCO-Confusion.
| 2-object | 3-object | |
|---|---|---|
| Total pairs | 950 | 1,740 |
Table 6: COCO-Confusion final statistics.
Figure 4: 2-object COCO-Confusion example. Query: “a cat is present but no dog.” Top row: correct images, containing a cat but no dog. Bottom row: confusing images, containing both a cat and a dog. All ten images contain a cat—only the negated concept differs across rows. A method that merely dilutes the “dog” direction in the text embedding scores all ten images similarly. Only a method that genuinely rejects separates correct from confusing.
Figure 5: 3-object COCO-Confusion example. Query: “an apple and banana are present but no bowl.” The query requires affirming both apple and banana while rejecting bowl. All shown images contain an apple and a banana. Only the bottom row also contains a bowl.
More in this topic
Vision-Language Models
Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.
See all 722 papers