Probing Negation Understanding in Vision-Language Models With Object Attention

Beth Pearson; Michael Wray; Martha Lewis

cs.CVOct 14, 2026

Probing Negation Understanding in Vision-Language Models With Object Attention

Authors: Beth Pearson, Michael Wray, Martha Lewis

Organizations: University of Bristol · ILLC, University of Amsterdam

Abstract

Negation is a fundamental aspect of language understanding and it is important vision-language models (VLMs) are able to process it effectively. We investigate how VLMs process negation by analysing attention patterns of different models. We draw comparisons between human gaze and attention in models when processing affirmative vs negated statements. We compare object level model attention to human eye-fixations in both binary and multary settings, evaluating with both synthetic and realistic datasets. We find models generally do not follow human-like attention patterns when resolving negation, suggesting they rely on mechanisms outside of attention. While layerwise analysis shows that PaliGemma2-3B shifts object attention to the alternative under negation, our headwise ablations suggest this is a byproduct rather than a negation processing mechanism. Additionally, we find that LLaVA-1.5-7B displays a spatial bias towards left-side objects. Our evaluation of models on downstream tasks shows that all models have a discrepancy between affirmative and negated questions highlighting the need to better understand the underlying processing of negation to close this performance gap. Code available at https://github.com/otmive/negation_attention

Figures & tables

Panel of Figure 1 — Figure 1: Dataset example images: (a) and (b) show images from the Binary and Multary datasets, while (c) and (d) show examples from the What’sUp dataset.

Model	Layers	Attn. Heads	Visual Tokens
LLaVA-1.5-7B	32	32	576
LLaVA-1.5-13B	40	40	576
InternVL3.5-4B	36	32	256
InternVL3.5-8B	32	32	256
PaliGemma2-3B	26	8	256
PaliGemma2-10B	42	16	256

Table 1: Configurations of VLMs

Figure 2: Mean attention to objects for models across datasets. For a human-like response we expect Mentioned $>$ Alternative throughout, Alternative Negated $>$ Mentioned Negated for the Binary and What’sUp datasets, and Mentioned Negated $>$ Alternative Negated for the Multary dataset. Error bars represent the 95% confidence interval.

Figure 3: Layer-wise attention for What’sUp dataset

Panel of Figure 4 — Figure 4: Layer-wise attention patterns for left-shape vs. right-shape colour mentions in both positive and negated contexts for smaller models.

	Binary				Multary				What’sUp
Model	A.T	N.T	A.F.	N.F.	A.T	N.T	A.F.	N.F.	A.T	N.T.	A.F.	N.F.
LLaVA-7B	100.0	22.7	53.6	99.3	93.5	45.5	31.7	73.9	88.7	3.5	29.0	95.7
LLaVA-13B	100.0	39.5	79.1	88.5	100.0	66.5	24.6	51.7	85.1	22.7	40.7	86.3
InternVL-4B	100.0	100.0	100.0	99.8	92.7	77.1	99.9	97.7	83.3	69.9	75.8	78.5
InternVL-8B	100.0	78.2	93.5	98.9	99.2	81.8	91.1	81.4	78.5	29.0	59.4	91.2
PGemma-3B	100.0	76.2	100.0	96.6	100.0	65.6	65.0	95.0	90.3	20.8	61.1	97.6

Table 2: Downstream Accuracy For Models On All Datasets. We report mean accuracy for four question types: affirmative-true (A.T.), negated-true (N.T.), affirmative-false (A.F.) and negated-false (N.F.)

Figure 5: Effect layer ablations on binary dataset. A positive value means the targeted layer caused a decrease in correct prediction.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Comparison of binary results based on relative object sizes.

Figure 8: Layer-wise attention for models in the Binary setting.

Figure 9: Layer-wise attention for models in the Multary setting.

Figure 10: Effect of each layer on correct model output for affirmative and negated prompts across binary and multary datasets. A positive value means the targeted layer caused a decrease in correct prediction.

Figure 11: Layer-wise attention plots for PaliGemma2-3B on downstream questions for correct and incorrect responses.

Figure 12: Layerwise attention plots for each head in PaliGemma2-3B on the WhatsUp dataset.

Figure 13: Headwise total attention to objects for PaliGemma2-3B on the WhatsUp dataset

		Binary				Multary				WhatsUp
Model	Pr.	A.T	N.T	A.F.	N.F.	A.T	N.T	A.F.	N.F.	A.T	N.T.	A.F.	N.F.
LLaVA-7B	1	100.0 0.0	19.4 0.9	91.8 0.6	97.8 0.2	95.7 0.4	1.4 0.3	41.6 1.7	99.0 0.3	90.7 2.2	2.3 1.2	33.3 2.9	97.4 1.2
	2	100.0 0.0	62.6 0.8	41.7 1.6	100.0 0.0	100.0 0.0	74.3 2.5	12.6 1.5	59.0 0.9	84.5 2.3	4.9 1.5	30.8 1.3	93.8 0.8
	3	100.0 0.0	0.0 0.0	82.5 0.8	100.0 0.0	71.9 1.8	80.3 1.2	78.1 1.1	54.0 1.5	87.9 2.3	0.3 0.2	38.6 3.3	99.2 0.6
	4	100.0 0.0	31.6 1.0	3.9 0.5	98.8 0.1	100.0 0.0	71.2 1.5	8.5 1.5	57.6 1.6	91.8 2.2	8.8 1.1	23.9 1.5	88.8 1.6
	5	100.0 0.0	0.0 0.0	48.0 1.9	100.0 0.0	99.9 0.1	0.02 0.0	17.9 1.6	100.0 0.0	88.5 2.2	1.3 0.6	18.5 2.2	99.0 0.3

Appendix

Table 3: Downstream Accuracy (Mean %

\pm

SD) across datasets for each prompt template

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 846 papers

Disparities In Negation Understanding Across Languages In Vision-Language Models
Apr 21, 2026Charikleia Moraitaki, Sarah Pan, Skyler Pulling +3VLM Evaluation Vision-Language Models
Massachusetts Institute of Technology
When Negation Is a Geometry Problem in Vision-Language Models
Mar 20, 2026Fawaz Sammani, Tzoulio Chamiti, Paul Gavrikov +1VLM Reasoning VLM Evaluation
ETRO Department, Vrije Universiteit Brussel, Belgium · imec, Kapeldreef 75, B-3001 Leuven, Belgium · Independent Researcher
How Language Models Process Negation
May 4, 2026Zhejian Zhou, Tianyi Zhou, Robin Jia +1LLM Interpretability Language Modeling
Information Sciences Institute, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California

Apr 21, 2026J/K move · Enter open · S save

Disparities In Negation Understanding Across Languages In Vision-Language Models

Charikleia Moraitaki, Sarah Pan, Skyler Pulling, Gwendolyn Flusche+2

Massachusetts Institute of Technology

VLM Evaluation Vision-Language Models Multilingual VLM Evaluation VLM Reasoning

Open paper AskPDF arXiv

Vision-language models (VLMs) exhibit affirmation bias: a systematic tendency to select positive captions ("X is present") even when the correct description contains negation ("no X"). While prior work has documented this failure mode in English and proposed solutions, negation manifests differently across languages through varying morphology, word order, and cliticization patterns, raising the question of whether these solutions serve all linguistic communities equitably. We introduce the first human-verified multilingual negation benchmark, spanning seven typologically diverse languages: English, Mandarin Chinese, Arabic, Greek, Russian, Tagalog, and Spanish. Evaluating three VLMs - CLIP, SigLIP, and MultiCLIP - we find that standard CLIP performs at or below chance on non-Latin-script languages, while MultiCLIP achieves the highest and most uniform accuracy. We also evaluate SpaceVLM, a proposed negation correction, and find that it produces substantial improvements for several languages - particularly English, Greek, Spanish, and Tagalog - while showing varied effectiveness across typologically different languages. This variation reveals that linguistic properties like morphology, script, and negation structure interact with model improvements in fairness-relevant ways. As VLMs are deployed globally, multilingual benchmarks are essential for understanding not just whether solutions work, but for whom.

Abstract

VLM Reasoning

Explore similar work

Disparities In Negation Understanding Across Languages In Vision-Language Models

When Negation Is a Geometry Problem in Vision-Language Models

How Language Models Process Negation