Skeleton-and-Strategy Prompting: Training-Free Negation Understanding for Vision-Language Models

Yuliang Cai; Mohammad Rostami; Jesse Thomason

cs.CVOct 1, 2026

Skeleton-and-Strategy Prompting: Training-Free Negation Understanding for Vision-Language Models

Authors: Yuliang Cai, Mohammad Rostami, Jesse Thomason

Organizations: Viterbi School of Engineering University of Southern California · Amazon Inc.

Abstract

Despite the strong performance of Vision-Language Models (VLMs) on a wide range of visual question answering (VQA) tasks, these models consistently struggle to understand negation and produce incorrect answers when questions involve negated clauses. To address this limitation, we propose Skeleton-and-Strategy Prompting (\textbf{SSP}), a training-free, in-context learning method that improves VLM negation understanding capabilities without any parameter updates. Given a negation question, our method first abstracts the underlying question structure into a skeleton, retrieves a small set of same-skeleton questions from a lightweight question pool, then prompts the VLM to analyze their shared negation pattern and synthesize a single-sentence answering strategy. The skeleton and strategy are prepended to the test sample to guide the model correctly tackle the negation problems. Experiments on multiple negation VQA benchmarks show that SSP achieves state-of-the-art performance on negation-focused VQA tasks while remaining computationally efficient.

Figures & tables

Figure 1: Comparison of prompting methods under the negation scenario. Given the same negation question to VLMs, the plain prompt and rule-based prompt both produce the answers which align with the visual content. Our Skeleton-and-Strategy Prompting (SSP) extracts the question skeleton and applies a negation-aware strategy, guiding the model to choose the option that differs from the visual content, which is the correct answer for the negation question.

Figure 2: Overview of Skeleton and Strategy Prompting (SSP). SSP first builds a negative example pool offline (top). At inference time, SSP extracts the question skeleton, retrieves or synthesizes a reusable reasoning strategy, and constructs the final prompt for the target VLM (bottom).

Figure 3: Comparison of prompting methods on the NegVQA benchmark. Accuracy (%) is reported for six VLMs spanning commercial, 8B-level, and 2B-level models. SSP achieves the highest accuracy for every model, outperforming plain prompt, CoT negation and four-example in-context prompting, providing consistent gains across all evaluated model scales.

Figure 4: Comparison of prompting methods on the DNVQA benchmark. DNVQA evaluates implicit negation expressed without explicit markers such as not , no , or without . Accuracy (%) is reported for the same six VLMs evaluated on DNVQA. SSP achieves the highest accuracy for every model. These consistent gains show that SSP remains effective when negation is conveyed through diverse semantic expressions rather than explicit lexical cues.

Method	InternVL3-8B	Qwen3VL-8B	LLaVA-OV2-8B	Avg.
Skeleton Only	72.3	70.0	73.7	72.0
Strategy Only	77.6	72.4	75.5	75.2
Random Skeleton + Strategy	76.1	71.0	75.4	74.2
Skeleton + Strategy, one per sample	78.7	75.8	76.3	76.9
SSP	79.1	75.5	76.3	77.0

Table 1: Ablation study on 8B-level VLMs using the NegVQA benchmark. SSP achieves the best average performance, showing that combining matched skeletons with reusable strategies is more effective than using either component alone or using randomly selected skeleton-strategy pairs.

Model	Time (s)	Tokens	Time (s)	Tokens
	Reusable strategy		Per-sample strategy
GPT-4o	51.45	489	2,237.84	20,317
Qwen3VL-8B	35.97	463	1,564.76	20,114
Qwen3VL-2B	27.19	423	1,182.17	18,914

Table 2: Efficiency of reusable strategy pairs on 1,000 NegVQA samples. Total strategy-generation time and generated-token count are reported when SSP reuses cached strategy pairs and when it generates a new strategy for every sample. Lower values indicate greater efficiency.

Pool size	Accuracy (%)
32	76.2
64	76.8
128	78.6
256	79.1
512	79.1

Table 3: Sensitivity of SSP to example-pool size and strategy-generation sample count on NegVQA using InternVL3-8B. Left: Accuracy increases as the example pool grows from 32 to 256 examples and saturates thereafter. Right: Accuracy reaches its maximum when three examples are used to generate each strategy, with no further improvement from additional examples.

Skeleton	Generated Strategy
What [X] is not [Y]	The correct answer to a negative question is the option that specifies the incorrect attribute or action.
How many [X] is not [Y]	Choose the option that represents the incorrect or less accurate number to the question.
Why [X] is not [Y]	The correct answer is that identifies the reason why the situation described in the question does not occur, which can be determined by selecting the option that describes an incorrect or unrelated circumstance.
Which [X] is not [Y]	Identifies the elements or elements that does not match the condition described in the question, and it can be determined by selecting the option that, if negated, would make the statement in the question true.
Is it not [X]	The correct answer is the option that contradicts the statement in the question, indicating the absence or incorrectness of the described condition.

Table 4: Visualization of generated strategies for different negation-aware question skeletons. Each strategy is generated once for a skeleton and reused for later questions with the same question type.

Explore similar work

HANCLIP: A Family of Hyperbolic Angular Negation Vision Language Models
Jun 22, 2026Hoang-Bao Le, Aiden Durrant, Thai Son Mai +3Negation Image-Text Pairs
ADAPT Centre, Dublin City University, Dublin, Ireland · University of East Anglia, Norwich, UK · Queen’s University Belfast, Belfast, UK +1
What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features
Jul 25, 2026Chen-Yi Lu, Yueh-Shao Chen, Somali ChaterjiNegation Representational Collapse
Purdue University, West Lafayette, IN 47907, USA
Disparities In Negation Understanding Across Languages In Vision-Language Models
Apr 21, 2026Charikleia Moraitaki, Sarah Pan, Skyler Pulling +3Negation Linguistics
Massachusetts Institute of Technology

Jun 22, 2026, cs.CVJ/K move · Enter open · S save

HANCLIP: A Family of Hyperbolic Angular Negation Vision Language Models

Hoang-Bao Le, Aiden Durrant, Thai Son Mai, Binh T. Nguyen+2

ADAPT Centre, Dublin City University, Dublin, Ireland · University of East Anglia, Norwich, UK · Queen’s University Belfast, Belfast, UK+1

Negation Image-Text Pairs Hyperbolic Space Token Co-Occurrence Graphs

Open paper AskPDF arXiv

Vision-language models (VLMs) achieve strong cross-modal alignment but remain brittle to negation, often relying on shallow word associations rather than compositional reasoning. Fine-tuning on negation-specific data can also compromise their general purpose capabilities through catastrophic forgetting. We introduce HANCLIP (Hyperbolic, Angular, and Negation), a geometry-aware framework that improves negation sensitivity while preserving the structure of the pretrained joint embedding space. HANCLIP combines a hyperbolic contrastive objective, which models hierarchical relations and semantic asymmetries, with an angular triplet loss that separates negated descriptions from their affirmative counterparts. Using only 20,000 image-text quadruplets, HANCLIP consistently improves performance across CLIP, LongCLIP, and SmartCLIP backbones on the NegBench benchmark, while maintaining or improving zero-shot classification and image-text retrieval performance. These results show that lightweight, geometry-guided objectives can enhance negation understanding without large-scale retraining.

Abstract

Explore similar work

HANCLIP: A Family of Hyperbolic Angular Negation Vision Language Models

What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features

Disparities In Negation Understanding Across Languages In Vision-Language Models