Not What You Meant: Can LLMs Follow a Specified Negation Semantics?

Qiming Bao; Agnieszka Mensfelt; Michael J. Witbrock; Kostas Stathis

cs.AISep 23, 2026

Not What You Meant: Can LLMs Follow a Specified Negation Semantics?

Authors: Qiming Bao, Agnieszka Mensfelt, Michael J. Witbrock, Kostas Stathis

Organizations: University of Auckland, New Zealand · Royal Holloway, University of London, United Kingdom

Abstract

Negation does not carry a uniform interpretation across domains. In legal, regulatory, and medical reasoning, the intended interpretation depends on the reading in force -- open- versus closed-world, two- versus three-valued, and credulous versus skeptical. We study which reading of negation large language models adopt by default and whether they can override that preference when a different reading is explicitly specified. To this end, we introduce NAFBench, a procedural generator of solver-certified instances spanning four semantic viewpoints: SLDNF, well-founded semantics (WFS), and credulous and skeptical reasoning under stable-model semantics. The generator emits ground normal logic programs with controlled depth, width, and cycle structure. Each program is solved under all four viewpoints using SWI-Prolog, a well-founded semantics solver, and clingo, yielding up to four divergent labels. The programs are then verbalized into natural language under multiple framings and rule orderings that leave the answer invariant. The results expose a consistent gap. Across open-source models, following a specified negation semantics remains unsolved: the strongest models score 59--74% across the four semantic viewpoints, while the weakest score 31--67%. All models are order-sensitive on more than half of logically identical rule shufflings, while the two weaker models frequently overcommit on well-founded "undefined." Two frontier models reach 100% on the main fixed-complexity evaluation set, and a third, o4-mini, is near-perfect, falling only to 81% on well-founded "undefined." Delegating reasoning to a solver, fine-tuning on certified traces, or forcing an explicit three-valued verdict each partly closes the gap.

Figures & tables

Panel of Figure 2 — Figure 2: The NAF-Bench generation-and-verification pipeline. A generated program is (i) solved under all four semantic viewpoints to obtain certified labels and a divergence bin, and (ii) rendered to natural language under several framings and rule orders. The model is prompted under conditions C0 (no semantics named), C1 (semantics named, agreeing instance), and C2 (semantics named, divergent instance); answers are scored against the certified label.

Frontier
Model	cred.	skept.	WFS	SLDNF	cor.
Claude Sonnet 5	100	100	100	100	n/a
GPT-5.6 Sol	100	100	100	100	n/a
o4-mini	100	98	81	100	0 4
Open-source
Qwen2.5-coder 32B	72	61	74	59	55

Table 1: Semantic-following accuracy (%) per semantics and the default-correspondence rate, main evaluation set. “cred.”/“skept.” are credulous/skeptical stable. “cor.” is a default-correspondence measure: of a model’s incorrect answers, the percentage that coincide with its own no-instruction default reading (lower is better; n/a when a model makes no errors). 95% CIs (clustered over the 120 programs) span roughly

\pm 8

–

11

points on accuracy for the open panel.

Model	control	even-1	odd	even-2
Qwen2.5-coder 32B	65	67	62	72
Llama3-8B	78	30	37	51
DeepSeek-R1 32B	81	38	52	35

Table 2: Accuracy (%) by divergence bin, semantics named (excludes the no-instruction baseline). “control” = all four semantic viewpoints agree; “even-1” even-one-sided, “even-2” even-both-sided. Weaker models are strong on control and collapse on divergent bins despite identical program size. Each bin has

30

programs

\times\,4

semantics (

120

prompts, minus a few unparsed).

	negation depth (accuracy %)
Model	$d{=}0$	$2$	$4$	$6$	$8$
Qwen2.5-coder 32B	71	67	67	64	64
Llama3-8B	44	43	45	44	51
DeepSeek-R1 32B	42	61	68	41	51
	shared-subgoal width (accuracy %)
Model	$w{=}0$	$2$	$4$	$6$	$8$

Table 3: Marginal accuracy along the two nominal-complexity axes (depth sweep averaged over width, and vice versa) on the larger grid (

222

prompts,

74

programs, even-one-sided bin): each of the

25

depth

\times

width cells contributes up to three distinct programs

\times

three conditions (credulous/skeptical/WFS).

Model	accuracy (%)	flip (%)	cov.
Frontier
Claude Sonnet 5	99.7	0 1	640/640
GPT-5.6 Sol	100	0 0	640/640
o4-mini	94.8	15	640/640
Open-source
Qwen2.5-coder 32B	68.1	54	640/640

Table 4: Rule-order robustness: the main evaluation programs re-rendered in four logically-equivalent rule orders (

640

prompts

=

160

(program, condition) groups

\times 4

). Accuracy is over all

640

prompts; the order-flip rate is the fraction of groups whose answer is not constant across the four orders (group counted if

\geq 2

orders parse); “cov.” is parsed prompts.

Model	definite (A/B)	correct “undef.”
Qwen2.5-coder 32B	9%	91%
Llama3-8B	74%	26%
DeepSeek-R1 32B	71%	29%

Table 5: Behaviour on the main evaluation WFS-condition items whose certified answer is undefined (

\approx 85

parsed per model; same programs as E1). “definite” is the share answered A/B.

Model	direct WFS	$+$ solver (translate-then-solve)
Qwen2.5-coder 32B	59	95
Llama3-8B	49	74
DeepSeek-R1 32B	62	89

Table 6: Translate-then-solve on

44

structurally-diverse WFS programs (accuracy % of parsed answers). Per-model nulls, strict accuracy, and the verify-scaffold breakdown are in the appendix.

Model	base	+SFT	+SFT+DPO
gemma-3-4b-it	41	91	—
qwen2.5-7b	59	93	93

Table 7: WFS accuracy (%) after LoRA fine-tuning on solver-certified reasoning, held-out 44-item set; the qwen

+

SFT cell is the mean of

3

seeds (

91

–

96\%

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	control			ev_one			odd			ev_both
Claude Sonnet 5	30A					30C			30C	30A
GPT-5.6 Sol	30A					30C			30C	30A
o4-mini	12A		18C		1B	29C	1A	2B	27C	30A
Qwen2.5-coder 32B	10A		20C	19A	2B	9C	15A	2B	12C	22A		8C
DeepSeek-R1 32B	7A	1B	22C	17A	5B	6C	8A	10B	8C	23A	1B	1C
Llama3-8B	19A		11C	27A	1B	2C	21A		9C	22A	1B	7C

Appendix

Table 9: Answers under the unspecified-semantics condition ( none ) per divergence bin, with 30 programs in each bin. A

=

definitely yes, B

=

definitely no, and C

=

cannot be determined; blank subcolumns are zero, and row totals below 30 indicate unparsed answers.

Model	cred	skept	wfs	c-world
Claude Sonnet 5	60	60	90	90
GPT-5.6 Sol	60	60	90	90
o4-mini	44	44	68	68
Qwen2.5-coder 32B	53	49	39	39
DeepSeek-R1 32B	57	43	22	22
Llama3-8B	68	63	37	37

Appendix

Table 10: Number of answers, out of 120 prompts, agreeing with each candidate reading under the unspecified-semantics condition. Unparsed answers agree with no reading.

Model	correct	auditable	sound	contradicted
Claude Sonnet 5	480	219	219	0 0 (0.0%)
GPT-5.6 Sol	480	262	262	0 0 (0.0%)
o4-mini	455	243	243	0 0 (0.0%)
Qwen2.5-coder 32B	314	150	129	21 (14.0%)
DeepSeek-R1 32B	224	105	0 81	24 (22.9%)
Llama3-8B	224	0 71	0 43	28 (39.4%)

Appendix

Table 11: Among correct answers, the frequency with which an auditable trace states a verdict that contradicts solver certification.

Model	sldnf	cred	skept	wfs
Claude Sonnet 5	42/42	48/48	59/59	70/70
GPT-5.6 Sol	27/27	42/42	95/95	98/98
o4-mini	42/42	67/67	61/61	74/96
Qwen2.5-coder 32B	17/54	37/55	34/76	46/67
DeepSeek-R1 32B	26/95	12/25	23/42	24/94
Llama3-8B	0 6/43	10/33	17/59	15/62

Appendix

Table 12: Sound derivations over auditable traces, by prompting condition.

	odd $+$ skept	odd $+$ cred
Model	correct / earned	correct / earned
Claude Sonnet 5	30 / 30	30 / 30
GPT-5.6 Sol	30 / 30	30 / 30
o4-mini	30 / 29	30 / 30
Qwen2.5-coder 32B	23 / 0 1	0 3 / 1
DeepSeek-R1 32B	14 / 10	18 / 5

Appendix

Table 13: “Earned” means that the trace states that the program has no stable model—the reason the skeptical answer is vacuously yes. Earned status is detected by pattern-matching the trace for an explicit no-stable-model statement, so an unmatched phrasing reads as unearned; the earned counts are lower bounds.

Model	Metric	Direct	Verify	T2S
Qwen2.5-coder 32B	Parsed accuracy (%)	59	82	95
Qwen2.5-coder 32B	Strict accuracy (%; nulls)	59 (0)	82 (0)	95 (0)
DeepSeek-R1 32B	Parsed accuracy (%)	62	69	89
DeepSeek-R1 32B	Strict accuracy (%; nulls)	45 (12)	57 (8)	77 (6)
Llama3-8B	Parsed accuracy (%)	49	62	74
Llama3-8B	Strict accuracy (%; nulls)	48 (1)	48 (10)	70 (2)

Appendix

Table 14: Mitigation results on the 44-program WFS set. Null counts are shown in parentheses for strict accuracy. Translation success for T2S was

44/44

for Qwen2.5-coder,

38/44

for DeepSeek-R1, and

42/44

for Llama3; unsuccessful translations are counted as incorrect under strict scoring.

Nonmonotonic Reasoning

Nonmonotonic reasoning studies formalisms in which conclusions can be withdrawn when new information reveals exceptions or conflicts.

See all 17 papers

How Language Models Process Negation
May 4, 2026Zhejian Zhou, Tianyi Zhou, Robin Jia +1LLM Interpretability Language Modeling
Information Sciences Institute, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California
How Do LLMs Change Predictions Under Negation?
Oct 7, 2026Jongwook Yoon, Jongwon Lim, Sungjib Lim +2LLM Reliability LLM Evaluation
Graduate School of Data Science, Seoul National University · Department of Computer Science and Engineering, Seoul National University
Same Formulas, Different Semantics: Do Language Models Follow Modal Logic Specifications?
Aug 5, 2026Réemi Andrieu, Damien SileoLogical Reasoning LLM Evaluation
Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 - CRIStAL, F-59000 Lille, France

May 4, 2026J/K move · Enter open · S save

How Language Models Process Negation

Zhejian Zhou, Tianyi Zhou, Robin Jia, Jonathan May

Information Sciences Institute, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California

LLM Interpretability Language Modeling Attention Mechanisms

Open paper AskPDF arXiv

We study how Large Language Models (LLMs) process negation mechanistically. First, we establish that even though open-weight models often provide wrong answers to questions involving negation, they do possess internal components that process negation correctly. Their poor accuracy is due to late-layer attention behavior that promotes simple shortcuts; ablating those attention modules greatly improves accuracy on negation-related questions. Second, we uncover how models process negation. We consider two hypotheses: models could use attention heads that attend to the phrase being negated and suppress related concepts, or they could directly construct a representation of the entire negative phrase (e.g., representing "not gas" as a vector that promotes liquids and solids). We apply a range of observational and causal interpretability techniques on Mistral-7B and Llama-3.1-8B to show that models implement both mechanisms, with the "constructive" mechanism being more prominent. Combined, our work deepens the understanding of LLMs' internals, highlighting construction-dominant computations and the coexistence of competing mechanisms within LLMs.

Abstract

Nonmonotonic Reasoning

Explore similar work

How Language Models Process Negation

How Do LLMs Change Predictions Under Negation?

Same Formulas, Different Semantics: Do Language Models Follow Modal Logic Specifications?