Negation does not carry a uniform interpretation across domains. In legal, regulatory, and medical reasoning, the intended interpretation depends on the reading in force -- open- versus closed-world, two- versus three-valued, and credulous versus skeptical. We study which reading of negation large language models adopt by default and whether they can override that preference when a different reading is explicitly specified. To this end, we introduce NAFBench, a procedural generator of solver-certified instances spanning four semantic viewpoints: SLDNF, well-founded semantics (WFS), and credulous and skeptical reasoning under stable-model semantics. The generator emits ground normal logic programs with controlled depth, width, and cycle structure. Each program is solved under all four viewpoints using SWI-Prolog, a well-founded semantics solver, and clingo, yielding up to four divergent labels. The programs are then verbalized into natural language under multiple framings and rule orderings that leave the answer invariant. The results expose a consistent gap. Across open-source models, following a specified negation semantics remains unsolved: the strongest models score 59--74% across the four semantic viewpoints, while the weakest score 31--67%. All models are order-sensitive on more than half of logically identical rule shufflings, while the two weaker models frequently overcommit on well-founded "undefined." Two frontier models reach 100% on the main fixed-complexity evaluation set, and a third, o4-mini, is near-perfect, falling only to 81% on well-founded "undefined." Delegating reasoning to a solver, fine-tuning on certified traces, or forcing an explicit three-valued verdict each partly closes the gap.
Figures & tables
Figure 2: The NAF-Bench generation-and-verification pipeline. A generated program is (i) solved under all four semantic viewpoints to obtain certified labels and a divergence bin, and (ii) rendered to natural language under several framings and rule orders. The model is prompted under conditions C0 (no semantics named), C1 (semantics named, agreeing instance), and C2 (semantics named, divergent instance); answers are scored against the certified label.
Model
cred.
skept.
WFS
SLDNF
cor.
Frontier
Claude Sonnet 5
100
100
100
100
n/a
GPT-5.6 Sol
100
100
100
100
n/a
o4-mini
100
98
81
100
0 4
Open-source
Qwen2.5-coder 32B
72
61
74
59
55
Table 1: Semantic-following accuracy (%) per semantics and the default-correspondence rate, main evaluation set. “cred.”/“skept.” are credulous/skeptical stable. “cor.” is a default-correspondence measure: of a model’s incorrect answers, the percentage that coincide with its own no-instruction default reading (lower is better; n/a when a model makes no errors). 95% CIs (clustered over the 120 programs) span roughly ±8 – 11 points on accuracy for the open panel.
Model
control
even-1
odd
even-2
Qwen2.5-coder 32B
65
67
62
72
Llama3-8B
78
30
37
51
DeepSeek-R1 32B
81
38
52
35
Table 2: Accuracy (%) by divergence bin, semantics named (excludes the no-instruction baseline). “control” = all four semantic viewpoints agree; “even-1” even-one-sided, “even-2” even-both-sided. Weaker models are strong on control and collapse on divergent bins despite identical program size. Each bin has 30 programs ×4 semantics ( 120 prompts, minus a few unparsed).
negation depth (accuracy %)
Model
d=0
2
4
6
8
Qwen2.5-coder 32B
71
67
67
64
64
Llama3-8B
44
43
45
44
51
DeepSeek-R1 32B
42
61
68
41
51
shared-subgoal width (accuracy %)
Model
w=0
2
4
6
8
Table 3: Marginal accuracy along the two nominal-complexity axes (depth sweep averaged over width, and vice versa) on the larger grid ( 222 prompts, 74 programs, even-one-sided bin): each of the 25 depth × width cells contributes up to three distinct programs × three conditions (credulous/skeptical/WFS).
Model
accuracy (%)
flip (%)
cov.
Frontier
Claude Sonnet 5
99.7
0 1
640/640
GPT-5.6 Sol
100
0 0
640/640
o4-mini
94.8
15
640/640
Open-source
Qwen2.5-coder 32B
68.1
54
640/640
Table 4: Rule-order robustness: the main evaluation programs re-rendered in four logically-equivalent rule orders ( 640 prompts =160 (program, condition) groups ×4 ). Accuracy is over all 640 prompts; the order-flip rate is the fraction of groups whose answer is not constant across the four orders (group counted if ≥2 orders parse); “cov.” is parsed prompts.
Model
definite (A/B)
correct “undef.”
Qwen2.5-coder 32B
9%
91%
Llama3-8B
74%
26%
DeepSeek-R1 32B
71%
29%
Table 5: Behaviour on the main evaluation WFS-condition items whose certified answer is undefined ( ≈85 parsed per model; same programs as E1). “definite” is the share answered A/B.
Model
direct WFS
+ solver (translate-then-solve)
Qwen2.5-coder 32B
59
95
Llama3-8B
49
74
DeepSeek-R1 32B
62
89
Table 6: Translate-then-solve on 44 structurally-diverse WFS programs (accuracy % of parsed answers). Per-model nulls, strict accuracy, and the verify-scaffold breakdown are in the appendix.
Model
base
+SFT
+SFT+DPO
gemma-3-4b-it
41
91
—
qwen2.5-7b
59
93
93
Table 7: WFS accuracy (%) after LoRA fine-tuning on solver-certified reasoning, held-out 44-item set; the qwen + SFT cell is the mean of 3 seeds ( 91 – 96% ).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Model
control
ev_one
odd
ev_both
Claude Sonnet 5
30A
30C
30C
30A
GPT-5.6 Sol
30A
30C
30C
30A
o4-mini
12A
18C
1B
29C
1A
2B
27C
30A
Qwen2.5-coder 32B
10A
20C
19A
2B
9C
15A
2B
12C
22A
8C
DeepSeek-R1 32B
7A
1B
22C
17A
5B
6C
8A
10B
8C
23A
1B
1C
Llama3-8B
19A
11C
27A
1B
2C
21A
9C
22A
1B
7C
Appendix
Table 9: Answers under the unspecified-semantics condition ( none ) per divergence bin, with 30 programs in each bin. A = definitely yes, B = definitely no, and C = cannot be determined; blank subcolumns are zero, and row totals below 30 indicate unparsed answers.
Model
cred
skept
wfs
c-world
Claude Sonnet 5
60
60
90
90
GPT-5.6 Sol
60
60
90
90
o4-mini
44
44
68
68
Qwen2.5-coder 32B
53
49
39
39
DeepSeek-R1 32B
57
43
22
22
Llama3-8B
68
63
37
37
Appendix
Table 10: Number of answers, out of 120 prompts, agreeing with each candidate reading under the unspecified-semantics condition. Unparsed answers agree with no reading.
Model
correct
auditable
sound
contradicted
Claude Sonnet 5
480
219
219
0 0 (0.0%)
GPT-5.6 Sol
480
262
262
0 0 (0.0%)
o4-mini
455
243
243
0 0 (0.0%)
Qwen2.5-coder 32B
314
150
129
21 (14.0%)
DeepSeek-R1 32B
224
105
0 81
24 (22.9%)
Llama3-8B
224
0 71
0 43
28 (39.4%)
Appendix
Table 11: Among correct answers, the frequency with which an auditable trace states a verdict that contradicts solver certification.
Model
sldnf
cred
skept
wfs
Claude Sonnet 5
42/42
48/48
59/59
70/70
GPT-5.6 Sol
27/27
42/42
95/95
98/98
o4-mini
42/42
67/67
61/61
74/96
Qwen2.5-coder 32B
17/54
37/55
34/76
46/67
DeepSeek-R1 32B
26/95
12/25
23/42
24/94
Llama3-8B
0 6/43
10/33
17/59
15/62
Appendix
Table 12: Sound derivations over auditable traces, by prompting condition.
odd + skept
odd + cred
Model
correct / earned
correct / earned
Claude Sonnet 5
30 / 30
30 / 30
GPT-5.6 Sol
30 / 30
30 / 30
o4-mini
30 / 29
30 / 30
Qwen2.5-coder 32B
23 / 0 1
0 3 / 1
DeepSeek-R1 32B
14 / 10
18 / 5
Appendix
Table 13: “Earned” means that the trace states that the program has no stable model—the reason the skeptical answer is vacuously yes. Earned status is detected by pattern-matching the trace for an explicit no-stable-model statement, so an unmatched phrasing reads as unearned; the earned counts are lower bounds.
Model
Metric
Direct
Verify
T2S
Qwen2.5-coder 32B
Parsed accuracy (%)
59
82
95
Strict accuracy (%; nulls)
59 (0)
82 (0)
95 (0)
DeepSeek-R1 32B
Parsed accuracy (%)
62
69
89
Strict accuracy (%; nulls)
45 (12)
57 (8)
77 (6)
Llama3-8B
Parsed accuracy (%)
49
62
74
Strict accuracy (%; nulls)
48 (1)
48 (10)
70 (2)
Appendix
Table 14: Mitigation results on the 44-program WFS set. Null counts are shown in parentheses for strict accuracy. Translation success for T2S was 44/44 for Qwen2.5-coder, 38/44 for DeepSeek-R1, and 42/44 for Llama3; unsuccessful translations are counted as incorrect under strict scoring.
We study how Large Language Models (LLMs) process negation mechanistically. First, we establish that even though open-weight models often provide wrong answers to questions involving negation, they do possess internal components that process negation correctly. Their poor accuracy is due to late-layer attention behavior that promotes simple shortcuts; ablating those attention modules greatly improves accuracy on negation-related questions. Second, we uncover how models process negation. We consider two hypotheses: models could use attention heads that attend to the phrase being negated and suppress related concepts, or they could directly construct a representation of the entire negative phrase (e.g., representing "not gas" as a vector that promotes liquids and solids). We apply a range of observational and causal interpretability techniques on Mistral-7B and Llama-3.1-8B to show that models implement both mechanisms, with the "constructive" mechanism being more prominent. Combined, our work deepens the understanding of LLMs' internals, highlighting construction-dominant computations and the coexistence of competing mechanisms within LLMs.
Zhejian Zhou, Tianyi Zhou, Robin Jia +1
Information Sciences Institute, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California
Negation is an essential feature of human language, yet large language models (LLMs) remain unreliable in processing it. We evaluate recent open-source and closed-source LLMs on our negation benchmark and find that, in 37-71% of cases, they repeat the same answer under negation (e.g., "Madrid" for "What is not the capital of Spain?"). To understand and address this brittleness, we mechanistically examine how models operate under negation. Our main finding is that specialized attention heads and MLP neurons jointly implement negation by (1) suppressing retrieval of the original answer (e.g., "Madrid") while (2) promoting a favored candidate within the answer category (e.g., "Paris"). This contrasts with accounts of human negation processing, in which information about the original answer helps to determine what should be excluded. Furthermore, we find that this difference from human processing is a key source of negation failures: the model's mechanism relies on suppressing the original answer rather than using it to determine what to exclude, so the model can repeat the original answer when suppression is too weak or when a bias toward particular answers prevents it from selecting an alternative. To address this weakness in the model's negation mechanism, we propose a training objective that requires larger shifts in answer preference for more confident original predictions, and show that it reduces negation failures with less degradation of general capabilities than standard fine-tuning baselines. Together, our results demonstrate how mechanistic analysis can reveal why a linguistic capability fails and guide training that targets the underlying limitation.
Jongwook Yoon, Jongwon Lim, Sungjib Lim +2
Graduate School of Data Science, Seoul National University · Department of Computer Science and Engineering, Seoul National University
Reasoning about necessity and possibility depends on assumptions about accessibility between worlds and about which objects exist at each one. The same inference may therefore hold under one modal system and fail under another. Evaluating language models on such problems requires testing whether their judgments follow the stated semantics rather than a familiar logic. We construct paired modal problems with identical premises and conjecture but different frame or domain conditions; automated reasoning verifies opposite labels. A balanced core prevents the semantic condition alone from revealing the answer. On this core, four of five recent models perform below the condition-only baseline under direct prompting. Yet enabling reasoning mode raises DeepSeek V4 Flash from 4.4% to 88.1% on unchanged prompts. Following stipulated modal semantics thus depends strongly on inference mode as well as model identity. When frame conditions are omitted, models often agree but fit different familiar logics best. We release the formulas, oracle artifacts, countermodels, and responses.
Réemi Andrieu, Damien Sileo
Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 - CRIStAL, F-59000 Lille, France