Negation remains a longstanding challenge for both language models (LMs) and large language models (LLMs). Prior work mainly focuses on a small set of high-frequency single-word negation cues, such as not and never, with limited exploration of broader negation types and modern LLMs. To address this gap, we construct NegCue, a large-scale dataset containing over 1.8M samples spanning single-word, multi-word, and affixal negation with more than 200 unique cues. We further pre-train both encoder-only LMs and LLMs on NegCue to investigate how different negation types affect negation understanding. Experiments on five downstream benchmarks show that negation types contribute unevenly to performance gains under the same training scale. In particular, affixal negation yields the largest improvements, while the gains from the commonly studied single-word negation remain modest. Moreover, our results demonstrate that further pre-training improves negation understanding for both LMs and LLMs.
Figures & tables
Samples from NegCue:
S11 :
The room was extremely hot and crowded.
S21 :
Everyone felt [uncomfortable] affixal cue inside.
S12 :
The big gate was locked from the outside.
S22 :
She did [not] single-word cue manage to open it.
S13 :
The public garden is now private property.
S23 :
It is [no longer] multi-word cue open to the public.
Figure 1: Four examples from NegCue, the corpus we create for Next Sentence Polarity Prediction, our self-supervised task to improve negation understanding. Given a sentence ( S1i ), the task is to predict whether the next sentence ( S2i ) contains negation. While previous work is limited to a handful of cues (e.g., not , never ), we work with all negation types (affixal, single- and multi-word) and over 200 negation cues.
Figure 2: Frequency distribution of the negation cues in the NegCue training splits. Each cue type (inner ring) accounts for 1/3 of NegCue. The three most frequent cues per type (outer ring) account for 71% and 65% of multi- and single-word cues, but only 24% of affixal cues. Unlabeled segments indicate the remaining cues.
Affixal
Single
Multi
Avg. sent. length
22.3
22.2
22.9
# negation cues
167
36
11
most frequent
unable
not
rather than
frequency (%)
10.29
41.81
31.05
least frequent
uncarved
forbid
be deprived of
frequency (%)
0.004
0.032
0.045
Table 1: Descriptive statistics of the NegCue training splits. There are more affixal cues, but most are infrequent. On the other hand, there are fewer single- and multi-word cues, but they are relatively frequent ( not : 41.81%, rather than : 31.05%). Sentence length (# words) is roughly the same across cue types.
Accuracy
Group Consistency
# Pars.
All
( Δ %)
Affixal
Single
Multi
All
Par.
Sco.
Aff.
Fine-tuned encoder-only LMs
Previous Work
BERT-large [1]
340M
46.3
n/a
n/a
n/a
2.2
44.2
14.8
12.4
RoBERTa-large [1]
355M
54.1
n/a
n/a
n/a
13.6
51.6
26.5
27.2
+ affirmative interpretations [2]
67.1
n/a
n/a
n/a
31.4
61.9
43.8
50.7
Table 2: Results on CondaQA with encoder-only LMs and decoder-only LLMs and zero-shot prompts. [1] refers to Ravichander et al. (2022) , [2] to Rezaei and Blanco (2024) , and [3] to Rezaei and Blanco (2025) . Pars. denotes parameters; Par., Sco., and Aff. denote paraphrase, scope edit, and affirmative edit, respectively. Δ indicates the relative performance with respect to the off-the-shelf models, and an asterisk ( ∗ ) indicates statistical significance for Accuracy (All) only (McNemar’s test McNemar (1947) , p<0.05 ). Both LMs and LLMs benefit from further NSPP pre-training over their off-the-shelf counterparts, especially with affixal negation cues.
Accuracy
Group Consistency
Q1
( Δ %)
Q2
( Δ %)
Q3
( Δ %)
Q4
( Δ %)
Q1
Q2
Q3
Q4
BERT-large
46.7
47.1
45.9
48.8
2.5
1.7
1.8
0.0
+ NSPP, affixal
50.0
(+7.1)
50.3
(+6.8)
49.5
(+7.8)
52.2
(+7.0)
7.4
4.6
7.1
2.0
RoBERTa-large
55.3
52.0
48.6
51.4
14.3
8.0
8.8
12.0
+ NSPP, affixal
68.7
(+24.2)
67.4
(+29.6)
66.6
(+37.0)
62.1
(+20.8)
34.5
32.1
31.0
22.0
Llama3.2-1B
48.9
48.3
47.4
48.0
2.3
2.6
0.9
0.0
Table 3: Results with the best NSPP pre-training per model (Table 2 ) on four CondaQA subsets grouped by cue frequency (quartiles; Q1 contains the top-25% most frequent cues). Δ indicates relative change with respect to the off-the-shelf model. NSPP pre-training yields stronger performance gains on moderately frequent cues ( Q2 , Q3 ).
NeQA
NevIR
NMoNLI
ScoNe-NLI
# Pars.
Acc.
( Δ %)
Acc.
( Δ %)
Acc.
( Δ %)
Acc.
( Δ %)
Fine-tuned encoder-only LMs
Previous Work
BERT-SNLI [1]
n/a
n/a
2.2
n/a
RoBERTa-large-MAF-NLI [2]
355M
n/a
n/a
n/a
65.8
Our results
Table 4: Results on four NLU benchmarks. Same notation as Table 2 . [1]: Geiger et al. (2020) (BERT fine-tuned on SNLI); [2]: She et al. (2023) (fine-tuned on the MAF-NLI datasets); [3]: Zhang et al. (2023) . “n/a” indicates that comparable baselines were not reported. Further pre-training with NegCue yields better results across the four corpora. We also outperform results by previous work (when available) by substantial margins despite experimenting with much smaller models (e.g., on NeQA, off-the-shelf Llama3.2-1B achieves 38% accuracy, substantially below Cohere-medium (44%), while our further pre-trained version reaches 45% despite being six times smaller.).
Benchmark
Setting
BERT-Large
RoBERTa-Large
Llama3.2-1B
Llama3.2-3B
CondaQA
Off-the-shelf
46.8
54.1
48.6
58.9
NSP
46.0
63.9
32.3
51.8
NSPP, affixal
50.1
68.0
50.1
59.6
NSPP, single
48.3
65.2
48.8
59.1
NSPP, multi
48.9
65.2
49.3
59.3
NeQA
Off-the-shelf
70.0
42.0
38.0
63.0
Table 5: Performance on CondaQA and NeQA after continued pre-training with NSP or NSPP using different negation cue types. The reported metric is accuracy, and the best results are shown in bold.
Benchmark
Setting
BERT-Large
RoBERTa-Large
Llama3.2-1B
Llama3.2-3B
CondaQA
NSPP, affixal
47.9
66.4
52.3
59.6
NSPP, single
44.3
66.1
49.3
55.6
NSPP, multi
46.0
65.8
52.0
59.5
NeQA
NSPP, affixal
82.0
55.0
46.0
63.0
NSPP, single
79.0
54.0
38.0
62.0
NSPP, multi
74.0
52.0
45.0
63.0
Table 6: Performance on CondaQA and NeQA after NSPP continued pre-training on the balanced affixal, single-word, and multi-word corpora. We report accuracy, with the best results shown in bold.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Random
Acc.
Macro F1
Polarity
50.0
57.7 ± 0.45
57.7 ± 0.44
Negation Type
33.3
39.6 ± 0.24
39.6 ± 0.24
Appendix
Table 7: Linear probing results using only frozen S1 embeddings to predict the polarity or negation type of the following sentence S2 .
Figure 3: Confusion matrices for the two linear probing experiments. Left: polarity prediction (AFF vs. NEG) using only S1 embeddings. Right: negation type prediction (affixal, multi-word, and single-word negation) using only S1 embeddings from NEG samples.
Figure 4: Real examples from the NegCue corpus.
Accuracy
Group Consistency
All
( Δ %)
Affixal
Single
Multi
All
Par.
Sco.
Aff.
Qwen2-0.5B
49.6
48.4
50.9
49.6
3.8
47.0
18.4
15.2
further pre-trained:
affixal cues
49.9
( + 0.7)
49.6
50.7
46.2
5.1
45.1
17.4
17.4
single-word cues
48.4
( − 2.5)
48.1
49.0
44.2
4.1
44.9
16.1
15.6
multi-word cues
50.0
( + 0.8)
49.8
50.2
48.9
4.7
46.1
17.6
17.4
Appendix
Table 8: Evaluation results of Qwen2-0.5B on CondaQA (top), and cue-frequency analysis for the best-performing further pre-trained model (bottom). Par., Sco., and Aff. denote paraphrase, scope edit, and affirmative edit. The lower section reports results on four CondaQA cue-frequency quartiles ( Q1 – Q4 ), where Q1 contains the top 25% most frequent negation cues. Δ indicates the relative change with respect to the off-the-shelf model.
NeQA
NevIR
NMoNLI
ScoNe-NLI
# Pars.
Acc.
( Δ %)
Acc.
( Δ %)
Acc.
( Δ %)
Acc.
( Δ %)
Qwen2-0.5B (off-the-shelf)
0.5B
16.0
49.3
49.5
50.0
further pre-trained, NSPP with
affixal cues
18.0
(+12.5)
49.8
(+1.0)
49.5
(+0.0)
50.1
(+0.2)
single-word cues
17.0
(+6.3)
49.6
(+0.6)
50.0
(+1.0)
49.7
(−0.8)
multi-word cues
16.0
(+0.0)
49.1
(−0.4)
49.5
(+0.0)
50.1
(+0.2)
Appendix
Table 9: Qwen results on four NLU Benchmarks. Δ indicates improvements with respect to the off-the-shelf model.
Figure 5: Zero-shot prompt used to further pre-train LLMs on the NSPP task with our NegCue corpus.
Figure 6: NSPP training loss curves of the three LLMs across training steps.
Training
Validation
Test
CondaQA
5,832
1,110
7,240
NeQA
374
54
100
NevIR
948
225
1,383
NMoNLI
1,002
n/a
200
ScoNe-NLI
6,012
n/a
1,200
SNLI
366,180
6,564
n/a
Appendix
Table 10: The number of samples in the training, validation, and test sets of the benchmarks.
Learning Rate
Batch Size
Epochs
CondaQA
5e-6, 1e-5
8, 16
10
NeQA
1e-5
16
8
NevIR
2e-5
16
20
SNLI
2e-5
16
1
Appendix
Table 11: Hyperparameters used for fully supervised fine-tuning of LMs across benchmarks. For CondaQA, we perform grid search to select the optimal hyperparameter configuration. For NMoNLI and ScoNe-NLI, models are instead fine-tuned on SNLI, as in-domain fine-tuning leads to near-saturated performance.
No
One
Two
Two
One
One Scoped
Negation
Not Scoped
Not Scoped
Scoped
Scoped
One Not Scoped
BERT-large (off-the-shelf )
0 93.5
0 97.0
0 94.0
0 97.5
0 98.5
0 98.5
+ NSPP, Affixal
0 99.0
0 99.0
0 98.0
0 99.5
0 99.0
100.0
+ NSPP, Single-word
0 98.5
0 99.5
0 99.5
100.0
0 99.0
0 99.0
+ NSPP, Multi-word
0 99.0
0 99.0
0 99.0
100.0
0 99.5
0 99.0
+ NSPP, All cues
0 99.0
0 99.5
0 98.0
0 99.5
0 99.0
100.0
Appendix
Table 12: Accuracy (%) on the six ScoNe-NLI test splits for BERT-large and RoBERTa-large fine-tuned on the ScoNe-NLI training set. All models, including off-the-shelf baselines, achieve near-perfect results across all splits.
Model
Entailment
Not Entailment
All
Strict Accuracy
RoBERTa-MNLI
0.648
0.684
0.670
0.250
RoBERTa-NegCue-MNLI
0.710
0.708
0.709
0.271
BERT-NegCue-MNLI
0.639
0.607
0.623
0.104
Appendix
Table 13: Results on NaN-NLI. We report F1 scores for Entailment, Not Entailment, and All under the Binary setting, together with Strict Accuracy. Best results are shown in bold.
Figure 7: Zero-shot prompt example used to evaluate LLMs with CondaQA.
Figure 8: Zero-shot prompt example used to evaluate LLMs on NevIR.
Figure 9: Zero-shot prompt examples used for evaluation on NeQA, ScoNe-NLI, and NMoNLI.
We study how Large Language Models (LLMs) process negation mechanistically. First, we establish that even though open-weight models often provide wrong answers to questions involving negation, they do possess internal components that process negation correctly. Their poor accuracy is due to late-layer attention behavior that promotes simple shortcuts; ablating those attention modules greatly improves accuracy on negation-related questions. Second, we uncover how models process negation. We consider two hypotheses: models could use attention heads that attend to the phrase being negated and suppress related concepts, or they could directly construct a representation of the entire negative phrase (e.g., representing "not gas" as a vector that promotes liquids and solids). We apply a range of observational and causal interpretability techniques on Mistral-7B and Llama-3.1-8B to show that models implement both mechanisms, with the "constructive" mechanism being more prominent. Combined, our work deepens the understanding of LLMs' internals, highlighting construction-dominant computations and the coexistence of competing mechanisms within LLMs.
Zhejian Zhou, Tianyi Zhou, Robin Jia +1
Information Sciences Institute, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California
Negation is an essential feature of human language, yet large language models (LLMs) remain unreliable in processing it. We evaluate recent open-source and closed-source LLMs on our negation benchmark and find that, in 37-71% of cases, they repeat the same answer under negation (e.g., "Madrid" for "What is not the capital of Spain?"). To understand and address this brittleness, we mechanistically examine how models operate under negation. Our main finding is that specialized attention heads and MLP neurons jointly implement negation by (1) suppressing retrieval of the original answer (e.g., "Madrid") while (2) promoting a favored candidate within the answer category (e.g., "Paris"). This contrasts with accounts of human negation processing, in which information about the original answer helps to determine what should be excluded. Furthermore, we find that this difference from human processing is a key source of negation failures: the model's mechanism relies on suppressing the original answer rather than using it to determine what to exclude, so the model can repeat the original answer when suppression is too weak or when a bias toward particular answers prevents it from selecting an alternative. To address this weakness in the model's negation mechanism, we propose a training objective that requires larger shifts in answer preference for more confident original predictions, and show that it reduces negation failures with less degradation of general capabilities than standard fine-tuning baselines. Together, our results demonstrate how mechanistic analysis can reveal why a linguistic capability fails and guide training that targets the underlying limitation.
Jongwook Yoon, Jongwon Lim, Sungjib Lim +2
Graduate School of Data Science, Seoul National University · Department of Computer Science and Engineering, Seoul National University
Negation is a common and important semantic feature in natural language, yet Large Language Models (LLMs) struggle when negation is involved in natural language understanding tasks. Commonsense knowledge, on the other hand, despite being a well-studied topic, lacks investigations involving negation. In this work, we show that commonsense knowledge with negation is challenging for models to understand. We present a novel approach to automatically augment existing commonsense knowledge corpora with negation, yielding two new corpora containing over 2M triples with if-then relations. In addition, pre-training LLMs on our corpora benefits negation understanding.