Not all Negation Cues are Equal: Affixal Negations Yield Better Negation Understanding

Tian Tan; Eduardo Blanco

cs.CLSep 12, 2026

Not all Negation Cues are Equal: Affixal Negations Yield Better Negation Understanding

Authors: Tian Tan, Eduardo Blanco

Organizations: Department of Computer Science, University of Arizona

Abstract

Negation remains a longstanding challenge for both language models (LMs) and large language models (LLMs). Prior work mainly focuses on a small set of high-frequency single-word negation cues, such as not and never, with limited exploration of broader negation types and modern LLMs. To address this gap, we construct NegCue, a large-scale dataset containing over 1.8M samples spanning single-word, multi-word, and affixal negation with more than 200 unique cues. We further pre-train both encoder-only LMs and LLMs on NegCue to investigate how different negation types affect negation understanding. Experiments on five downstream benchmarks show that negation types contribute unevenly to performance gains under the same training scale. In particular, affixal negation yields the largest improvements, while the gains from the commonly studied single-word negation remain modest. Moreover, our results demonstrate that further pre-training improves negation understanding for both LMs and LLMs.

Figures & tables

Samples from NegCue:
$S_{1}^{1}$ :	The room was extremely hot and crowded.
$S_{2}^{1}$ :	Everyone felt [uncomfortable] affixal cue inside.
$S_{1}^{2}$ :	The big gate was locked from the outside.
$S_{2}^{2}$ :	She did [not] single-word cue manage to open it.
$S_{1}^{3}$ :	The public garden is now private property.
$S_{2}^{3}$ :	It is [no longer] multi-word cue open to the public.

Figure 1: Four examples from NegCue, the corpus we create for Next Sentence Polarity Prediction, our self-supervised task to improve negation understanding. Given a sentence (

S_{1}^{i}

), the task is to predict whether the next sentence (

S_{2}^{i}

) contains negation. While previous work is limited to a handful of cues (e.g., not , never ), we work with all negation types (affixal, single- and multi-word) and over 200 negation cues.

Figure 2: Frequency distribution of the negation cues in the NegCue training splits. Each cue type (inner ring) accounts for 1/3 of NegCue. The three most frequent cues per type (outer ring) account for 71% and 65% of multi- and single-word cues, but only 24% of affixal cues. Unlabeled segments indicate the remaining cues.

	Affixal	Single	Multi
Avg. sent. length	22.3	22.2	22.9
# negation cues	167	36	11
most frequent	unable	not	rather than
frequency (%)	10.29	41.81	31.05
least frequent	uncarved	forbid	be deprived of
frequency (%)	0.004	0.032	0.045

Table 1: Descriptive statistics of the NegCue training splits. There are more affixal cues, but most are infrequent. On the other hand, there are fewer single- and multi-word cues, but they are relatively frequent ( not : 41.81%, rather than : 31.05%). Sentence length (# words) is roughly the same across cue types.

		Accuracy					Group Consistency
	# Pars.	All	( $\Delta$ %)	Affixal	Single	Multi	All	Par.	Sco.	Aff.
Fine-tuned encoder-only LMs
Previous Work
BERT-large [1]	340M	46.3		n/a	n/a	n/a	2.2	44.2	14.8	12.4
RoBERTa-large [1]	355M	54.1		n/a	n/a	n/a	13.6	51.6	26.5	27.2
+ affirmative interpretations [2]		67.1		n/a	n/a	n/a	31.4	61.9	43.8	50.7

Table 2: Results on CondaQA with encoder-only LMs and decoder-only LLMs and zero-shot prompts. [1] refers to Ravichander et al. (2022) , [2] to Rezaei and Blanco (2024) , and [3] to Rezaei and Blanco (2025) . Pars. denotes parameters; Par., Sco., and Aff. denote paraphrase, scope edit, and affirmative edit, respectively.

\Delta

indicates the relative performance with respect to the off-the-shelf models, and an asterisk ( ∗ ) indicates statistical significance for Accuracy (All) only (McNemar’s test McNemar (1947) ,

p<0.05

). Both LMs and LLMs benefit from further NSPP pre-training over their off-the-shelf counterparts, especially with affixal negation cues.

	Accuracy								Group Consistency
	$Q_{1}$	( $\Delta$ %)	$Q_{2}$	( $\Delta$ %)	$Q_{3}$	( $\Delta$ %)	$Q_{4}$	( $\Delta$ %)	$Q_{1}$	$Q_{2}$	$Q_{3}$	$Q_{4}$
BERT-large	46.7		47.1		45.9		48.8		2.5	1.7	1.8	0.0
+ NSPP, affixal	50.0	$(+7.1)$	50.3	$(+6.8)$	49.5	$(+7.8)$	52.2	$(+7.0)$	7.4	4.6	7.1	2.0
RoBERTa-large	55.3		52.0		48.6		51.4		14.3	8.0	8.8	12.0
+ NSPP, affixal	68.7	$(+24.2)$	67.4	$(+29.6)$	66.6	$(+37.0)$	62.1	$(+20.8)$	34.5	32.1	31.0	22.0
Llama3.2-1B	48.9		48.3		47.4		48.0		2.3	2.6	0.9	0.0

Table 3: Results with the best NSPP pre-training per model (Table 2 ) on four CondaQA subsets grouped by cue frequency (quartiles;

Q_{1}

contains the top-25% most frequent cues).

\Delta

indicates relative change with respect to the off-the-shelf model. NSPP pre-training yields stronger performance gains on moderately frequent cues (

Q_{2}

Q_{3}

		NeQA		NevIR		NMoNLI		ScoNe-NLI
	# Pars.	Acc.	( $\Delta$ %)	Acc.	( $\Delta$ %)	Acc.	( $\Delta$ %)	Acc.	( $\Delta$ %)
Fine-tuned encoder-only LMs
Previous Work
BERT-SNLI [1]		n/a		n/a		2.2		n/a
RoBERTa-large-MAF-NLI [2]	355M	n/a		n/a		n/a		65.8
Our results

Table 4: Results on four NLU benchmarks. Same notation as Table 2 . [1]: Geiger et al. (2020) (BERT fine-tuned on SNLI); [2]: She et al. (2023) (fine-tuned on the MAF-NLI datasets); [3]: Zhang et al. (2023) . “n/a” indicates that comparable baselines were not reported. Further pre-training with NegCue yields better results across the four corpora. We also outperform results by previous work (when available) by substantial margins despite experimenting with much smaller models (e.g., on NeQA, off-the-shelf Llama3.2-1B achieves 38% accuracy, substantially below Cohere-medium (44%), while our further pre-trained version reaches 45% despite being six times smaller.).

Benchmark	Setting	BERT-Large	RoBERTa-Large	Llama3.2-1B	Llama3.2-3B
CondaQA	Off-the-shelf	46.8	54.1	48.6	58.9
	NSP	46.0	63.9	32.3	51.8
	NSPP, affixal	50.1	68.0	50.1	59.6
	NSPP, single	48.3	65.2	48.8	59.1
	NSPP, multi	48.9	65.2	49.3	59.3
NeQA	Off-the-shelf	70.0	42.0	38.0	63.0

Table 5: Performance on CondaQA and NeQA after continued pre-training with NSP or NSPP using different negation cue types. The reported metric is accuracy, and the best results are shown in bold.

Benchmark	Setting	BERT-Large	RoBERTa-Large	Llama3.2-1B	Llama3.2-3B
CondaQA	NSPP, affixal	47.9	66.4	52.3	59.6
	NSPP, single	44.3	66.1	49.3	55.6
	NSPP, multi	46.0	65.8	52.0	59.5
NeQA	NSPP, affixal	82.0	55.0	46.0	63.0
	NSPP, single	79.0	54.0	38.0	62.0
	NSPP, multi	74.0	52.0	45.0	63.0

Table 6: Performance on CondaQA and NeQA after NSPP continued pre-training on the balanced affixal, single-word, and multi-word corpora. We report accuracy, with the best results shown in bold.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Random	Acc.	Macro F1
Polarity	50.0	57.7 $\pm$ 0.45	57.7 $\pm$ 0.44
Negation Type	33.3	39.6 $\pm$ 0.24	39.6 $\pm$ 0.24

Appendix

Table 7: Linear probing results using only frozen

S_{1}

embeddings to predict the polarity or negation type of the following sentence

S_{2}

Figure 3: Confusion matrices for the two linear probing experiments. Left: polarity prediction (AFF vs. NEG) using only $S_{1}$ embeddings. Right: negation type prediction (affixal, multi-word, and single-word negation) using only $S_{1}$ embeddings from NEG samples.

Panel of Figure 4 — Figure 4: Real examples from the NegCue corpus.

	Accuracy					Group Consistency
	All	( $\Delta$ %)	Affixal	Single	Multi	All	Par.	Sco.	Aff.
Qwen2-0.5B	49.6		48.4	50.9	49.6	3.8	47.0	18.4	15.2
further pre-trained:
affixal cues	49.9	( $+$ 0.7)	49.6	50.7	46.2	5.1	45.1	17.4	17.4
single-word cues	48.4	( $-$ 2.5)	48.1	49.0	44.2	4.1	44.9	16.1	15.6
multi-word cues	50.0	( $+$ 0.8)	49.8	50.2	48.9	4.7	46.1	17.6	17.4

Appendix

Table 8: Evaluation results of Qwen2-0.5B on CondaQA (top), and cue-frequency analysis for the best-performing further pre-trained model (bottom). Par., Sco., and Aff. denote paraphrase, scope edit, and affirmative edit. The lower section reports results on four CondaQA cue-frequency quartiles (

Q_{1}

–

Q_{4}

), where

Q_{1}

contains the top 25% most frequent negation cues.

\Delta

indicates the relative change with respect to the off-the-shelf model.

		NeQA		NevIR		NMoNLI		ScoNe-NLI
	# Pars.	Acc.	( $\Delta$ %)	Acc.	( $\Delta$ %)	Acc.	( $\Delta$ %)	Acc.	( $\Delta$ %)
Qwen2-0.5B (off-the-shelf)	0.5B	16.0		49.3		49.5		50.0
further pre-trained, NSPP with
affixal cues		18.0	$(+12.5)$	49.8	$(+1.0)$	49.5	$(+0.0)$	50.1	$(+0.2)$
single-word cues		17.0	$(+6.3)$	49.6	$(+0.6)$	50.0	$(+1.0)$	49.7	$(-0.8)$
multi-word cues		16.0	$(+0.0)$	49.1	$(-0.4)$	49.5	$(+0.0)$	50.1	$(+0.2)$

Appendix

Table 9: Qwen results on four NLU Benchmarks.

\Delta

indicates improvements with respect to the off-the-shelf model.

Panel of Figure 5 — Figure 5: Zero-shot prompt used to further pre-train LLMs on the NSPP task with our NegCue corpus.

Figure 6: NSPP training loss curves of the three LLMs across training steps.

	Training	Validation	Test
CondaQA	5,832	1,110	7,240
NeQA	374	54	100
NevIR	948	225	1,383
NMoNLI	1,002	n/a	200
ScoNe-NLI	6,012	n/a	1,200
SNLI	366,180	6,564	n/a

Appendix

Table 10: The number of samples in the training, validation, and test sets of the benchmarks.

	Learning Rate	Batch Size	Epochs
CondaQA	5e-6, 1e-5	8, 16	10
NeQA	1e-5	16	8
NevIR	2e-5	16	20
SNLI	2e-5	16	1

Appendix

Table 11: Hyperparameters used for fully supervised fine-tuning of LMs across benchmarks. For CondaQA, we perform grid search to select the optimal hyperparameter configuration. For NMoNLI and ScoNe-NLI, models are instead fine-tuned on SNLI, as in-domain fine-tuning leads to near-saturated performance.

	No	One	Two	Two	One	One Scoped
	Negation	Not Scoped	Not Scoped	Scoped	Scoped	One Not Scoped
BERT-large (off-the-shelf )	0 93.5	0 97.0	0 94.0	0 97.5	0 98.5	0 98.5
+ NSPP, Affixal	0 99.0	0 99.0	0 98.0	0 99.5	0 99.0	100.0
+ NSPP, Single-word	0 98.5	0 99.5	0 99.5	100.0	0 99.0	0 99.0
+ NSPP, Multi-word	0 99.0	0 99.0	0 99.0	100.0	0 99.5	0 99.0
+ NSPP, All cues	0 99.0	0 99.5	0 98.0	0 99.5	0 99.0	100.0

Appendix

Table 12: Accuracy (%) on the six ScoNe-NLI test splits for BERT-large and RoBERTa-large fine-tuned on the ScoNe-NLI training set. All models, including off-the-shelf baselines, achieve near-perfect results across all splits.

Model	Entailment	Not Entailment	All	Strict Accuracy
RoBERTa-MNLI	0.648	0.684	0.670	0.250
RoBERTa-NegCue-MNLI	0.710	0.708	0.709	0.271
BERT-NegCue-MNLI	0.639	0.607	0.623	0.104

Appendix

Table 13: Results on NaN-NLI. We report F1 scores for Entailment, Not Entailment, and All under the Binary setting, together with Strict Accuracy. Best results are shown in bold.

Panel of Figure 7 — Figure 7: Zero-shot prompt example used to evaluate LLMs with CondaQA.

Panel of Figure 8 — Figure 8: Zero-shot prompt example used to evaluate LLMs on NevIR.

Panel of Figure 9 — Figure 9: Zero-shot prompt examples used for evaluation on NeQA, ScoNe-NLI, and NMoNLI.

How Language Models Process Negation
May 4, 2026Zhejian Zhou, Tianyi Zhou, Robin Jia +1LLM Interpretability Language Modeling
Information Sciences Institute, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California
How Do LLMs Change Predictions Under Negation?
Oct 7, 2026Jongwook Yoon, Jongwon Lim, Sungjib Lim +2LLM Reliability LLM Evaluation
Graduate School of Data Science, Seoul National University · Department of Computer Science and Engineering, Seoul National University
Commonsense Knowledge with Negation: A Resource to Enhance Negation Understanding
Apr 21, 2026Zijie Wang, MohammadHossein Rezaei, Farzana Rashid +1Knowledge Augmentation for Language Models LLM Training
University of Arizona · University of North Carolina Asheville

May 4, 2026J/K move · Enter open · S save

How Language Models Process Negation

Zhejian Zhou, Tianyi Zhou, Robin Jia, Jonathan May

Information Sciences Institute, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California

LLM Interpretability Language Modeling Attention Mechanisms

Open paper AskPDF arXiv

We study how Large Language Models (LLMs) process negation mechanistically. First, we establish that even though open-weight models often provide wrong answers to questions involving negation, they do possess internal components that process negation correctly. Their poor accuracy is due to late-layer attention behavior that promotes simple shortcuts; ablating those attention modules greatly improves accuracy on negation-related questions. Second, we uncover how models process negation. We consider two hypotheses: models could use attention heads that attend to the phrase being negated and suppress related concepts, or they could directly construct a representation of the entire negative phrase (e.g., representing "not gas" as a vector that promotes liquids and solids). We apply a range of observational and causal interpretability techniques on Mistral-7B and Llama-3.1-8B to show that models implement both mechanisms, with the "constructive" mechanism being more prominent. Combined, our work deepens the understanding of LLMs' internals, highlighting construction-dominant computations and the coexistence of competing mechanisms within LLMs.

Abstract

Explore similar work

How Language Models Process Negation

How Do LLMs Change Predictions Under Negation?

Commonsense Knowledge with Negation: A Resource to Enhance Negation Understanding