"Not" is Not Lost in Translation: A Causal Analysis of Negation Representations in Multilingual LLMs

Giang Son Nguyen; Wray Buntine; Dung D. Le; Wenya Wang

cs.CLOct 23, 2026

"Not" is Not Lost in Translation: A Causal Analysis of Negation Representations in Multilingual LLMs

Authors: Giang Son Nguyen, Wray Buntine, Dung D. Le, Wenya Wang

Organizations: Nanyang Technological University · VinUniversity · Monash University

Abstract

Negation occurs in every language, yet its surface realization differs drastically. Despite this, multilingual large language models (LLMs) translate negation robustly, raising the question of whether they share an underlying representation of negation across languages. We study this question in six open models across English, French, Vietnamese, and Chinese, using causal interventions on a synthetic contrastive dataset and a negation translation dataset derived from FLORES-200. Across models, we find strong evidence that negation representations are causally controllable and largely shared across languages. In particular, we show that a contrastive activation direction extracted from one language can induce or remove negation in the others, and directions extracted from different languages are substantially aligned. We further find that negation is represented in sparse features: a single sparse autoencoder (SAE) feature per model suffices to control polarity across languages, while responding selectively to negated inputs. Finally, both dense and sparse interventions generalize to four additional languages and FLORES-derived sentences. We release data and code to support future work: https://anonymous.4open.science/r/negation-translation/.

Figures & tables

Figure 1: Overview of causal interventions on negation representations . Adding or subtracting a contrastive direction or one SAE feature’s decoder vector during translation can reverse polarity across languages.

Lang.	Negative	Positive
EN	We will not finish the chapter tonight.	We will finish the chapter tonight.
VI	Tôi không đeo khẩu trang khi ra ngoài.	Tôi đeo khẩu trang khi ra ngoài.
FR	Mon père ne taille pas le rosier.	Mon père taille le rosier.
ZH	我不打算骑摩托车去郊外。	我打算骑摩托车去郊外。

Figure 2: Examples of Mini-minus-4 contrastive pairs and activation extraction positions. Each pair differs only in polarity. The negated predicate (NP) position is bold and underlined. The final source token (FST) is typically the period.

Figure 3: Cross-language transfer flip rates (off-diagonal cells) for CAA steering (top: addition, bottom: removal). Flip rates remain moderate to high when applying negation directions from any language to the others, indicating negation directions transfer strongly across languages .

Model	EN	VI	FR	ZH
Gemma-3-1B	86.9 / 77.0	65.0 / 81.8	82.0 / 99.0	85.0 / 95.5
Gemma-3-4B	98.0 / 88.0	83.0 / 95.9	92.0 / 94.0	99.0 / 97.9
Gemma-3-12B	99.0 / 95.0	93.0 / 97.0	84.0 / 100.0	98.0 / 96.9
Qwen3-4B	90.9 / 92.7	78.0 / 95.9	60.0 / 98.0	97.0 / 95.7
Qwen3.5-9B	98.0 / 93.0	97.0 / 93.9	95.0 / 98.0	99.0 / 94.5
Llama-3.1-8B	94.9 / 91.0	86.0 / 74.5	87.0 / 95.0	95.0 / 95.8

Table 1: Within-language CAA addition/removal rates. (

l,p

) is chosen once, while

\alpha

is chosen for each language and flip mode by value sweeps over the evaluation set. Values

\geq 80\%

are bolded.

Figure 4: Per-layer comparison of within and cross-language flip rates. Cross-language lines are means of all 12 transfer directions’ flip rates, within-language lines are means of 4 within-language flip rates. Across all models, cross-language transfer strength follows within-language control and remains strong in multiple layers .

Negation Removal beyond negation markers (ZH

\to

EN)

Ig SRC 我们不愿意参加讨论。 (unwilling)

We are unwilling to participate…

\to

We want to participate…

Ig SRC 同事们不常点外卖。 (rarely)

Colleagues rarely order takeout.

\to

Colleagues often order takeout.

Figure 5: Intervention flips negation without explicit markers (Qwen3-4B). Baseline

\to

steered translation.

Figure 6: Pairwise cosine similarities between negation directions extracted from different languages at the same $(l,p)$ .

Figure 7: Representative activation patterns of the selected SAE features. Darker red indicates higher SAE activation. Gemma-3-4B L4 f331 fires strongly on positions of negation markers and Llama-3.1-8B L13 f12611 activates on tokens in negated contexts, while Qwen3.5-9B L10 f43677 fires on both.

Feature	EN	VI	FR	ZH
G4B L4 f331	96.0 / 91.0	91.0 / 95.9	97.0 / 95.0	95.0 / 94.8
Q9B L10 f43677	97.0 / 95.0	93.0 / 93.9	99.0 / 97.0	96.0 / 97.8
L8B L13 f12611	83.8 / 91.0	72.0 / 57.1	87.0 / 92.0	90.0 / 82.3

Table 2: Addition/removal rates using one top-performing EN-selected SAE feature (

K=1

) per model. Values

\geq 80\%

are bolded.

Table 3: Generalization of dense and sparse negation interventions. Each cell reports addition/removal rates. CAA uses the

\mathbf{d}^{\text{EN}}

for each model, while SAE uses the single EN-selected feature. Values

\geq 80\%

are bolded.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Dim. Index	Ratio	Mass range	Raw cos.	Zeroed cos.
Gemma-3-1B	1038	$525\times$	0.2– 52.7 %	0.540	0.642
Gemma-3-4B	443	$2093\times$	85.5–92.6 %	0.960	0.643
Gemma-3-12B	2339	$2198\times$	83.9–93.2 %	0.965	0.673
Qwen3-4B	0	$54\times$	0.1–3.8%	0.679	0.677
Qwen3.5-9B	3994	$166\times$	0.0–1.3%	0.667	0.668
Llama-3.1-8B	4055	$38\times$	0.1–1.9%	0.658	0.657

Appendix

Table 4: Massive-activation dimensions and their effect on cross-lingual similarity. Ratio is the mean absolute magnitude of the highest-ranked dimension relative to the median dimension. Mass range is the fraction of the direction’s squared norm carried by that dimension across languages. Raw cos. and Zeroed cos. are the mean pairwise cross-lingual cosine similarities before and after zeroing the identified dimension.

Model	Mean $\|\Delta\|$ flip	Max. $\|\Delta\|$ flip
Gemma-3-1B	0.7 pp	3 pp
Gemma-3-4B	0.4 pp	2 pp
Gemma-3-12B	0.5 pp	4 pp

Appendix

Table 5: Effect of removing the massive-activation dimension on causal steering. Changes are computed across the full cross-language transfer matrices for both addition and removal.

Table 6: Target-language ablation with EN as the fixed source language. Top: pairwise cosine similarity between negation directions extracted with VI, FR, and ZH as translation targets. Bottom: addition and removal rates using the corresponding target-specific direction.

Model	EN	VI	FR	ZH
Gemma-3-1B	0 / 0	0 / 2	0 / 1	0 / 8
Gemma-3-4B	0 / 1	0 / 6	0 / 0	0 / 6
Gemma-3-12B	0 / 4	0 / 2	0 / 5	0 / 5
Qwen3-4B	0 / 0	0 / 1	0 / 0	0 / 1
Qwen3.5-9B	0 / 0	0 / 0	0 / 0	0 / 1
Llama-3.1-8B	0 / 0	0 / 0	0 / 0	0 / 1

Appendix

Table 7: Random-direction control: addition / removal rates (%). Median over 5 seeds of norm-matched isotropic random directions, applied with the same

(l,p)

and

\alpha

as § 4.1 .

Training Activations	SAE Release	Activation	Width	Site	L0
Gemma-3-4B-IT	Gemma Scope 2 1 1 footnotemark: 1	JumpReLU	16k	Residual stream	“big” (90-150)
Qwen3.5-9B-Base	Qwen SAE-Res 2 2 footnotemark: 2	TopK	64k	Residual stream	100
Llama-3.1-8B-Base	Llama Scope 3 3 footnotemark: 3	JumpReLU	32k (8 $\times$ )	Residual stream	50

Appendix

Table 8: Sparse autoencoder releases used in this work. Width = dictionary size; L0 = mean number of active features per token at convergence.

Panel of Figure 8 — Figure 8: LLM-as-judge prompt for evaluating steering interventions on FLORES-minus-4 .

Figure 9: Example of the annotation interface. Human evaluators are shown one example at a time, including the source sentence, the reference translation and the un-intervened baseline, and the intervened output. The evaluator is asked to judge if the intervened output is a correct polarity flip, with meaning preserved otherwise.

Table 9: Human validation of automatic intervention evaluation. Agreement (Agree.), precision (Prec.), and recall (Rec.) are reported in percentages. FP (False Positives) and FN (False Negatives) denote automatic evaluation errors relative to human judgments.

Mini-minus-4
Lang.	Affirmative	Negative
EN	She is at the store.	She is not at the store.
VI	Tôi thích phở.	Tôi không thích phở.
FR	Nous aimons cette chanson.	Nous n’aimons pas cette chanson.
ZH	他喜欢跑步。	他不喜欢跑步。
FLORES-minus-4

Appendix

Table 10: Examples from Mini-minus-4 and FLORES-minus-4. Each row shows an affirmative/negative contrast in one of the four languages. Mini-minus-4 consists of controlled minimal pairs, whereas FLORES-minus-4 contains polarity contrasts constructed from parallel FLORES-200 sentences.

Panel of Figure 10 — Figure 10: LLM-as-judge prompt for evaluating whether translation preserves the polarity of the source main clause.

Figure 11: Baseline translation evaluation on FLORES-minus-4. Top : Negation preservation (%) via LLM-as-judge. Rows 2–4 : Translation quality measured by COMET-Kiwi (reference-free), COMET (reference-based), and chrF. All six models preserve negation at

{\geq}97\%

from English and

{\geq}93\%

into English, with no substantial quality gap between polarities.

Table 11: Number of baseline translations with correctly preserved polarity.

Negation Removal (ZH

\to

EN)

Ig SRC 这台投影仪不清楚。 (unclear)

Ig BASE This projector is unclear .

Ig INT This projector is clear .

Ig SRC 我们不愿意参加讨论。 (unwilling)

Ig BASE We are unwilling to participate in the discussion.

Ig INT We want to participate in the discussion.

Appendix

Table 12: Examples of activation steering flipping morphological and lexical negations on Qwen3-4B. SRC : Source sentence; BASE : Un-intervened baseline translation; INT : Model output after steering. Colors indicate affirmative states and negations .

Model	Ineligible	Morph./Ant.	Flipped
Gemma-3-1B	11	8	7
Gemma-3-4B	6	6	5
Gemma-3-12B	5	4	4
Qwen3-4B	12	11	5
Qwen3.5-9B	11	11	6
Llama-3.1-8B	4	4	3

Appendix

Table 13: Analysis of ineligible ZH negative-source examples on the Mini-minus-4 evaluation set. Morph./Ant. Ineligible examples that express negation through negative-prefix morphology or a lexical antonym. Flipped Morph./Ant. cases where intervention successfully flips the baseline output.

Figure 12: Accuracies for probes trained at the NP position.

Figure 13: Accuracies for probes trained at the FST position.

Figure 14: Accuracies for cross-language probe transfers.

Model	$l$	$p$	EN Flip Rate
Gemma-3-1B	L8	NP	92.0 / 83.0
Gemma-3-4B	L10	NP	99.0 / 88.0
Gemma-3-12B	L17	NP	100.0 / 97.0
Qwen3-4B	L13	NP	93.8 / 92.8
Qwen3.5-9B	L9	NP	98.0 / 97.0
Llama-3.1-8B	L10	NP	95.9 / 92.0

Appendix

Table 14: Selected

(l,p)

configurations and their EN addition/removal rates. A single

(l,p)

is selected per model for both intervention modes. Rates are reported as addition/removal.

Figure 15: Addition rates using the direction extracted at the NP position.

Figure 16: Removal rates using the direction extracted at the NP position.

Figure 17: Addition rates using the direction extracted at the FST position.

Figure 18: Removal rates using the direction extracted at the FST position.

Model	$l$	$\boldsymbol{\alpha=2}$	$\boldsymbol{\alpha=4}$	$\boldsymbol{\alpha=6}$
Gemma-3-4B	L4	9.2 / 81.0	96.9 / 86.0	67.3 / 80.0
Qwen3.5-9B	L10	95.0 / 95.0	97.0 / 95.0	95.0 / 92.0
Llama-3.1-8B	L13	62.2 / 77.0	89.8 / 84.0	61.2 / 74.0

Appendix

Table 15: EN CAA sweep at the layers of the selected SAE features. Each cell reports addition/removal flip rates (%) on the EN evaluation set. Bold indicates the

\alpha^{*}

selected for SAE norm matching.

Feature	EN	VI	FR	ZH
Gemma L4 f331	not (132/0); cannot (18/0); the (23/3)	không (150/0); tôi (29/29); Tôi (15/15)	pas (150/0); ne (118/0); n (32/0)	不 (78/0); 我不 (18/0); 喜欢 (11/0); 很 (3/41)
Qwen L10 f43677	not (132/0); . (107/0); the (37/0)	không (150/0); đọc (13/0); . (11/0)	pas (150/0); . (125/0); ne (118/0)	不 (37/0); 。 (18/0); 不爱 (11/0); 喜欢 (8/0)
Llama L13 f12611	. (83/0); the (19/0); read (5/0); easy (2/0)	thích (3/0); đủ (1/0); nghe (1/0)	pas (69/0); . (31/0); le (9/0)	喜欢 (18/0); 爱 (13/0); 。 (12/0); 常 (10/0)

Appendix

Table 16: Representative activating tokens for the selected

K=1

SAE features. Parenthesized counts denote firing occurrences in negative/affirmative examples. Gemma-3-4B f331 responds primarily at explicit negation markers; Qwen3.5-9B f43677 responds both at negation markers and downstream tokens; Llama-3.1-8B f12611 predominantly responds to tokens in negated contexts.

Figure 19: Flip rates by layer for

K

=10 SAE features. Only a few layers per model show the expected effects when steering with SAEs. We choose these layers to search for single effective SAE features.

Figure 20: Flip rates for individual SAE features at selected layers. Every model contains at least one feature exceeding 80% addition/removal rates.

Cross-Lingual Consistency in Language Models

Cross-lingual consistency in language models studies whether equivalent inputs across languages produce comparable knowledge, reasoning, and safety behavior.

See all 71 papers

How Language Models Process Negation
May 4, 2026Zhejian Zhou, Tianyi Zhou, Robin Jia +1LLM Interpretability Language Modeling
Information Sciences Institute, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California
How Do LLMs Change Predictions Under Negation?
Oct 7, 2026Jongwook Yoon, Jongwon Lim, Sungjib Lim +2LLM Reliability LLM Evaluation
Graduate School of Data Science, Seoul National University · Department of Computer Science and Engineering, Seoul National University
Disparities In Negation Understanding Across Languages In Vision-Language Models
Apr 21, 2026Charikleia Moraitaki, Sarah Pan, Skyler Pulling +3VLM Evaluation Vision-Language Models
Massachusetts Institute of Technology

May 4, 2026J/K move · Enter open · S save

How Language Models Process Negation

Zhejian Zhou, Tianyi Zhou, Robin Jia, Jonathan May

Information Sciences Institute, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California

LLM Interpretability Language Modeling Attention Mechanisms

Open paper AskPDF arXiv

We study how Large Language Models (LLMs) process negation mechanistically. First, we establish that even though open-weight models often provide wrong answers to questions involving negation, they do possess internal components that process negation correctly. Their poor accuracy is due to late-layer attention behavior that promotes simple shortcuts; ablating those attention modules greatly improves accuracy on negation-related questions. Second, we uncover how models process negation. We consider two hypotheses: models could use attention heads that attend to the phrase being negated and suppress related concepts, or they could directly construct a representation of the entire negative phrase (e.g., representing "not gas" as a vector that promotes liquids and solids). We apply a range of observational and causal interpretability techniques on Mistral-7B and Llama-3.1-8B to show that models implement both mechanisms, with the "constructive" mechanism being more prominent. Combined, our work deepens the understanding of LLMs' internals, highlighting construction-dominant computations and the coexistence of competing mechanisms within LLMs.

Abstract

Cross-Lingual Consistency in Language Models

Explore similar work

How Language Models Process Negation

How Do LLMs Change Predictions Under Negation?

Disparities In Negation Understanding Across Languages In Vision-Language Models