Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearning

Pengyang Shao; Chuanpeng Lu; Wei Qin; Yanzheng Jin; Xiaohao Liu; Xi Ai; Kenji Kawaguchi; Richang Hong

cs.CLSep 5, 2026

Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearning

Authors: Pengyang Shao, Chuanpeng Lu, Wei Qin, Yanzheng Jin, Xiaohao Liu, Xi Ai, Kenji Kawaguchi, Richang Hong

Organizations: National University of Singapore Singapore · Hefei University of Technology Anhui, China

Abstract

Large Language Model (LLM) unlearning aims to suppress target knowledge while preserving general capabilities. In multilingual settings, unlearning must additionally propagate within its intended linguistic scope. However, existing evaluations mainly measure cross-lingual transfer and cannot distinguish insufficient from excessive propagation. We introduce CLLPU (Cross-Lingual and Language-Bound Protocol for LLM Unlearning), a multilingual benchmark that formulates this problem through two settings: common-goal forgetting, where target knowledge should be suppressed across all languages, and language-conditioned forgetting, where suppression should remain confined to a designated language. CLLPU combines goal-guided topic pairing, schema-aware relation matching, and dual-anchor multilingual translation to construct 800 matched knowledge-unit pairs and 72,000 QA instances across ten languages. Experiments with six representative methods on Llama-3.1-8B-Instruct reveal opposite failure modes: forgetting remains incomplete when universal suppression is required, yet spreads beyond the intended boundary when language-conditioned confinement is required. We further find that general multilingual utility can conceal damage to neighbor knowledge. These findings establish propagation control as a central challenge for multilingual LLM unlearning. We publicly release CLLPU together with its construction pipeline.

Figures & tables

Figure 1. Overview of the CLLPU construction pipeline.

Panel of Table 1 — Table 1. Overall performance in the common-goal setting. Lower forget accessibility and higher retain accessibility indicate better performance. The best result is shown in bold and the second-best result is underlined .

Panel of Table 2 — Table 2. Overall performance in the language-conditioned setting. Lower forget accessibility and higher retain accessibility indicate better performance. The best result is shown in bold and the second-best result is underlined .

Figure 2. ROUGE-L in the common-goal setting. Diagonal cells report source accessibility $A_{z}^{\mathrm{RL}}(s,s)$ , while off-diagonal cells report cross accessibility $A_{z}^{\mathrm{RL}}(s,t)$ , where $t\neq s$ . Lower forget accessibility and higher retain accessibility indicate better performance.

Figure 3. ROUGE-L in the language-conditioned setting. Diagonal cells report source accessibility $A_{z}^{\mathrm{RL}}(s,s)$ , while off-diagonal cells report cross accessibility $A_{z}^{\mathrm{RL}}(s,t)$ , where $t\neq s$ . Successful unlearning requires low forget accessibility on the diagonal, high forget accessibility off the diagonal, and high retain accessibility across all languages.

Method	Common-goal					Language-conditioned
	Membership Inference (AUC $\rightarrow 0.5$ )				Belebele $\uparrow$	Membership Inference (AUC $\rightarrow 0.5$ )				Belebele $\uparrow$
	Loss	Min-K%	Min-K%++	Zlib	Accuracy	Loss	Min-K%	Min-K%++	Zlib	Accuracy
Original	0.9706	0.9683	0.9344	0.9673	0.8086	0.9727	0.9716	0.9393	0.9712	0.8086
Retrain	0.4631	0.4669	0.4797	0.4402	0.8060	–	–	–	–	–
GA	0.5033	0.4927	0.5042	0.4307	0.5597	0.1523	0.1449	0.2156	0.1767	0.7868
GD	0.3246	0.3074	0.3340	0.3286	0.7186	0.1630	0.1488	0.1626	0.1671	0.6510

Table 3. Overall performance on membership inference and multilingual utility. The best result is shown in bold and the second-best result is underlined .

Figure 4. Performance on core QAs versus surface variants. Each surface-variant bar reports the mean ROUGE-L score across three variants. Values adjacent to the bars denote the surface-variant score minus the core-QA score in percentage points.

Figure 5. ROUGE-L accessibility under forced response-language switching. Given a query in language $s$ , we instruct the model to respond in a designated language $u$ .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Data Product	Common-goal	Language-conditioned	Total
Stage 1	Curated pairs of target-neighbor topics	50	30	80
	Canonical Wikipedia pages	100	60	160
Stage 2	Extracted knowledge units	2,888	1,761	4,649
	Selected relation-matched knowledge-unit pairs	500	300	800
	Sampled holdout knowledge units	500	300	800
	Canonical English QAs (core + variants)	4,000	2,400	6,400

Appendix

Table 4. Statistics of the benchmark construction pipeline. Each matched knowledge-unit pair contains one forget unit and one retain unit. Each unit is instantiated as one core QA and three semantically equivalent surface variants, whereas each holdout unit produces only one core QA. Multilingual counts include English and its nine translated versions. In the common-goal setting, all language realizations of forget knowledge are targeted; in the language-conditioned setting, only the designated source-language realization is targeted.

Figure 6. EM accessibility in the common-goal setting. Diagonal cells report source accessibility $A_{z}^{\mathrm{EM}}(s,s)$ , while off-diagonal cells report cross accessibility $A_{z}^{\mathrm{EM}}(s,t)$ , where $t\neq s$ . Lower forget accessibility and higher retain accessibility indicate better performance.

Figure 7. EM accessibility in the language-conditioned setting. Diagonal cells report source accessibility $A_{z}^{\mathrm{EM}}(s,s)$ , while off-diagonal cells report cross accessibility $A_{z}^{\mathrm{EM}}(s,t)$ , where $t\neq s$ . Successful unlearning requires low forget accessibility on the diagonal, high forget accessibility off the diagonal, and high retain accessibility across all languages.

Figure 8. SS accessibility in the common-goal setting. Diagonal cells report source accessibility $A_{z}^{\mathrm{SS}}(s,s)$ , while off-diagonal cells report cross accessibility $A_{z}^{\mathrm{SS}}(s,t)$ , where $t\neq s$ . Lower forget accessibility and higher retain accessibility indicate better performance.

Figure 9. SS accessibility in the language-conditioned setting. Diagonal cells report source accessibility $A_{z}^{\mathrm{SS}}(s,s)$ , while off-diagonal cells report cross accessibility $A_{z}^{\mathrm{SS}}(s,t)$ , where $t\neq s$ . Successful unlearning requires low forget accessibility on the diagonal, high forget accessibility off the diagonal, and high retain accessibility across all languages.

Figure 10. Cosine similarity between the expected answers of paired forget- and retain-set QAs.

Method	Common-goal				Language-conditioned
	Forget Accessibility		Retain Accessibility		Forget Accessibility		Retain Accessibility
	EN $\downarrow$	Avg. non-EN $\downarrow$	EN $\uparrow$	Avg. non-EN $\uparrow$	EN $\downarrow$	Avg. non-EN $\uparrow$	EN $\uparrow$	Avg. non-EN $\uparrow$
Original	0.8238	0.7676	0.8315	0.7747	0.9188	0.8461	0.8896	0.7900
Retrain	0.1668	0.1301	0.8261	0.7738	–	–	–	–
GA	0.0000	0.0002	0.0015	0.0005	0.0813	0.0875	0.1521	0.1516
GD	0.0114	0.0165	0.4920	0.2611	0.0208	0.0002	0.3542	0.2201

Appendix

Table 5. LLM-as-a-Judge accessibility under English-source unlearning. EN reports evaluation in the source language, while Avg. non-EN averages the other nine evaluation languages. These results describe a single English-source run rather than a macro-average over source languages. Arrows indicate the desired direction.

	Forget Accessibility		Retain Accessibility
Method	Source $\downarrow$	Cross $\uparrow$	Source $\uparrow$	Cross $\uparrow$
Original	$0.818{\pm}0.134$	$0.774{\pm}0.079$	$0.829{\pm}0.091$	$0.772{\pm}0.097$
GA	$0.037{\pm}0.050$	$0.099{\pm}0.074$	$0.060{\pm}0.047$	$0.141{\pm}0.099$
GD	$\mathbf{0.036}{\pm}0.048$	$0.093{\pm}0.131$	$0.389{\pm}0.235$	$0.197{\pm}0.173$
NPO	$0.056{\pm}0.099$	$0.045{\pm}0.039$	$0.829{\pm}0.112$	$\mathbf{0.286}{\pm}0.096$
SimNPO	$0.098{\pm}0.126$	$\mathbf{0.108}{\pm}0.062$	$\mathbf{0.882}{\pm}0.105$	$0.278{\pm}0.079$

Appendix

Table 6. Topic-level ROUGE-L accessibility in the language-conditioned setting. Values are the mean

\pm

standard deviation over nine topic pairs. Standard deviations characterize topic-level heterogeneity.

LLM Unlearning

Large language model unlearning studies how to remove specified knowledge or behaviors while preserving a model’s unrelated capabilities.

See all 191 papers

Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility
Jun 2, 2026Chaoyi Xiang, Olga Ohrimenko, Benjamin I. P. Rubinstein +1Machine Unlearning LLM Unlearning
School of Computing and Information Systems, The University of Melbourne, Melbourne, Australia.
Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearning
Sep 30, 2026Tyler Skow, Shravan Chaudhari, Rama Chellappa +1Machine Unlearning LLM Unlearning
Johns Hopkins University
$μ^2$ -Bench: A Multilingual Machine Unlearning Benchmark
Sep 17, 2026Kyomin Hwang, Hyeonjin Kim, Hyunho Lee +3Machine Unlearning LLM Unlearning
GSCST, Seoul National University · AIIS, Seoul National University

Jun 2, 2026J/K move · Enter open · S save

Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility

Chaoyi Xiang, Olga Ohrimenko, Benjamin I. P. Rubinstein, Lea Frermann

School of Computing and Information Systems, The University of Melbourne, Melbourne, Australia.

Machine Unlearning LLM Unlearning Cross-Lingual Transfer

Open paper AskPDF arXiv

Large language models (LLMs) can memorize sensitive facts, motivating unlearning methods that remove targeted knowledge without costly retraining. However, unlearning research remains heavily English-centric. We study multilingual unlearning by extending the TOFU benchmark to five languages, and fine-tune, unlearn, and query our models with different permutations of languages. We find that unlearning transfer, the ability of an unlearned model to "forget" facts in languages other than the unlearning language, is highly variable: e.g., it is strongest between languages sharing scripts and families, and we show that the unlearning language predicts which query languages are most likely to yield the strongest transfer. Layer-wise analysis reveals that unlearning leaves the shared cross-lingual latent space largely intact in early layers, instead operating primarily in later decoding layers. This suggests that unlearning does not truly erase knowledge, but rather induces superficial suppression. Exploiting this structure, a single inference-time steering direction reverses much of this suppression across languages, recovering 50% (Qwen) and 90% (Gemma) of the unlearned knowledge.

Abstract

LLM Unlearning

Explore similar work

Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility

Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearning

μ2μ^2μ2-Bench: A Multilingual Machine Unlearning Benchmark

$μ^2$ -Bench: A Multilingual Machine Unlearning Benchmark