Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearning

Pengyang Shao; Chuanpeng Lu; Wei Qin; Yanzheng Jin; Xiaohao Liu; Xi Ai; Kenji Kawaguchi; Richang Hong

cs.CLSep 5, 2026

Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearning

Authors: Pengyang Shao, Chuanpeng Lu, Wei Qin, Yanzheng Jin, Xiaohao Liu, Xi Ai, Kenji Kawaguchi, Richang Hong

Organizations: National University of Singapore Singapore · Hefei University of Technology Anhui, China

Abstract

Large Language Model (LLM) unlearning aims to suppress target knowledge while preserving general capabilities. In multilingual settings, unlearning must additionally propagate within its intended linguistic scope. However, existing evaluations mainly measure cross-lingual transfer and cannot distinguish insufficient from excessive propagation. We introduce CLLPU (Cross-Lingual and Language-Bound Protocol for LLM Unlearning), a multilingual benchmark that formulates this problem through two settings: common-goal forgetting, where target knowledge should be suppressed across all languages, and language-conditioned forgetting, where suppression should remain confined to a designated language. CLLPU combines goal-guided topic pairing, schema-aware relation matching, and dual-anchor multilingual translation to construct 800 matched knowledge-unit pairs and 72,000 QA instances across ten languages. Experiments with six representative methods on Llama-3.1-8B-Instruct reveal opposite failure modes: forgetting remains incomplete when universal suppression is required, yet spreads beyond the intended boundary when language-conditioned confinement is required. We further find that general multilingual utility can conceal damage to neighbor knowledge. These findings establish propagation control as a central challenge for multilingual LLM unlearning. We publicly release CLLPU together with its construction pipeline.

Figures & tables

Figure 1. Overview of the CLLPU construction pipeline.

Panel of Table 1 — Table 1. Overall performance in the common-goal setting. Lower forget accessibility and higher retain accessibility indicate better performance. The best result is shown in bold and the second-best result is underlined .

Panel of Table 2 — Table 2. Overall performance in the language-conditioned setting. Lower forget accessibility and higher retain accessibility indicate better performance. The best result is shown in bold and the second-best result is underlined .

Figure 2. ROUGE-L in the common-goal setting. Diagonal cells report source accessibility $A_{z}^{\mathrm{RL}}(s,s)$ , while off-diagonal cells report cross accessibility $A_{z}^{\mathrm{RL}}(s,t)$ , where $t\neq s$ . Lower forget accessibility and higher retain accessibility indicate better performance.

Figure 3. ROUGE-L in the language-conditioned setting. Diagonal cells report source accessibility $A_{z}^{\mathrm{RL}}(s,s)$ , while off-diagonal cells report cross accessibility $A_{z}^{\mathrm{RL}}(s,t)$ , where $t\neq s$ . Successful unlearning requires low forget accessibility on the diagonal, high forget accessibility off the diagonal, and high retain accessibility across all languages.

Method	Common-goal					Language-conditioned
	Membership Inference (AUC $\rightarrow 0.5$ )				Belebele $\uparrow$	Membership Inference (AUC $\rightarrow 0.5$ )				Belebele $\uparrow$
	Loss	Min-K%	Min-K%++	Zlib	Accuracy	Loss	Min-K%	Min-K%++	Zlib	Accuracy
Original	0.9706	0.9683	0.9344	0.9673	0.8086	0.9727	0.9716	0.9393	0.9712	0.8086
Retrain	0.4631	0.4669	0.4797	0.4402	0.8060	–	–	–	–	–
GA	0.5033	0.4927	0.5042	0.4307	0.5597	0.1523	0.1449	0.2156	0.1767	0.7868
GD	0.3246	0.3074	0.3340	0.3286	0.7186	0.1630	0.1488	0.1626	0.1671	0.6510

Table 3. Overall performance on membership inference and multilingual utility. The best result is shown in bold and the second-best result is underlined .

Figure 4. Performance on core QAs versus surface variants. Each surface-variant bar reports the mean ROUGE-L score across three variants. Values adjacent to the bars denote the surface-variant score minus the core-QA score in percentage points.

Figure 5. ROUGE-L accessibility under forced response-language switching. Given a query in language $s$ , we instruct the model to respond in a designated language $u$ .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Data Product	Common-goal	Language-conditioned	Total
Stage 1	Curated pairs of target-neighbor topics	50	30	80
	Canonical Wikipedia pages	100	60	160
Stage 2	Extracted knowledge units	2,888	1,761	4,649
	Selected relation-matched knowledge-unit pairs	500	300	800
	Sampled holdout knowledge units	500	300	800
	Canonical English QAs (core + variants)	4,000	2,400	6,400

Appendix

Table 4. Statistics of the benchmark construction pipeline. Each matched knowledge-unit pair contains one forget unit and one retain unit. Each unit is instantiated as one core QA and three semantically equivalent surface variants, whereas each holdout unit produces only one core QA. Multilingual counts include English and its nine translated versions. In the common-goal setting, all language realizations of forget knowledge are targeted; in the language-conditioned setting, only the designated source-language realization is targeted.

Figure 6. EM accessibility in the common-goal setting. Diagonal cells report source accessibility $A_{z}^{\mathrm{EM}}(s,s)$ , while off-diagonal cells report cross accessibility $A_{z}^{\mathrm{EM}}(s,t)$ , where $t\neq s$ . Lower forget accessibility and higher retain accessibility indicate better performance.

Figure 7. EM accessibility in the language-conditioned setting. Diagonal cells report source accessibility $A_{z}^{\mathrm{EM}}(s,s)$ , while off-diagonal cells report cross accessibility $A_{z}^{\mathrm{EM}}(s,t)$ , where $t\neq s$ . Successful unlearning requires low forget accessibility on the diagonal, high forget accessibility off the diagonal, and high retain accessibility across all languages.

Figure 8. SS accessibility in the common-goal setting. Diagonal cells report source accessibility $A_{z}^{\mathrm{SS}}(s,s)$ , while off-diagonal cells report cross accessibility $A_{z}^{\mathrm{SS}}(s,t)$ , where $t\neq s$ . Lower forget accessibility and higher retain accessibility indicate better performance.

Figure 9. SS accessibility in the language-conditioned setting. Diagonal cells report source accessibility $A_{z}^{\mathrm{SS}}(s,s)$ , while off-diagonal cells report cross accessibility $A_{z}^{\mathrm{SS}}(s,t)$ , where $t\neq s$ . Successful unlearning requires low forget accessibility on the diagonal, high forget accessibility off the diagonal, and high retain accessibility across all languages.

Figure 10. Cosine similarity between the expected answers of paired forget- and retain-set QAs.

Method	Common-goal				Language-conditioned
	Forget Accessibility		Retain Accessibility		Forget Accessibility		Retain Accessibility
	EN $\downarrow$	Avg. non-EN $\downarrow$	EN $\uparrow$	Avg. non-EN $\uparrow$	EN $\downarrow$	Avg. non-EN $\uparrow$	EN $\uparrow$	Avg. non-EN $\uparrow$
Original	0.8238	0.7676	0.8315	0.7747	0.9188	0.8461	0.8896	0.7900
Retrain	0.1668	0.1301	0.8261	0.7738	–	–	–	–
GA	0.0000	0.0002	0.0015	0.0005	0.0813	0.0875	0.1521	0.1516
GD	0.0114	0.0165	0.4920	0.2611	0.0208	0.0002	0.3542	0.2201

Appendix

Table 5. LLM-as-a-Judge accessibility under English-source unlearning. EN reports evaluation in the source language, while Avg. non-EN averages the other nine evaluation languages. These results describe a single English-source run rather than a macro-average over source languages. Arrows indicate the desired direction.

	Forget Accessibility		Retain Accessibility
Method	Source $\downarrow$	Cross $\uparrow$	Source $\uparrow$	Cross $\uparrow$
Original	$0.818{\pm}0.134$	$0.774{\pm}0.079$	$0.829{\pm}0.091$	$0.772{\pm}0.097$
GA	$0.037{\pm}0.050$	$0.099{\pm}0.074$	$0.060{\pm}0.047$	$0.141{\pm}0.099$
GD	$\mathbf{0.036}{\pm}0.048$	$0.093{\pm}0.131$	$0.389{\pm}0.235$	$0.197{\pm}0.173$
NPO	$0.056{\pm}0.099$	$0.045{\pm}0.039$	$0.829{\pm}0.112$	$\mathbf{0.286}{\pm}0.096$
SimNPO	$0.098{\pm}0.126$	$\mathbf{0.108}{\pm}0.062$	$\mathbf{0.882}{\pm}0.105$	$0.278{\pm}0.079$

Appendix

Table 6. Topic-level ROUGE-L accessibility in the language-conditioned setting. Values are the mean

\pm

standard deviation over nine topic pairs. Standard deviations characterize topic-level heterogeneity.

LLM Unlearning

Large language model unlearning studies how to remove specified knowledge or behaviors while preserving a model’s unrelated capabilities.

See all 191 papers

Jun 2, 2026

Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility

Large language models (LLMs) can memorize sensitive facts, motivating unlearning methods that remove targeted knowledge without costly retraining. However, unlearning research remains heavily English-centric. We study multilingual unlearning by extending the TOFU benchmark to five languages, and fine-tune, unlearn, and query our models with different permutations of languages. We find that unlearning transfer, the ability of an unlearned model to "forget" facts in languages other than the unlearning language, is highly variable: e.g., it is strongest between languages sharing scripts and families, and we show that the unlearning language predicts which query languages are most likely to yield the strongest transfer. Layer-wise analysis reveals that unlearning leaves the shared cross-lingual latent space largely intact in early layers, instead operating primarily in later decoding layers. This suggests that unlearning does not truly erase knowledge, but rather induces superficial suppression. Exploiting this structure, a single inference-time steering direction reverses much of this suppression across languages, recovering 50% (Qwen) and 90% (Gemma) of the unlearned knowledge.

Chaoyi Xiang, Olga Ohrimenko, Benjamin I. P. Rubinstein +1

School of Computing and Information Systems, The University of Melbourne, Melbourne, Australia.

Machine Unlearning LLM Unlearning Cross-Lingual Transfer

Ask PDF

Sep 30, 2026

Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearning

Unlearning a fact in one language does not guarantee its removal in others as changing the query or even the requested answer language can reopen seemingly forgotten knowledge -- a cross-lingual loophole. The most straightforward solution to this challenge -- unlearning in all languages -- is neither scalable nor desirable as it amplifies damage to unrelated model capabilities. We introduce the task of language budgeted multilingual unlearning where the goal is to select a subset of languages that maximizes cross-lingual erasure. To study this task we introduce the Cross-Lingual Unlearning Tensor, an unlearning benchmark that spans 174 language--script pairs and 25 atomic paraphrase types to examine when forgetting generalizes across linguistic expressions of the same knowledge. We further propose COVER, which selects source languages to maximize predicted COVERage of languages receiving no forget supervision, enabling unlearning on a language budget. Surprisingly, we find naively selecting strong individual sources does not reliably compose into strong source sets motivating our development of COVER. At deployment COVER only requires benign calibration data and access to the frozen model. Across three model families and two disjoint forget sets, COVER reduces mean held-out residual access by 7.8--27.3% relative to uniform source selection. We find these gains extend beyond synthetic benchmarks to real news documents in low-resource language settings using human translated data from the Low Resource Languages for Emergent Incidents (LORELEI) corpus.

Tyler Skow, Shravan Chaudhari, Rama Chellappa +1

Johns Hopkins University

Machine Unlearning LLM Unlearning Machine Unlearning Evaluation

Ask PDF

Sep 17, 2026

$μ^2$ -Bench: A Multilingual Machine Unlearning Benchmark

Undesired information such as harmful content and private data propagates through Multilingual Large Language Models (LLMs) via direct training and indirect cross-linguistic spread. Multilingual Machine Unlearning (MMU) aims to remove such information, yet its evaluation remains underexplored, leaving unclear whether unlearning truly eliminates target knowledge across all languages. To bridge this gap, we introduce

μ^2

-Bench, an MMU benchmark that simulates the full pipeline of memorization, unlearning, and evaluation across diverse languages. It 1) spans a broad set of languages, 2) evaluates on both training and hold-out languages, and 3) assesses knowledge as dispersed across multiple languages. We show that successful MMU requires methods that reflect multilingual characteristics, and conduct analysis to provide deeper insights into MMU.

Kyomin Hwang, Hyeonjin Kim, Hyunho Lee +3

GSCST, Seoul National University · AIIS, Seoul National University

Machine Unlearning LLM Unlearning Machine Unlearning Evaluation Multilingual Language Model Evaluation

Ask PDF

Abstract

LLM Unlearning

Explore similar work

Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility

Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearning

μ2μ^2μ2-Bench: A Multilingual Machine Unlearning Benchmark

$μ^2$ -Bench: A Multilingual Machine Unlearning Benchmark