$μ^2$-Bench: A Multilingual Machine Unlearning Benchmark

Kyomin Hwang; Hyeonjin Kim; Hyunho Lee; Yearim Kim; Yeji Song; Nojun Kwak

cs.CLSep 17, 2026

$μ^2$ -Bench: A Multilingual Machine Unlearning Benchmark

Authors: Kyomin Hwang, Hyeonjin Kim, Hyunho Lee, Yearim Kim, Yeji Song, Nojun Kwak

Organizations: GSCST, Seoul National University · AIIS, Seoul National University

Abstract

Undesired information such as harmful content and private data propagates through Multilingual Large Language Models (LLMs) via direct training and indirect cross-linguistic spread. Multilingual Machine Unlearning (MMU) aims to remove such information, yet its evaluation remains underexplored, leaving unclear whether unlearning truly eliminates target knowledge across all languages. To bridge this gap, we introduce $μ^2$ -Bench, an MMU benchmark that simulates the full pipeline of memorization, unlearning, and evaluation across diverse languages. It 1) spans a broad set of languages, 2) evaluates on both training and hold-out languages, and 3) assesses knowledge as dispersed across multiple languages. We show that successful MMU requires methods that reflect multilingual characteristics, and conduct analysis to provide deeper insights into MMU.

Figures & tables

Figure 1: Illustration of knowledge after unlearning. Columns correspond to languages, and rows correspond to knowledge instances expressed across multiple languages. Gray cells indicate knowledge forgotten in the corresponding language, whereas colored cells indicate the retained ones, either acquired directly via training ( blue ) or indirectly via cross-linguistic spread ( red ).

Figure 2: Overview of the $\bm{\mu^{2}}$ -Bench: (Left) a multilingual dataset is constructed, (Middle) memorization and unlearning are performed using only the training languages of each setting, (Right) and the unlearned model is evaluated on both the training and hold-out languages. Latin and non-Latin languages are denoted in red and blue .

Figure 3: The construction of the translation prompt. To guarantee the translation quality, the prompt includes the definition of each English attribute along with its translation into the target language.

Setting	Script	Training	Hold-out
Setting 1	Latin	EN, DE , ID, AF	ES , LV
Setting 1	Non-Latin	—	ZH , UR
Setting 2	Latin	—	EN , LV
Setting 2	Non-Latin	ZH, EL , MK, TH	KO , UR

Table 1: Language configurations used in

\bm{\mu^{2}}

-Bench across two evaluation settings, organized by writing script (Latin vs. non-Latin) and resource level (High vs. Low). Bold indicates high-resource languages.

				Train							Holdout							Gen. Know.
Model	Set	$p$	Method	PS	SE	KSS ${}_{\text{prob}}^{\text{ROC}}$	KSS ${}_{\text{prob}}^{\text{PR}}$	KSS ${}_{\text{gen}}^{\text{ROC}}$	KSS ${}_{\text{gen}}^{\text{PR}}$	PrivLeak	PS	SE	KSS ${}_{\text{prob}}^{\text{ROC}}$	KSS ${}_{\text{prob}}^{\text{PR}}$	KSS ${}_{\text{gen}}^{\text{ROC}}$	KSS ${}_{\text{gen}}^{\text{PR}}$	PrivLeak	Gen. Know.
Gemma3-12B	S1	p1	Mem	99.99	100.00	-53.18	-99.06	-50.00	-99.00	-107.90	55.82	76.88	-58.37	-99.17	-50.17	-98.99	-86.85	30.19
			GA	72.44	91.88	-14.24	-77.47	-41.73	-93.94	-90.57	37.45	68.75	-42.05	-97.21	-44.63	-98.68	-60.75	32.85
			GAGDR	75.34	94.38	-20.39	-76.45	-42.89	-94.68	-97.23	45.03	70.00	-49.86	-98.61	-44.79	-98.75	-75.34	32.83
			NPO	47.89	83.12	-4.65	-43.28	-33.69	-86.37	-68.61	26.09	67.50	-26.60	-92.70	-45.54	-98.59	-41.32	32.49
			LingTea	45.32	91.88	-20.23	-41.52	-38.96	-89.00	-1.07	25.92	73.12	-28.49	-47.51	-46.95	-98.88	+9.18	30.84

Table 2: MMU performance on Gemma3-12B and Qwen2.5-7B. LingTea generally performs well on both training and hold-out languages.

		$\Delta$ PS		$\Delta$ SE
Cell	Method	High	Low	High	Low
G/S1	LingTea	+53.8	+55.5	+7.5	+8.8
	NPO	+40.0	+64.2	+16.2	+17.5
	GAGDR	+37.8	+11.5	+6.2	+5.0
	GA	+36.4	+18.7	+7.5	+8.8
G/S2	LingTea	+58.6	+32.9	+27.5	+15.0

Table 3: Training language drop rates per unlearning method. For each metric, bold marks the higher value within the High/Low pair. G and Q denote Gemma3 and Qwen2.5, respectively.

		$\Delta$ PS				$\Delta$ SE
		resource		script		resource		script
Cell	Method	High	Low	Lat	nLat	High	Low	Lat	nLat
G/S1	LingTea	+54.8	+55.8	+56.6	+54.0	+4.2	+5.9	+8.1	+1.6
	NPO	+52.1	+48.7	+65.7	+35.1	+13.9	+9.8	+11.3	+13.1
	GAGDR	+21.8	+29.6	+29.1	+22.2	+2.8	+17.6	+16.1	+1.6
	GA	+27.1	+41.9	+36.9	+32.1	+5.6	+17.6	+4.8	+16.4

Table 4: Hold-out language drop rates per unlearning method. Bold marks the higher value within each comparison pair (High/Low and Lat/nLat). G and Q denote Gemma3 and Qwen2.5, respectively.

Figure 4: Example of the model’s response in a hold-out language after MMU. The numbers in the figure denote the probability that the LLM assigns to the predicted next token, given the preceding context up to “of”. The memorized model outputs the English word wolves , whereas after unlearning the Spanish lobos ( wolves ) receives the highest probability.

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

Table 5: Valid values for the attributes used to build synthetic profiles (1/4)

Table 6: (continued) Valid values for the attributes used to build synthetic profiles (2/4)

Table 7: (continued) Valid values for the attributes used to build synthetic profiles (3/4)

Table 8: (continued) Valid values for the attributes used to build synthetic profiles (4/4)

Figure 5: Example of Generated Synthetic Profile

Figure 6: Examples of the generated English QA dataset. Left shows the naive QA dataset generated directly from templates, while Right shows the QA dataset refined using GPT.

Figure 7: Examples of code-mixed responses after NPO unlearning on Gemma3-12b model. Left shows the question, while Right shows the gold answer alongside the model’s code-mixed response.

Table 9: QA dataset template for each attribute (1/2)

Table 10: (continued) QA dataset template for each attribute (2/2)

Figure 8: Prompt used to refine the English QA dataset using GPT

Figure 9: Examples of the translated Multilingual QA dataset.

Figure 10: Prompt used to translate the English QA dataset to each language using GPT

Table 11: Description of each value in English (1/20)

Table 12: (continued) Description of each value in English (2/20)

Table 13: (continued) Description of each value in English (3/20)

Table 14: (continued) Description of each value in English (4/20)

Table 15: (continued) Description of each value in English (5/20)

Table 16: (continued) Description of each value in English (6/20)

Table 17: (continued) Description of each value in English (7/20)

Table 18: (continued) Description of each value in English (8/20)

Table 19: (continued) Description of each value in English (9/20)

Table 20: (continued) Description of each value in English (10/20)

Table 21: (continued) Description of each value in English (11/20)

Table 22: (continued) Description of each value in English (12/20)

Table 23: (continued) Description of each value in English (13/20)

Table 24: (continued) Description of each value in English (14/20)

Table 25: (continued) Description of each value in English (15/20)

Table 26: (continued) Description of each value in English (16/20)

Table 27: (continued) Description of each value in English (17/20)

Table 28: (continued) Description of each value in English (18/20)

Table 29: (continued) Description of each value in English (19/20)

Table 30: (continued) Description of each value in English (20/20)

Gemma3-12B

Qwen2.5-7B

Setting 1

Setting 2

Setting 1

Setting 2

Method

4.32e-5

7.60e-6

5.00e-6

4.19e-5

6.70e-6

2.90e-6

4.10e-5

9.10e-6

5.50e-6

4.41e-5

1.41e-5

8.10e-6

GAGDR

9.05e-5

3.04e-5

2.47e-5

1.005e-4

3.50e-5

2.04e-5

9.32e-5

3.00e-5

1.70e-5

1.110e-4

3.69e-5

2.29e-5

LingTea

1.302e-4

6.02e-5

3.72e-5

1.005e-4

4.12e-5

3.26e-5

1.699e-4

7.11e-5

5.00e-5

1.530e-4

1.425e-4

5.79e-5

NPO

4.00e-5

7.50e-6

4.60e-6

4.05e-5

7.20e-6

3.30e-6

4.90e-5

9.90e-6

5.80e-6

5.81e-5

1.70e-5

9.50e-6

Appendix

Table 31: Learning rates selected by an LR sweep with increments of

1\times 10^{-7}

across models, settings, forget ratios, and unlearning methods.

Figure 11: Prompt used to judge semantic equivalence of model responses

		$\Delta$ PS		$\Delta$ SE
Cell	Method	High	Low	High	Low
G/S1	LingTea	+53.9	+46.4	+5.4	+10.4
	NPO	+23.4	+0.9	+0.0	+0.4
	GAGDR	+30.0	+10.3	+0.0	+0.0
	GA	+12.7	+0.5	+0.0	+0.0
G/S2	LingTea	+23.8	+29.9	+3.7	+2.9

Appendix

Table 32: Training-language drop rates per unlearning method for

p3

. For each metric, bold marks the higher value within the High/Low pair. G and Q denote Gemma3 and Qwen2.5, respectively.

		$\Delta$ PS		$\Delta$ SE
Cell	Method	High	Low	High	Low
G/S1	LingTea	+29.3	+26.4	+5.0	+6.8
	NPO	+20.7	+1.3	+0.0	+0.0
	GAGDR	+20.8	+30.5	+2.7	+2.2
	GA	+24.1	+1.0	+0.0	+0.0
G/S2	LingTea	+26.2	+36.5	+1.2	+3.7

Appendix

Table 33: Training-language drop rates per unlearning method for

p5

. For each metric, bold marks the higher value within the High/Low pair. G and Q denote Gemma3 and Qwen2.5, respectively.

		$\Delta$ PS				$\Delta$ SE
		resource		script		resource		script
Cell	Method	High	Low	Lat	nLat	High	Low	Lat	nLat
G/S1	LingTea	+51.0	+51.8	+49.5	+53.4	+10.3	+12.4	+11.1	+11.3
	NPO	+3.0	+2.8	+3.7	+2.1	+0.0	+5.0	+1.6	+2.6
	GAGDR	+7.1	+14.2	+10.2	+11.2	+0.0	+5.0	+3.7	+0.5
	GA	+2.7	+2.2	+2.9	+2.0	-0.4	+3.7	+0.0	+2.6

Appendix

Table 34: Hold-out-language drop rates per unlearning method for

p3

. Bold marks the higher value within each comparison pair (High/Low and Lat/nLat). G and Q denote Gemma3 and Qwen2.5, respectively.

		$\Delta$ PS				$\Delta$ SE
		resource		script		resource		script
Cell	Method	High	Low	Lat	nLat	High	Low	Lat	nLat
G/S1	LingTea	+30.1	+24.7	+28.6	+26.2	+10.4	+10.1	+9.6	+10.8
	NPO	+2.5	+1.3	+2.3	+1.5	+1.1	+3.7	+1.9	+2.5
	GAGDR	+35.2	+28.5	+32.1	+31.6	+4.4	+6.7	+4.2	+6.5
	GA	+6.7	+3.8	+4.9	+5.6	+0.5	+4.1	+1.9	+2.2

Appendix

Table 35: Hold-out-language drop rates per unlearning method for

p5

. Bold marks the higher value within each comparison pair (High/Low and Lat/nLat). G and Q denote Gemma3 and Qwen2.5, respectively.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

May 14, 2026

Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation

While LLMs are increasingly used in commercial services, they pose privacy risks such as leakage of sensitive personally identifiable information (PII). For LLMs trained on multilingual corpora, Multilingual Machine Unlearning (MMU) aims to remove information across multiple languages. However, prior MMU evaluations fail to capture such cross-linguistic distribution of information, being largely limited to direct extensions of per-language evaluation protocols. To this end, we propose two metrics to evaluate the information spread across languages: the Knowledge Separability Score (KSS) and the Knowledge Persistence Score (KPS). KSS measures the overall unlearning quality across multiple languages, while KPS more specifically aims to assess consistent removal of information among different language pairs. We evaluated various unlearning methods in the multilingual setting with these metrics and conducted comprehensive analyses. Through our investigation, we provide insights into unique phenomena exclusive to MMU and offer a new perspective on MMU evaluation.

Kyomin Hwang, Hyeonjin Kim, Sangyeon Cho +1

GSCST, Seoul National University · Department of Artificial Intelligence, Chung-Ang University · Korean Surgical Researcher Foundation, Republic of Korea +1

Multilingual Language Model Evaluation Privacy Leakage in Language Models Machine Unlearning LLM Unlearning Machine Unlearning Evaluation

Ask PDF

Sep 5, 2026

Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearning

Large Language Model (LLM) unlearning aims to suppress target knowledge while preserving general capabilities. In multilingual settings, unlearning must additionally propagate within its intended linguistic scope. However, existing evaluations mainly measure cross-lingual transfer and cannot distinguish insufficient from excessive propagation. We introduce CLLPU (Cross-Lingual and Language-Bound Protocol for LLM Unlearning), a multilingual benchmark that formulates this problem through two settings: common-goal forgetting, where target knowledge should be suppressed across all languages, and language-conditioned forgetting, where suppression should remain confined to a designated language. CLLPU combines goal-guided topic pairing, schema-aware relation matching, and dual-anchor multilingual translation to construct 800 matched knowledge-unit pairs and 72,000 QA instances across ten languages. Experiments with six representative methods on Llama-3.1-8B-Instruct reveal opposite failure modes: forgetting remains incomplete when universal suppression is required, yet spreads beyond the intended boundary when language-conditioned confinement is required. We further find that general multilingual utility can conceal damage to neighbor knowledge. These findings establish propagation control as a central challenge for multilingual LLM unlearning. We publicly release CLLPU together with its construction pipeline.

Pengyang Shao, Chuanpeng Lu, Wei Qin +5

National University of Singapore Singapore · Hefei University of Technology Anhui, China

LLM Unlearning Machine Unlearning Evaluation Multilingual Language Model Evaluation

Ask PDF

Jun 2, 2026

Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility

Large language models (LLMs) can memorize sensitive facts, motivating unlearning methods that remove targeted knowledge without costly retraining. However, unlearning research remains heavily English-centric. We study multilingual unlearning by extending the TOFU benchmark to five languages, and fine-tune, unlearn, and query our models with different permutations of languages. We find that unlearning transfer, the ability of an unlearned model to "forget" facts in languages other than the unlearning language, is highly variable: e.g., it is strongest between languages sharing scripts and families, and we show that the unlearning language predicts which query languages are most likely to yield the strongest transfer. Layer-wise analysis reveals that unlearning leaves the shared cross-lingual latent space largely intact in early layers, instead operating primarily in later decoding layers. This suggests that unlearning does not truly erase knowledge, but rather induces superficial suppression. Exploiting this structure, a single inference-time steering direction reverses much of this suppression across languages, recovering 50% (Qwen) and 90% (Gemma) of the unlearned knowledge.

Chaoyi Xiang, Olga Ohrimenko, Benjamin I. P. Rubinstein +1

School of Computing and Information Systems, The University of Melbourne, Melbourne, Australia.

Machine Unlearning LLM Unlearning Cross-Lingual Transfer

Ask PDF

Abstract

Machine Unlearning

Explore similar work

Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation

Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearning

Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility