$μ^2$-Bench: A Multilingual Machine Unlearning Benchmark

Kyomin Hwang; Hyeonjin Kim; Hyunho Lee; Yearim Kim; Yeji Song; Nojun Kwak

cs.CLSep 17, 2026

$μ^2$ -Bench: A Multilingual Machine Unlearning Benchmark

Authors: Kyomin Hwang, Hyeonjin Kim, Hyunho Lee, Yearim Kim, Yeji Song, Nojun Kwak

Organizations: GSCST, Seoul National University · AIIS, Seoul National University

Abstract

Undesired information such as harmful content and private data propagates through Multilingual Large Language Models (LLMs) via direct training and indirect cross-linguistic spread. Multilingual Machine Unlearning (MMU) aims to remove such information, yet its evaluation remains underexplored, leaving unclear whether unlearning truly eliminates target knowledge across all languages. To bridge this gap, we introduce $μ^2$ -Bench, an MMU benchmark that simulates the full pipeline of memorization, unlearning, and evaluation across diverse languages. It 1) spans a broad set of languages, 2) evaluates on both training and hold-out languages, and 3) assesses knowledge as dispersed across multiple languages. We show that successful MMU requires methods that reflect multilingual characteristics, and conduct analysis to provide deeper insights into MMU.

Figures & tables

Figure 1: Illustration of knowledge after unlearning. Columns correspond to languages, and rows correspond to knowledge instances expressed across multiple languages. Gray cells indicate knowledge forgotten in the corresponding language, whereas colored cells indicate the retained ones, either acquired directly via training ( blue ) or indirectly via cross-linguistic spread ( red ).

Figure 2: Overview of the $\bm{\mu^{2}}$ -Bench: (Left) a multilingual dataset is constructed, (Middle) memorization and unlearning are performed using only the training languages of each setting, (Right) and the unlearned model is evaluated on both the training and hold-out languages. Latin and non-Latin languages are denoted in red and blue .

Figure 3: The construction of the translation prompt. To guarantee the translation quality, the prompt includes the definition of each English attribute along with its translation into the target language.

Setting	Script	Training	Hold-out
Setting 1	Latin	EN, DE , ID, AF	ES , LV
Setting 1	Non-Latin	—	ZH , UR
Setting 2	Latin	—	EN , LV
Setting 2	Non-Latin	ZH, EL , MK, TH	KO , UR

Table 1: Language configurations used in

\bm{\mu^{2}}

-Bench across two evaluation settings, organized by writing script (Latin vs. non-Latin) and resource level (High vs. Low). Bold indicates high-resource languages.

				Train							Holdout							Gen. Know.
Model	Set	$p$	Method	PS	SE	KSS ${}_{\text{prob}}^{\text{ROC}}$	KSS ${}_{\text{prob}}^{\text{PR}}$	KSS ${}_{\text{gen}}^{\text{ROC}}$	KSS ${}_{\text{gen}}^{\text{PR}}$	PrivLeak	PS	SE	KSS ${}_{\text{prob}}^{\text{ROC}}$	KSS ${}_{\text{prob}}^{\text{PR}}$	KSS ${}_{\text{gen}}^{\text{ROC}}$	KSS ${}_{\text{gen}}^{\text{PR}}$	PrivLeak	Gen. Know.
Gemma3-12B	S1	p1	Mem	99.99	100.00	-53.18	-99.06	-50.00	-99.00	-107.90	55.82	76.88	-58.37	-99.17	-50.17	-98.99	-86.85	30.19
			GA	72.44	91.88	-14.24	-77.47	-41.73	-93.94	-90.57	37.45	68.75	-42.05	-97.21	-44.63	-98.68	-60.75	32.85
			GAGDR	75.34	94.38	-20.39	-76.45	-42.89	-94.68	-97.23	45.03	70.00	-49.86	-98.61	-44.79	-98.75	-75.34	32.83
			NPO	47.89	83.12	-4.65	-43.28	-33.69	-86.37	-68.61	26.09	67.50	-26.60	-92.70	-45.54	-98.59	-41.32	32.49
			LingTea	45.32	91.88	-20.23	-41.52	-38.96	-89.00	-1.07	25.92	73.12	-28.49	-47.51	-46.95	-98.88	+9.18	30.84

Table 2: MMU performance on Gemma3-12B and Qwen2.5-7B. LingTea generally performs well on both training and hold-out languages.

		$\Delta$ PS		$\Delta$ SE
Cell	Method	High	Low	High	Low
G/S1	LingTea	+53.8	+55.5	+7.5	+8.8
	NPO	+40.0	+64.2	+16.2	+17.5
	GAGDR	+37.8	+11.5	+6.2	+5.0
	GA	+36.4	+18.7	+7.5	+8.8
G/S2	LingTea	+58.6	+32.9	+27.5	+15.0

Table 3: Training language drop rates per unlearning method. For each metric, bold marks the higher value within the High/Low pair. G and Q denote Gemma3 and Qwen2.5, respectively.

		$\Delta$ PS				$\Delta$ SE
		resource		script		resource		script
Cell	Method	High	Low	Lat	nLat	High	Low	Lat	nLat
G/S1	LingTea	+54.8	+55.8	+56.6	+54.0	+4.2	+5.9	+8.1	+1.6
	NPO	+52.1	+48.7	+65.7	+35.1	+13.9	+9.8	+11.3	+13.1
	GAGDR	+21.8	+29.6	+29.1	+22.2	+2.8	+17.6	+16.1	+1.6
	GA	+27.1	+41.9	+36.9	+32.1	+5.6	+17.6	+4.8	+16.4

Table 4: Hold-out language drop rates per unlearning method. Bold marks the higher value within each comparison pair (High/Low and Lat/nLat). G and Q denote Gemma3 and Qwen2.5, respectively.

Figure 4: Example of the model’s response in a hold-out language after MMU. The numbers in the figure denote the probability that the LLM assigns to the predicted next token, given the preceding context up to “of”. The memorized model outputs the English word wolves , whereas after unlearning the Spanish lobos ( wolves ) receives the highest probability.

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

Table 5: Valid values for the attributes used to build synthetic profiles (1/4)

Table 6: (continued) Valid values for the attributes used to build synthetic profiles (2/4)

Table 7: (continued) Valid values for the attributes used to build synthetic profiles (3/4)

Table 8: (continued) Valid values for the attributes used to build synthetic profiles (4/4)

Figure 5: Example of Generated Synthetic Profile

Figure 6: Examples of the generated English QA dataset. Left shows the naive QA dataset generated directly from templates, while Right shows the QA dataset refined using GPT.

Figure 7: Examples of code-mixed responses after NPO unlearning on Gemma3-12b model. Left shows the question, while Right shows the gold answer alongside the model’s code-mixed response.

Table 9: QA dataset template for each attribute (1/2)

Table 10: (continued) QA dataset template for each attribute (2/2)

Figure 8: Prompt used to refine the English QA dataset using GPT

Figure 9: Examples of the translated Multilingual QA dataset.

Figure 10: Prompt used to translate the English QA dataset to each language using GPT

Table 11: Description of each value in English (1/20)

Table 12: (continued) Description of each value in English (2/20)

Table 13: (continued) Description of each value in English (3/20)

Table 14: (continued) Description of each value in English (4/20)

Table 15: (continued) Description of each value in English (5/20)

Table 16: (continued) Description of each value in English (6/20)

Table 17: (continued) Description of each value in English (7/20)

Table 18: (continued) Description of each value in English (8/20)

Table 19: (continued) Description of each value in English (9/20)

Table 20: (continued) Description of each value in English (10/20)

Table 21: (continued) Description of each value in English (11/20)

Table 22: (continued) Description of each value in English (12/20)

Table 23: (continued) Description of each value in English (13/20)

Table 24: (continued) Description of each value in English (14/20)

Table 25: (continued) Description of each value in English (15/20)

Table 26: (continued) Description of each value in English (16/20)

Table 27: (continued) Description of each value in English (17/20)

Table 28: (continued) Description of each value in English (18/20)

Table 29: (continued) Description of each value in English (19/20)

Table 30: (continued) Description of each value in English (20/20)

Gemma3-12B

Qwen2.5-7B

Setting 1

Setting 2

Setting 1

Setting 2

Method

4.32e-5

7.60e-6

5.00e-6

4.19e-5

6.70e-6

2.90e-6

4.10e-5

9.10e-6

5.50e-6

4.41e-5

1.41e-5

8.10e-6

GAGDR

9.05e-5

3.04e-5

2.47e-5

1.005e-4

3.50e-5

2.04e-5

9.32e-5

3.00e-5

1.70e-5

1.110e-4

3.69e-5

2.29e-5

LingTea

1.302e-4

6.02e-5

3.72e-5

1.005e-4

4.12e-5

3.26e-5

1.699e-4

7.11e-5

5.00e-5

1.530e-4

1.425e-4

5.79e-5

NPO

4.00e-5

7.50e-6

4.60e-6

4.05e-5

7.20e-6

3.30e-6

4.90e-5

9.90e-6

5.80e-6

5.81e-5

1.70e-5

9.50e-6

Appendix

Table 31: Learning rates selected by an LR sweep with increments of

1\times 10^{-7}

across models, settings, forget ratios, and unlearning methods.

Figure 11: Prompt used to judge semantic equivalence of model responses

		$\Delta$ PS		$\Delta$ SE
Cell	Method	High	Low	High	Low
G/S1	LingTea	+53.9	+46.4	+5.4	+10.4
	NPO	+23.4	+0.9	+0.0	+0.4
	GAGDR	+30.0	+10.3	+0.0	+0.0
	GA	+12.7	+0.5	+0.0	+0.0
G/S2	LingTea	+23.8	+29.9	+3.7	+2.9

Appendix

Table 32: Training-language drop rates per unlearning method for

p3

. For each metric, bold marks the higher value within the High/Low pair. G and Q denote Gemma3 and Qwen2.5, respectively.

		$\Delta$ PS		$\Delta$ SE
Cell	Method	High	Low	High	Low
G/S1	LingTea	+29.3	+26.4	+5.0	+6.8
	NPO	+20.7	+1.3	+0.0	+0.0
	GAGDR	+20.8	+30.5	+2.7	+2.2
	GA	+24.1	+1.0	+0.0	+0.0
G/S2	LingTea	+26.2	+36.5	+1.2	+3.7

Appendix

Table 33: Training-language drop rates per unlearning method for

p5

. For each metric, bold marks the higher value within the High/Low pair. G and Q denote Gemma3 and Qwen2.5, respectively.

		$\Delta$ PS				$\Delta$ SE
		resource		script		resource		script
Cell	Method	High	Low	Lat	nLat	High	Low	Lat	nLat
G/S1	LingTea	+51.0	+51.8	+49.5	+53.4	+10.3	+12.4	+11.1	+11.3
	NPO	+3.0	+2.8	+3.7	+2.1	+0.0	+5.0	+1.6	+2.6
	GAGDR	+7.1	+14.2	+10.2	+11.2	+0.0	+5.0	+3.7	+0.5
	GA	+2.7	+2.2	+2.9	+2.0	-0.4	+3.7	+0.0	+2.6

Appendix

Table 34: Hold-out-language drop rates per unlearning method for

p3

. Bold marks the higher value within each comparison pair (High/Low and Lat/nLat). G and Q denote Gemma3 and Qwen2.5, respectively.

		$\Delta$ PS				$\Delta$ SE
		resource		script		resource		script
Cell	Method	High	Low	Lat	nLat	High	Low	Lat	nLat
G/S1	LingTea	+30.1	+24.7	+28.6	+26.2	+10.4	+10.1	+9.6	+10.8
	NPO	+2.5	+1.3	+2.3	+1.5	+1.1	+3.7	+1.9	+2.5
	GAGDR	+35.2	+28.5	+32.1	+31.6	+4.4	+6.7	+4.2	+6.5
	GA	+6.7	+3.8	+4.9	+5.6	+0.5	+4.1	+1.9	+2.2

Appendix

Table 35: Hold-out-language drop rates per unlearning method for

p5

. Bold marks the higher value within each comparison pair (High/Low and Lat/nLat). G and Q denote Gemma3 and Qwen2.5, respectively.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation
May 14, 2026Kyomin Hwang, Hyeonjin Kim, Sangyeon Cho +1Multilingual Language Model Evaluation Privacy Leakage in Language Models
GSCST, Seoul National University · Department of Artificial Intelligence, Chung-Ang University · Korean Surgical Researcher Foundation, Republic of Korea +1
Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearning
Sep 5, 2026Pengyang Shao, Chuanpeng Lu, Wei Qin +5LLM Unlearning Machine Unlearning Evaluation
National University of Singapore Singapore · Hefei University of Technology Anhui, China
Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility
Jun 2, 2026Chaoyi Xiang, Olga Ohrimenko, Benjamin I. P. Rubinstein +1Machine Unlearning LLM Unlearning
School of Computing and Information Systems, The University of Melbourne, Melbourne, Australia.

May 14, 2026J/K move · Enter open · S save

Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation

Kyomin Hwang, Hyeonjin Kim, Sangyeon Cho, Nojun Kwak

GSCST, Seoul National University · Department of Artificial Intelligence, Chung-Ang University · Korean Surgical Researcher Foundation, Republic of Korea+1

Multilingual Language Model Evaluation Privacy Leakage in Language Models Machine Unlearning LLM Unlearning Machine Unlearning Evaluation

Open paper AskPDF arXiv

While LLMs are increasingly used in commercial services, they pose privacy risks such as leakage of sensitive personally identifiable information (PII). For LLMs trained on multilingual corpora, Multilingual Machine Unlearning (MMU) aims to remove information across multiple languages. However, prior MMU evaluations fail to capture such cross-linguistic distribution of information, being largely limited to direct extensions of per-language evaluation protocols. To this end, we propose two metrics to evaluate the information spread across languages: the Knowledge Separability Score (KSS) and the Knowledge Persistence Score (KPS). KSS measures the overall unlearning quality across multiple languages, while KPS more specifically aims to assess consistent removal of information among different language pairs. We evaluated various unlearning methods in the multilingual setting with these metrics and conducted comprehensive analyses. Through our investigation, we provide insights into unique phenomena exclusive to MMU and offer a new perspective on MMU evaluation.

Abstract

Machine Unlearning

Explore similar work

Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation

Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearning

Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility