UnlearningSoup: Is Repeated Tuning Necessary for Large Language Model Unlearning?

Puning Yang; Qizhou Wang; Junchi Yu; Bo Han; Xiuying Chen

cs.LGSep 29, 2026

UnlearningSoup: Is Repeated Tuning Necessary for Large Language Model Unlearning?

Authors: Puning Yang, Qizhou Wang, Junchi Yu, Bo Han, Xiuying Chen

Organizations: Division of Computing and Mathematical Sciences, MBZUAI · RIKEN Center for Advanced Intelligence Project · TMLR Group, Department of Computer Science, Hong Kong Baptist University · Torr Vision Group, Department of Engineering Science, University of Oxford

Abstract

Large language models trained on vast corpora inherently risk memorizing harmful content that may later re-emerge in their outputs. To mitigate this issue, existing unlearning methods typically rely on training-based parameter updates, such as gradient ascent and its variants, to delete targeted content while preserving other knowledge. However, balancing the competing goals of forgetting and retention makes hyperparameter choices for these methods particularly difficult, often requiring repeated tuning to obtain a strong model that still leaves substantial room for improvement and transfers poorly across models and datasets. To address this challenge, we investigate whether unlearning runs exhibit exploitable structure in weight space, and observe that models from different runs still lie in a shared evaluation-performance basin. This suggests that stronger models may be recovered through an unlearning-tailored soup strategy, reducing the need for repeated tuning for further improvement or new settings. Motivated by this, we propose UnlearningSoup, a unified framework that provides two strategies: EfficientSoup uses binary-search-based interpolation to quickly discover a well-performing model in the early stage, where repeated tuning would otherwise make strong model selection costly. PerformanceSoup uses reweighted souping to efficiently unlock the remaining performance potential in the later stage, where repeated tuning becomes increasingly inefficient. Extensive experiments across diverse datasets and models show that UnlearningSoup delivers 2.4x to 3.3x efficiency gains in hyperparameter selection, while consistently improving performance across settings.

Figures & tables

Figure 1: Motivation of our paper: Existing unlearning methods rely heavily on hyper-parameter tuning. As shown in the Background, small changes in $\beta$ and $\gamma$ cause dramatic performance swings. I. Finding the optimal setup requires up to 7 tuning iterations. II. Worse, the optimal hyper-parameters do not transfer across datasets or models, forcing another 6 iterations of re-tuning.

Figure 2: The significant mismatch between training loss (Negative Log Likelihood, NLL) and test metrics (Extraction Strength, ES, and ROUGE-L) explains why repeated tuning strategy is needed: training loss can not reliably predict the performance, therefore introduing the repeat process to approach better-performing regions of the test-performance basin. However, the shared basin encourage a more efficient solution in the weight space to replace this costly tuning-based process.

Figure 3: Strong correlation between the ES metric and overall performance (SBQ in Sec. 5.1 ).

Method	LLaMA-3.2-1B			LLaMA-3.2-3B			Qwen2.5-1.5B			Qwen2.5-3B
Method	T $\downarrow$	EAM $\downarrow$	ESM $\downarrow$	T $\downarrow$	EAM $\downarrow$	ESM $\downarrow$	T $\downarrow$	EAM $\downarrow$	ESM $\downarrow$	T $\downarrow$	EAM $\downarrow$	ESM $\downarrow$
GradDiff	10.45			18.00			18.70			23.79
NPO	20.48	2.67	0.05	31.20	3.65	0.11	31.15	3.41	0.10	37.42	4.42	0.14
WGA	10.66			19.75			19.05			24.2

Table 1: Comparison of the time efficiency of tuning (T), evaluating on all metrics (EAM), and evaluating on a single metric (ESM) on TOFU-10%. All times are reported in minutes.

Figure 4: Illustration of UnlearningSoup . (a) EfficientSoup identifies a better-performing model within the model-triad region through binary-search-style interpolation over mixing coefficients, locating a point in a better-performing region of the shared basin. (b) PerformanceSoup performs reweighted souping over sorted candidates, greedily steering interpolation toward a better-performing region of the shared basin and thereby efficiently unlocking additional performance potential.

Figure 5: EfficientSoup achieves 3.3× speedup over repeated tuning by replacing most training runs with lightweight interpolation search (TOFU-10%, LLaMA-3.1-8B).

Method	LLaMA-3.2-1B			LLaMA-3.2-3B			LLaMA-3.1-8B
Method	SBQ $\uparrow$	LBQ $\uparrow$	GPU-Hours $\downarrow$	SBQ $\uparrow$	LBQ $\uparrow$	GPU-Hours $\downarrow$	SBQ $\uparrow$	LBQ $\uparrow$	GPU-Hours $\downarrow$
Original	5.216	5.049	-	5.574	6.253	-	5.272	6.144	-
GradDiff	6.118	0.230	1.269	5.635	0.049	2.172	6.230	0.385	4.774
+EfSoup	7.355 ( $+20.2\%$ )	1.021 ( $+344.5\%$ )	0.400 ( $-68.5\%$ )	7.776 ( $+38.0\%$ )	2.150 ( $+4326.2\%$ )	0.675 ( $-68.9\%$ )	7.187 ( $+15.4\%$ )	2.145 ( $+457.7\%$ )	1.464 ( $-69.3\%$ )
NPO	5.722	4.452	2.439	6.190	4.372	3.712	6.621	1.006	8.408
+EfSoup	7.083 ( $+23.8\%$ )	5.272 ( $+18.4\%$ )	0.734 ( $-69.9\%$ )	7.294 ( $+17.8\%$ )	5.349 ( $+22.4\%$ )	1.115 ( $-69.9\%$ )	6.817 ( $+3.0\%$ )	1.630 ( $+62.0\%$ )	2.502 ( $-70.2\%$ )

Table 2: EfficientSoup (EfSoup) performance comparisons with training-based unlearning methods on the TOFU-Forget 10% setting.

\uparrow/\downarrow

indicates larger/smaller values are preferable, respectively. Statistic-Based Quality (SBQ), LaaJ-Based Quality (LBQ), and GPU-Hours are reported.

Strategy	$\mathrm{DS}_{\mathrm{ES}}\downarrow$	SBQ $\uparrow$
Uniform Wortsman et al. (2022)	80.05	5.982
Uniform&Greedy Wortsman et al. (2022)	46.69	6.952
Reweighted&Greedy	42.58	7.108

Table 3: Ablation study about souping strategies on TOFU-10

\%

with Qwen-2.5-7B.

Method	LLaMA-3.2-1B			LLaMA-3.2-3B			LLaMA-3.1-8B
Method	SBQ $\uparrow$	LBQ $\uparrow$	GPU-Hours $\downarrow$	SBQ $\uparrow$	LBQ $\uparrow$	GPU-Hours $\downarrow$	SBQ $\uparrow$	LBQ $\uparrow$	GPU-Hours $\downarrow$
Original	5.216	5.049	-	5.574	6.253	-	5.272	6.144	-
GradDiff	6.118	0.230	1.269	5.635	0.049	2.172	6.230	0.385	4.774
+PeSoup	7.485 ( $+22.3\%$ )	1.204 ( $+424.0\%$ )	1.274 ( $+0.5\%$ )	7.454 ( $+32.3\%$ )	0.516 ( $+962.1\%$ )	2.185 ( $+0.6\%$ )	7.251 ( $+16.4\%$ )	1.574 ( $+309.4\%$ )	4.819 ( $+0.9\%$ )
NPO	5.722	4.452	2.439	6.190	4.372	3.712	6.621	1.006	8.408
+PeSoup	6.920 ( $+20.9\%$ )	5.361 ( $+20.4\%$ )	2.445 ( $+0.2\%$ )	7.083 ( $+14.4\%$ )	5.003 ( $+14.4\%$ )	3.725 ( $+0.3\%$ )	6.931 ( $+4.7\%$ )	1.671 ( $+66.1\%$ )	8.452 ( $+0.5\%$ )

Table 4: PerformanceSoup (PeSoup) performance comparisons with training-based unlearning methods on the TOFU-Forget 10% setting.

\uparrow/\downarrow

indicates larger/smaller values are preferable, respectively. Statistic-Based Quality (SBQ), LaaJ-Based Quality (LBQ), and GPU-Hours are reported.

Figure 6: The robustness evaluations of UnlearningSoup on TOFU-10

\%

with LLaMA-3.2-3B.

Method	WMDP			MUSE-Books			MUSE-News
Method	Forget Acc. $\downarrow$	MMLU Acc. $\uparrow$	GPU-Hours $\downarrow$	MemQ $\downarrow$	UtilPres $\uparrow$	GPU-Hours $\downarrow$	MemQ $\downarrow$	UtilPres $\uparrow$	GPU-Hours $\downarrow$
Original	0.528	0.585	-	81.589	67.010	-	60.654	54.310	-
GradDiff	0.270	0.444	6.724	0.000	28.598	9.861	22.615	28.341	8.042
+EfSoup	0.260 ( $-3.6\%$ )	0.493 ( $+10.9\%$ )	2.491 ( $-63.0\%$ )	0.000 ( N/A )	34.482 ( $+20.6\%$ )	3.843 ( $-61.0\%$ )	22.354 ( $-1.2\%$ )	34.321 ( $+21.1\%$ )	3.237 ( $-59.8\%$ )
+PeSoup	0.269 ( $-0.2\%$ )	0.462 ( $+3.9\%$ )	7.036 ( $+4.6\%$ )	0.000 ( N/A )	31.450 ( $+10.0\%$ )	10.556 ( $+7.0\%$ )	22.596 ( $-0.1\%$ )	30.333 ( $+7.0\%$ )	8.737 ( $+8.6\%$ )
NPO	0.286	0.501	7.554	11.713	37.362	12.984	29.429	34.787	10.301

Table 5: Performance comparisons with training-based unlearning methods on the WMDP and MUSE benchmarks.

\uparrow/\downarrow

indicates larger/smaller values are preferable, respectively.

Model	$\mathrm{DS}_{\mathrm{ES}}\downarrow$	SBQ $\uparrow$
Seed=1, 42	25.19	7.776
Lr=1e-5, 5e-6	28.44	7.691
$\lambda=1.0,\ 0.5$	24.90	7.779
Search Depth	$\mathrm{DS}_{\mathrm{ES}}\downarrow$	SBQ $\uparrow$
Depth=2	46.34	7.144
Depth=3	29.03	7.654

Table 6: Ablation study about binary search details on TOFU-10

\%

with LLaMA-3.2-3B.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Weight space landscape for LLaMA-3.2-1B on TOFU 10

\%

. NPO and WGA are reported.

Figure 8: Weight-space landscape for Qwen2.5-1.5B on TOFU 10

\%

. Seeds and learning rates vary.

Figure 9: Weight space landscape for Qwen2.5-1.5B on TOFU 10

\%

. NPO and WGA are reported.

Panel of Figure 10 — Figure 10: Correlation between single metrics (ROUGE-L, Truth Ratio) and overall performance.

Figure 11: Different resulting models in the weight space across souping strategies. ModelSoups (Orange) typically results in less effective points due to the performance-irrelevant souping strategy.

Strategy	Statistic-Based Quality (SBQ)
Strategy	Prob. $\downarrow$	ROUGE-L $\downarrow$	ES Unlearn $\downarrow$	Truth Ratio $\downarrow$	EQ $\uparrow$	Model Utility $\uparrow$	ES Retain $\uparrow$	RQ $\uparrow$	SBQ $\uparrow$
GradDiff	0.0000	0.0000	0.0000	0.4390	8.3637	0.4946	0.3477	4.0835	6.5813
+Uniform	0.0000	0.0000	0.0028	0.4395	8.3559	0.4968	0.3721	4.2553	6.6306
+Uniform Greedy	0.0000	0.0000	0.0028	0.4558	8.2636	0.4971	0.4565	4.7597	6.7432
+Reweighted Greedy	0.0000	0.0025	0.0028	0.4666	8.1967	0.5179	0.6094	5.5995	7.0193
NPO	0.1460	0.3977	0.0869	0.6080	6.1751	0.4965	0.3415	4.0467	5.2205

Appendix

Table 7: Comparsions between PerformanceSoup and ModelSoups on TOFU-5

\%

with Qwen2.5-1.5B.

\uparrow/\downarrow

indicates that larger / smaller values are preferable.

Method	LLaMA-3.2-1B			LLaMA-3.2-3B			LLaMA-3.1-8B
Method	SBQ $\uparrow$	LBQ $\uparrow$	GPU-Hours $\downarrow$	SBQ $\uparrow$	LBQ $\uparrow$	GPU-Hours $\downarrow$	SBQ $\uparrow$	LBQ $\uparrow$	GPU-Hours $\downarrow$
Original	5.216	5.326	-	5.574	6.478	-	5.272	5.825	-
GradDiff	5.695	0.152	0.704	6.251	0.260	1.194	6.792	0.136	2.598
+EfSoup	7.351 ( $+29.1\%$ )	0.164 ( $+8.3\%$ )	0.237 ( $-66.4\%$ )	7.670 ( $+22.7\%$ )	0.450 ( $+73.1\%$ )	0.392 ( $-67.2\%$ )	7.319 ( $+7.8\%$ )	0.087 ( $-36.3\%$ )	0.838 ( $-67.8\%$ )
NPO	4.977	6.052	1.335	5.294	4.616	1.972	7.056	1.000	4.341
+EfSoup	7.185 ( $+44.4\%$ )	6.320 ( $+4.4\%$ )	0.417 ( $-68.8\%$ )	7.343 ( $+38.7\%$ )	5.949 ( $+28.9\%$ )	0.614 ( $-68.8\%$ )	7.665 ( $+8.6\%$ )	4.298 ( $+330.0\%$ )	1.336 ( $-69.2\%$ )

Appendix

Table 8: EfficientSoup (EfSoup) performance comparisons with training-based unlearning methods on the TOFU-Forget 5% setting.

\uparrow/\downarrow

indicates larger/smaller values are preferable, respectively. Statistic-Based Quality (SBQ), LaaJ-Based Quality (LBQ), and GPU-Hours are reported.

Method	LLaMA-3.2-1B			LLaMA-3.2-3B			LLaMA-3.1-8B
Method	SBQ $\uparrow$	LBQ $\uparrow$	GPU-Hours $\downarrow$	SBQ $\uparrow$	LBQ $\uparrow$	GPU-Hours $\downarrow$	SBQ $\uparrow$	LBQ $\uparrow$	GPU-Hours $\downarrow$
Original	5.216	5.326	-	5.574	6.478	-	5.272	5.825	-
GradDiff	5.695	0.152	0.704	5.635	0.049	1.194	6.792	0.136	2.598
+PeSoup	7.349 ( $+29.0\%$ )	1.869 ( $+1133.1\%$ )	0.710 ( $+0.8\%$ )	7.454 ( $+32.3\%$ )	0.516 ( $+962.1\%$ )	1.207 ( $+1.1\%$ )	7.236 ( $+6.5\%$ )	0.240 ( $+76.1\%$ )	2.643 ( $+1.7\%$ )
NPO	4.977	6.052	1.335	6.190	4.372	1.972	7.056	1.000	4.341
+PeSoup	7.309 ( $+46.9\%$ )	6.187 ( $+2.2\%$ )	1.341 ( $+0.4\%$ )	7.083 ( $+14.4\%$ )	5.003 ( $+14.4\%$ )	1.984 ( $+0.7\%$ )	7.554 ( $+7.1\%$ )	3.614 ( $+261.6\%$ )	4.386 ( $+1.0\%$ )

Appendix

Table 9: PerformanceSoup (PeSoup) performance comparisons with training-based unlearning methods on the TOFU-Forget 5% setting.

\uparrow/\downarrow

indicates larger/smaller values are preferable, respectively. Statistic-Based Quality (SBQ), LaaJ-Based Quality (LBQ), and GPU-Hours are reported.

GradDiff
Attack	Tuning-Based					+EfficientSoup					+PerformanceSoup
Attack	Prob. $\downarrow$	ROUGE-L $\downarrow$	ES Unlearn $\downarrow$	Truth Ratio $\downarrow$	EQ $\uparrow$	Prob. $\downarrow$	ROUGE-L $\downarrow$	ES Unlearn $\downarrow$	Truth Ratio $\downarrow$	EQ $\uparrow$	Prob. $\downarrow$	ROUGE-L $\downarrow$	ES Unlearn $\downarrow$	Truth Ratio $\downarrow$	EQ $\uparrow$
Original	0.0306	0.2838	0.0785	0.4350	7.5713	0.0468	0.0452	0.0191	0.2844	8.8627	0.0000	0.1492	0.0000	0.3726	8.3870
Cross-lingual	0.0623	0.2786	0.0331	0.4911	7.3370	0.0814	0.0465	0.0191	0.2956	8.7403	0.0000	0.1399	0.0000	0.3975	8.2949
Jailbreaking	0.0349	0.2849	0.0334	0.4806	7.4149	0.0815	0.0717	0.0193	0.2868	8.7191	0.0001	0.1815	0.0049	0.3846	8.2445
Relearn	0.1660	0.4058	0.2500	0.5573	6.1784	0.1703	0.1789	0.2037	0.5318	6.8790	0.1503	0.2585	0.2064	0.5323	6.7527

Appendix

Table 10: Attack results on TOFU-10

\%

benchmark with LLaMA-3.2-3B.

\uparrow/\downarrow

indicates that larger / smaller values are preferable.

Method	W/o Relearning Attacks		W/ Relearning Attacks
Method	VerbMem $\downarrow$	KnowMem $\downarrow$	VerbMem $\downarrow$	KnowMem $\downarrow$
GradDiff	0.0000	0.0000	36.4200	43.0300
+EfSoup	0.0000	0.0000	36.2335	41.0889
+PeSoup	0.0000	0.0000	36.8505	42.0792
NPO	10.4016	12.8913	68.5800	46.0600
+EfSoup	7.5604	13.6783	66.8434	47.1971

Appendix

Table 11: Attack results on MUSE-Books benchmark.

\downarrow

indicates smaller values are preferable.

Strategy	Statistic-Based Quality (SBQ)
Strategy	Prob. $\downarrow$	ROUGE-L $\downarrow$	ES Unlearn $\downarrow$	Truth Ratio $\downarrow$	EQ $\uparrow$	Model Utility $\uparrow$	ES Retain $\uparrow$	RQ $\uparrow$	$\mathrm{DS}_{\mathrm{ES}}\downarrow$	SBQ $\uparrow$
GradDiff	0.0000	0.0085	0.0000	0.4936	8.0266	0.4637	0.2016	2.8102	0.7984	6.0135
Uniform	0.0000	0.0094	0.0000	0.5003	7.9829	0.4701	0.1995	2.8013	0.8005	5.9822
Uniform&Greedy	0.0000	0.0080	0.0000	0.4284	8.4080	0.4879	0.5331	5.0952	0.4669	6.9518
Reweight&Greedy (PeSoup)	0.0000	0.0078	0.0000	0.4031	8.5413	0.4922	0.5742	5.3002	0.4258	7.1079

Appendix

Table 12: Detailed PerformanceSoup ablation study results.

Strategy	Statistic-Based Quality (SBQ)
Strategy	Prob. $\downarrow$	ROUGE-L $\downarrow$	ES Unlearn $\downarrow$	Truth Ratio $\downarrow$	EQ $\uparrow$	Model Utility $\uparrow$	ES Retain $\uparrow$	RQ $\uparrow$	$\mathrm{DS}_{\mathrm{ES}}\downarrow$	SBQ $\uparrow$
GradDiff	0.0306	0.2838	0.0785	0.4350	7.5713	0.4763	0.1682	2.4866	0.8355	5.6351
Seed=1,42	0.0468	0.0452	0.0191	0.2844	8.8627	0.5758	0.7488	6.5098	0.2519	7.7758
Lr=1e-5,5e-6	0.0366	0.0583	0.0036	0.2979	8.8343	0.5699	0.7156	6.3447	0.2844	7.6909
lambda=1,0.5	0.0360	0.0505	0.0122	0.2884	8.8727	0.5734	0.7513	6.5043	0.2490	7.7792
depth=2	0.0596	0.0739	0.0387	0.3350	8.5338	0.5434	0.5382	5.4080	0.4634	7.1440

Appendix

Table 13: Detailed Efficient ablation study results.

Strategy	Statistic-Based Quality (SBQ)
Strategy	Prob. $\downarrow$	ROUGE-L $\downarrow$	ES Unlearn $\downarrow$	Truth Ratio $\downarrow$	EQ $\uparrow$	Model Utility $\uparrow$	ES Retain $\uparrow$	RQ $\uparrow$	SBQ $\uparrow$
GradDiff	0.0000	0.0126	0.0000	0.4645	8.1964	0.4326	0.3012	3.5514	6.3164
$\mathrm{DS}_{\mathrm{ES}}$ +EfSoup	0.0000	0.0089	0.0000	0.4218	8.4415	0.5453	0.5857	5.6480	7.1819
$\mathrm{DS}_{\mathrm{ES}}$ +PeSoup	0.0000	0.0089	0.0000	0.3940	8.5855	0.5017	0.6203	5.5470	7.2277
$\mathrm{DS}_{\mathrm{ROUGE-L}}$ +EfSoup	0.0000	0.0089	0.0000	0.4218	8.4415	0.5453	0.5857	5.6480	7.1819
$\mathrm{DS}_{\mathrm{ROUGE-L}}$ +PeSoup	0.0000	0.0113	0.0000	0.4115	8.4911	0.5132	0.6087	5.5691	7.1803

Appendix

Table 14: Ablation study on different proxy metrics. Results on TOFU-10

\%

with Qwen2.5-1.5B are reported.

\uparrow/\downarrow

indicates that larger / smaller values are preferable.

Method	LLaMA-3.2-1B	LLaMA-3.2-3B	LLaMA-3.1-8B	Qwen2.5-1.5B	Qwen2.5-3B	Qwen2.5-7B
GradDiff	91.840	151.550	314.370	154.770	197.470	334.880
+EfSoup	44.930 ( $-51.1\%$ )	68.850 ( $-54.6\%$ )	125.030 ( $-60.2\%$ )	68.090 ( $-56.0\%$ )	87.360 ( $-55.8\%$ )	135.160 ( $-59.6\%$ )
+PeSoup	107.860 ( $+17.4\%$ )	173.450 ( $+14.5\%$ )	344.550 ( $+9.6\%$ )	175.230 ( $+13.2\%$ )	223.990 ( $+13.4\%$ )	368.720 ( $+10.1\%$ )
NPO	162.050	243.950	532.350	241.920	292.880	592.200
+EfSoup	64.990 ( $-59.9\%$ )	95.250 ( $-61.0\%$ )	187.310 ( $-64.8\%$ )	92.990 ( $-61.6\%$ )	114.620 ( $-60.9\%$ )	208.680 ( $-64.8\%$ )
+PeSoup	178.070 ( $+9.9\%$ )	265.850 ( $+9.0\%$ )	562.530 ( $+5.7\%$ )	262.380 ( $+8.5\%$ )	319.400 ( $+9.1\%$ )	626.040 ( $+5.7\%$ )

Appendix

Table 15: PerformanceSoup (PeSoup) performance comparisons with training-based unlearning methods on the TOFU-Forget 5% setting.

\uparrow/\downarrow

indicates larger/smaller values are preferable, respectively. Statistic-Based Quality (SBQ), LaaJ-Based Quality (LBQ), and GPU-Hours are reported.

Method	WMDP				MUSE-Books				MUSE-News
Method	Cyber Acc. $\downarrow$	Bio Acc. $\downarrow$	Forget Acc. $\downarrow$	MMLU Acc. $\uparrow$	VerbMem $\downarrow$	KnowMem $\downarrow$	MemQ $\downarrow$	UtilPres $\uparrow$	VerbMem $\downarrow$	KnowMem $\downarrow$	MemQ $\downarrow$	UtilPres $\uparrow$
Original	0.3924	0.6354	0.5281	0.5853	99.5600	58.3200	81.5887	67.0100	58.2900	62.9300	60.6544	54.3100
GradDiff	0.2657	0.2739	0.2698	0.4442	0.0000	0.0000	0.0000	28.5984	4.0600	31.7238	22.6151	28.3406
+EfSoup	0.2567	0.2635	0.2601	0.4927	0.0000	0.0000	0.0000	34.4819	2.0053	31.5490	22.3535	34.3208
+PeSoup	0.2613	0.2772	0.2694	0.4617	0.0000	0.0000	0.0000	31.4498	3.8907	31.7176	22.5958	30.3334
NPO	0.3067	0.2647	0.2865	0.5011	10.4016	12.8913	11.7128	37.3619	16.4346	38.2363	29.4288	34.7869

Appendix

Table 16: Detailed results on WMDP and MUSE benchmarks.

Method	Statistic-Based Quality (SBQ)									LaaJ-Based Quality (LBQ)
Method	Prob. $\downarrow$	ROUGE-L $\downarrow$	ES Unlearn $\downarrow$	Truth Ratio $\downarrow$	EQ $\uparrow$	Model Utility $\uparrow$	ES Retain $\uparrow$	RQ $\uparrow$	SBQ $\uparrow$	Fluency $\uparrow$	Relevance $\uparrow$	Hallucination $\uparrow$	Correctness $\uparrow$	LQ $\uparrow$
LLaMA-3.2-1B, Forget 5 $\%$
Original	0.9882	0.7827	0.9735	0.6865	0.3072	0.5891	0.9840	7.3701	5.2160	8.6748	7.6544	8.9612	2.5412	5.3260
GradDiff	0.0357	0.3478	0.0830	0.3560	7.6722	0.4527	0.1680	2.4509	5.6952	0.0685	0.0880	3.7300	6.4920	0.1516
+EfSoup	0.0000	0.0886	0.0000	0.3377	8.6825	0.5282	0.6229	5.7167	7.3507	0.0628	0.1245	3.8410	6.7703	0.1642
NPO	0.2218	0.2632	0.0880	0.5570	6.6713	0.4620	0.1483	2.2450	4.9772	8.3434	5.6316	4.5251	7.0138	6.0515

Appendix

Table 17: Detailed EfficientSoup results on TOFU benchmark with LLaMA series LLMs.

Method	Statistic-Based Quality (SBQ)									LaaJ-Based Quality (LBQ)
Method	Prob. $\downarrow$	ROUGE-L $\downarrow$	ES Unlearn $\downarrow$	Truth Ratio $\downarrow$	EQ $\uparrow$	Model Utility $\uparrow$	ES Retain $\uparrow$	RQ $\uparrow$	SBQ $\uparrow$	Fluency $\uparrow$	Relevance $\uparrow$	Hallucination $\uparrow$	Correctness $\uparrow$	LQ $\uparrow$
Qwen2.5-1.5B, Forget 5 $\%$
Original	0.9803	0.9209	0.9536	0.4376	0.4605	0.5332	0.9550	6.8435	4.8500	8.7391	7.5376	9.3561	2.4393	5.2361
GradDiff	0.0000	0.0000	0.0000	0.4390	8.3637	0.4946	0.3477	6.5813	7.5254	0.0040	0.0040	1.1500	6.9120	0.0080
+EfSoup	0.0524	0.0023	0.0371	0.4447	8.1685	0.5403	0.6255	7.0831	7.6451	0.0626	0.0832	3.7650	6.4320	0.1408
NPO	0.1460	0.3977	0.0869	0.6080	6.1751	0.4965	0.3415	5.2205	5.7177	8.0583	6.1414	3.1397	7.3711	5.3974

Appendix

Table 18: Detailed EfficientSoup results on TOFU benchmark with Qwen series LLMs.

Method	Statistic-Based Quality (SBQ)									LaaJ-Based Quality (LBQ)
Method	Prob. $\downarrow$	ROUGE-L $\downarrow$	ES Unlearn $\downarrow$	Truth Ratio $\downarrow$	EQ $\uparrow$	Model Utility $\uparrow$	ES Retain $\uparrow$	RQ $\uparrow$	SBQ $\uparrow$	Fluency $\uparrow$	Relevance $\uparrow$	Hallucination $\uparrow$	Correctness $\uparrow$	LQ $\uparrow$
LLaMA-3.2-1B, Forget 5 $\%$
Original	0.9882	0.7827	0.9735	0.6865	0.3072	0.5891	0.9840	7.3701	5.2160	8.6748	7.6544	8.9612	2.5412	5.3260
GradDiff	0.0357	0.3478	0.0830	0.3560	7.6722	0.4527	0.1680	2.4509	5.6952	0.0685	0.0880	3.7300	6.4920	0.1516
+PeSoup	0.0043	0.2049	0.0094	0.3365	8.3707	0.5688	0.6720	6.1608	7.3493	1.2270	1.0642	4.3031	6.5503	1.8694
NPO	0.2218	0.2632	0.0880	0.5570	6.6713	0.4620	0.1483	2.2450	4.9772	8.3434	5.6316	4.5251	7.0138	6.0515

Appendix

Table 19: Detailed PerformanceSoup results on TOFU benchmark with LLaMA series LLMs.

Method	Statistic-Based Quality (SBQ)									LaaJ-Based Quality (LBQ)
Method	Prob. $\downarrow$	ROUGE-L $\downarrow$	ES Unlearn $\downarrow$	Truth Ratio $\downarrow$	EQ $\uparrow$	Model Utility $\uparrow$	ES Retain $\uparrow$	RQ $\uparrow$	SBQ $\uparrow$	Fluency $\uparrow$	Relevance $\uparrow$	Hallucination $\uparrow$	Correctness $\uparrow$	LQ $\uparrow$
Qwen2.5-1.5B, Forget 5 $\%$
Original	0.9803	0.9209	0.9536	0.4376	0.4605	0.5332	0.9550	6.8435	4.8500	8.7391	7.5376	9.3561	2.4393	5.2361
GradDiff	0.0000	0.0000	0.0000	0.4390	8.3637	0.4946	0.3477	4.0835	6.5813	0.0040	0.0040	1.1500	6.9120	0.0080
+PeSoup	0.0000	0.0025	0.0028	0.4666	8.1967	0.5179	0.6094	5.5995	7.0193	0.0527	0.0332	2.9183	6.7982	0.0807
NPO	0.1460	0.3977	0.0869	0.6080	6.1751	0.4965	0.3415	4.0467	5.2205	8.0583	6.1414	3.1397	7.3711	5.3974

Appendix

Table 20: Detailed PerformanceSoup results on TOFU benchmark with Qwen series LLMs.

LLM Unlearning

Large language model unlearning studies how to remove specified knowledge or behaviors while preserving a model’s unrelated capabilities.

See all 190 papers

Dec 4, 2025

RapidUn: Influence-Driven Parameter Reweighting for Efficient Large Language Model Unlearning

Machine unlearning for large language models (LLMs) remains challenging because full retraining is costly, while approximate methods often struggle to remove targeted behaviors without degrading retained utility, especially under limited post-deployment supervision. We consider a practical PEFT setting for targeted behavioral contamination removal with a small forget set, a limited retain buffer, and LoRA-only updates, and propose RapidUn, an influence-guided framework that converts cross-sample influence estimates into fixed sample-specific weights for weighted LoRA unlearning. Across Llama-3-8B on Dolly-15k and Alpaca-57k, with cross-model validation on Mistral-7B + Dolly-15k, RapidUn achieves lower seen-trigger and OOD-trigger-family ASR than Fisher, GA, and LoReUn while maintaining competitive clean utility. On Llama-3-8B + Alpaca-57k, it achieves a 77x wall-clock speedup over the clean-corpus LoRA retraining reference. Complementary TOFU, semantic LLM-judge, and IFEval evaluations further support the effectiveness of influence-guided sample reweighting beyond the controlled trigger benchmark.

Guoshenghui Zhao, Huawei Lin, Weijie Zhao

Golisano College of Computing and Information Sciences, Rochester Institute of Technology

Machine Unlearning LLM Unlearning

Ask PDF

Jun 1, 2026

Fast Unlearning at Scale via Margin Self-Correction

Language-model unlearning updates a trained model to behave as if it had not seen selected training examples, while preserving utility and avoiding costly retraining. Existing approaches typically fine-tune the pretrained model with a fixed training budget and select the final model afterwards by evaluating several saved checkpoints on downstream validation data. Two sources of unnecessary computation limit scalability: training beyond the desired forget-retain trade-off, and checkpoint selection that requires extra storage and repeated evaluations. To address these limitations, we introduce MArgin Self-Correction (MASC), an efficient unlearning method with an online stopping rule that does not require downstream evaluation. Given a text sequence to be forgotten, MASC actively reduces the logit gap between the original next token and the most likely alternatives. It outputs a final model once this gap is small on average over a sufficiently large proportion of token positions across all forget sequences. On TOFU, MUSE News, and MUSE Books, MASC achieves a competitive forget-retain trade-off at a fraction of the computational cost of existing baselines. We further observe that as we increase model size (a.k.a. number of parameters), the trade-offs improve for both MASC and SimNPO -- the forget metrics remain comparable while retain utility increases.

Federico Di Gennaro, Alexander Shevchenko, Fanny Yang

ETH Zürich

Machine Unlearning LLM Unlearning

Ask PDF

Apr 19, 2026

Representation-Guided Parameter-Efficient LLM Unlearning

Large Language Models (LLMs) often memorize sensitive or harmful information, necessitating effective machine unlearning techniques. While existing parameter-efficient unlearning methods have shown promise, they still struggle with the forget-retain trade-off. This can be attributed to their reliance on parameter importance metrics to identify parameters that are important exclusively for the forget set, which is fundamentally limited by the superposition phenomenon. Due to the polysemantic nature of LLM parameters, such an importance metric may struggle to disentangle parameters associated with the forget and retain sets. In this work, we propose Representation-Guided Low-rank Unlearning (REGLU), a novel approach that leverages the geometric properties of representation spaces to achieve robust and precise unlearning. First, we develop a representation-guided initialization for LoRA that identifies the optimal subspace for selective forgetting. Second, we introduce a regularization loss that constrains the outputs of the LoRA update to lie in the orthogonal complement of the retain set's representation subspace, thereby minimizing interference with the model's performance on the retain set. We evaluate REGLU on the TOFU and WMDP benchmarks across multiple models. Our results demonstrate that REGLU consistently outperforms state-of-the-art baselines, achieving superior unlearning quality while maintaining higher model utility.

Zeguan Xiao, Lang Mo, Yun Chen +4

Shanghai University of Finance and Economics · Southern University of Science and Technology · Deepexi Technology Co. Ltd.

Machine Unlearning LLM Unlearning Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

LLM Unlearning

Explore similar work

RapidUn: Influence-Driven Parameter Reweighting for Efficient Large Language Model Unlearning

Fast Unlearning at Scale via Margin Self-Correction

Representation-Guided Parameter-Efficient LLM Unlearning