BLADE: Bilevel Low-rank Augmented-Lagrangian Erasure for LLM Unlearning

Md Toufikuzzaman; Ahmad Mousavi; Dongwon Lee

cs.LGAug 23, 2026

BLADE: Bilevel Low-rank Augmented-Lagrangian Erasure for LLM Unlearning

Authors: Md Toufikuzzaman, Ahmad Mousavi, Dongwon Lee

Organizations: The Pennsylvania State University, USA · American University

Abstract

Existing LLM unlearning methods struggle with robustness: unbounded forget losses degrade model coherence, fixed-weight balancing cannot adapt as retain difficulty shifts mid-training, and methods that work on one benchmark falter under scaling or repeated application. We propose BLADE, a constrained bilevel framework whose three mechanisms give smooth, predictable control over the optimization landscape: a clamped-entropy forget loss whose gradient is exactly zero once a token reaches sufficient uncertainty; an asymmetric augmented Lagrangian that permanently ratchets retain protection after any violation; and a bilevel structure confined to LoRA adapters that repairs retain damage before each forgetting step. BLADE dominates across three benchmark families, improving average composite scores over the strongest baselines by $6$ % on TOFU, $9$ % on MUSE Books, and $7$ % on KnowUndo, and it remains stable under $4\times$ scaling and $4$ sequential unlearning steps on MUSE News where the best competing method collapses entirely.

Figures & tables

Forget 1%

Forget 5%

Forget 10%

Method

\uparrow

Prob

\downarrow

\downarrow

\uparrow

\uparrow

Prob

\downarrow

\downarrow

\uparrow

\uparrow

Prob

\downarrow

\downarrow

\uparrow

Gold (retrain)

.597

.166

.414

.655

.599

.127

.383

.676

.591

.116

.379

.677

GradAscent

.596 ±.00

.477 ±.01

.456 ±.01

.553 ±.01

.008 ±.01

.003 ±.01

.112 ±.08

.022 ±.04

.000 ±.00

.001 ±.00

.000 ±.00

GradDiff

.581 ±.01

.421 ±.04

.464 ±.02

.564 ±.01

.453 ±.00

.073 ±.00

.375 ±.01

.614 ±.01

.435 ±.00

.047 ±.00

.339 ±.01

.617 ±.00

NPO

.596 ±.00

.474 ±.01

.438 ±.02

.560 ±.01

.454 ±.01

.245 ±.01

.308 ±.01

.603 ±.01

.393 ±.03

.213 ±.01

.210 ±.01

.590 ±.02

SimNPO

.594 ±.00

.858 ±.01

.734 ±.02

.240 ±.01

.596 ±.00

.848 ±.00

.741 ±.01

.248 ±.00

.597 ±.00

.842 ±.00

.734 ±.01

.255 ±.00

Table 1: TOFU results on Llama-3.2-1B-Instruct (5 seeds). HM = hmean(MU, 1

-

Prob, 1

-

RG). Best per column in bold (excl. Gold and collapsed models).

	Books				News
Method	fk $\downarrow$	vm $\downarrow$	rk $\uparrow$	HM $\uparrow$	fk $\downarrow$	vm $\downarrow$	rk $\uparrow$	HM $\uparrow$
Gold (retrain)	.303	.145	.687	.739	.324	.204	.552	.660
GradAscent	.000 ±.00	.000 ±.00	.000 ±.00	.000 ±.00	.001 ±.00	.020 ±.02	.003 ±.00	.009 ±.01
GradDiff	.000 ±.00	.000 ±.00	.004 ±.00	.011 ±.01	.329 ±.03	.043 ±.02	.269 ±.02	.479 ±.02
NPO	.303 ±.02	.342 ±.03	.574 ±.02	.638 ±.01	.517 ±.02	.274 ±.02	.435 ±.01	.522 ±.01
SimNPO	.238 ±.02	.002 ±.00	.600 ±.01	.753 ±.01	.628 ±.01	.542 ±.01	.513 ±.01	.440 ±.00

Table 2: MUSE results (5 seeds). HM = hmean(1

-

fk, 1

-

vm, rk). Best per column in bold (excl. Gold and collapsed models).

Figure 1: KnowUndo results (5 seeds). Error bars:

\pm

1 std. Full tables in Appendix D .

Figure 2: LLM judge scores (HM

{}_{\text{J}}

) across all benchmarks and settings. Error bars:

\pm

1 std over 5 seeds. BLADE achieves the highest or near-highest HM

{}_{\text{J}}

on every setting except MUSE News, where PDU leads. KnowUndo uses Opus 4.6 as judge (Opus 4.7 safety guardrails refuse on copyrighted content).

Figure 3: Scalability and sustainability on MUSE News. PDU collapses at 4

\times

scale and after 2 sequential steps. BLADE degrades gracefully.

Benchmark	TF-IDF cosine	Vocab Jaccard	NE Jaccard
MUSE News	0.957	0.486	0.191
MUSE Books	0.391	0.329	0.266
TOFU forget01	0.172	0.035	0.006
TOFU forget05	0.347	0.130	0.032
TOFU forget10	0.449	0.220	0.047
KnowUnDo Copyright	0.441	0.389	0.121

Table 3: Corpus-level entanglement proxies between the forget and retain splits of each benchmark.

Configuration	fk $\downarrow$	vm $\downarrow$	rk $\uparrow$	HM $\uparrow$	$\Delta$ rk	FL ${}_{\text{J}}$ $\downarrow$	RA ${}_{\text{J}}$ $\uparrow$	rRQ ${}_{\text{J}}$ $\uparrow$	HM ${}_{\text{J}}$ $\uparrow$
Full BLADE	.110	.000	.658	.823	—	0.14	1.40	1.69	.814
$K\!=\!0$ (no inner loop)	.072	.000	.631	.819	$-$ .027	0.10	1.37	1.68	.811
$\tau\!=\!1.0$ (unclamped entropy)	.161	.003	.604	.780	$-$ .054	0.20	1.34	1.64	.785
No LoRA (full fine-tuning)	.002	.000	.306	.459	$-$ .352	0.00	0.70	1.25	.550
ALM off ( $\lambda\!=\!0$ , $\rho\!=\!0$ )	.000	.002	.092	.233	$-$ .566	0.01	0.24	0.28	.117
Swapped bilevel (forget inner)	.372	.643	.651	.506	$-$ .007	0.99	1.36	1.77	.655

Table 4: Ablation on MUSE Books (seed=42). Each row modifies one design choice from the full BLADE.

Figure 4: Training dynamics on MUSE Books. Top: BLADE (warm-up

\to

spike-and-ratchet

\to

smooth convergence). Bottom: PDU (recurring oscillations). Full dynamics across all benchmarks in Appendix F .

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
Model and data
$M_{\boldsymbol{\theta}}$	Model parameterized by $\boldsymbol{\theta}$
$\boldsymbol{\theta},\boldsymbol{\theta}_{0}$	Trainable / frozen pretrained parameters
$\mathcal{D}_{\text{forget}},\mathcal{D}_{\text{retain}}$	Forget / retain datasets
$\mathcal{B}_{\text{ret}},\mathcal{B}^{\prime}_{\text{ret}}$	Retain mini-batches (inner / outer)
Vocabulary and tokens

Appendix

Table 5: Symbols used in the main paper. Overloaded symbols (

r

\alpha

) marked ∗ ; meaning is fixed by context in each equation.

	TOFU 1B	TOFU 3B	MUSE	KnowUnDo
$\eta_{\text{out}}$	$5{\times}10^{-5}$	$5{\times}10^{-5}$	$3{\times}10^{-5}$	$3{\times}10^{-5}$
$\varepsilon_{\text{mul}}$	0.85	0.85	3.2	3.2
LoRA rank	8	16	16	16
LoRA $\alpha$	16	32	32	32
Safety cap $T$	250/500 †	250/500 †	250/300 ‡	150

Appendix

Table 6: Benchmark-specific BLADE hyperparameters. † T=500 for forget10. ‡ T=300 for News.

Method	LR	Eff. BS	Epochs	Key params
GradAscent	$1{\times}10^{-5}$	32	10	—
GradDiff	$1{\times}10^{-5}$	32	10	$\gamma=1.0$
NPO	$3{\times}10^{-5}$	32	10	$\beta=0.1$
SimNPO	$1{\times}10^{-5}$	32	10	$\beta=0.7$
RMU	$5{\times}10^{-5}$	32	80 steps	layer 7, $\gamma_{s}=2$
BLURNPO	see Table 8

Appendix

Table 7: Baseline hyperparameters (shared across benchmarks).

Method	Benchmark	LR	Key params
PDU	TOFU	$1{\times}10^{-5}$	$\alpha{=}100$ , $\varepsilon{=}0.3$ , dual_lr ${=}5$
	MUSE	$1{\times}10^{-5}$	$\alpha{=}50$ , $\varepsilon{=}1.5$ , dual_lr ${=}1$
	KnowUndo	$1{\times}10^{-5}$	$\alpha{=}50$ , $\varepsilon{=}1.5$ , dual_lr ${=}1$
BLURNPO	TOFU	$1{\times}10^{-5}$	$\beta=0.1$
	MUSE News	$2.5{\times}10^{-5}$	$\beta=0.05$
	MUSE Books	$1{\times}10^{-5}$	$\beta=0.4$

Appendix

Table 8: Benchmark-specific parameters for PDU and BLURNPO. All use eff. batch size 32 and 10 epochs.

Benchmark	GradAscent	GradDiff	NPO	SimNPO	RMU	BLURNPO	PDU	BLADE
TOFU 1B fgt10	1.1	2.2	5.9	4.1	1.6	6.0	2.0	12.8
TOFU 3B fgt10	3.3	15.4	23.9	20.6	3.8	27.3	15.3	38.2
MUSE Books	22.2	42.6	93.8	80.8	39.2	96.0	75.1	111.7
MUSE News	59.4	102.8	119.3	104.8	47.2	118.3	106.4	133.5
KnowUnDo Copyright	6.7	40.2	35.5	13.9	10.9	46.7	40.2	43.5
KnowUnDo Privacy	3.0	34.4	26.9	6.3	8.1	44.0	26.8	43.0

Appendix

Table 9: Average training time (minutes) across methods and benchmarks, extracted from training logs.

Benchmark	GradAscent	GradDiff	NPO	SimNPO	RMU	BLURNPO	PDU	BLADE
TOFU 1B fgt10	18.4	23.6	29.2	24.7	25.4	36.5	20.9	12.2
TOFU 3B fgt10	39.7	40.7	47.0	40.7	27.9	66.3	39.8	13.4
MUSE Books	79.2	39.5	79.2	79.2	42.1	79.2	39.5	19.3
MUSE News	79.2	39.5	79.2	79.2	42.1	79.2	39.5	19.3
KnowUnDo Copyright	77.3	39.5	79.2	78.2	32.1	79.2	39.5	14.5
KnowUnDo Privacy	76.4	39.5	79.2	77.5	30.9	79.2	39.5	14.2

Appendix

Table 10: Peak GPU memory usage (GB) across methods and benchmarks.

Split	Method	MU $\uparrow$	Prob $\downarrow$	RG $\downarrow$	HM $\uparrow$
fgt01	Gold (retrain)	.663	.179	.409	.656
	GradAscent	.668 ±.001	.564 ±.009	.525 ±.014	.509 ±.008
	GradDiff	.659 ±.003	.569 ±.019	.578 ±.027	.483 ±.020
	NPO	.668 ±.000	.553 ±.009	.494 ±.008	.525 ±.005
	SimNPO	.656 ±.002	.920 ±.007	.830 ±.021	.151 ±.012
	RMU	.657 ±.001	.861 ±.002	.715 ±.006	.246 ±.003

Appendix

Table 11: TOFU (Llama-3.2-3B), 5 seeds. Best per column in bold (excl. Gold and collapsed models).

Split	Method	FL $\downarrow$	RA $\uparrow$	rRQ $\uparrow$	HM ${}_{\text{J}}$ $\uparrow$
fgt01	GradAscent	1.335 ±.086	1.849 ±.003	1.998 ±.000	.587 ±.045
	GradDiff	1.500 ±.066	1.818 ±.017	1.997 ±.001	.490 ±.041
	NPO	1.230 ±.041	1.843 ±.004	1.998 ±.001	.640 ±.019
	SimNPO	1.825 ±.053	1.839 ±.008	1.995 ±.001	.220 ±.057
	RMU	1.585 ±.038	1.807 ±.002	1.995 ±.001	.432 ±.027
	BLURNPO	1.730 ±.122	1.853 ±.003	1.998 ±.001	.308 ±.109

Appendix

Table 12: LLM Judge on TOFU 3B (5 seeds, Claude Opus 4.7). Best HM

{}_{\text{J}}

in bold .

	Copyright						LLM Judge
Method	fgt_Acc $\downarrow$	fgt_R $\downarrow$	ret_Acc $\uparrow$	ret_R $\uparrow$	MMLU $\uparrow$	HM $\uparrow$	FL $\downarrow$	RA $\uparrow$	rRQ $\uparrow$	HM $\uparrow$
FT (target)	.854	.244	.894	.336	.439	.456	1.50	1.80	1.13	.44
Gold (retrain)	.666	.205	.890	.343	.447	.468	0.96	1.79	1.13	.62
GradAscent	.002 ±.00	.000 ±.00	.002 ±.00	.000 ±.00	.250 ±.02	.000 ±.00	0.00 ±.00	0.00 ±.00	0.00 ±.00	.00 ±.00
GradDiff	.002 ±.00	.000 ±.00	.642 ±.04	.199 ±.01	.386 ±.01	.348 ±.01	0.00 ±.00	0.81 ±.10	0.53 ±.06	.41 ±.04
NPO	.678 ±.02	.189 ±.01	.763 ±.02	.289 ±.00	.445 ±.00	.433 ±.00	0.77 ±.10	1.74 ±.02	1.15 ±.03	.66 ±.02

Appendix

Table 13: KnowUndo full results (5-fold). HM = hmean(1

-

fgt_R, ret_R, MMLU). Judge HM = hmean(1

-

FL/2, RA/2, rRQ/2). Best per column in bold (excl. Gold and collapsed models).

	Llama-3.2-1B-Instruct			Llama-3.2-3B-Instruct
Method	fgt01	fgt05	fgt10	fgt01	fgt05	fgt10
GradAscent	.786 ±.01	.028 ±.04	.000 ±.00	.800 ±.00	.760 ±.01	.000 ±.00
GradDiff	.794 ±.00	.752 ±.00	.742 ±.00	.788 ±.01	.806 ±.00	.798 ±.01
NPO	.790 ±.00	.731 ±.01	.689 ±.02	.801 ±.00	.786 ±.00	.787 ±.01
SimNPO	.684 ±.01	.648 ±.00	.657 ±.00	.616 ±.02	.557 ±.01	.588 ±.00
RMU	.790 ±.00	.794 ±.00	.824 ±.00	.708 ±.00	.803 ±.00	.836 ±.00

Appendix

Table 14: Adversarial extraction robustness (Adv_HM) on TOFU (5 seeds). Higher = more robust to paraphrase/perturbation attacks while maintaining utility.

Method	fgt01	fgt05	fgt10
BLADE	0.025	0.022	0.026
PDU	0.087	0.062	0.026
GradAscent	0.263	0.048	0.000 ∗
NPO	0.225	0.130	0.020
RMU	0.263	0.245	0.109
GradDiff	0.300	0.188	0.154

Appendix

Table 15: Jailbreak ASR (

\downarrow

) averaged over two OpenUnlearning prompt templates on TOFU (Llama-3.2-1B-Instruct). ∗ Model collapsed for GradAscent fgt10.

Method	fgt01	fgt05	fgt10
BLADE	88.4	50.5	56.5
PDU	$-$ 39.3	4.9	58.5
GradDiff	$-$ 85.0	$-$ 43.4	$-$ 32.3
GradAscent	$-$ 83.8	$-$ 24.1	$-$ 6.9
RMU	$-$ 86.8	$-$ 84.7	23.1
NPO	$-$ 88.3	$-$ 69.5	$-$ 58.7

Appendix

Table 16: MIA PrivLeak on TOFU (Llama-3.2-1B-Instruct); composite of LOSS, ZLib, Min-K% Prob, Min-K++, GradNorm, and Reference-based scores. Values close to zero or positive indicate no leakage.

Method	fgt01	fgt05	fgt10
BLADE	0.00 / 0.08	0.08 / 0.21	0.08 / 0.15
PDU	0.04 / 0.17	0.08 / 0.22	0.04 / 0.14
GradAscent	0.12 / 0.29	0.12 / 0.17	0.08 / 0.08
RMU	0.16 / 0.33	0.12 / 0.26	0.12 / 0.25
GradDiff	0.16 / 0.37	0.28 / 0.35	0.08 / 0.22
BLURNPO	0.16 / 0.33	0.36 / 0.38	0.12 / 0.24

Appendix

Table 17: GCG attack (20-token adversarial suffix, 200 steps): ASR / mean post-attack ROUGE-L on TOFU (Llama-3.2-1B-Instruct). Lower is better on both.

Panel of Figure 5 — Figure 5: Training dynamics across benchmarks (MUSE Books in main paper, Figure 4 ). Top: BLADE shows consistent three-phase convergence. Bottom: PDU exhibits recurring oscillations.

Figure 6: Inner loop ablation on MUSE News across 8

\varepsilon

-multiplier settings.

K{=}3

consistently outperforms

K{=}0

;

K{=}6

provides no additional gain.

Figure 7: Hyperparameter robustness: HM across 8 sweep values per parameter on TOFU 1B forget01 (blue), KnowUnDo Privacy (red), and MUSE Books (green). All use

K{=}3

Figure 8: Representative

\lambda

trajectories under hyperparameter sweeps. Top: TOFU. Middle: KnowUnDo Privacy. Bottom: MUSE Books. Each panel shows three values (low/mid/high) for one parameter. The dual variable adapts its trajectory to compensate for parameter changes, explaining the robustness in Figure 7 .

Table 26

Table 27

Table 28

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 343 papers

ALTER: Asymmetric LoRA for Token-Entropy-Guided Unlearning of LLMs
Mar 2, 2026Xunlei Chen, Jinyu Guo, Yuang Li +5Machine Unlearning LLM Unlearning
School of Information and Software Engineering, University of Electronic Science and Technology of China · School of Computer Science and Engineering, University of Electronic Science and Technology of China
RA $^2$ : Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning
Oct 14, 2026Zinan Ling, Yang Xiao, Ruimeng Ye +2Machine Unlearning LLM Unlearning
The University of Tulsa · Michigan State University
BalDRO: A Distributionally Robust Optimization based Framework for Large Language Model Unlearning
Jan 14, 2026Pengyang Shao, Naixin Zhai, Lei Chen +4Distributionally Robust Optimization LLM Unlearning
National University of Singapore Singapore · University of Science and Technology of China Hefei, China · Hefei University of Technology Hefei, China

Mar 2, 2026J/K move · Enter open · S save

ALTER: Asymmetric LoRA for Token-Entropy-Guided Unlearning of LLMs

Xunlei Chen, Jinyu Guo, Yuang Li, Zhaokun Wang+4

School of Information and Software Engineering, University of Electronic Science and Technology of China · School of Computer Science and Engineering, University of Electronic Science and Technology of China

Machine Unlearning LLM Unlearning Low-Rank Adaptation

Open paper AskPDF arXiv

Large language models (LLMs) have advanced to encompass extensive knowledge across diverse domains. Yet controlling what a LLMs should not know is important for ensuring alignment and thus safe use. However, effective unlearning in LLMs is difficult due to the fuzzy boundary between knowledge retention and forgetting. This challenge is exacerbated by entangled parameter spaces from continuous multi-domain training, often resulting in collateral damage, especially under aggressive unlearning strategies. Furthermore, the computational overhead required to optimize State-of-the-Art (SOTA) models with billions of parameters poses an additional barrier. In this work, we present ALTER, a lightweight unlearning framework for LLMs to address both the challenges of knowledge entanglement and unlearning efficiency. ALTER operates through two phases: (I) high entropy tokens are captured and learned via the shared A matrix in LoRA, followed by (II) an asymmetric LoRA architecture that achieves a specified forgetting objective by parameter isolation and unlearning tokens within the target subdomains. Serving as a new research direction for achieving unlearning via token-level isolation in the asymmetric framework. ALTER achieves SOTA performance on TOFU, WMDP, and MUSE benchmarks with over 95% forget quality and shows minimal side effects through preserving foundational tokens. By decoupling unlearning from LLMs' billion-scale parameters, this framework delivers excellent efficiency while preserving over 90% of model utility, exceeding baseline preservation rates of 47.8-83.6%.

Abstract

Machine Unlearning

Explore similar work

ALTER: Asymmetric LoRA for Token-Entropy-Guided Unlearning of LLMs

RA2^22: Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning

BalDRO: A Distributionally Robust Optimization based Framework for Large Language Model Unlearning

RA $^2$ : Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning