ALTER: Asymmetric LoRA for Token-Entropy-Guided Unlearning of LLMs
Authors: Xunlei Chen, Jinyu Guo, Yuang Li, Zhaokun Wang, Yi Gong, Jie Zou, Jiwei Wei, Wenhong Tian
Organizations: School of Information and Software Engineering, University of Electronic Science and Technology of China · School of Computer Science and Engineering, University of Electronic Science and Technology of China
Large language models (LLMs) have advanced to encompass extensive knowledge across diverse domains. Yet controlling what a LLMs should not know is important for ensuring alignment and thus safe use. However, effective unlearning in LLMs is difficult due to the fuzzy boundary between knowledge retention and forgetting. This challenge is exacerbated by entangled parameter spaces from continuous multi-domain training, often resulting in collateral damage, especially under aggressive unlearning strategies. Furthermore, the computational overhead required to optimize State-of-the-Art (SOTA) models with billions of parameters poses an additional barrier. In this work, we present ALTER, a lightweight unlearning framework for LLMs to address both the challenges of knowledge entanglement and unlearning efficiency. ALTER operates through two phases: (I) high entropy tokens are captured and learned via the shared A matrix in LoRA, followed by (II) an asymmetric LoRA architecture that achieves a specified forgetting objective by parameter isolation and unlearning tokens within the target subdomains. Serving as a new research direction for achieving unlearning via token-level isolation in the asymmetric framework. ALTER achieves SOTA performance on TOFU, WMDP, and MUSE benchmarks with over 95% forget quality and shows minimal side effects through preserving foundational tokens. By decoupling unlearning from LLMs' billion-scale parameters, this framework delivers excellent efficiency while preserving over 90% of model utility, exceeding baseline preservation rates of 47.8-83.6%.
Figures & tables
Figure 1: (a) The impact of corpus heterogeneity on the performance of FT/PEFT. (b) The chaos in the LoRA parameter space caused by corpus heterogeneity in the WMDP dataset.
Figure 2: Conceptual illustration of our unlearning framework. After achieving explicit parameter isolation with the AsymLoRA structure, word clouds from the WMDP dataset show that task-specific forgetting experts Bi and the retention expert Br process low entropy tokens (left), whereas the shared matrix A processes high entropy tokens (right).
Figure 3: Architecture and workflow of our unlearning framework. During fine-tuning, ALTER first automatically identifies and initializes N intrinsic components (without requiring domain-specific knowledge). Then, guided by entropy, the architecture uses a trainable MoE router that treats each intrinsic component as an expert, automatically assigning training samples to the corresponding component. High entropy tokens (red fire), inherently adapted to the shared A matrix, are processed jointly, while low entropy tokens (blue fire) are directed to specialized B experts for fine-tuning. During inference, ALTER dynamically combines multiple B matrices using the trained router for flexible and adaptive unlearning.
Model/Category
FT & Model Editing
LoRA Variations (r=8)
Llama3-8B
Base
RMU
ELM
GA
RL
NPO
NPO_KL
NPO_GD
LoRA
AsymLoRA
Ours
WMDP-Bio↓
71.2
49.4
33.3
23.3
24.7
58.1
64.3
56.2
28.7
25.7
24.4
WMDP-Cyber ↓
45.3
37.0
26.6
24.0
26.6
34.4
41.3
33.1
32.1
28.8
25.6
MMLU↑
62.1
40.1
57.2
24.8
23.0
50.1
56.0
51.9
39.6
55.3
57.8
Flu-mean ↑
2.97
2.96
3.07
1.00
1.00
3.07
2.97
3.03
2.23
2.23
3.46
Flu-var↓
1.91
1.88
2.18
0.00
0.00
1.86
1.96
2.08
1.42
1.42
1.17
Table 1: Multiple-choice accuracy of five LLMs on the WMDP benchmark (forget) and the full MMLU (retain) after unlearning. Our method enables model-agnostic unlearning, reducing WMDP accuracy to around 25% while preserving MMLU performance, and achieves stable forgetting with superior fluency by avoiding entangled errors in knowledge domains.
Figure 4: Utility-forgetting trade-off at 1%/5%/10% unlearning ratios for Llama2-7B (top) and Llama3-8B (bottom). GradDiff/Ascent and KLMin show low forgetting efficacy or severe utility loss. NPO incurs utility drops. Standard LoRA maintains utility but minimal forgetting gain. Our AsymLoRA/ALTER achieve near-complete forgetting with Retain-matched utility.
Figure 5: Average model utility of baselines across Sequential Unlearning rounds for TOFU-injected Llama3-8B, with the forgetting set expanded from 1% to 10%.
Method
Harry Potter
Forget Perf.
Retain Perf.
BLEU
R-L
ASG↓
MMLU↑
Ful.↑
Original
74.8
85.1
74.1
46.3
4.0
Retain
1.9
9.8
0
47.8
2.8
Fine-tune
6.4
17.2
5.9
46.0
1.9
GA
0
0
6.0
26.9
1.0
Table 2: Performance on HarryPotter dataset. R-L and Ful. denote the ROUGE-L score and fluency-mean, respectively. A-LoRA denotes the AsymLoRA. Instead, average similarity gap (ASG) serves as the target for forget performance.
Figure 6: Trade-off between forget quality and relative training time for Llama2-7B on TOFU-10%. The top-left corner indicates better forgetting performance and efficiency.
Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.
Existing LLM unlearning methods struggle with robustness: unbounded forget losses degrade model coherence, fixed-weight balancing cannot adapt as retain difficulty shifts mid-training, and methods that work on one benchmark falter under scaling or repeated application. We propose BLADE, a constrained bilevel framework whose three mechanisms give smooth, predictable control over the optimization landscape: a clamped-entropy forget loss whose gradient is exactly zero once a token reaches sufficient uncertainty; an asymmetric augmented Lagrangian that permanently ratchets retain protection after any violation; and a bilevel structure confined to LoRA adapters that repairs retain damage before each forgetting step. BLADE dominates across three benchmark families, improving average composite scores over the strongest baselines by 6% on TOFU, 9% on MUSE Books, and 7% on KnowUndo, and it remains stable under 4× scaling and 4 sequential unlearning steps on MUSE News where the best competing method collapses entirely.
Md Toufikuzzaman, Ahmad Mousavi, Dongwon Lee
The Pennsylvania State University, USA · American University
Large Language Models (LLMs) often memorize sensitive or harmful information, necessitating effective machine unlearning techniques. While existing parameter-efficient unlearning methods have shown promise, they still struggle with the forget-retain trade-off. This can be attributed to their reliance on parameter importance metrics to identify parameters that are important exclusively for the forget set, which is fundamentally limited by the superposition phenomenon. Due to the polysemantic nature of LLM parameters, such an importance metric may struggle to disentangle parameters associated with the forget and retain sets. In this work, we propose Representation-Guided Low-rank Unlearning (REGLU), a novel approach that leverages the geometric properties of representation spaces to achieve robust and precise unlearning. First, we develop a representation-guided initialization for LoRA that identifies the optimal subspace for selective forgetting. Second, we introduce a regularization loss that constrains the outputs of the LoRA update to lie in the orthogonal complement of the retain set's representation subspace, thereby minimizing interference with the model's performance on the retain set. We evaluate REGLU on the TOFU and WMDP benchmarks across multiple models. Our results demonstrate that REGLU consistently outperforms state-of-the-art baselines, achieving superior unlearning quality while maintaining higher model utility.
Zeguan Xiao, Lang Mo, Yun Chen +4
Shanghai University of Finance and Economics · Southern University of Science and Technology · Deepexi Technology Co. Ltd.
Large Language Models (LLMs) inevitably internalize substantial amounts of sensitive or private information during pre-training, while LLM unlearning aims to selectively erase specific knowledge to prevent privacy leakage with minimal loss of model utility. However, existing methods struggle to balance forget quality with utility, and typically incur substantial computational costs due to parameter fine-tuning. To address this, we propose Nullify, a training-free, non-destructive activation steering method for LLM unlearning. Nullify employs steering vectors during inference to redirect privacy-related activations away from their memorized answers, while satisfying a null-space constraint that leaves retained-query activations essentially unaffected to maintain utility. Evaluations on TOFU and MUSE show that Nullify matches or surpasses established baselines in forget quality while achieving near-lossless preservation of model utility. By avoiding weight updates entirely, Nullify serves as an efficient, plug-and-play inference-time intervention framework.
Wei Zhai, Xiang Liu, Qiang Huang +6
College of Computer Science and Artificial Intelligence, Fudan University · BEDI Cloud · School of Electrical and Information Engineering, Tianjin University +3