Storage Is Not Strategy: State-Conditioned Support Control for LLM Unlearning

Tianhao Qian; Ziming Hong; Chongyang Gao; Kezhen Chen; Lixu Wang

cs.LGSep 29, 2026

Storage Is Not Strategy: State-Conditioned Support Control for LLM Unlearning

Authors: Tianhao Qian, Ziming Hong, Chongyang Gao, Kezhen Chen, Lixu Wang

Organizations: Southeast University (seu.edu.cn) · University of Sydney (usyd.edu.au) · Northwestern University (northwestern.edu) · Together AI (together.ai) · The Chinese University of Hong Kong Shenzhen (cuhk.edu.cn)

Abstract

Many localized large language model (LLM) unlearning methods select a small parameter subset from a localization signal and keep it fixed during optimization. The parameters most associated with a target, however, need not be the best ones to update, and candidate interventions can change value as optimization proceeds. In a controlled experiment, a storage-localization score reaches an area under the receiver operating characteristic curve (AUROC) of 0.981, yet storage identity agrees with the better intervention on only 17/36 targets, while low-rank adaptation (LoRA) wins 35/36. We introduce Intervention Score, which ranks editable groups by the predicted effect of the actual unlearning update while accounting for collateral damage, and use it to form the static intervention-value baseline (Static-IV). We then introduce selective dynamic intervention re-ranking (DIR-R), which revisits that subset only when a calibrated probe justifies the comparison. On the Natural-TOFU dataset, our method has positive descriptive margins in 19/20 comparisons between methods and objectives, although several are near zero. On the LACUNA localization-precision benchmark, our mean terminal utility is higher in all six negative preference optimization (NPO) and SimNPO comparisons: NPO margins range from +0.431 to +0.848, and SimNPO margins range from +0.503 to +0.571. The gradient-difference (GradDiff) objective reveals substantial field dependence. Relative to Static-IV, the primary four-field GradDiff evaluation has six wins, six ties, and no losses, with mean and median paired gains of +0.165 and +0.0025. The evidence supports separating localization, initial intervention selection, and checkpoint-dependent support revision.

Figures & tables

Quantity	Result
Storage AUROC	$0.98148$
Residualized storage AUROC	$0.92593$
Correct within-pair storage direction	$17/18$
LoRA intervention wins	$35/36$
Storage/best-intervention agreement	$17/36$ ( $47.2\%$ )
LoRA $-$ backbone paired difference	$+0.1905$ $[0.1570,0.2242]$

Table 1: Controlled separation between localization and intervention.

Baseline	NPO $\Delta J$	SimNPO $\Delta J$
Random	$+0.004799$	$+0.006974$
Activation	$+0.041006$	$+0.154743$
WAGLE	$+0.136450$	$+0.297768$
Storage	$+0.032802$	$+0.162268$
MemFlex	$+0.094780$	$+0.210191$
FILA-relative Fisher	$+0.003390$	$-0.000799$

Table 2: Natural-TOFU margins (

\Delta J

); positive values favor ours.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning	Symbol	Meaning
$f_{{\bm{\theta}}}$ , ${\bm{\theta}}_{0}$	Model and pre-unlearning parameters.	$T$ , $t$	Total steps and DIR-R decision checkpoint.
$\mathcal{D}_{f},\mathcal{D}_{r}$	Forget and retain sets.	$\mathcal{L}_{\mathrm{unl}}$	Fixed unlearning objective.
${\mathbb{G}}$ , $g$	Editable groups and one group.	${\bm{\theta}}_{g}$	Parameter coordinates in group $g$ .
${\mathbb{S}}$	Active editable support.	$c({\mathbb{S}})$ , $B$	Editable-parameter cost and budget.
$\mathcal{F}_{B}$	Supports satisfying $c({\mathbb{S}})\leq B$ .	$J({\bm{\theta}})$	Forgetting and retention evaluation utility.
$V_{0}({\mathbb{S}})$	Value of using ${\mathbb{S}}$ from the start.	${\mathbb{S}}_{0}^{\star}$ , ${\mathbb{S}}_{0}$	Ideal and actually selected initial supports.

Appendix

Table 5: Core notation used in the problem formulation and method.

Baseline	NPO mean $J$	SimNPO mean $J$
Random	5.105164	1.035321
Activation exact	6.174900	1.586493
WAGLE	6.253223	1.470818
Storage	5.839900	1.489109
MemFlex	6.374923	1.745046
FILA	5.362761	1.605025

Appendix

Table 7: LACUNA NPO/SimNPO parameter-selection comparison. Each mean for a method and objective uses

n=12

unique runs after deduplication.

Field	Static-IV $J$	DIR-R $J$	DIR-R $-\textsc{Static-IV}$	Triggered
Birth City	90.536120	90.737786	$+0.201667$	1/3
Driver’s License	111.466042	111.839375	$+0.373333$	2/3
Email Address	118.499479	118.559479	$+0.060000$	1/3
Phone Number	89.353444	89.376777	$+0.023333$	2/3
Overall	102.463771	102.628354	$+0.164583$	6/12

Appendix

Table 10: GradDiff primary four-field DIR-R decomposition. The final column gives triggered runs among the three final seeds.

Quantity	Value
Frozen threshold $\tau$	0.021250
Calibration instances	12
Triggered instances	5/12
Trigger rate	41.67%
Trigger precision	1.000
Captured positive gain	0.9025

Appendix

Table 11: GradDiff primary calibration summary. Gain captured is the fraction of total positive available gain over the frozen continuation candidate set.

Objective	Baseline	Baseline $J$	DIR-R $J$	DIR-R $-C$	Wins/Losses
NPO	Projected Causal	5.763156	6.193908	+0.430752	8/4
NPO	MemFlex	5.502749	6.193908	+0.691159	10/2
NPO	Activation exact	5.346050	6.193908	+0.847858	9/3
SimNPO	MemFlex	1.271154	1.776698	+0.505544	11/1
SimNPO	FILA-relative Fisher	1.205905	1.776698	+0.570793	8/4
SimNPO	Activation-Down	1.273278	1.776698	+0.503420	12/0

Appendix

Table 14: LACUNA comparison of Intervention Score + DIR-R with the selected baselines.

Explore similar work

Null-Space Constrained Low-Rank Adaptation for Response-Specified Large Language Model Unlearning
Jun 9, 2026Bocheng Ju, Jianhua Wang, Chengliang Liu +1Large Language Model Unlearning Exact Unlearning
Beijing Key Laboratory of Security and Privacy in Intelligent Transportation, Beijing Jiaotong University, P.R.China · College of Computer Science and Technology, Taiyuan University of Technology, Taiyuan, China, 030024 · Institute of Computing Technologies, China Academy of Railway Sciences Corporation Limited, Beijing 100081, China
Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning
Oct 6, 2025Kai Qin, Jiaqi Wu, Jianxiang He +8Large Language Model Unlearning Forgetting
1Tsinghua University · 2The Hong Kong University of Science and Technology · 3Jianghuai Advanced Technology Center +1
Randomized Antipodal Search Done Right for Data Pareto Improvement of LLM Unlearning
Apr 17, 2026Ziwen Liu, Huawei Lin, Yide Ran +5Large Language Model Unlearning Pareto Frontier
Rice University · Rochester Institute of Technology · Stevens Institute of Technology +1

Jun 9, 2026, cs.AIJ/K move · Enter open · S save

Null-Space Constrained Low-Rank Adaptation for Response-Specified Large Language Model Unlearning

Bocheng Ju, Jianhua Wang, Chengliang Liu, Xiaolin Chang

Beijing Key Laboratory of Security and Privacy in Intelligent Transportation, Beijing Jiaotong University, P.R.China · College of Computer Science and Technology, Taiyuan University of Technology, Taiyuan, China, 030024 · Institute of Computing Technologies, China Academy of Railway Sciences Corporation Limited, Beijing 100081, China

Large Language Model Unlearning Exact Unlearning Suppression Approximate Nearest-Neighbor Search

Open paper AskPDF arXiv

Large language model unlearning aims to suppress designated undesirable knowledge while preserving benign capabilities. Many unlearning objectives focus on suppressing undesired answers, while recent target-guided variants specify replacement behavior but still leave update locality largely unconstrained. This paper introduces \emph{Null-Space Constrained Response-Specified Unlearning} (NSRU), a projection-constrained low-rank framework for controlled LLM unlearning. NSRU uses an explicitly structured safe target response to specify the desired behavior for each forget query, while suppressing the original undesired content. To localize adaptation, NSRU estimates per-module retain subspaces from benign hidden representations and uses an orthogonal-projected low-rank parameterization to confine LoRA updates to the null space of the retain subspace. The resulting objective jointly optimizes safe-target learning, undesired-response suppression, and retention preservation under this constrained parameterization. We provide a local first-order analysis showing that the projected update reduces retain-side perturbations while preserving editable directions for shaping forget-query behavior. Experiments on TOFU show that NSRU effectively suppresses extractable forget-set knowledge while improving retain QA performance, model utility, and safe-target alignment over representative baselines. On WMDP, NSRU keeps hazardous-domain accuracy near the random-choice region while preserving broad and domain-adjacent MMLU utility. Ablation studies support the complementary roles of safe-target supervision, undesired-response suppression, retention loss, and null-space projected updates, while sensitivity and robustness analyses indicate stable behavior across the tested hyperparameter and prompt variations.

Abstract

Explore similar work

Null-Space Constrained Low-Rank Adaptation for Response-Specified Large Language Model Unlearning

Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning

Randomized Antipodal Search Done Right for Data Pareto Improvement of LLM Unlearning