Preserving Utility in Continual LLM Unlearning via Orthogonal Projections

Juan Belieni; Ana Carolina Erthal; Eliezer de Souza da Silva; Diego Mesquita

cs.LGOct 29, 2026

Preserving Utility in Continual LLM Unlearning via Orthogonal Projections

Authors: Juan Belieni, Ana Carolina Erthal, Eliezer de Souza da Silva, Diego Mesquita

Organizations: School of Applied Mathematics, Getulio Vargas Foundation · University of Coimbra, CISUC/LASI, DEI

Abstract

Unlearning concerns the task of removing the influence of a sensitive portion of the training data, such as protected, copyrighted or private text, from a model without retraining it from scratch. While unlearning requests happen continuously during a model's lifetime, the vast majority of LLM unlearning methods are ill-suited for iterated removal, yielding steep decay in forgetting and utility upon sequential unlearning. This decay stems from interference between successive updates, as each new deletion can undo earlier ones or push the model further along directions already degraded by them. To address this, we introduce Continual Orthogonal Projection (COP), a gradient-projection scheme that restricts each update to the orthogonal complement of activation subspaces cached from prior forget sets. This approach provably guarantees first-order non-interference with earlier deletions. COP stores only a low-rank basis per layer, requiring neither past gradients nor previous forget sets, and plugs into unlearning methods that update the model through gradient steps on a differentiable loss without modifying its objective. Experiments on standard LLM unlearning benchmarks (TOFU and RWKU) show that COP preserves utility across sequential requests without weakening forgetting, preventing the collapse of standard objectives and forgetting more effectively than continual-specific baselines.

Figures & tables

Figure 1: Overview of COP for one linear layer with weights

W

. The input activations of earlier forget sets are summarized by a compact basis

V_{k-1}

, which marks the directions where earlier forgetting lives. At request

k

, the gradient of the unlearning objective on

\mathcal{F}_{k}

, optionally combined with a retain term on

\mathcal{R}

, is projected away from these directions, so the update cannot change how the layer responds to earlier forget inputs. After training, the dominant new activation directions of

\mathcal{F}_{k}

are compressed and added to the basis, giving

V_{k}

for the next request.

Figure 2: COP preserves utility as forget requests accumulate. Model utility on TOFU ( forget05 ) with Llama-3.2-1B-Instruct after each of the five forget requests, for each base objective (dashed, hollow markers) and its COP-augmented variant (solid, filled markers). The dotted line marks the utility of the original model before unlearning. Utility under COP stays close to its early level as requests accumulate, whereas base GA and IDK collapse and base NPO and SimNPO fall increasingly behind their COP variants in later requests.

Method	Task 1				Task 2				Task 3				Task 4				Task 5
	FR $\downarrow$	ES $\downarrow$	EM $\downarrow$	MU $\uparrow$	FR $\downarrow$	ES $\downarrow$	EM $\downarrow$	MU $\uparrow$	FR $\downarrow$	ES $\downarrow$	EM $\downarrow$	MU $\uparrow$	FR $\downarrow$	ES $\downarrow$	EM $\downarrow$	MU $\uparrow$	FR $\downarrow$	ES $\downarrow$	EM $\downarrow$	MU $\uparrow$
Original	0.92	0.81	0.98	0.599	0.91	0.83	0.98	0.599	0.81	0.73	0.97	0.599	0.82	0.72	0.97	0.599	0.83	0.73	0.97	0.599
GA	0.47	0.10	0.71	0.524	0.40	0.09	0.66	0.467	0.30	0.06	0.52	0.288	0.25	0.05	0.46	0.274	0.17	0.03	0.30	0.159
GA+COP	0.47	0.10	0.71	0.526	0.43	0.10	0.69	0.521	0.35	0.08	0.65	0.493	0.35	0.08	0.63	0.478	0.34	0.07	0.63	0.461
NPO	0.44	0.09	0.67	0.584	0.38	0.09	0.67	0.568	0.35	0.09	0.65	0.535	0.34	0.08	0.64	0.529	0.32	0.08	0.63	0.496
NPO+COP	0.40	0.06	0.57	0.575	0.36	0.06	0.57	0.565	0.32	0.05	0.55	0.560	0.31	0.05	0.55	0.546	0.30	0.05	0.56	0.532

Table 1: TOFU results on Llama-3.2-1B-Instruct ( forget05 ). After each task, forget metrics are computed over all requests seen so far, and model utility (MU) is measured on the retain set and general utility datasets. Bold marks the higher MU within each pair of a base objective and its COP-augmented variant. From the third request onward, every COP variant retains higher utility than its base objective. COP prevents the utility collapse of GA and IDK, whose base versions reach lower EM only as part of that collapse, while forget metrics for NPO and SimNPO remain comparable with and without COP. O 3 attains the highest utility but forgets weakly, with EM close to that of the original model. ALKN’s forget metrics fall together with its utility, which collapses as requests accumulate.

Method	Task 1				Task 2				Task 3				Task 4				Task 5
	FR $\downarrow$	ES $\downarrow$	EM $\downarrow$	MU $\uparrow$	FR $\downarrow$	ES $\downarrow$	EM $\downarrow$	MU $\uparrow$	FR $\downarrow$	ES $\downarrow$	EM $\downarrow$	MU $\uparrow$	FR $\downarrow$	ES $\downarrow$	EM $\downarrow$	MU $\uparrow$	FR $\downarrow$	ES $\downarrow$	EM $\downarrow$	MU $\uparrow$
Original	0.97	0.93	0.99	0.667	0.96	0.93	0.99	0.667	0.90	0.87	0.99	0.667	0.92	0.88	0.99	0.667	0.93	0.89	0.99	0.667
GA	0.55	0.23	0.80	0.641	0.34	0.11	0.66	0.517	0.19	0.05	0.35	0.380	0.14	0.04	0.23	0.393	0.07	0.03	0.12	0.224
GA+COP	0.53	0.24	0.80	0.641	0.52	0.20	0.80	0.634	0.43	0.18	0.77	0.616	0.41	0.15	0.74	0.594	0.38	0.12	0.71	0.566
NPO	0.36	0.07	0.60	0.658	0.30	0.07	0.62	0.640	0.24	0.07	0.57	0.592	0.23	0.06	0.54	0.548	0.21	0.05	0.52	0.456
NPO+COP	0.37	0.07	0.59	0.614	0.37	0.07	0.57	0.603	0.32	0.06	0.55	0.593	0.33	0.05	0.55	0.583	0.33	0.06	0.56	0.571

Table 2: TOFU results on Llama-3.2-3B-Instruct ( forget05 ), with the same metrics and notation as Table 1 . COP improves utility at every request for SimNPO and IDK, from the second request onward for GA, and from the third onward for NPO. Base GA and IDK drive forget ROUGE close to zero only as their utility collapses, while their COP variants remain stable at the cost of higher EM. O 3 attains the highest utility but fails to forget effectively, with EM above 0.9 at every request. ALKN’s utility falls sharply as requests accumulate.

Figure 3: COP maintains higher utility on real-world knowledge unlearning. Model utility on RWKU with Llama-3.2-3B-Instruct after each forget request, for each base objective (dashed, hollow markers) and its COP-augmented variant (solid, filled markers). Utility under COP remains stable as requests accumulate, while base objectives fluctuate or degrade.

Method	Task 1				Task 2				Task 3				Task 4				Task 5
	FR $\downarrow$	FP $\downarrow$	EM $\downarrow$	MU $\uparrow$	FR $\downarrow$	FP $\downarrow$	EM $\downarrow$	MU $\uparrow$	FR $\downarrow$	FP $\downarrow$	EM $\downarrow$	MU $\uparrow$	FR $\downarrow$	FP $\downarrow$	EM $\downarrow$	MU $\uparrow$	FR $\downarrow$	FP $\downarrow$	EM $\downarrow$	MU $\uparrow$
Original	0.78	0.34	0.70	0.497	0.73	0.31	0.67	0.497	0.74	0.30	0.66	0.497	0.75	0.32	0.68	0.497	0.72	0.32	0.66	0.497
GA	0.34	0.00	0.36	0.569	0.20	0.02	0.26	0.484	0.08	0.01	0.21	0.482	0.03	0.00	0.11	0.004	0.02	0.00	0.09	0.366
GA+COP	0.33	0.00	0.36	0.571	0.24	0.02	0.28	0.571	0.18	0.01	0.26	0.544	0.19	0.02	0.25	0.498	0.15	0.01	0.21	0.485
NPO	0.28	0.03	0.39	0.706	0.24	0.03	0.37	0.730	0.19	0.02	0.34	0.677	0.16	0.01	0.32	0.630	0.12	0.01	0.30	0.603
NPO+COP	0.04	0.01	0.32	0.689	0.07	0.00	0.27	0.659	0.05	0.00	0.23	0.641	0.05	0.00	0.23	0.619	0.04	0.00	0.20	0.605

Table 3: RWKU results on Llama-3.2-3B-Instruct . After each task, forget metrics are computed over all requests seen so far, and model utility (MU) is measured on the retained subjects and world facts. Bold marks the higher MU within each pair of a base objective and its COP-augmented variant. COP keeps utility stable across requests, while O 3 forgets at the cost of most of its utility and ALKN retains much of the targeted knowledge.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Larger protected subspaces improve utility without weakening forgetting. Final model utility (left) and extraction strength (right) on TOFU with Llama-3.2-1B-Instruct, as a function of the COP rank cap

r_{\max}

. “Base” denotes each objective without COP, and dotted lines mark the original model. Utility grows with rank for all methods, sharply for GA and IDK and moderately for NPO and SimNPO, while extraction strength remains far below original model at every rank.

LLM Unlearning

Large language model unlearning studies how to remove specified knowledge or behaviors while preserving a model’s unrelated capabilities.

See all 192 papers

Trajectory-Guided Forget-Recover Network for Continual LLM Unlearning
Aug 4, 2026Zezheng Wu, Xinghe Cheng, Qinggang Zhang +4Machine Unlearning LLM Unlearning
Guilin University of Electronic Technology · Jinan University · Jilin University +2
Towards Mitigating Excessive Forgetting in LLM Unlearning via Entanglement-Guidance with Proxy Constraint
Aug 28, 2025Zhihao Liu, Jian Lou, Yuke Hu +6Machine Unlearning LLM Unlearning
Zhejiang University · Sun Yat-sen University · UNC Greensboro +1
Nullify: Null-Space Activation Steering for Training-Free LLM Unlearning
Oct 7, 2026Wei Zhai, Xiang Liu, Qiang Huang +6LLM Unlearning Privacy Leakage in Language Models
College of Computer Science and Artificial Intelligence, Fudan University · BEDI Cloud · School of Electrical and Information Engineering, Tianjin University +3

Aug 4, 2026J/K move · Enter open · S save

Trajectory-Guided Forget-Recover Network for Continual LLM Unlearning

Zezheng Wu, Xinghe Cheng, Qinggang Zhang, Haoran Luo+3

Guilin University of Electronic Technology · Jinan University · Jilin University+2

Machine Unlearning LLM Unlearning Catastrophic Forgetting

Open paper AskPDF arXiv

Machine unlearning aims to eliminate the influence of sensitive data on a model. In the real world, unlearning requests arrive continually, which gives rise to two challenges. First, an unlearning intervention may redistribute target-related computation across remaining pathways, allowing previously forgotten knowledge to re-emerge. Second, repeated unlearning interventions may progressively reduce the model capacity needed to preserve retained utility. To address these challenges, we propose the Trajectory-guided Forget-Recover Network (TFR-Net). TFR-Net tracks channel-level risk across requests. It separates persistent target-related channels from transient hotspots and suppresses only the persistent ones. TFR-Net also recovers model capacity by reactivating dormant channels. These channels make strong contributions to retained utility and show low current and historical forget risk. The recovery is accepted only when retained-utility degradation remains within a predefined tolerance. Experiments on four datasets show that TFR-Net consistently achieves a more favorable trade-off between unlearning effectiveness and retained utility than representative baselines.

Abstract

LLM Unlearning

Explore similar work

Trajectory-Guided Forget-Recover Network for Continual LLM Unlearning

Towards Mitigating Excessive Forgetting in LLM Unlearning via Entanglement-Guidance with Proxy Constraint

Nullify: Null-Space Activation Steering for Training-Free LLM Unlearning