Unlearning aims to make a large language model behave like one retrained without its forget set. Prevailing likelihood-suppression objectives, however, lack an explicit forgetting target: they continually reward lower forget-set likelihood, making forgetting sensitive to the optimization budget. We address this limitation by estimating the direction toward the oracle without training it. Our key idea is to shrink the fine-tuned model toward its pre-fine-tuning weights and briefly re-learn the retain set, thereby removing the forget-specific contribution; in the linearized regime, this shrink-and-heal lookahead moves the model a fixed fraction of the way to the retrained model, its unique fixed point. Our method, SHIFT, extrapolates the lookahead's per-token changes into distillation targets and iteratively refines them. Evaluated against eleven baselines under a unified OpenUnlearning protocol, SHIFT achieves the highest aggregate score across all TOFU settings and the closest oracle alignment, 3.7x closer on average than the next-best method; on MUSE, it preserves the most retain knowledge. Under larger optimization budgets, baselines overshoot the oracle, while SHIFT remains near it, even under relearning attacks.
Figures & tables
Figure 1: (a) Privacy leakage on TOFU forget05 (Llama-3.2-1B) across learning rates under a fixed 10-epoch budget ( 0 : oracle; positive: the forget set looks less familiar than never-seen data). Likelihood-suppression methods overshoot the oracle, whereas Shift stabilizes near it. (b) In the linearized regime (Prop. 1 ), shrinking θ0 toward the base θb and healing on the retain set removes a fraction ρ of the forget-specific contribution; Shift extrapolates and iterates this step, which stops at the oracle θr , whereas likelihood suppression has no reason to stop there.
Figure 2: One round of Shift . (1) Shrink toward the pre-fine-tuning weights and re-fit the retain set. (2) On forget answers, the lookahead lowers the memorized token and raises the alternatives the retain set supports. (3) Extrapolating by s/ρ gives the teacher, the oracle’s next-token distribution in the linearized regime. (4) Distill it on forget tokens, anchored to θ0 on retain data; the next round looks ahead from the new model.
Figure 3: What happens when training continues, on TOFU forget05 (Llama-3.2-1B, learning rate 2⋅10−5 ): the baselines train for 20 epochs (warm-up, then linear decay over the 20 epochs) and Shift for 8 rounds. The dashed line marks the end of a standard 10-epoch run (step 60); Shift ’s standard 4 rounds end at step 48. Top: distance to the oracle (log scale); bottom: privacy leakage (0 is the oracle, positive is over-forgetting). Shift : mean over three seeds, shaded from minimum to maximum; baselines: one run, evaluated every second epoch.
Figure 4: Relearning attacks of increasing strength on Llama-3.2-1B forget05 : distance to the equally attacked oracle (mean of three attack seeds). Shift is the closest at every strength (results for all baselines refer to Appendix Table 11 ).
Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.
Machine unlearning for large language models (LLMs) aims to selectively remove memorized content such as private data, copyrighted text, or hazardous knowledge, without costly full retraining. Most existing methods require a retain set of curated examples to prevent catastrophic degradation of general model utility, creating an extra data dependency that complicates deployment. We propose SHRED (Self-distillation via High-surprisal-only Retain-set-free Entropy Demotion), a retain-set-free unlearning method built on a key insight: not all tokens within a forget set instance carry memorized information equally. High-information tokens concentrate the model's memorized knowledge, while low-information tokens reflect general language competence. SHRED operates in two stages. (1) Selection: We perform a forward pass on a forget set instance, collect per-token autoregressive probabilities, and select the bottom (lowest probability, highest Shannon information) as forget positions; the remaining positions are retained as benign anchors. (2) Training: We construct modified KL targets that demote the memorized token's logit at forget positions while preserving the original distribution at benign positions. The model is then trained via a single top KL self-distillation objective that simultaneously drives forgetting and utility preservation. We evaluate SHRED across four standard unlearning benchmarks and demonstrate that it establishes a new Pareto-optimal trade-off between forget efficacy and model utility, outperforming retain-set-dependent methods. Our analysis shows that SHRED is robust against relearning attacks and membership-inference attacks, and it maintains stable utility even after many sequential unlearning runs.
Zizhao Hu, Ameya Godbole, Johnny Tian-Zheng Wei +3
University of Southern California · USC Information Sciences Institute
Language-model unlearning updates a trained model to behave as if it had not seen selected training examples, while preserving utility and avoiding costly retraining. Existing approaches typically fine-tune the pretrained model with a fixed training budget and select the final model afterwards by evaluating several saved checkpoints on downstream validation data. Two sources of unnecessary computation limit scalability: training beyond the desired forget-retain trade-off, and checkpoint selection that requires extra storage and repeated evaluations. To address these limitations, we introduce MArgin Self-Correction (MASC), an efficient unlearning method with an online stopping rule that does not require downstream evaluation. Given a text sequence to be forgotten, MASC actively reduces the logit gap between the original next token and the most likely alternatives. It outputs a final model once this gap is small on average over a sufficiently large proportion of token positions across all forget sequences. On TOFU, MUSE News, and MUSE Books, MASC achieves a competitive forget-retain trade-off at a fraction of the computational cost of existing baselines. We further observe that as we increase model size (a.k.a. number of parameters), the trade-offs improve for both MASC and SimNPO -- the forget metrics remain comparable while retain utility increases.
Federico Di Gennaro, Alexander Shevchenko, Fanny Yang
Unlearning aims to remove selected knowledge from large language models to mitigate privacy and safety risks while preserving useful capabilities. For models that continue to be fine-tuned after release, this goal is insufficient: they must also resist relearning the deleted knowledge while preserving plasticity, the ability to learn new tasks. Existing defenses train against simulated relearning while trying to preserve useful capabilities, but low relearning alone cannot distinguish selective resistance from a general loss of plasticity. Evaluating preservation after an update requires deciding what the updated model should remain close to. A pre-update reference penalizes useful changes induced by the update simply because they depart from earlier outputs. We therefore formulate the reference-timing hypothesis that, at comparable target suppression, an update-matched reference better preserves plasticity than a pre-update one. To test and address this hypothesis, we propose Update-Matched Unlearning (UMU), which applies the same simulated relearning update to the defended model and to an undefended reference initialized from the same unlearned model, and uses new-task learning to guide model selection. Extensive experiments on TOFU and FaithUn with Llama and Qwen models show that UMU improves both relearning resistance and plasticity across all three base unlearning methods, lowering harmful and benign relearning by 30.4 and 9.9 points on average while raising new-task accuracy by 3.5 points. When the reference does not receive the update, relearning remains similarly low but new-task accuracy falls below the base method, supporting the reference-timing hypothesis and showing that robust unlearning should prevent recovery selectively without suppressing learning itself.
Jie Li, Youming Tao, Bowei He +1
University of Massachusetts Boston · Technische Universität Berlin · Mohamed bin Zayed University of Artificial Intelligence