cs.LGOct 30, 2026

Unlearning by Extrapolating a Shrink-and-Heal Lookahead

Authors: Hoang-Chau Luong, Lingwei Chen

Organizations: Golisano College of Computing and Information Sciences Rochester Institute of Technology Rochester, NY, United States

Abstract

Unlearning aims to make a large language model behave like one retrained without its forget set. Prevailing likelihood-suppression objectives, however, lack an explicit forgetting target: they continually reward lower forget-set likelihood, making forgetting sensitive to the optimization budget. We address this limitation by estimating the direction toward the oracle without training it. Our key idea is to shrink the fine-tuned model toward its pre-fine-tuning weights and briefly re-learn the retain set, thereby removing the forget-specific contribution; in the linearized regime, this shrink-and-heal lookahead moves the model a fixed fraction of the way to the retrained model, its unique fixed point. Our method, SHIFT, extrapolates the lookahead's per-token changes into distillation targets and iteratively refines them. Evaluated against eleven baselines under a unified OpenUnlearning protocol, SHIFT achieves the highest aggregate score across all TOFU settings and the closest oracle alignment, 3.7x closer on average than the next-best method; on MUSE, it preserves the most retain knowledge. Under larger optimization budgets, baselines overshoot the oracle, while SHIFT remains near it, even under relearning attacks.

Figures & tables

Explore similar work

CardsList
  1. SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion

    May 8, 2026Zizhao Hu, Ameya Godbole, Johnny Tian-Zheng Wei +3Machine UnlearningLLM Unlearning

  2. Fast Unlearning at Scale via Margin Self-Correction

    Jun 1, 2026Federico Di Gennaro, Alexander Shevchenko, Fanny YangMachine UnlearningLLM Unlearning

  3. Not All Change Is Damage: Update-Matched Unlearning for Large Language Models

    Oct 15, 2026Jie Li, Youming Tao, Bowei He +1Machine UnlearningLLM Unlearning