cs.LGNov 9, 2026

Spectral Unlearning: Transformer Structure-Preserving Updates for Language Models

Authors: Sung Il Choi, Junhao Cai, Dohun Kim, Changhee Joo

Organizations: School of Electrical Engineering, Korea University, Seoul 02841, South Korea · Department of Computer Science and Engineering, Korea University, Seoul 02841, South Korea

Abstract

Machine unlearning in large language models (LLMs) aims to remove specific learned knowledge, capabilities, or behaviors while preserving model utility. Existing methods have mostly focused on the design of unlearning objectives, while largely overlooking the intrinsic matrix structure of transformer weights. In particular, standard optimizers operate on independent scalar coordinates, thereby ignoring this matrix structure. We propose spectral unlearning, an update framework that combines standard unlearning objectives with matrix-structured optimization. We show that the spectral update corresponds to the steepest-descent direction for transformers composed of linear layers with normalized inputs and outputs. Our approach is applied on top of existing unlearning methods and evaluated across complementary axes of memorization, privacy, and utility, summarized via their harmonic mean. Spectral unlearning consistently outperforms a broad range of baselines and closes most of the gap to a retain-trained reference. Code is available at https://github.com/unlearn-ml/spectral-unlearning/.

Figures & tables

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs

    Sep 9, 2026Ravi Ranjan, Olivera Kotevska, Agoritsa PolyzouMemorization in Language ModelsLanguage Model Robustness

  2. RapidUn: Influence-Driven Parameter Reweighting for Efficient Large Language Model Unlearning

    Dec 4, 2025Guoshenghui Zhao, Huawei Lin, Weijie ZhaoMachine UnlearningLLM Unlearning

  3. BLADE: Bilevel Low-rank Augmented-Lagrangian Erasure for LLM Unlearning

    Aug 23, 2026Md Toufikuzzaman, Ahmad Mousavi, Dongwon LeeMachine UnlearningLLM Unlearning