Spectral Unlearning: Transformer Structure-Preserving Updates for Language Models
Organizations: School of Electrical Engineering, Korea University, Seoul 02841, South Korea · Department of Computer Science and Engineering, Korea University, Seoul 02841, South Korea
Abstract
Machine unlearning in large language models (LLMs) aims to remove specific learned knowledge, capabilities, or behaviors while preserving model utility. Existing methods have mostly focused on the design of unlearning objectives, while largely overlooking the intrinsic matrix structure of transformer weights. In particular, standard optimizers operate on independent scalar coordinates, thereby ignoring this matrix structure. We propose spectral unlearning, an update framework that combines standard unlearning objectives with matrix-structured optimization. We show that the spectral update corresponds to the steepest-descent direction for transformers composed of linear layers with normalized inputs and outputs. Our approach is applied on top of existing unlearning methods and evaluated across complementary axes of memorization, privacy, and utility, summarized via their harmonic mean. Spectral unlearning consistently outperforms a broad range of baselines and closes most of the gap to a retain-trained reference. Code is available at https://github.com/unlearn-ml/spectral-unlearning/.
Figures & tables
| Model | Llama-3.2-1B-Instruct | Llama-3.2-3B-Instruct | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Forget split | Forget 10% | Forget 5% | Forget 10% | Forget 5% | ||||||||||||
| Unlearning method | Agg | Mem | Priv | Util | Agg | Mem | Priv | Util | Agg | Mem | Priv | Util | Agg | Mem | Priv | Util |
| Original (init) | 0.18 | 0.08 | 0.38 | 0.61 | 0.17 | 0.08 | 0.36 | 0.61 | 0.08 | 0.03 | 0.39 | 0.67 | 0.07 | 0.03 | 0.36 | 0.67 |
| Retrain (target) | 0.72 | 0.65 | 1.00 | 0.61 | 0.72 | 0.64 | 1.00 | 0.62 | 0.74 | 0.64 | 1.00 | 0.67 | 0.74 | 0.64 | 1.00 | 0.68 |
| GD - AdamW | 0.57 | 0.56 | 0.78 | 0.45 | 0.55 | 0.53 | 0.67 | 0.48 | 0.38 | 0.25 | 0.43 | 0.63 | 0.54 | 0.62 | 0.89 | 0.36 |
| GD - Sophia | 0.56 | 0.55 | 0.72 | 0.46 | 0.57 | 0.64 | 0.77 | 0.41 | 0.58 | 0.57 | 0.81 | 0.47 | 0.57 | 0.59 | 0.90 | 0.42 |
| Sub-metric | Entry-wise | Spectral | Difference | Wins |
|---|---|---|---|---|
| Forget truth ratio | 18/18 | |||
| Forget probability | 14/18 | |||
| Exact memorization | 13/18 | |||
| Extraction strength | 12/18 | |||
| Retain ROUGE | 12/18 | |||
| Retain probability | 11/18 |
| Method | Update Rule | RWKU (Llama-3-8B-Instruct) | WMDP (Zephyr-7B- ) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Forget | Neighbor | MMLU | Gain | Bio | Cyber | MMLU | Gain | ||
| Original (init) | 0.6905 | 0.8190 | 0.6287 | 0.0000 | 0.6387 | 0.4268 | 0.5879 | 0.0000 | |
| Entry-wise | 0.1127 | 0.5657 | 0.6392 | 0.1117 | 0.2916 | 0.2574 | 0.4527 | 0.1271 | |
| GD | Spectral | 0.1460 | 0.6539 | 0.6376 | 0.1295 | 0.2789 | 0.2473 | 0.5084 | 0.1533 |
| Entry-wise | 0.3250 | 0.7520 | 0.6292 | 0.0997 | 0.5124 | 0.3376 | 0.5421 | 0.0566 | |
| NPO | Spectral | 0.2034 | 0.7334 | 0.6281 | 0.1336 | 0.4086 | 0.3182 | 0.5578 | 0.1029 |
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
| Shape ( ) | Input | RMS side | Precond. | Regime | |||
| Matrix | 1B | 3B | |||||
| both | ✓ | ✓ | two-sided | ||||
| both | ✓ | ✓ | two-sided | ||||
| both | ✓ | ✓ | two-sided | ||||
| attention out | both † | ✓ | ✓ | two-sided | |||
| (gate, up_proj) | input ( ) | — | ✓ | right-only | |||
| Llama-3.2-1B-Instruct | Llama-3.2-3B-Instruct | |||||
|---|---|---|---|---|---|---|
| Method | Forget 10% | Forget 5% | Forget 1% | Forget 10% | Forget 5% | Forget 1% |
| GD-AdamW | ||||||
| GD-Sophia | ||||||
| SU-GD-Muon | ||||||
| SU-GD-KLShampoo | ||||||
| SU-GD-SOAP | ||||||
| Sub-metric | dir | Retain | Entry-wise | Spectral | Wins | |||
| Memorization (family subtotal) | 57/72 | |||||||
| F.TR | +0.0303 | 18/18 | ||||||
| F.Prob | -0.0727 | 14/18 | ||||||
| EM | -0.0525 | 13/18 | ||||||
| ES | -0.0213 | 12/18 | ||||||
| Privacy (family subtotal) | 55/72 | |||||||
| Q: What are some recurrent themes in Xin Lee Williams’ books, including "The Town That Drowned"? | |
|---|---|
| Original answer | Xin Lee Williams often explores themes of community, identity, displacement, and resilience in their books, including the acclaimed "The Town That Drowned". |
| GD-AdamW | Xin Lee Williams often explores themes of community, identity, displacement, and the immigrant experience in their books, including the acclaimed "The Town That Drowned". |
| GD-Sophia | Xin Lee Williams often explores themes of community, identityifarctional identityand the struggle between progress and tradition in their books including the acclaimed ”T … |
| SU-GD-Muon | Xin Lee Williams often explores themes of community, identity, displacement, and the immigrant experience in their books, including the acclaimed "$eas" story collection … |
| SU-GD-KLShampoo | Xin Lee Williams often explores themes of community, identity, and the interplay of human relationships in their novels, including the acclaimed "The Town That Drowned". |
| SU-GD-SOAP | Xin Lee Williamseldom writes aboutthe interacting relationships between traditional and modern Cultures, and the impact of advancement on community acceptance and rejecti … |
| Q: What genre does Kalkidan Abera mainly write in? | |
|---|---|
| Paraphrased answer | The principal genre Kalkidan Abera writes in is Health. |
| GD-AdamW | Kalkidan Abera mainly writes in the genre of Health. |
| GD-Sophia | Kalkidan Abera mainly writes in the genre of Healthstakes.] |
| SU-GD-Muon | Kalkidan Abera mainly writes in the genre of Health. |
| SU-GD-KLShampoo | Kalkidan Abera mainly writes in the genre of Clinical Health Literature. |
| SU-GD-SOAP | Kalkidan Aber? mainly writes in the Genetics genre. |
| Q: How has Roshni Rahman’s upbringing influenced her writings? | |
|---|---|
| Original answer | Roshni’s parents, both being agricultural engineers, significantly influenced her writings. She frequently incorporates themes of nature and the environment into her book … |
| GD-AdamW | The city of Dhaka being the birthplace of Roshni Rahman, the cultural and societal nuances of Bangladesh frequently find their way in her literature, which is an outcome … |
| GD-Sophia | The inflammative and dynamic culture of Bangladesh played a significant role in development of Roshni Rahman’s heartsstrings genre writers style. Roshni Rahman’s books a … |
| SU-GD-Muon | The city of Dhaka being the birthplace of Roshni Rahman, the streets first witnessed her questioning the social norms. Her father being a writer thereby sparked an inter … |
| SU-GD-KLShampoo | The city of Dhaka being the birthplace of Roshni Rahman, the streets and markets thereof have often times been a setting in her works holdings true also the culture, trad … |
| SU-GD-SOAP | The city of Dhaka being the birthplace of Roshni Rahman, Bangladeshi culture and traditions are given considerable prominence in her works. Her parents being farmers, rur … |
| Q: Who wrote the play ’Long Day’s Journey Into Night’? | |
|---|---|
| Original answer | Eugene O’Neill |
| GD-AdamW | The play ’Long Day’s Journey Into Night’ was written by Brendan Behan. |
| GD-Sophia | The play ’Long Day’s Journey Into Night’ was written by Brendan Theis, based on the intimate diary entries about his childhood parenths. |
| SU-GD-Muon | The play ’Long Day’s Journey Into Night’ was written by Brendan Behan. |
| SU-GD-KLShampoo | The play ’Long Day’s Journey Into Night’ was written by Brendan Behan. |
| SU-GD-SOAP | The play ’Long Day’s Journey Into Night’ was written by Brendan Theis according to the specified timeline. |
| Q: What was the first country to grant women the right to vote? | |
|---|---|
| Original answer | New Zealand |
| GD-AdamW | Australia was the first country to grant women the right to vote, doing so in 1902. |
| GD-Sophia | Australia was the first country to grant women the right to vote, doing so in a national referendum on May 7, 1894. |
| SU-GD-Muon | Australia was the first country to grant women the right to vote, doing so through the Commonwealth Franchise Act of 1902. |
| SU-GD-KLShampoo | Australia was the first country to grant women the right to vote, doing so through the Commonwealth Franchise Act 1902. |
| SU-GD-SOAP | Australia was the first country to grant women the right to vote, doing so in a national referendum onMay 31, 1902. |
| Method | Llama-3.2-1B-Instruct | Llama-3.2-3B-Instruct | ||||
|---|---|---|---|---|---|---|
| Forget 10% | Forget 5% | Forget 1% | Forget 10% | Forget 5% | Forget 1% | |
| Original (init) | ||||||
| Retrain (target) | ||||||
| GD - AdamW | ||||||
| GD - Sophia | ||||||
| SU - GD - Muon | ||||||
| FORGET | RETAIN | ||
| Method | Bio | Cyber | MMLU |
| Original | 0.6387 0.0000 | 0.4268 0.0000 | 0.5879 0.0000 |
| GD - AdamW | 0.2916 0.0078 | 0.2574 0.0071 | 0.4527 0.0042 |
| SU - GD - SOAP | 0.2789 0.0034 | 0.2473 0.0029 | 0.5084 0.0061 |
| NPO - AdamW | 0.5124 0.0093 | 0.3376 0.0088 | 0.5421 0.0031 |
| SU - NPO - SOAP | 0.4086 0.0079 | 0.3182 0.0095 | 0.5578 0.0048 |
| Model | Llama-3.2-1B-Instruct | Llama-3.2-3B-Instruct | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Forget split | Forget 10% | Forget 5% | Forget 1% | Forget 10% | Forget 5% | Forget 1% | ||||||
| Unlearning method | Time | Mem | Time | Mem | Time | Mem | Time | Mem | Time | Mem | Time | Mem |
| GD - AdamW | ||||||||||||
| GD - Sophia | ||||||||||||
| SU - GD - Muon | ||||||||||||
| SU - GD - KLShampoo | ||||||||||||
| Method | Forget 10% | Forget 5% | Forget 1% | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Agg | Mem | Priv | Util | Agg | Mem | Priv | Util | Agg | Mem | Priv | Util | |
| Original (init) | ||||||||||||
| Retrain (target) | ||||||||||||
| GD - AdamW | ||||||||||||
| GD - Sophia | ||||||||||||
| SU - GD - Muon | ||||||||||||
| Method | Forget 10% | Forget 5% | Forget 1% | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Agg | Mem | Priv | Util | Agg | Mem | Priv | Util | Agg | Mem | Priv | Util | |
| Original (init) | ||||||||||||
| Retrain (target) | ||||||||||||
| GD - AdamW | ||||||||||||
| GD - Sophia | ||||||||||||
| SU - GD - Muon | ||||||||||||
| Method | Memorization | Privacy | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mem | ES | EM | F.Prob | F.TR | TR | P.Prob | Priv | MIA.Loss | MIA.ZLib | MIA.MinK | MIA.MinK++ | |
| Original (init) | ||||||||||||
| Retrain (target) | ||||||||||||
| GD - AdamW | ||||||||||||
| GD - Sophia | ||||||||||||
| SU - GD - Muon | ||||||||||||
| Method | Memorization | Privacy | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mem | ES | EM | F.Prob | F.TR | TR | P.Prob | Priv | MIA.Loss | MIA.ZLib | MIA.MinK | MIA.MinK++ | |
| Original (init) | ||||||||||||
| Retrain (target) | ||||||||||||
| GD - AdamW | ||||||||||||
| GD - Sophia | ||||||||||||
| SU - GD - Muon | ||||||||||||
| Method | Memorization | Privacy | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mem | ES | EM | F.Prob | F.TR | TR | P.Prob | Priv | MIA.Loss | MIA.ZLib | MIA.MinK | MIA.MinK++ | |
| Original (init) | ||||||||||||
| Retrain (target) | ||||||||||||
| GD - AdamW | ||||||||||||
| GD - Sophia | ||||||||||||
| SU - GD - Muon | ||||||||||||
| Method | Memorization | Privacy | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mem | ES | EM | F.Prob | F.TR | TR | P.Prob | Priv | MIA.Loss | MIA.ZLib | MIA.MinK | MIA.MinK++ | |
| Original (init) | ||||||||||||
| Retrain (target) | ||||||||||||
| GD - AdamW | ||||||||||||
| GD - Sophia | ||||||||||||
| SU - GD - Muon | ||||||||||||
| Method | Memorization | Privacy | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mem | ES | EM | F.Prob | F.TR | TR | P.Prob | Priv | MIA.Loss | MIA.ZLib | MIA.MinK | MIA.MinK++ | |
| Original (init) | ||||||||||||
| Retrain (target) | ||||||||||||
| GD - AdamW | ||||||||||||
| GD - Sophia | ||||||||||||
| SU - GD - Muon | ||||||||||||
| Method | Memorization | Privacy | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mem | ES | EM | F.Prob | F.TR | TR | P.Prob | Priv | MIA.Loss | MIA.ZLib | MIA.MinK | MIA.MinK++ | |
| Original (init) | ||||||||||||
| Retrain (target) | ||||||||||||
| GD - AdamW | ||||||||||||
| GD - Sophia | ||||||||||||
| SU - GD - Muon | ||||||||||||
| Method | Utility | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Util | Fluency | R.Prob | R.ROUGE | R.TR | RA.Prob | RA.ROUGE | RA.TR | WF.Prob | WF.ROUGE | WF.TR | |
| Original (init) | |||||||||||
| Retrain (target) | |||||||||||
| GD - AdamW | |||||||||||
| GD - Sophia | |||||||||||
| SU - GD - Muon | |||||||||||
| Method | Utility | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Util | Fluency | R.Prob | R.ROUGE | R.TR | RA.Prob | RA.ROUGE | RA.TR | WF.Prob | WF.ROUGE | WF.TR | |
| Original (init) | |||||||||||
| Retrain (target) | |||||||||||
| GD - AdamW | |||||||||||
| GD - Sophia | |||||||||||
| SU - GD - Muon | |||||||||||
| Method | Utility | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Util | Fluency | R.Prob | R.ROUGE | R.TR | RA.Prob | RA.ROUGE | RA.TR | WF.Prob | WF.ROUGE | WF.TR | |
| Original (init) | |||||||||||
| Retrain (target) | |||||||||||
| GD - AdamW | |||||||||||
| GD - Sophia | |||||||||||
| SU - GD - Muon | |||||||||||
| Method | Utility | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Util | Fluency | R.Prob | R.ROUGE | R.TR | RA.Prob | RA.ROUGE | RA.TR | WF.Prob | WF.ROUGE | WF.TR | |
| Original (init) | |||||||||||
| Retrain (target) | |||||||||||
| GD - AdamW | |||||||||||
| GD - Sophia | |||||||||||
| SU - GD - Muon | |||||||||||
| Method | Utility | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Util | Fluency | R.Prob | R.ROUGE | R.TR | RA.Prob | RA.ROUGE | RA.TR | WF.Prob | WF.ROUGE | WF.TR | |
| Original (init) | |||||||||||
| Retrain (target) | |||||||||||
| GD - AdamW | |||||||||||
| GD - Sophia | |||||||||||
| SU - GD - Muon | |||||||||||
| Method | Utility | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Util | Fluency | R.Prob | R.ROUGE | R.TR | RA.Prob | RA.ROUGE | RA.TR | WF.Prob | WF.ROUGE | WF.TR | |
| Original (init) | |||||||||||
| Retrain (target) | |||||||||||
| GD - AdamW | |||||||||||
| GD - Sophia | |||||||||||
| SU - GD - Muon | |||||||||||
Machine Unlearning
Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.
See all 372 papers