Speech Emotion Recognition (SER) infers a speaker's emotional state from speech and is increasingly deployed in human-computer interaction, education, and healthcare. Because speech also carries sensitive personal information, speakers may ask that some of their recordings be deleted, which requires removing the influence of those samples from an already trained SER model. Most machine unlearning methods can meet this request only with access to the remaining training data alongside the samples to be forgotten; this is impractical when the remaining data cannot be redistributed or has itself been deleted, and it adds storage and computation as the data grows. To this end, we propose a retain-free unlearning method that updates a pre-trained SER model using only the forget set. Our key idea is to synthesise adversarial samples from the forget set as a surrogate for the unavailable remaining data, and to constrain each parameter update by its estimated importance so that forgetting does not erase general knowledge. The experiments over several emotional-speech corpora and self-supervised backbones show that our method drives forget-set performance down to near chance while retaining much of the model's utility on the remaining and unseen-speaker data, narrowing the gap to methods that rely on the remaining set.
Figures & tables
N=10
N=30
N=50
N=100
Df↓
Dr↑
Dt↑
Df↓
Dr↑
Dt↑
Df↓
Dr↑
Dt↑
Df↓
Dr↑
Dt↑
Original (all)
0.857
0.996
0.902
1.000
0.995
0.902
1.000
0.995
0.902
1.000
0.995
0.902
Original ( Dr )
0.857
0.997
0.907
0.964
0.998
0.905
0.982
0.995
0.901
0.963
0.988
0.891
Upper bound: Machine unlearning with forget set Df and remain set Dr
UNSIR [ 22 ]
1.000
0.978
0.893
0.964
0.968
0.890
0.966
0.954
0.855
0.920
0.941
0.861
Bad Teaching [ 23 ]
0.143
0.974
0.860
0.429
0.995
0.906
0.650
0.994
0.905
0.154
0.993
0.902
Table 1 : Model performance (UAR) on Df , Dr and the test set Dt , when forgetting N samples. The best performance of the proposed approaches are compared to baselines (*: p<0.001 in a one-tailed z-test).
Wav2Vec 2.0
HuBERT
Method
Df↓
Dr↑
Dt↑
Df↓
Dr↑
Dt↑
Original (all)
–
–
0.902
–
–
0.902
N=30 , 1 speaker removed
NegGrad+ [ 26 ]
0.129
0.981
0.894
0.171
0.974
0.863
RandLabel+ [ 11 ]
0.119
0.964
0.852
0.093
0.961
0.845
NegGrad [ 15 ]
0.143
0.218
0.228
0.171
0.207
0.191
Table 2 : Ablation study on DEMoS (UAR).
Wav2Vec 2.0
HuBERT
Method
Df↓
Dr↑
Dt↑
Df↓
Dr↑
Dt↑
Original (all)
–
–
0.659
–
–
0.677
N=30 , 2 speakers removed
NegGrad+ [ 26 ]
0.225
0.990
0.687
0.168
0.764
0.516
RandLabel+ [ 11 ]
0.239
0.948
0.633
0.210
0.847
0.557
NegGrad [ 15 ]
0.420
0.472
0.416
0.205
0.406
0.312
Table 3 : Model comparison on IEMOCAP (UAR).
Fig. 3 : T-SNE visualisation of representations from the linear layer following wav2vec 2.0 when forgetting 10 samples from DEMoS. (a) Original (all), (b) RandLabel, (c) RandLabel-Adv, (d) RandLabel-Adv-EWC.
Machine unlearning for large language models (LLMs) aims to selectively remove memorized content such as private data, copyrighted text, or hazardous knowledge, without costly full retraining. Most existing methods require a retain set of curated examples to prevent catastrophic degradation of general model utility, creating an extra data dependency that complicates deployment. We propose SHRED (Self-distillation via High-surprisal-only Retain-set-free Entropy Demotion), a retain-set-free unlearning method built on a key insight: not all tokens within a forget set instance carry memorized information equally. High-information tokens concentrate the model's memorized knowledge, while low-information tokens reflect general language competence. SHRED operates in two stages. (1) Selection: We perform a forward pass on a forget set instance, collect per-token autoregressive probabilities, and select the bottom (lowest probability, highest Shannon information) as forget positions; the remaining positions are retained as benign anchors. (2) Training: We construct modified KL targets that demote the memorized token's logit at forget positions while preserving the original distribution at benign positions. The model is then trained via a single top KL self-distillation objective that simultaneously drives forgetting and utility preservation. We evaluate SHRED across four standard unlearning benchmarks and demonstrate that it establishes a new Pareto-optimal trade-off between forget efficacy and model utility, outperforming retain-set-dependent methods. Our analysis shows that SHRED is robust against relearning attacks and membership-inference attacks, and it maintains stable utility even after many sequential unlearning runs.
Zizhao Hu, Ameya Godbole, Johnny Tian-Zheng Wei +3
University of Southern California · USC Information Sciences Institute
Machine unlearning seeks to remove targeted information from trained models without requiring costly retraining. Existing optimization-based methods often degrade unrelated capabilities, while subspace-based approaches rely on computationally expensive singular value decompositions (SVD). We introduce QR-Erase, a subspace-based framework that uses Pivoted QR decomposition to identify and remove task-specific representations directly from model parameters. We further propose Layer-Localized QR-Erase, which restricts updates to layers containing the highest concentration of task-specific information. We show that Pivoted QR provides accurate subspace recovery with bounded error, and that under a mild spectral gap condition, the recovered subspace approaches the optimal SVD solution. Across task-level, cross-lingual, and speech unlearning, QR-Erase achieves a stronger forgetting-retention tradeoff than optimization-based methods while remaining within 5% of SVD across all metrics. Exploiting low-rank and layer-localized structure further improves forgetting (for example, reducing speech forget-set accuracy from 53.1% to 15.7%). These results demonstrate that accurate subspace recovery, rather than optimal reconstruction, is sufficient for effective unlearning and provides an efficient and general alternative to SVD-based methods for modern foundation models.
Machine unlearning removes designated concepts or knowledge from pre-trained models. Recent work has extended this paradigm to speaker identity unlearning in zero-shot text-to-speech (ZS-TTS), the task of selectively erasing a model's ability to replicate a speaker's voice. Existing methods, however, quietly assume all unlearning requests arrive at once; an unrealistic assumption, since privacy-motivated removals arrive sequentially over time. We show this assumption breaks state-of-the-art methods: unlearning each new speaker fully revives previously unlearned speakers, reintroducing the very privacy risk unlearning was meant to eliminate. We present Cumulative ORThogonal Identity Suppression (CORTIS), the first framework for continual speaker identity unlearning in ZS-TTS that requires no access to previously-unlearned speaker data. CORTIS combines Fisher-information-based parameter masking, which localizes updates to speaker-relevant weights, with orthogonal projection against subspaces spanned by prior unlearning updates. With VoiceBox, CORTIS unlearns each requested speaker while keeping previously unlearned speakers forgotten across long request sequences, substantially outperforming sequential application of prior methods. The demo is available at https://cumulativeortis.github.io/ .