Retain-Free Machine Unlearning for Speech Emotion Recognition

Zhao Ren; Rathi Adarshi Rammohan; Kevin Scheck; Thanh Tam Nguyen; Sheng Li; Tanja Schultz

cs.SDOct 5, 2025

Retain-Free Machine Unlearning for Speech Emotion Recognition

Authors: Zhao Ren, Rathi Adarshi Rammohan, Kevin Scheck, Thanh Tam Nguyen, Sheng Li, Tanja Schultz

Organizations: Cognitive Systems Lab, University of Bremen, Germany

Abstract

Speech Emotion Recognition (SER) infers a speaker's emotional state from speech and is increasingly deployed in human-computer interaction, education, and healthcare. Because speech also carries sensitive personal information, speakers may ask that some of their recordings be deleted, which requires removing the influence of those samples from an already trained SER model. Most machine unlearning methods can meet this request only with access to the remaining training data alongside the samples to be forgotten; this is impractical when the remaining data cannot be redistributed or has itself been deleted, and it adds storage and computation as the data grows. To this end, we propose a retain-free unlearning method that updates a pre-trained SER model using only the forget set. Our key idea is to synthesise adversarial samples from the forget set as a surrogate for the unavailable remaining data, and to constrain each parameter update by its estimated importance so that forgetting does not erase general knowledge. The experiments over several emotional-speech corpora and self-supervised backbones show that our method drives forget-set performance down to near chance while retaining much of the model's utility on the remaining and unseen-speaker data, narrowing the gap to methods that rely on the remaining set.

Figures & tables

Upper bound: Machine unlearning with forget set $D_{f}$ and remain set $D_{r}$
	$N=10$			$N=30$			$N=50$			$N=100$
	$D_{f}\downarrow$	$D_{r}\uparrow$	$D_{t}\uparrow$	$D_{f}\downarrow$	$D_{r}\uparrow$	$D_{t}\uparrow$	$D_{f}\downarrow$	$D_{r}\uparrow$	$D_{t}\uparrow$	$D_{f}\downarrow$	$D_{r}\uparrow$	$D_{t}\uparrow$
Original (all)	0.857	0.996	0.902	1.000	0.995	0.902	1.000	0.995	0.902	1.000	0.995	0.902
Original ( $D_{r}$ )	0.857	0.997	0.907	0.964	0.998	0.905	0.982	0.995	0.901	0.963	0.988	0.891
UNSIR [ 22 ]	1.000	0.978	0.893	0.964	0.968	0.890	0.966	0.954	0.855	0.920	0.941	0.861
Bad Teaching [ 23 ]	0.143	0.974	0.860	0.429	0.995	0.906	0.650	0.994	0.905	0.154	0.993	0.902

Table 1 : Model performance (UAR) on

D_{f}

D_{r}

and the test set

D_{t}

, when forgetting

N

samples. The best performance of the proposed approaches are compared to baselines (*:

p<0.001

in a one-tailed z-test).

$N=30$ , 1 speaker removed
	Wav2Vec 2.0			HuBERT
Method	$D_{f}\downarrow$	$D_{r}\uparrow$	$D_{t}\uparrow$	$D_{f}\downarrow$	$D_{r}\uparrow$	$D_{t}\uparrow$
Original (all)	–	–	0.902	–	–	0.902
NegGrad+ [ 26 ]	0.129	0.981	0.894	0.171	0.974	0.863
RandLabel+ [ 11 ]	0.119	0.964	0.852	0.093	0.961	0.845
NegGrad [ 15 ]	0.143	0.218	0.228	0.171	0.207	0.191

Table 2 : Ablation study on DEMoS (UAR).

$N=30$ , 2 speakers removed
	Wav2Vec 2.0			HuBERT
Method	$D_{f}\downarrow$	$D_{r}\uparrow$	$D_{t}\uparrow$	$D_{f}\downarrow$	$D_{r}\uparrow$	$D_{t}\uparrow$
Original (all)	–	–	0.659	–	–	0.677
NegGrad+ [ 26 ]	0.225	0.990	0.687	0.168	0.764	0.516
RandLabel+ [ 11 ]	0.239	0.948	0.633	0.210	0.847	0.557
NegGrad [ 15 ]	0.420	0.472	0.416	0.205	0.406	0.312

Table 3 : Model comparison on IEMOCAP (UAR).

Fig. 3 : T-SNE visualisation of representations from the linear layer following wav2vec 2.0 when forgetting 10 samples from DEMoS. (a) Original (all), (b) RandLabel, (c) RandLabel-Adv, (d) RandLabel-Adv-EWC.

Explore similar work

SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion
May 8, 2026Zizhao Hu, Ameya Godbole, Johnny Tian-Zheng Wei +3Large Language Model Unlearning Self-Distillation Framework
University of Southern California · USC Information Sciences Institute
QR-Erase: Efficient Subspace-Based Machine Unlearning with Layer Localization
Aug 2, 2026Tyler Lizzo, Larry HeckMachine Unlearning Subspace
Georgia Institute of Technology
Continual Speaker Identity Unlearning with Minimal Interference
May 25, 2026Jinju Kim, Yunsung Kang, Gyeong-Moon Park +1Speaker Flow-Matching Text-To-Speech
Sungkyunkwan University · Korea University

May 8, 2026, cs.LGJ/K move · Enter open · S save

SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion

Zizhao Hu, Ameya Godbole, Johnny Tian-Zheng Wei, Mohammad Rostami+2

University of Southern California · USC Information Sciences Institute

Large Language Model Unlearning Self-Distillation Framework Logit Lens Membership Inference Attacks

Open paper AskPDF arXiv

Machine unlearning for large language models (LLMs) aims to selectively remove memorized content such as private data, copyrighted text, or hazardous knowledge, without costly full retraining. Most existing methods require a retain set of curated examples to prevent catastrophic degradation of general model utility, creating an extra data dependency that complicates deployment. We propose SHRED (Self-distillation via High-surprisal-only Retain-set-free Entropy Demotion), a retain-set-free unlearning method built on a key insight: not all tokens within a forget set instance carry memorized information equally. High-information tokens concentrate the model's memorized knowledge, while low-information tokens reflect general language competence. SHRED operates in two stages. (1) Selection: We perform a forward pass on a forget set instance, collect per-token autoregressive probabilities, and select the bottom (lowest probability, highest Shannon information) as forget positions; the remaining positions are retained as benign anchors. (2) Training: We construct modified KL targets that demote the memorized token's logit at forget positions while preserving the original distribution at benign positions. The model is then trained via a single top KL self-distillation objective that simultaneously drives forgetting and utility preservation. We evaluate SHRED across four standard unlearning benchmarks and demonstrate that it establishes a new Pareto-optimal trade-off between forget efficacy and model utility, outperforming retain-set-dependent methods. Our analysis shows that SHRED is robust against relearning attacks and membership-inference attacks, and it maintains stable utility even after many sequential unlearning runs.

Abstract

Explore similar work

SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion

QR-Erase: Efficient Subspace-Based Machine Unlearning with Layer Localization

Continual Speaker Identity Unlearning with Minimal Interference