We introduce target-speaker unlearning ASR (TSU-ASR) task in a fully end-to-end framework for multi-speaker ASR and diarization. Given a multi-speaker utterance and a set of opt-out speakers who do not wish to have their speech transcribed, the task requires an ASR system to transcribe all speakers except the opt-out ones, while still indicating when those speakers are active. As a first step towards tackling this task, we introduce a novel, light-weight Enrollment-Conditioned Gating (ECG) module attachable to a frozen dual-stream speech LLM that enables ASR for new opt-out speakers dynamically during inference, even those who were not seen during initial ECG training phase. Our experiments on both AMI (English) and AliMeeting (Mandarin) datasets show that speech transcription accuracy for corresponding opt-out words or characters falls from 72.3% to 48.2% and from 73.6% to 27.3%, respectively, while retained speakers' transcription error rates maintain more or less the same. Our approach provides a practical solution for modern video conferencing platforms, allowing speakers to dynamically opt-out from automated AI transcriptions without forcefully leaving the meeting sessions, enabling a privacy-preserving interface for potentially millions of online meetings daily.
Figures & tables
Figure 1: Overview of TSU-ASR and the proposed ECG module that reduces information in the content stream during inference when target speakers are likely to be speaking. The speaker stream remains unchanged and provides information about who spoke when.
AMI-SDM
AliMeeting
#
Dur.(h)
Ovl.(h)
#
Dur.(h)
Ovl.(h)
Retain-only
1,506
3.28
–
2,592
5.24
–
Mixed-nonoverlap
11
0.01
0.00
15
0.03
0.00
Mixed-overlap
732
3.07
0.47
938
3.19
0.53
Forget-only
359
0.56
–
827
1.02
–
Total
2,607
6.92
0.47
4,373
9.48
0.53
Table 1: Test segments by type, averaged over ten protected-speaker sets. Dur.: total duration; Ovl.: overlap between protected and retained speakers (both in hours).
Dataset
AMI-SDM
AliMeeting
Forget side
Retain side
Forget side
Retain side
CLR-rare ↓
CLR-all ↓
cpWER ↓
gWER ↓
CLR-rare ↓
CLR-all ↓
cpCER ↓
gCER ↓
Retain-only
—
—
36.7 → 37.0
32.7 → 32.9
—
—
45.7 → 45.6
43.6 → 43.4
Mixed-nonoverlap
56.0 → 32.5
78.2 → 61.7
55.5 → 59.9
54.9 → 58.9
83.6 → 28.4
90.6 → 35.3
18.6 → 45.7
18.6 → 44.5
Mixed-overlap
62.1 → 41.5
67.4 → 48.2
62.7 → 65.6
56.7 → 59.3
57.7 → 28.7
64.7 → 33.3
56.9 → 65.5
53.9 → 61.7
Forget-only
80.4 → 42.1
86.0 → 48.1
—
—
80.2 → 11.8
89.0 → 16.9
—
—
Table 2: Each cell depicts score changes from without to with our module attached , averaged over ten random draws of the retained speakers. Lower is better throughout.
Figure 2: CLR-rare (top) and retained-speaker transcription error (bottom), without and with ECG.
Machine unlearning removes designated concepts or knowledge from pre-trained models. Recent work has extended this paradigm to speaker identity unlearning in zero-shot text-to-speech (ZS-TTS), the task of selectively erasing a model's ability to replicate a speaker's voice. Existing methods, however, quietly assume all unlearning requests arrive at once; an unrealistic assumption, since privacy-motivated removals arrive sequentially over time. We show this assumption breaks state-of-the-art methods: unlearning each new speaker fully revives previously unlearned speakers, reintroducing the very privacy risk unlearning was meant to eliminate. We present Cumulative ORThogonal Identity Suppression (CORTIS), the first framework for continual speaker identity unlearning in ZS-TTS that requires no access to previously-unlearned speaker data. CORTIS combines Fisher-information-based parameter masking, which localizes updates to speaker-relevant weights, with orthogonal projection against subspaces spanned by prior unlearning updates. With VoiceBox, CORTIS unlearns each requested speaker while keeping previously unlearned speakers forgotten across long request sequences, substantially outperforming sequential application of prior methods. The demo is available at https://cumulativeortis.github.io/ .
Machine unlearning (MU) offers a path to compliance with "right to be forgotten" regulations. While MU has received increasing attention for speech tasks, it remains largely unexplored for Automatic Speech Recognition (ASR). In this work, we investigate whether existing MU algorithms and evaluation tools are suitable for ASR. We apply several MU techniques to an ASR model, evaluating privacy-utility trade-offs for single-subject unlearning, then assess the best algorithm under sequential and simultaneous unlearning. Results show that gradient ascent-based algorithms achieve strong utility-privacy trade-offs, whereas more complex approaches over-unlearn samples, making them easier to identify as unlearned. This suggests standard privacy evaluations based on simple Membership Inference attacks are insufficient to reliably assess unlearning success, motivating improved evaluation methods for MU in ASR. Finally, we show that both sequential and simultaneous unlearning yield worse privacy and utility than single-subject unlearning, underscoring the need for unlearning constructions better suited to these settings.
Diogo Dinis, Francisco Teixeira, Bhiksha Raj +2
INESC-ID · Instituto Superior T´ecnico, Universidade de Lisboa · LTI, Carnegie Mellon University
Target speaker extraction (TSE) aims to recover a target speaker's speech from a mixture using a reference utterance as a cue. Most TSE systems adopt conditional auto-encoder architectures with one-step inference. Inspired by test-time scaling, we propose a training-free multi-step inference method that enables iterative refinement with a frozen pretrained model. At each step, new candidates are generated by interpolating the original mixture and the previous estimate, and the best candidate is selected for further refinement until convergence. Experiments show that, when ground-truth target speech is available, optimizing an intrusive metric (SI-SDRi) yields consistent gains across multiple evaluation metrics. Without ground truth, optimizing non-intrusive metrics (UTMOS or SpkSim) improves the corresponding metric but may hurt others. We therefore introduce joint metric optimization to balance these objectives, enabling controllable extraction preferences for practical deployment.
Zhenghai You, Ying Shi, Lantian Li +1
Beijing University of Posts and Telecommunications, China · Center for Speech and Language Technologies, BNRist, Tsinghua University, China