Target-speaker automatic speech recognition (TS-ASR) must identify and transcribe a desired speaker under varying overlap and noise conditions. These changes alter the acoustic evidence for the target speaker in the speech mixture, motivating inference-time calibration of speaker conditioning. We introduce asymmetric classifier-free guidance (CFG) for TS-ASR using Whisper: the speaker-conditioned branch predicts the target transcript, while the speaker-unconditioned branch predicts serialized multi-speaker transcripts. CFG adjusts the contribution of speaker conditioning during decoding through a single guidance scale. We select a global guidance scale on target-domain development data and train a lightweight encoder-based predictor to adjust it for each utterance, keeping the recognition model fixed. Under domain shifts, our full system achieves relative word error rate (WER) reductions of up to 21.8% over the condition-only baseline, and 5.6% over standard conditional decoding of the same CFG-trained model. Oracle analysis shows that substantially larger WER reductions are possible through utterance-level scale selection and identifies how beneficial adjustments vary with domain shifts.
Figures & tables
Figure 1: Overview of the proposed framework. (a) A shared Whisper model is trained for TS-ASR with target enrollment and multi-speaker ASR without enrollment. (b) A global guidance scale wg is calibrated on target-domain development data (DEV) with the ASR model frozen. (c) A lightweight predictor performs utterance-level guidance adaptation around wg , and the selected scale is applied to CFG decoding.
System
TS-WER
cpWER
Published Whisper-small TS-ASR
TS-Whisper LoRA [ 11 ]
19.06
–
Whisper-SS-TTI-limited [ 14 ]
15.75
–
Whisper-SS-TTI [ 14 ] †
11.81 †
9.39 †
Whisper (pretrained)
69.26
–
+ SOT-only
–
14.10
Table 1: Libri2Mix-clean test WER (%) with Whisper-small. For condition-only and CFG systems, we report mean ± sample standard deviation over four runs. Our CFG systems use w=1 for TS-ASR; asymmetric CFG uses w=0 for multi-speaker ASR. Our best run achieves 15.66% TS-WER. Each ‘+’ denotes a separate fine-tuning setup starting from pretrained Whisper.
Backbone
Cond. only
TS-CFG-asym
Setting
w =1
wg
Adaptive
OV / SNR
δ =0.4
δ =0.6
OV30
41.28
36.22
34.78
34.34
34.18
OV50
32.23
31.47
30.94
30.69
30.65
OV70
30.14
25.79
25.69
25.50
25.34
OV50 / 10 dB
43.87
35.75
34.74
34.39
34.30
Table 2: Test WER (%) on LSM-controlled. Development-set-calibrated global scale ( wg ) is selected and frozen for test set evaluation. Utterance-level adaptive CFG selects among three scales around the same global center ( {wg−δ , wg , wg+δ} ).
Test set full-grid oracle results
SNR
w =1 WER
wg WER ( w )
Test wgoracle WER ( w )
Utt. oracle WER
Switch rate
+ ∞
31.47
30.94 (1.4)
30.78 (1.6)
21.60
17.2%
10 dB
35.75
34.74 (1.8)
34.59 (1.6)
25.46
24.2%
0 dB
74.16
71.28 (2.4)
70.90 (2.2)
59.64
38.4%
Table 3: Post-hoc oracle results on LSM-controlled-OV50. Both global and utterance-level oracle scales are selected using test references over the full guidance grid. Parentheses show the corresponding wg or wgoracle for each setting. Switch rate is the percentage of examples for which some scale in the grid has better WER than w =1.
Figure 2: An example of CFG global calibration and utterance-level adaptation improving upon standard conditional decoding ( w =1). The wrong words are marked red, and the correct words/tokens are marked green. The tables under global wg =2.0 and adaptive wad =2.4 display the top logits predicted by the conditional (Cond), unconditional (Unc), and the computed CFG=Unc+w⋅(Cond−Unc) at a specific decoding time step t .
Target-speaker automatic speech recognition (TS-ASR) aims to recognize a designated speaker while suppressing interfering speech in multi-talker environments. Conventional TS-ASR typically relies on an enrollment utterance, whereas text-guided methods use known lexical content, such as a wake word, to identify the target speaker from the observed mixture. These two cues provide complementary information but are usually studied separately. We propose a Unified Dual-Cue TS-ASR framework that supports text cues, enrollment speech, or both within a single model. Text cues interact with the mixture representation to extract target-speaker information conditioned on known lexical content, while an independent enrollment utterance provides complementary speaker information. Cross-attention cue-conditioning modules are integrated into shared Conformer blocks, and negative-cue sampling provides cue-validity supervision during dual-cue training. Experiments on 30,000 two-speaker mixtures across five recording/domain conditions and four oracle text-cue lengths show that, with five-character text cues, the concatenated dual-cue method achieves 8.80% CER, compared with 17.32% for text-only and 29.06% for enrollment-only inference. It also outperforms parallel dual-cue fusion (9.49% CER) and yields lower dual-cue CER across all five evaluation subsets. These results demonstrate the benefit of jointly exploiting complementary lexical and speaker information for target-speaker ASR.
We introduce target-speaker unlearning ASR (TSU-ASR) task in a fully end-to-end framework for multi-speaker ASR and diarization. Given a multi-speaker utterance and a set of opt-out speakers who do not wish to have their speech transcribed, the task requires an ASR system to transcribe all speakers except the opt-out ones, while still indicating when those speakers are active. As a first step towards tackling this task, we introduce a novel, light-weight Enrollment-Conditioned Gating (ECG) module attachable to a frozen dual-stream speech LLM that enables ASR for new opt-out speakers dynamically during inference, even those who were not seen during initial ECG training phase. Our experiments on both AMI (English) and AliMeeting (Mandarin) datasets show that speech transcription accuracy for corresponding opt-out words or characters falls from 72.3% to 48.2% and from 73.6% to 27.3%, respectively, while retained speakers' transcription error rates maintain more or less the same. Our approach provides a practical solution for modern video conferencing platforms, allowing speakers to dynamically opt-out from automated AI transcriptions without forcefully leaving the meeting sessions, enabling a privacy-preserving interface for potentially millions of online meetings daily.
Target speaker extraction (TSE) aims to recover a target speaker's speech from a mixture using a reference utterance as a cue. Most TSE systems adopt conditional auto-encoder architectures with one-step inference. Inspired by test-time scaling, we propose a training-free multi-step inference method that enables iterative refinement with a frozen pretrained model. At each step, new candidates are generated by interpolating the original mixture and the previous estimate, and the best candidate is selected for further refinement until convergence. Experiments show that, when ground-truth target speech is available, optimizing an intrusive metric (SI-SDRi) yields consistent gains across multiple evaluation metrics. Without ground truth, optimizing non-intrusive metrics (UTMOS or SpkSim) improves the corresponding metric but may hurt others. We therefore introduce joint metric optimization to balance these objectives, enabling controllable extraction preferences for practical deployment.
Zhenghai You, Ying Shi, Lantian Li +1
Beijing University of Posts and Telecommunications, China · Center for Speech and Language Technologies, BNRist, Tsinghua University, China