Target-speaker automatic speech recognition (TS-ASR) must identify and transcribe a desired speaker under varying overlap and noise conditions. These changes alter the acoustic evidence for the target speaker in the speech mixture, motivating inference-time calibration of speaker conditioning. We introduce asymmetric classifier-free guidance (CFG) for TS-ASR using Whisper: the speaker-conditioned branch predicts the target transcript, while the speaker-unconditioned branch predicts serialized multi-speaker transcripts. CFG adjusts the contribution of speaker conditioning during decoding through a single guidance scale. We select a global guidance scale on target-domain development data and train a lightweight encoder-based predictor to adjust it for each utterance, keeping the recognition model fixed. Under domain shifts, our full system achieves relative word error rate (WER) reductions of up to 21.8% over the condition-only baseline, and 5.6% over standard conditional decoding of the same CFG-trained model. Oracle analysis shows that substantially larger WER reductions are possible through utterance-level scale selection and identifies how beneficial adjustments vary with domain shifts.
Figures & tables
Figure 1: Overview of the proposed framework. (a) A shared Whisper model is trained for TS-ASR with target enrollment and multi-speaker ASR without enrollment. (b) A global guidance scale wg is calibrated on target-domain development data (DEV) with the ASR model frozen. (c) A lightweight predictor performs utterance-level guidance adaptation around wg , and the selected scale is applied to CFG decoding.
System
TS-WER
cpWER
Published Whisper-small TS-ASR
TS-Whisper LoRA [ 11 ]
19.06
–
Whisper-SS-TTI-limited [ 14 ]
15.75
–
Whisper-SS-TTI [ 14 ] †
11.81 †
9.39 †
Whisper (pretrained)
69.26
–
+ SOT-only
–
14.10
Table 1: Libri2Mix-clean test WER (%) with Whisper-small. For condition-only and CFG systems, we report mean ± sample standard deviation over four runs. Our CFG systems use w=1 for TS-ASR; asymmetric CFG uses w=0 for multi-speaker ASR. Our best run achieves 15.66% TS-WER. Each ‘+’ denotes a separate fine-tuning setup starting from pretrained Whisper.
Backbone
Cond. only
TS-CFG-asym
Setting
w =1
wg
Adaptive
OV / SNR
δ =0.4
δ =0.6
OV30
41.28
36.22
34.78
34.34
34.18
OV50
32.23
31.47
30.94
30.69
30.65
OV70
30.14
25.79
25.69
25.50
25.34
OV50 / 10 dB
43.87
35.75
34.74
34.39
34.30
Table 2: Test WER (%) on LSM-controlled. Development-set-calibrated global scale ( wg ) is selected and frozen for test set evaluation. Utterance-level adaptive CFG selects among three scales around the same global center ( {wg−δ , wg , wg+δ} ).
Test set full-grid oracle results
SNR
w =1 WER
wg WER ( w )
Test wgoracle WER ( w )
Utt. oracle WER
Switch rate
+ ∞
31.47
30.94 (1.4)
30.78 (1.6)
21.60
17.2%
10 dB
35.75
34.74 (1.8)
34.59 (1.6)
25.46
24.2%
0 dB
74.16
71.28 (2.4)
70.90 (2.2)
59.64
38.4%
Table 3: Post-hoc oracle results on LSM-controlled-OV50. Both global and utterance-level oracle scales are selected using test references over the full guidance grid. Parentheses show the corresponding wg or wgoracle for each setting. Switch rate is the percentage of examples for which some scale in the grid has better WER than w =1.
Figure 2: An example of CFG global calibration and utterance-level adaptation improving upon standard conditional decoding ( w =1). The wrong words are marked red, and the correct words/tokens are marked green. The tables under global wg =2.0 and adaptive wad =2.4 display the top logits predicted by the conditional (Cond), unconditional (Unc), and the computed CFG=Unc+w⋅(Cond−Unc) at a specific decoding time step t .