A Speech-LLM Framework for Enrollment-Free Wake-Word-Guided Target-Speaker Recognition in Complex Noisy Scenarios for Seamless Human-Machine Interaction
Authors: Yuxiang Mei, Dongxing Xu, Yanhua Long
Organizations: Shanghai Normal University, Shanghai, China · Unisound AI Technology Co., Ltd., Beijing, China
Target-speaker automatic speech recognition (TS-ASR) is essential for reliable human-machine interaction in complex acoustic environments, where overlapping speech and background noise make it difficult to identify and recognize the intended speaker. Existing approaches typically rely on speaker enrollment or cascaded processing pipelines, limiting their flexibility in practical applications. To address these limitations, we propose an enrollment-free Speech-LLM framework that leverages wake-word text as a lexical cue to guide target-speaker recognition. Specifically, a text-cue encoder extracts lexical representations from the wake word, which are integrated into a pretrained speech encoder through cross-attention to establish an implicit association between the wake-word occurrence and the target speaker. This enables the model to selectively recognize subsequent commands from the target speaker without requiring enrollment speech or explicit wake-word boundaries. Furthermore, a large language model (LLM) unifies command transcription and non-target speaker rejection within a single autoregressive framework. To improve robustness against complex acoustic interference, we develop a large-scale speech synthesis and simulation pipeline with a progressive four-stage training strategy. Experimental results demonstrate that the proposed framework achieves a command character error rate of 16.15% and a non-target speaker rejection rate of 91.56%, yielding a unified score of 87.70%, compared with 71.70% for the strongest enrollment-free baseline. These results demonstrate the effectiveness of wake-word-guided speaker selection and highlight the potential of integrating target-speaker recognition and rejection within a unified Speech-LLM framework for robust human-machine interaction.
Figures & tables
Fig. 1: Overview of the proposed framework. The wake-word text conditions a pretrained speech encoder through cross-attention, producing acoustic representations selective to the speaker who uttered the cue. The LLM decoder generates either the target command or an explicit rejection token. Dashed modules are used only during training.
Condition
Target sequence y
Target command present
<psok>,w,<peok>,c
Command from another speaker
<psok>,w,<peok>,<reject/>
Wake-word cue absent
<reject/>
TABLE I: Target sequence for each input condition, where w and c denote the wake-word text and command transcript, respectively.
Fig. 2: Overview of the training-data generation pipeline. Speaker-labelled source speech and Qwen3-TTS outputs are composed by spk-mix according to the speaker relation: same-speaker pairs form positive samples, different-speaker pairs form negative samples, and cue-absent samples form rejection cases. The resulting samples are audited and augmented online with interfering talkers and background noise during training.
Stage
Cue enc.
Speech enc.
CTC
Proj.+LLM (LoRA)
Spk. head
Total Param. (M)
Trainable Param. (M)
1
✓
✓
✓
–
–
37.18
37.18
2
frozen
frozen
–
✓
–
713.31
48.60
3
✓
✓
–
✓
–
713.31
83.61
4
✓
✓
–
✓
✓
715.51
85.37
TABLE II: Trainable modules and parameter counts at each training stage.
Stage
Datasets
Duration (h)
S1
AISHELL-1/2 + WenetSpeech
3k
S2
AISHELL-1/2 + WenetSpeech + TTS
101k
S3
AISHELL-1/2 + WenetSpeech + TTS
101k
S4
AISHELL-1/2 + TTS
30k
TABLE III: Training data composition and nominal corpus duration at each stage. The reported durations refer to the listed stored training corpora rather than cumulative exposure after online resampling. Online speaker mixing and noise augmentation are applied during training and are not counted as additional stored hours.
ID
System
pos-CER ↓
FR ↓
RR ↑
U↑
Notes
B1
Qwen3-ASR-1.7B [ 7 ]
32.21
0.00
0.00
33.89
Direct ASR
B2
CAM++ [ 40 ] + Qwen3-ASR [ 7 ]
46.26
27.05
89.66
71.70
Speaker Gating & ASR; τ=0.28
B3
Qwen2.5-Omni-7B [ 41 ]
82.08
28.08
27.00
22.46
Audio-LLM Prompting
B4
MOSS-Transcribe-Diarize [ 30 ]
51.44
35.92
93.46
71.01
Multi-talker ASR & match
B5
WeSep [ 42 ] + Qwen3-ASR [ 7 ]
73.99
1.25
29.11
27.56
Speech separation & ASR
B6
CocktailASR-1 [ 18 ]
29.19
9.60
66.67
68.74
Wake as ref.
TABLE IV: Results on midea-test . All metric values are percentages, and U is computed from unrounded rates before being rounded to two decimal places. Qwen3-ASR components use Qwen3-ASR-1.7B. The B2 cosine-similarity threshold τ is selected by scanning this same test set.
Fig. 3: pos-CER, RR, and unified score U of the six baselines and of the proposed model after Stage 1 and Stage 4 on midea-test . B2 is shown at its best threshold.
Fig. 4: Effect of the training stages on midea-test . (a) Fraction of positives accepted and rejection rate on negatives. (b) Unified score U .
Fig. 5: Threshold scan of the speaker-gated cascade B2 on midea-test . The dotted line marks the unified score of the proposed model after Stage 4.
Stage
Acc.
CER acc
SER acc
pos-CER
RR
1
54.18 (739)
21.81
57.92
43.47
98.10
2
79.33 (1082)
12.49
25.51
29.15
93.25
3
86.14 (1175)
7.71
13.96
20.47
90.72
4
90.54 (1235)
6.85
10.53
16.15
91.56
TABLE V: Stage-wise breakdown on midea-test . “Acc.” Is the Fraction (Number) of Accepted Positives; CER and SER Are Computed on Accepted Positives Only.
Subset
N
CER ↓
FR ↓
Standby, quiet, 5 m
298
0.18
0.00
Fan level 3, quiet, 3 m
300
2.72
0.00
Fan level 5, quiet, 5 m
299
0.86
0.00
Fan level 6, quiet, 5 m
298
2.89
0.00
Fan level 7, quiet, 4 m
300
5.14
0.00
Standby, external noise, 3 m
300
7.41
0.00
TABLE VI: Additional validation results on xiaoji-test , grouped by acoustic condition and ordered by fan level. N denotes the number of recordings. All metric values are percentages.
Fig. 6: CER on the TS-AISHELL target-speaker test sets as a function of the text-cue length (6000 evaluation instances per point).
Target speech extraction (TSE) typically relies on pre-recorded high-quality enrollment speech, which disrupts user experience and limits feasibility in spontaneous interaction. In this paper, we propose Enroll-on-Wakeup (EoW), a novel framework where the wake-word segment, captured naturally during human-machine interaction, is automatically utilized as the enrollment reference. This eliminates the need for pre-collected speech to enable a seamless experience. We perform the first systematic study of EoW-TSE, evaluating advanced discriminative and generative models under real diverse acoustic conditions. Given the short and noisy nature of wake-word segments, we investigate enrollment augmentation using LLM-based TTS. Results show that while current TSE models face performance degradation in EoW-TSE, TTS-based assistance significantly enhances the listening experience, though gaps remain in speech recognition accuracy.
Yiming Yang, Guangyong Wang, Haixin Guan +1
Shanghai Normal University, Shanghai, China · Unisound AI Technology Co., Ltd., Beijing, China
Target-speaker automatic speech recognition (TS-ASR) aims to recognize a designated speaker while suppressing interfering speech in multi-talker environments. Conventional TS-ASR typically relies on an enrollment utterance, whereas text-guided methods use known lexical content, such as a wake word, to identify the target speaker from the observed mixture. These two cues provide complementary information but are usually studied separately. We propose a Unified Dual-Cue TS-ASR framework that supports text cues, enrollment speech, or both within a single model. Text cues interact with the mixture representation to extract target-speaker information conditioned on known lexical content, while an independent enrollment utterance provides complementary speaker information. Cross-attention cue-conditioning modules are integrated into shared Conformer blocks, and negative-cue sampling provides cue-validity supervision during dual-cue training. Experiments on 30,000 two-speaker mixtures across five recording/domain conditions and four oracle text-cue lengths show that, with five-character text cues, the concatenated dual-cue method achieves 8.80% CER, compared with 17.32% for text-only and 29.06% for enrollment-only inference. It also outperforms parallel dual-cue fusion (9.49% CER) and yields lower dual-cue CER across all five evaluation subsets. These results demonstrate the benefit of jointly exploiting complementary lexical and speaker information for target-speaker ASR.
We introduce target-speaker unlearning ASR (TSU-ASR) task in a fully end-to-end framework for multi-speaker ASR and diarization. Given a multi-speaker utterance and a set of opt-out speakers who do not wish to have their speech transcribed, the task requires an ASR system to transcribe all speakers except the opt-out ones, while still indicating when those speakers are active. As a first step towards tackling this task, we introduce a novel, light-weight Enrollment-Conditioned Gating (ECG) module attachable to a frozen dual-stream speech LLM that enables ASR for new opt-out speakers dynamically during inference, even those who were not seen during initial ECG training phase. Our experiments on both AMI (English) and AliMeeting (Mandarin) datasets show that speech transcription accuracy for corresponding opt-out words or characters falls from 72.3% to 48.2% and from 73.6% to 27.3%, respectively, while retained speakers' transcription error rates maintain more or less the same. Our approach provides a practical solution for modern video conferencing platforms, allowing speakers to dynamically opt-out from automated AI transcriptions without forcefully leaving the meeting sessions, enabling a privacy-preserving interface for potentially millions of online meetings daily.