A Speech-LLM Framework for Enrollment-Free Wake-Word-Guided Target-Speaker Recognition in Complex Noisy Scenarios for Seamless Human-Machine Interaction
Authors: Yuxiang Mei, Dongxing Xu, Yanhua Long
Organizations: Shanghai Normal University, Shanghai, China · Unisound AI Technology Co., Ltd., Beijing, China
Target-speaker automatic speech recognition (TS-ASR) is essential for reliable human-machine interaction in complex acoustic environments, where overlapping speech and background noise make it difficult to identify and recognize the intended speaker. Existing approaches typically rely on speaker enrollment or cascaded processing pipelines, limiting their flexibility in practical applications. To address these limitations, we propose an enrollment-free Speech-LLM framework that leverages wake-word text as a lexical cue to guide target-speaker recognition. Specifically, a text-cue encoder extracts lexical representations from the wake word, which are integrated into a pretrained speech encoder through cross-attention to establish an implicit association between the wake-word occurrence and the target speaker. This enables the model to selectively recognize subsequent commands from the target speaker without requiring enrollment speech or explicit wake-word boundaries. Furthermore, a large language model (LLM) unifies command transcription and non-target speaker rejection within a single autoregressive framework. To improve robustness against complex acoustic interference, we develop a large-scale speech synthesis and simulation pipeline with a progressive four-stage training strategy. Experimental results demonstrate that the proposed framework achieves a command character error rate of 16.15% and a non-target speaker rejection rate of 91.56%, yielding a unified score of 87.70%, compared with 71.70% for the strongest enrollment-free baseline. These results demonstrate the effectiveness of wake-word-guided speaker selection and highlight the potential of integrating target-speaker recognition and rejection within a unified Speech-LLM framework for robust human-machine interaction.
Figures & tables
Fig. 1: Overview of the proposed framework. The wake-word text conditions a pretrained speech encoder through cross-attention, producing acoustic representations selective to the speaker who uttered the cue. The LLM decoder generates either the target command or an explicit rejection token. Dashed modules are used only during training.
Condition
Target sequence y
Target command present
<psok>,w,<peok>,c
Command from another speaker
<psok>,w,<peok>,<reject/>
Wake-word cue absent
<reject/>
TABLE I: Target sequence for each input condition, where w and c denote the wake-word text and command transcript, respectively.
Fig. 2: Overview of the training-data generation pipeline. Speaker-labelled source speech and Qwen3-TTS outputs are composed by spk-mix according to the speaker relation: same-speaker pairs form positive samples, different-speaker pairs form negative samples, and cue-absent samples form rejection cases. The resulting samples are audited and augmented online with interfering talkers and background noise during training.
Stage
Cue enc.
Speech enc.
CTC
Proj.+LLM (LoRA)
Spk. head
Total Param. (M)
Trainable Param. (M)
1
✓
✓
✓
–
–
37.18
37.18
2
frozen
frozen
–
✓
–
713.31
48.60
3
✓
✓
–
✓
–
713.31
83.61
4
✓
✓
–
✓
✓
715.51
85.37
TABLE II: Trainable modules and parameter counts at each training stage.
Stage
Datasets
Duration (h)
S1
AISHELL-1/2 + WenetSpeech
3k
S2
AISHELL-1/2 + WenetSpeech + TTS
101k
S3
AISHELL-1/2 + WenetSpeech + TTS
101k
S4
AISHELL-1/2 + TTS
30k
TABLE III: Training data composition and nominal corpus duration at each stage. The reported durations refer to the listed stored training corpora rather than cumulative exposure after online resampling. Online speaker mixing and noise augmentation are applied during training and are not counted as additional stored hours.
ID
System
pos-CER ↓
FR ↓
RR ↑
U↑
Notes
B1
Qwen3-ASR-1.7B [ 7 ]
32.21
0.00
0.00
33.89
Direct ASR
B2
CAM++ [ 40 ] + Qwen3-ASR [ 7 ]
46.26
27.05
89.66
71.70
Speaker Gating & ASR; τ=0.28
B3
Qwen2.5-Omni-7B [ 41 ]
82.08
28.08
27.00
22.46
Audio-LLM Prompting
B4
MOSS-Transcribe-Diarize [ 30 ]
51.44
35.92
93.46
71.01
Multi-talker ASR & match
B5
WeSep [ 42 ] + Qwen3-ASR [ 7 ]
73.99
1.25
29.11
27.56
Speech separation & ASR
B6
CocktailASR-1 [ 18 ]
29.19
9.60
66.67
68.74
Wake as ref.
TABLE IV: Results on midea-test . All metric values are percentages, and U is computed from unrounded rates before being rounded to two decimal places. Qwen3-ASR components use Qwen3-ASR-1.7B. The B2 cosine-similarity threshold τ is selected by scanning this same test set.
Fig. 3: pos-CER, RR, and unified score U of the six baselines and of the proposed model after Stage 1 and Stage 4 on midea-test . B2 is shown at its best threshold.
Fig. 4: Effect of the training stages on midea-test . (a) Fraction of positives accepted and rejection rate on negatives. (b) Unified score U .
Fig. 5: Threshold scan of the speaker-gated cascade B2 on midea-test . The dotted line marks the unified score of the proposed model after Stage 4.
Stage
Acc.
CER acc
SER acc
pos-CER
RR
1
54.18 (739)
21.81
57.92
43.47
98.10
2
79.33 (1082)
12.49
25.51
29.15
93.25
3
86.14 (1175)
7.71
13.96
20.47
90.72
4
90.54 (1235)
6.85
10.53
16.15
91.56
TABLE V: Stage-wise breakdown on midea-test . “Acc.” Is the Fraction (Number) of Accepted Positives; CER and SER Are Computed on Accepted Positives Only.
Subset
N
CER ↓
FR ↓
Standby, quiet, 5 m
298
0.18
0.00
Fan level 3, quiet, 3 m
300
2.72
0.00
Fan level 5, quiet, 5 m
299
0.86
0.00
Fan level 6, quiet, 5 m
298
2.89
0.00
Fan level 7, quiet, 4 m
300
5.14
0.00
Standby, external noise, 3 m
300
7.41
0.00
TABLE VI: Additional validation results on xiaoji-test , grouped by acoustic condition and ordered by fan level. N denotes the number of recordings. All metric values are percentages.
Fig. 6: CER on the TS-AISHELL target-speaker test sets as a function of the text-cue length (6000 evaluation instances per point).