Driven by recent advancements in hearing assistive devices, target speaker extraction (TSE) in multi-talker environments has drawn significant attention. To address this, hearing technologies identify attention using neural signals recorded via electroencephalography (EEG) to guide TSE, enabling neuro-steered auditory attention decoding. However, most existing techniques rely on time-frequency masking, which suffers from low fidelity in spectro-temporally overlapping regions and is often restricted to narrowband audio. In contrast, we introduce NeuroCodec, which, to the best of our knowledge, is the first generative EEG-guided TSE framework operating in the latent space of a neural audio codec. Specifically, NeuroCodec models spatial EEG dependencies via graph convolutions and fuses cross-modal EEG and speech representations through attention mechanisms and Mamba-based temporal modeling. We further introduce a six-phase training curriculum that progressively incorporates latent reconstruction, multi-scale spectral, and adversarial objectives to ensure stable, high-fidelity speech generation. Experiments on the public Cocktail Party and KUL datasets show NeuroCodec outperforms baselines, across several perceptual metrics. A listening experiment with 20 participants validated that NeuroCodec offers greater perceptual accuracy, achieving a mean score of 82.24 compared with 49.63 for M3ANet. Additionally, the proposed method is seen to enable 44.1 kHz reconstruction with spectral content extending up to 20 kHz, further supporting the perceptual results. Finally, we extend NeuroCodec with a causal streaming formulation suited for real-time processing, providing a basis for hearing assistive devices.
Figures & tables
Fig. 1: Illustration of NeuroCodec architecture. A frozen DAC encoder extracts mixture latents Zmix , which are fused with EEG embeddings through stacked CM-S3 blocks to predict target latents Ztgt . The predicted latents are quantized via codebooks ( Q1 – Q9 ), and a frozen DAC decoder reconstructs the target speech waveform s^ . The EEG encoder captures spatial-temporal neural features, and multi-period/multi-band discriminators provide adversarial supervision.
Model
STOI
ESTOI
PESQ
(a) Cocktail Party Dataset
BASEN [ 19 ]
0.790
0.715
1.91
M3ANet [ 14 ]
0.812
0.725
2.23
NeuroCodec
0.8273
0.721
2.31
(b) KUL Dataset
BISS [ 18 ]
0.106
–
-0.045
TABLE I: STOI, ESTOI, and PESQ results on Cocktail Party and KUL
Fig. 2: Qualitative spectrogram comparison on the Cocktail Party dataset: (i) mixture, (ii) ground-truth target, (iii) M3ANet output, and (iv) NeuroCodec output. Solid boxes highlight NeuroCodec’s preserved content above 7 kHz, while dashed region mark overlapping-speech regions with reduced interference. Note that M3ANet does not preserve content above 7 kHz, nor does it avoid speaker leakage within the dashed region.
TABLE II: Ablation studies of NeuroCodec on the KUL dataset
Fig. 3: MUSHRA listening-test results on the Cocktail Party dataset. Bandwidth-matched NeuroCodec outperforms M3ANet, while native NeuroCodec achieves a substantially larger perceptual gain.
Target speaker extraction (TSE) aims to recover a target speaker's speech from a speech mixture using an enrollment utterance. Recent codec-based generative approaches formulate TSE as conditional speech token generation and achieve promising perceptual quality. However, most existing systems rely on unified codec tokens that entangle linguistic content, speaker characteristics, and acoustic details, forcing a single model to jointly handle target selection, content semantic, and acoustic reconstruction. In this paper, we investigate whether disentangled semantic-acoustic codec representations are better suited to TSE. We first show that simply replacing unified codec tokens with semantic-acoustic tokens is insufficient and can even degrade performance when both remain handled by a unified generative model. Motivated by this observation, we propose a task-aware discriminative-generative framework, where a discriminative branch extracts target-related semantic information, while a generative branch predicts coarse acoustic tokens and refines them under semantic guidance. This specialization separates what is said from how it sounds while coordinating both factors in a factorized codec space. Experiments on Libri2Mix show that our method reduces dWER from 0.388 to 0.118 compared with a unified-codec baseline, while maintaining competitive perceptual quality and speaker consistency. Ablation studies confirm that the gains arise from task-aware semantic-acoustic modeling rather than representation disentanglement alone.
Wang Xiang, Bang Zeng, Ming Li
Wuhan University · The Chinese University of Hong Kong, Shenzhen
Recent end-to-end models for EEG-guided target speech extraction report impressive results, underscoring potential for neuro-steered hearing technologies. However, our analysis reveals that high within-trial performance can be driven by trial-specific EEG structure that acts as shortcuts for target selection, leading to poor generalization on unseen trials. To overcome this gap, we propose TRUST-TSE, a two-stage framework to mitigate shortcut learning. By introducing contrastive pretraining with attended-speaker negative sampling, we encourage the EEG encoder to capture fine-grained EEG--speech alignment while suppressing trial-identity cues. We also employ a confidence-weighted extraction objective based on EEG--source similarity to guide extraction using the learned representations. Experiments on KUL and DTU datasets show that TRUST-TSE outperforms end-to-end baselines under strict cross-trial protocols, addressing a key reliability bottleneck of existing approaches.
Wonchul Shin, Inyong Choi, Kyogu Lee
Department of Intelligence and Information, Seoul National University, Republic of Korea · Department of Communication Sciences and Disorders, University of Iowa, United States · Interdisciplinary Program in Artificial Intelligence, Seoul National University, Republic of Korea +1
Target speaker extraction (TSE) aims to recover a target speaker's speech from a mixture using a reference utterance as a cue. Most TSE systems adopt conditional auto-encoder architectures with one-step inference. Inspired by test-time scaling, we propose a training-free multi-step inference method that enables iterative refinement with a frozen pretrained model. At each step, new candidates are generated by interpolating the original mixture and the previous estimate, and the best candidate is selected for further refinement until convergence. Experiments show that, when ground-truth target speech is available, optimizing an intrusive metric (SI-SDRi) yields consistent gains across multiple evaluation metrics. Without ground truth, optimizing non-intrusive metrics (UTMOS or SpkSim) improves the corresponding metric but may hurt others. We therefore introduce joint metric optimization to balance these objectives, enabling controllable extraction preferences for practical deployment.
Zhenghai You, Ying Shi, Lantian Li +1
Beijing University of Posts and Telecommunications, China · Center for Speech and Language Technologies, BNRist, Tsinghua University, China