LL-SDR: Low-Latency Speech enhancement through Discrete Representations

Jingyi Li; Luca Della Libera; Mirco Ravanelli; Mingkun Xu; Cem Subakan

cs.SDMar 10, 2026

LL-SDR: Low-Latency Speech enhancement through Discrete Representations

Authors: Jingyi Li, Luca Della Libera, Mirco Ravanelli, Mingkun Xu, Cem Subakan

Organizations: Guangdong Institute of Intelligence Science and Technology · Concordia University · Mila – Quebec AI Institute · Laval University

Abstract

Many speech enhancement (SE) methods rely on continuous representations. Recently, discrete audio tokens have been explored to enable autoregressive generation for SE. However, it remains unclear whether discretization itself consistently improves SE performance. In this paper, we introduce LL-SDR, a token-based speech enhancement framework that explicitly leverages discretization to better separate speech and noise. Our first contribution is a Variance-Ordered Residual Vector Quantizer (VO-RVQ), designed to disentangle speech and noise distributions during tokenization. Second, we propose a latent-space discriminator to better align enhanced embeddings with semantic embeddings. Experiments show that LL-SDR outperforms continuous baselines and matches the performance of autoregressive token-based approaches. Despite its strong enhancement performance, LL-SDR remains lightweight and efficient, requiring only 40G MACs for a single forward pass on a 10-second 16 kHz speech segment and achieving low-latency inference with an RTF of 0.01 on GPU and 0.24 on CPU. Demos and source code are available at our project websites.

Figures & tables

Fig. 1: LL-SDR speech enhancement framework: a noisy waveform is encoded and enhanced in the discrete space via our variance-ordered quantizer where $N_{e}$ is four and $N_{n}$ is one, then decoded back to waveform. The triangle structure here illustrates that different quantizers have different dimensional sizes; a HuBERT-based semantic discriminator to align the enhanced representation with clean speech semantics.

Fig. 2: Visualization of latent embeddings using UMAP and t-SNE. Blue points denote noisy embeddings (the output of last Nn codebooks), and red points denote enhanced embeddings (the mean of first Ne codebooks). The top row shows the baseline model, where noisy and enhanced representations largely overlap. The bottom row corresponds to the proposed triangle (variance-ordered) quantization structure, which exhibits clearer structural separation and more organized distribution patterns.

Model	Type	Discrete	Reverb			No Reverb			Real
Model	Type	Discrete	OVRL $\uparrow$	SIG $\uparrow$	BAK $\uparrow$	OVRL $\uparrow$	SIG $\uparrow$	BAK $\uparrow$	OVRL $\uparrow$	SIG $\uparrow$	BAK $\uparrow$
Noisy	–	–	1.39	1.76	1.50	2.48	3.39	2.62	2.26	3.05	2.51
Conv-TasNet	NAR	✗	2.01	2.42	2.71	3.00	3.09	3.34	-	-	-
DEMUCS	NAR	✗	2.55	2.86	3.90	3.35	3.58	4.15	2.99	3.26	4.03
FRCRN	NAR	✗	2.28	2.93	2.92	3.34	3.58	4.13	3.04	3.37	3.98
VoiceFixer	NAR	✗	3.13	3.43	4.02	3.25	3.50	4.11	3.08*	3.37*	4.00*

TABLE I: DNSMOS (OVRL / SIG / BAK) under reverberant, non-reverberant, and real-recording conditions. Most values for compared methods are taken from prior work [ 6 ] and [ 23 ] . Values marked with * are obtained from our own evaluation. Best and second-best results are highlighted.

Model	Params (M) $\downarrow$	MACs (G) $\downarrow$	RTF (H100) $\downarrow$	RTF (CPU) $\downarrow$
FRCRN	53	77	0.04	0.99
VoiceFixer	117	777	0.01	0.52
LLaSE-G1 (single)	1319	145	0.37	2.43
LL-SDR (Ours)	74	40	0.01	0.24

TABLE II: Model complexity and runtime comparison. Best and second-best results are highlighted.

Model	Reverb			Non-reverb
Model	STFT $\downarrow$	Mel $\downarrow$	LPS $\downarrow$	STFT $\downarrow$	Mel $\downarrow$	LPS $\downarrow$
Noisy	244.41	90.78	20.43	182.97	73.15	13.97
FRCRN	216.18	76.37	13.72	125.17	44.39	3.60
VoiceFixer	192.51	68.98	9.77	180.40	63.22	8.24
LLaSE-G1	258.73	99.10	20.32	216.47	83.54	12.79
LL-SDR (Ours)	208.44	74.47	11.85	161.95	53.26	6.40

TABLE III: In-distribution Test:Distortion Metrics Under Reverberant and Non-Reverberant Conditions in DNS-Challenge 2020 Non-Blind Test Set.

Method	RTF (H100) $\downarrow$	RTF (CPU) $\downarrow$	STFT $\downarrow$	Mel $\downarrow$	LPS $\downarrow$
Noisy	-	-	155.72	54.23	14.76
FRCRN	0.04	0.99	100.10	29.81	4.93
VoiceFixer	0.01	0.52	123.53	30.86	6.56
LLaSE	0.37	2.43	183.25	60.55	13.94
LL–SDR	0.01	0.24	136.84	36.77	8.15

TABLE IV: Out-of-distribution distortion metrics on the VoiceBank-DEMAND test set.

Method	Accuracy $\uparrow$	Macro Recall $\uparrow$	Macro F1 $\uparrow$
RVQ	50.46	50.46	34.37
VO-RVQ	71.33	71.33	71.25

TABLE V: Spectral clustering results. Enhanced and noisy embeddings are first combined without labels, clustered using spectral clustering, and then compared with the true labels.

Method	Reverb			No Reverb
Method	OVRL $\uparrow$	SIG $\uparrow$	BAK $\uparrow$	OVRL $\uparrow$	SIG $\uparrow$	BAK $\uparrow$
Continuous	2.95	3.30	3.84	3.30	3.56	4.09
RVQ	3.13	3.43	4.01	3.36	3.59	4.16
VO-RVQ	3.14	3.45	3.99	3.39	3.62	4.17
VO-RVQ + HuBERT	3.22	3.51	4.03	3.40	3.63	4.16

TABLE VI: Ablation study One. Best and second-best results are highlighted.

Method	Reverb			No Reverb
Method	OVRL $\uparrow$	SIG $\uparrow$	BAK $\uparrow$	OVRL $\uparrow$	SIG $\uparrow$	BAK $\uparrow$
w/o $\mathcal{L}_{\text{NCE}}$	3.15	3.45	4.01	3.37	3.61	4.15
w/o $\mathcal{L}_{\text{sem}}$	3.13	3.45	3.96	3.38	3.61	4.16
LL-SDR (Ours)	3.27	3.55	4.08	3.39	3.62	4.16

TABLE VII: Ablation study Two. Best results are highlighted.

Method	Reverb			No Reverb			Real
Method	OVRL $\uparrow$	SIG $\uparrow$	BAK $\uparrow$	OVRL $\uparrow$	SIG $\uparrow$	BAK $\uparrow$	OVRL $\uparrow$	SIG $\uparrow$	BAK $\uparrow$
Shuffle One	3.19	3.50	4.00	3.38	3.61	4.15	3.12	3.44	3.99
Shuffle Two	3.12	3.44	3.97	3.38	3.62	4.15	3.12	3.43	4.00
Shuffle Three	3.18	3.48	4.00	3.36	3.60	4.14	3.12	3.44	3.99
LL-SDR (Ours)	3.27	3.55	4.08	3.39	3.62	4.16	3.16	3.45	4.05

TABLE VIII: Ablation study Three. Best results are highlighted.

Team/Model	Team Rank	OVRL	NISQA	UTMOS
Bobbsun	1	2.88	3.22	2.09
rc	2	2.83	2.92	2.03
FRCRN	NA	1.52	1.30	1.32
VoiceFixer	NA	2.83	3.40	1.98
LLaSE–G1	NA	2.62	3.02	1.94
LL-SDR (Ours)	NA	2.93	2.92	1.95

TABLE IX: Speech enhancement results on the 2025 URGENT Challenge blind test set.

Fig. 3: Left: DNSMOS OVRL scores for different Number of codebooks settings. Right: DNSMOS OVRL scores for different $N_{e}$ settings.OVRL scores are averaged over three independent runs for each setting. The solid lines indicate the mean OVRL scores under reverberant and non-reverberant conditions, while the shaded regions represent the standard deviation across the three runs.

Speech Enhancement

Speech enhancement improves intelligibility and listening quality by suppressing noise, reverberation, and competing speakers in recorded or live audio.

See all 128 papers

DriftSE: Speech Enhancement with Generative Drifting
Sep 10, 2026Liang Xu, Diego Caviedes-Nozal, W. Bastiaan Kleijn +2One-Step Generative Modeling Speech Enhancement
Victoria University of Wellington, New Zealand · GN Advanced Science, Denmark · Lincoln University
Absorbing Discrete Diffusion for Speech Enhancement
Feb 25, 2026Philippe GonzalezSpeech Enhancement Diffusion Models
Department of Health Technology, Technical University of Denmark
UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancement
Oct 23, 2025Haoyin Yan, Chengwei Liu, Shaofei Xue +4Target Speaker Extraction Decoder-Only Language Models
Qwen Business Unit of Alibaba, China · Tongyi AI Lab of Alibaba, China

Sep 10, 2026J/K move · Enter open · S save

DriftSE: Speech Enhancement with Generative Drifting

Liang Xu, Diego Caviedes-Nozal, W. Bastiaan Kleijn, Longfei Felix Yan+1

Victoria University of Wellington, New Zealand · GN Advanced Science, Denmark · Lincoln University

One-Step Generative Modeling Speech Enhancement Speech Dereverberation

Open paper AskPDF arXiv

We propose DriftSE, a novel one-step generative framework for speech enhancement formulated as a latent distribution equilibrium problem. During training, the drifting field aligns the generator's pushforward distribution with the clean speech manifold through drifting in a latent domain. During inference, the drifting process is discarded, enabling one-step generation. We establish that its enhancement quality depends fundamentally on the choice of latent representation. Semantic latents preserve phonetic structure but fail to capture physical acoustic cues, whereas acoustic latents reconstruct the physical signal but risk linguistic hallucination. Therefore, we introduce dual-latent drifting, performing parallel drifting in both semantic and acoustic latents to simultaneously preserve phonetic intelligibility and acoustic fidelity. Additionally, we demonstrate that DriftSE enables fully unpaired training by aligning latent distributions rather than exact point-wise targets. Consequently, DriftSE facilitates cross-dataset learning in the absence of paired noisy-clean samples. Moreover, DriftSE exhibits broad architectural flexibility across different generator backbones. Extensive evaluations on additive denoising and convolutive dereverberation demonstrate robust one-step enhancement across both offline and real-time causal settings. Notably, DriftSE achieves state-of-the-art word error rates across all four evaluated datasets while strictly operating at 1 NFE. Code and audio examples are available online.

Abstract

Speech Enhancement

Explore similar work

DriftSE: Speech Enhancement with Generative Drifting

Absorbing Discrete Diffusion for Speech Enhancement

UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancement