Organizations: Kyoto University · Meta · Brno University of Technology · School of Electronic Information, Wuhan University · Academia Sinica · National Institute of Information and Communications Technology, Japan
Large language model (LLM)-based autoregressive speech enhancement (SE) produces natural speech using learned clean-speech priors, but may hallucinate content unsupported by the input. Deterministic SE better preserves observation-coupled evidence, yet often retains residual noise or local distortion. We propose an evidence-grounded generative SE framework that uses a deterministic estimate as imperfect evidence. A Whisper-guided DPRNN produces an enhanced waveform, which is blended with the observation and tokenized into a discrete evidence sequence. The evidence conditions an autoregressive clean-speech token generator and is reused during decoding through Code-Space Grounding (CSG), which penalizes candidates according to their Hamming distance in the factorized finite-scalar-quantized (FSQ) space. Because the appropriate grounding strength depends on acoustic difficulty, we introduce SNR-Conditioned CSG (SNR-CSG), which maps a calibrated residual-SNR estimate to an utterance-level strength and constructs an adaptive grounded anchor. Although grounding improves content fidelity, the anchor may retain local acoustic defects inherited from the evidence. Since such defects are predominantly local in the FSQ space, nearby tokens may provide better acoustic realizations without large departures from the observation-supported trajectory. We therefore propose Grounded Neighborhood Refinement with LLM ranking (GNR-LLM). It performs one additional teacher-forced pass conditioned on the grounded-anchor history, intersects the LLM top-K candidates with a local FSQ Hamming neighborhood. Experiments on in-domain, controlled-SNR, and DNS no-reverb conditions show that SNR-CSG provides robust automatic grounding, while GNR-LLM substantially improves low-SNR perceptual quality without sacrificing content fidelity.
Figures & tables
Fig. 1: Overview of the proposed evidence-grounded generative SE framework. A Whisper-guided deterministic enhancer constructs an observation-coupled waveform estimate, which is blended with the noisy observation and tokenized into FSQ evidence. The evidence and continuous Whisper representations condition the autoregressive generator. During decoding, fixed CSG or SNR-CSG penalizes unsupported code-space departures; SNR-CSG selects one utterance-level grounding strength from calibrated residual SNR and applies it globally throughout decoding. The generated token sequence is converted to waveform speech by the common synthesis backend.
System [D,C,Y]
Development Set
Test Set
Faithfulness & Preservation
Perceptual Quality
Faithfulness & Preservation
Perceptual Quality
Text WER (%) ↓
WER (%) ↓
LPS ↑
SpBERT ↑
SECS ↑
P808 ↑
UTMOS ↑
Text WER (%) ↓
WER (%) ↓
LPS ↑
SpBERT ↑
SECS ↑
P808 ↑
UTMOS ↑
Reference / Input
Clean
-
3.7
1.000
1.000
1.000
3.862
4.088
-
3.2
1.000
1.000
1.000
3.843
4.086
Noisy
-
6.3
0.815
0.691
0.873
2.911
2.127
-
5.6
0.836
0.711
0.889
2.940
2.184
Discriminative SE
TABLE I: Comparison between discriminative and LLM-based generative speech enhancement on the development and test sets. Each LLM configuration is annotated as [D,C,Y] , denoting the discrete input, continuous input, and autoregressive target, respectively. Text WER and WER are reported in percentage (%). Text WER is applicable only to configurations with joint text prediction. The evaluation distinguishes faithfulness and preservation from perceptual quality . Boldface denotes the best result within each model subgroup, while underlining denotes the best result among all enhancement systems. Ties are marked consistently. Clean and noisy signals are shown as references and excluded from model ranking.
Split
Model
STOI ↑
ESTOI ↑
PESQ ↑
SI-SDR ↑
Dev
E1-DPRNN
0.919
0.829
2.253
16.52
E2-WhspDPRNN
0.933
0.855
2.457
17.32
Test
E1-DPRNN
0.924
0.836
2.311
16.50
E2-WhspDPRNN
0.937
0.861
2.522
17.30
TABLE II: Effect of Whisper-derived information injection on DPRNN-based discriminative speech enhancement. Higher values indicate better intelligibility, perceptual quality, and signal-level fidelity.
Fig. 2: Effect of the interpolation weight ρ on content faithfulness (based on system G1), speaker preservation, and perceptual quality. Increasing ρ generally improves perceptual quality but degrades speaker similarity, revealing a clear quality–preservation trade-off. The selected operating point, ρ=0.6 , is highlighted by the star marker and vertical dotted line.
Fig. 3: FSQ code-space analysis. (a) Frame-level Hamming-distance distributions for deterministic evidence distortion, generative deviation, and random token pairs. (b) AUC for separating deterministic distortion from generative deviation under the native 38 factorization, progressively coarser groupings, and a flat token-identity relation. All statistics are computed from archived frame-level token sequences.
Fig. 4: Reliability–naturalness trade-off on the test set. G5 (Blend) and B5 (Free) are ungrounded references, UD is the protocol-matched no-grounding anchor, and the fixed-CSG points trace increasing grounding strength. SNR-CSG is shown as the automatic utterance-adaptive operating point. Lower WER and bank WER (the WER computed on the bank subset used to evaluate severe content deviations) and higher UTMOS are better.
Method
Dev.
Test
WER ↓
P808 ↑
UTMOS ↑
WER ↓
P808 ↑
UTMOS ↑
B5 (Free)
10.1
3.575
3.779
9.0
3.559
3.754
UD
10.0
3.578
3.782
9.3
3.558
3.754
CSG ( λ=0.25 )
10.0
3.560
3.746
9.0
3.545
3.733
CSG ( λ=0.5 )
10.0
3.542
3.707
8.6
3.533
3.693
CSG ( λ=1 )
10.2
3.552
3.717
8.5
3.488
3.589
TABLE III: Main grounding results on the development and test sets. WER is reported in percentage (%).
Grounding strength
−5 dB
0 dB
5 dB
10 dB
15 dB
λ=0 (UD)
26.3
13.7
7.6
4.8
4.3
λ=0.1
26.5
14.4
7.5
4.9
4.3
λ=0.25
26.0
13.4
7.1
4.9
4.2
λ=0.5
26.0
13.0
6.8
5.0
4.3
λ=1.0
24.8
12.5
7.2
5.2
4.3
λ=2.0
24.0
12.9
7.4
5.1
4.2
TABLE IV: WER (%) of fixed-strength CSG under different SNR conditions. Only grounding strengths evaluated on all five conditions are included. The best result in each column is shown in bold. The base model is B5 (Free), and all CSG rows use the same decoding protocol.
Fig. 5: Adaptive grounding behavior across acoustic conditions. As SNR increases, both the mean grounding strength and the fraction of modified positions decrease, showing that SNR-CSG automatically relaxes grounding under cleaner conditions.
Fig. 6: Geometry of the grounded anchors across SNR conditions. (a) The grounded anchor is consistently closer to the clean-token trajectory than the raw evidence. (b) The anchor remains within one FSQ coordinate of the evidence on average, indicating that grounding improves the token realization without making a large departure from the observation-supported region.
Fig. 7: Distribution of the clean token relative to the grounded-neighborhood candidate set. Each bar separates positions where the grounded anchor is already clean, where the clean token is covered at different LLM ranks, and where the clean token is absent from the candidate set.
Fig. 8: Analysis of grounded-neighborhood search across SNR conditions. (a) Candidate coverage for two neighborhood configurations and the oracle-correction potential of the selected K=50,R=3 configuration. (b) Accepted refinement edits and the median LLM-score margin between the selected alternative and the grounded anchor.
Fig. 9: Tail robustness using raw utterance-level WER. The tail subsets are selected once according to the UD ranking and then frozen for all methods. (a) Mean raw WER on increasingly large worst-case subsets. (b) Full-test WER percentiles and the fraction of utterances with WER>0.5 . Lower values are better.
SNR
Metric
B5
SNR
G20
G50
Oracle
−5
WER
26.7
24.4
24.3
23.9
15.3
P808
3.453
3.254
3.353
3.362
3.397
UTMOS
3.592
2.993
3.267
3.299
3.352
0
WER
13.5
12.2
12.5
12.5
8.0
P808
3.528
3.430
3.490
3.500
3.508
UTMOS
3.715
3.469
3.590
3.602
3.595
TABLE V: Controlled-SNR results. Each entry reports WER (%), P808, or UTMOS. B5 denotes B5 (Free); SNR denotes SNR-CSG; G20 and G50 denote GNR-LLM with (K,R)=(20,2) and (50,3) , respectively. Oracle uses clean-token information and is not deployable; it is excluded from boldface comparisons. Averages are macro-averages over the five SNR conditions. ∗ The 10-dB G50 UTMOS value should be verified.
Method
WER ↓
P808 ↑
UTMOS ↑
G5
16.5
3.840
3.870
B5 (Free)
9.2
3.840
3.840
UD
9.2
3.851
3.840
Condition-tuned CSG
8.6
3.847
3.836
SNR-CSG
8.4
3.847
3.836
GNR-LLM
8.6
3.869
3.829
TABLE VI: Results on the DNS no-reverb condition. WER is reported in percentage (%). SNR-CSG and GNR-LLM use frozen configurations without DNS-specific tuning.
Speech enhancement improves intelligibility and listening quality by suppressing noise, reverberation, and competing speakers in recorded or live audio.