Noise-Aware Lombard Speech Synthesis via Flow-GRPO for Intelligibility Optimization
Authors: Zhaoyang Wang, Dingding Yao, Chengzhong Wang, Junfeng Li
Organizations: Key Laboratory of Speech Acoustics and Content Understanding, Institute of Acoustics, Chinese Academy of Sciences, Beijing 100190, China, and also with University of Chinese Academy of Sciences, Beijing 100049, China.
Voice-based conversational agents are increasingly deployed in real-world settings, where environmental noise often degrades the interactive experience. Therefore, robust acoustic adaptability is essential to maintain natural and intelligible human-computer communication. However, most existing systems remain acoustically insensitive and fail to emulate the human Lombard effect, in which speakers involuntarily adjust their vocal production to preserve intelligibility in the presence of noise. As a result, their communicative reliability degrades under adverse acoustic conditions. To remain intelligible in noisy environments, conversational agents require a mechanism to emulate noise-induced adaptive behavior during synthesis. We propose NIL-TTS, a noise-induced Lombard TTS framework. We integrate a feature-wise linear modulation (FiLM) mechanism into a flow-matching backbone for dynamic vocal adaptation. A two-stage training paradigm is utilized: initial supervised fine-tuning on paired Lombard corpora, followed by Flow-GRPO post-training to optimize for intelligibility. Evaluations confirm that NIL-TTS autonomously adjusts its output in response to varying noise conditions, exhibiting key correlates of the human Lombard effect. Driven by these adaptive modulations, the synthesized speech achieves significant intelligibility improvements that scale proportionally with noise intensity. Ultimately, this work presents a flow-matching and reinforcement learning-based advancement within the adaptive Lombard speech synthesis paradigm for challenging real-world scenarios.
Figures & tables
Fig. 1: An overview of the NIL-TTS pipeline. Left: The supervised fine-tuning (SFT) stage, where the model is trained with a conditional flow matching loss on paired plain and Lombard speech. Middle: The Flow-GRPO rollout process, which leverages an SDE solver to generate a group of exploratory sample trajectories for policy optimization. Right: The inference process, which uses an ODE solver for deterministic speech generation.
Fig. 2: Reward computation in Flow-GRPO. The objective function acts as a proxy for human auditory feedback. The intelligibility reward ( Racc ) mixes rollouts with ambient noise before ASR transcription to evaluate robustness. The speaker similarity reward ( Rspk ) utilizes a speaker encoder to penalize identity distortion against the ground truth ( xGT ).
Fig. 3: Objective ASR transcription error rates (WER) and objective intelligibility metrics (ESTOI) across four noise types (Babble, Factory, Pink, and White) and six SNR levels (6 dB to -9 dB). Error bars denote 95% confidence intervals. (a) Results from Seed-TTS test-zh . (b) Results from Seed-TTS test-en .
Conditions
Seed-TTS test-zh
Seed-TTS test-en
WER ↓
ESTOI ↑
WER ↓
ESTOI ↑
0 dB
-3 dB
-6 dB
0 dB
-3 dB
-6 dB
0 dB
-3 dB
-6 dB
0 dB
-3 dB
-6 dB
F5-TTS [ 27 ]
0.081
0.177
0.378
0.407
0.324
0.246
0.107
0.270
0.568
0.414
0.327
0.247
[2pt/2pt] F5-TTS + SS
0.057
0.123
0.278
0.420
0.339
0.264
0.086
0.183
0.498
0.394
0.321
0.254
F5-TTS + SSDRC
0.058
0.120
0.263
0.448
0.347
0.270
0.089
0.176
0.484
0.413
0.333
0.257
[2pt/2pt] MetaVoice-1B
-
-
-
-
-
-
0.419
0.651
0.895
0.420
0.331
0.248
TABLE I: Intelligibility performance in terms of WER and ESTOI, averaged across four noise types at SNRs of 0, −3 , and −6 dB on the Seed-TTS test set. † denotes models implemented by the authors.
Fig. 4: Subjective recognition accuracy across 3 SNR levels (0 dB, -3 dB, and -6 dB) under Babble noise. Error bars denote 95% confidence intervals.
Fig. 5: Comparison of acoustic features produced by the SFT-only and proposed NIL-TTS models across different SNR levels for female and male speakers. The panels show (a) fundamental frequency F0 , (b) first formant F1 , (c) second formant F2 , and (d) spectral tilt. Dashed lines with open markers denote the SFT-only model, whereas solid lines with filled markers denote the proposed model after Flow-GRPO post-training.
Fig. 6: Spectrogram comparison illustrating emergent Lombard characteristics. All samples are synthesized by the proposed NIL-TTS. Each row corresponds to a distinct test utterance generated using the same text and speaker reference, but conditioned on varying ambient noise levels (Plain, 0 dB, and -6 dB). The Plain condition represents standard, non-adapted speech generated by disabling the FiLM branch. Under noise, the model exhibits upward harmonic shifts and pronounced energy concentration in higher frequency bands.
Config
#Param .
RTF ↓
F5-TTS
336M
0.26
[2pt/2pt] MetaVoice-1B Lomb
1.2B
3.04
F5-TTS-Style Lomb
387M
0.27
NIL-TTS
382M
0.27
TABLE II: Comparison of computational costs. #Param. denotes the number of learnable parameters. The real-time factor (RTF) is measured for generating 10-s utterances on an NVIDIA RTX 3090 GPU.
TABLE III: WER obtained using different Flow-GRPO hyperparameter settings, averaged across four noise types at SNRs of 0, −3 , and −6 dB on the Seed-TTS test set.
Humans tend to speak louder and clearer in challenging environments, such as noisy conditions or when addressing hearingimpaired listeners, which is called Lombard effect. To simulate this behavior in speech synthesis systems, we introduce a flow-matching based text-to-speech (TTS) model trained with vocal effort and articulation pseudo-labels. The proposed model achieves continuous and disentangled control of vocal effort and articulation, while also enabling word-level emphasis for clarifying specific segments of an utterance. Experimental results show that these control mechanisms effectively improve clarityrelated acoustic features. Furthermore, speech-in-noise experiments demonstrate that our model successfully simulates the intelligibility gains of human clear speech in noisy conditions.
Seymanur Akti, Alexander Waibel
Karlsruhe Institute of Technology (KIT), Germany · KIT Campus Transfer (KCT), Germany · Carnegie Mellon University (CMU), USA
Speech becomes less intelligible in noisy environments, and humans naturally adapt their voice to compensate. Inspired by this behavior, we investigate whether a text-to-speech (TTS) model can be guided to produce more intelligible speech using activation steering, without retraining. We focus on two characteristics of the Lombard effect: increased vocal effort and hyper-articulation. We introduce a prompt-relative steering mechanism that prevents steering effects from accumulating during generation while allowing their strength to be adjusted dynamically. Across seen and unseen speakers and multiple languages, our method produces systematic changes in Lombard-related acoustic features, preserves speaker similarity (89-95%), and reduces WER under background noise by 7-22% at 1 dB SNR. These results show that pretrained TTS models can be dynamically controlled to generate more intelligible speech without retraining.
Seymanur Akti, Alexander Waibel
Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany · KIT Campus Transfer (KCT), Karlsruhe, Germany · Carnegie Mellon University (CMU), Pittsburgh, USA
The Lombard effect plays a key role in natural communication, particularly in noisy environments or when addressing hearing-impaired listeners. We present a controllable text-to-speech (TTS) system capable of synthesizing Lombard-like speech in a zero-shot manner without requiring Lombard-specific training data. Our approach extends F5-TTS with a learned style embedding representation and analyzes the resulting latent space using principal component analysis (PCA) to identify directions associated with Lombard-related attributes. By manipulating these directions, we obtain interpretable control over vocal effort and articulation and generate speech at different Lombard levels. Experimental results show that the proposed method preserves speaker identity and naturalness, improves intelligibility under noisy conditions, and generalizes to previously unseen speakers. These findings demonstrate that style-embedding manipulation provides an effective and scalable framework for controllable zero-shot Lombard speech synthesis.
Seymanur Akti, Alexander Waibel
KIT Campus Transfer GmbH (KCT) · Karlsruhe Institute of Technology (KIT) · Carnegie Mellon University (CMU)