Noise-Aware Lombard Speech Synthesis via Flow-GRPO for Intelligibility Optimization

Zhaoyang Wang; Dingding Yao; Chengzhong Wang; Junfeng Li

cs.SDOct 14, 2026

Noise-Aware Lombard Speech Synthesis via Flow-GRPO for Intelligibility Optimization

Authors: Zhaoyang Wang, Dingding Yao, Chengzhong Wang, Junfeng Li

Organizations: Key Laboratory of Speech Acoustics and Content Understanding, Institute of Acoustics, Chinese Academy of Sciences, Beijing 100190, China, and also with University of Chinese Academy of Sciences, Beijing 100049, China.

Abstract

Voice-based conversational agents are increasingly deployed in real-world settings, where environmental noise often degrades the interactive experience. Therefore, robust acoustic adaptability is essential to maintain natural and intelligible human-computer communication. However, most existing systems remain acoustically insensitive and fail to emulate the human Lombard effect, in which speakers involuntarily adjust their vocal production to preserve intelligibility in the presence of noise. As a result, their communicative reliability degrades under adverse acoustic conditions. To remain intelligible in noisy environments, conversational agents require a mechanism to emulate noise-induced adaptive behavior during synthesis. We propose NIL-TTS, a noise-induced Lombard TTS framework. We integrate a feature-wise linear modulation (FiLM) mechanism into a flow-matching backbone for dynamic vocal adaptation. A two-stage training paradigm is utilized: initial supervised fine-tuning on paired Lombard corpora, followed by Flow-GRPO post-training to optimize for intelligibility. Evaluations confirm that NIL-TTS autonomously adjusts its output in response to varying noise conditions, exhibiting key correlates of the human Lombard effect. Driven by these adaptive modulations, the synthesized speech achieves significant intelligibility improvements that scale proportionally with noise intensity. Ultimately, this work presents a flow-matching and reinforcement learning-based advancement within the adaptive Lombard speech synthesis paradigm for challenging real-world scenarios.

Figures & tables

Fig. 1: An overview of the NIL-TTS pipeline. Left: The supervised fine-tuning (SFT) stage, where the model is trained with a conditional flow matching loss on paired plain and Lombard speech. Middle: The Flow-GRPO rollout process, which leverages an SDE solver to generate a group of exploratory sample trajectories for policy optimization. Right: The inference process, which uses an ODE solver for deterministic speech generation.

Fig. 2: Reward computation in Flow-GRPO. The objective function acts as a proxy for human auditory feedback. The intelligibility reward ( $R_{acc}$ ) mixes rollouts with ambient noise before ASR transcription to evaluate robustness. The speaker similarity reward ( $R_{spk}$ ) utilizes a speaker encoder to penalize identity distortion against the ground truth ( $x_{GT}$ ).

Fig. 3: Objective ASR transcription error rates (WER) and objective intelligibility metrics (ESTOI) across four noise types (Babble, Factory, Pink, and White) and six SNR levels (6 dB to -9 dB). Error bars denote 95% confidence intervals. (a) Results from Seed-TTS test-zh . (b) Results from Seed-TTS test-en .

Conditions	Seed-TTS test-zh						Seed-TTS test-en
	WER $\downarrow$			ESTOI $\uparrow$			WER $\downarrow$			ESTOI $\uparrow$
	0 dB	-3 dB	-6 dB	0 dB	-3 dB	-6 dB	0 dB	-3 dB	-6 dB	0 dB	-3 dB	-6 dB
F5-TTS [ 27 ]	0.081	0.177	0.378	0.407	0.324	0.246	0.107	0.270	0.568	0.414	0.327	0.247
[2pt/2pt] F5-TTS + SS	0.057	0.123	0.278	0.420	0.339	0.264	0.086	0.183	0.498	0.394	0.321	0.254
F5-TTS + SSDRC	0.058	0.120	0.263	0.448	0.347	0.270	0.089	0.176	0.484	0.413	0.333	0.257
[2pt/2pt] MetaVoice-1B	-	-	-	-	-	-	0.419	0.651	0.895	0.420	0.331	0.248

TABLE I: Intelligibility performance in terms of WER and ESTOI, averaged across four noise types at SNRs of 0,

-3

, and

-6

dB on the Seed-TTS test set. † denotes models implemented by the authors.

Fig. 4: Subjective recognition accuracy across 3 SNR levels (0 dB, -3 dB, and -6 dB) under Babble noise. Error bars denote 95% confidence intervals.

Fig. 5: Comparison of acoustic features produced by the SFT-only and proposed NIL-TTS models across different SNR levels for female and male speakers. The panels show (a) fundamental frequency $F_{0}$ , (b) first formant $F_{1}$ , (c) second formant $F_{2}$ , and (d) spectral tilt. Dashed lines with open markers denote the SFT-only model, whereas solid lines with filled markers denote the proposed model after Flow-GRPO post-training.

Fig. 6: Spectrogram comparison illustrating emergent Lombard characteristics. All samples are synthesized by the proposed NIL-TTS. Each row corresponds to a distinct test utterance generated using the same text and speaker reference, but conditioned on varying ambient noise levels (Plain, 0 dB, and -6 dB). The Plain condition represents standard, non-adapted speech generated by disabling the FiLM branch. Under noise, the model exhibits upward harmonic shifts and pronounced energy concentration in higher frequency bands.

Config	#Param .	RTF $\downarrow$
F5-TTS	336M	0.26
[2pt/2pt] MetaVoice-1B Lomb	1.2B	3.04
F5-TTS-Style Lomb	387M	0.27
NIL-TTS	382M	0.27

TABLE II: Comparison of computational costs. #Param. denotes the number of learnable parameters. The real-time factor (RTF) is measured for generating 10-s utterances on an NVIDIA RTX 3090 GPU.

TABLE III: WER obtained using different Flow-GRPO hyperparameter settings, averaged across four noise types at SNRs of 0,

-3

, and

-6

dB on the Seed-TTS test set.

TTS Synthesis

Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.

See all 275 papers

Synthesizing the Lombard Effect: Multi-Level Control of Speech Clarity and Vocal Effort in TTS
Jun 22, 2026Seymanur Akti, Alexander WaibelTTS Synthesis Speech Prosody
Karlsruhe Institute of Technology (KIT), Germany · KIT Campus Transfer (KCT), Germany · Carnegie Mellon University (CMU), USA
Loud and Clear: Dynamic Activation Steering for Improving Speech Intelligibility in Noisy Environments
Oct 6, 2026Seymanur Akti, Alexander WaibelTTS Synthesis Controllable Speech Generation
Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany · KIT Campus Transfer (KCT), Karlsruhe, Germany · Carnegie Mellon University (CMU), Pittsburgh, USA
Zero-Shot Lombard Speech Synthesis with Controllable Style Embeddings
Jan 19, 2026Seymanur Akti, Alexander WaibelTTS Synthesis Zero-Shot TTS
KIT Campus Transfer GmbH (KCT) · Karlsruhe Institute of Technology (KIT) · Carnegie Mellon University (CMU)

Jun 22, 2026J/K move · Enter open · S save

Synthesizing the Lombard Effect: Multi-Level Control of Speech Clarity and Vocal Effort in TTS

Seymanur Akti, Alexander Waibel

Karlsruhe Institute of Technology (KIT), Germany · KIT Campus Transfer (KCT), Germany · Carnegie Mellon University (CMU), USA

TTS Synthesis Speech Prosody Controllable Speech Generation

Open paper AskPDF arXiv

Humans tend to speak louder and clearer in challenging environments, such as noisy conditions or when addressing hearingimpaired listeners, which is called Lombard effect. To simulate this behavior in speech synthesis systems, we introduce a flow-matching based text-to-speech (TTS) model trained with vocal effort and articulation pseudo-labels. The proposed model achieves continuous and disentangled control of vocal effort and articulation, while also enabling word-level emphasis for clarifying specific segments of an utterance. Experimental results show that these control mechanisms effectively improve clarityrelated acoustic features. Furthermore, speech-in-noise experiments demonstrate that our model successfully simulates the intelligibility gains of human clear speech in noisy conditions.

Abstract

TTS Synthesis

Explore similar work

Synthesizing the Lombard Effect: Multi-Level Control of Speech Clarity and Vocal Effort in TTS

Loud and Clear: Dynamic Activation Steering for Improving Speech Intelligibility in Noisy Environments

Zero-Shot Lombard Speech Synthesis with Controllable Style Embeddings