ProLombard: Structured Multi-Scale Modeling for Normal-to-Lombard Speech Conversion

Hongyang Chen; Xinmeng Xu; Youqiang Zheng; Xingyu Liu; Yuhong Yang; Zhongyuan Wang; Weiping Tu; Song Lin

cs.SDSep 4, 2026

ProLombard: Structured Multi-Scale Modeling for Normal-to-Lombard Speech Conversion

Authors: Hongyang Chen, Xinmeng Xu, Youqiang Zheng, Xingyu Liu, Yuhong Yang, Zhongyuan Wang, Weiping Tu, Song Lin

Organizations: National Engineering Research Center for Multimedia Software (NERCMS), School of Computer Science, Wuhan University, Wuhan 430072, China · Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, Wuhan 430072, China · Guangdong OPPO Mobile Telecommunications Corp., China

Abstract

Normal-to-Lombard (N2L) speech conversion aims to improve speech intelligibility in noisy environments by transforming normal speech into Lombard-style speech while preserving linguistic content, speaker identity, and speech quality. Despite recent progress, existing methods typically model the Lombard effect at the utterance level or the frame level, overlooking its hierarchical nature and its entanglement with both speaker identity and phoneme-level content. This limitation leads to Lombard leakage in speaker representations and incomplete separation between Lombard characteristics and linguistic content. In this work, we propose ProLombard, a structured multi-scale N2L framework that explicitly models the Lombard effect across utterance-, phoneme-, and frame-level representations. To address Lombard-speaker entanglement, we introduce an aligned speaker encoder (ASE) that suppresses Lombard leakage by aligning Lombard-speech speaker embeddings with their normal-speech counterparts. To achieve more complete Lombard-content disentanglement, we develop a phoneme-aware disentanglement and injection mechanism that extends conventional frame-level modeling to the phoneme level. Furthermore, we design a vector quantization (VQ)-median module that provides robust phoneme-level representations through VQ-based segmentation and median-frame-based aggregation. Extensive experiments on Mandarin and English Lombard datasets demonstrate that the proposed approach consistently improves speech intelligibility, Lombard similarity, and perceptual quality over baselines while maintaining speaker identity. These results highlight the importance of structured multi-scale modeling for effective N2L speech conversion.

Figures & tables

Fig. 1: Overview of ProLombard. The model is organized into three temporal scales: utterance level (top), frame level (middle), and phoneme level (bottom). The aligned speaker encoder achieves Lombard-speaker disentanglement and extracts Lombard-invariant speaker embeddings $\textbf{g}_{s}$ at the utterance level. The VQ-median module converts frame-level content features $\mathbf{Z}_{c}^{f}$ into phoneme-level features $\mathbf{Z}_{c}^{p}$ . Phoneme-level De-Lomb block (De-Lomb P ) and frame-level De-Lomb block (De-Lomb F ) remove Lombard-related information from content features, while phoneme-level En-Lomb block (En-Lomb P ) and frame-level En-Lomb block (En-Lomb F ) reintroduce Lombard characteristics conditioned on the style embedding g . This structured design enables effective modeling of the Lombard effect across temporal scales.

Fig. 2: Aligned Speaker Encoder. Speaker alignment loss and the Lombard remover module align the speaker embedding of input speech $x$ with the parallel normal speech $x_{N}$ (an utterance spoken by the same speaker with identical content but in a normal style) to remove Lombard-related information.

Fig. 3: Structure of the De-Lomb and En-Lomb blocks. The De-Lomb and En-Lomb blocks form a complementary pair for disentanglement and injection. The phoneme-level (De-Lomb P , En-Lomb P ) and frame-level (De-Lomb F , En-Lomb F ) blocks share the same architecture, differing only in their input features and training targets.

Model	LMUSHRA $\uparrow$	QMUSHRA $\uparrow$
Normal	46.54 $\pm$ 2.68 ∗	83.64 $\pm$ 1.74 ∗
Lombard	93.31 $\pm$ 0.86 ∗	89.60 $\pm$ 1.40 ∗
CycleGAN [ 14 ]	69.37 $\pm$ 2.40 ∗	61.55 $\pm$ 2.49 ∗
StarGAN [ 15 ]	59.86 $\pm$ 2.42 ∗	69.94 $\pm$ 2.18 ∗
DurFlex-Lomb [ 21 ]	58.27 $\pm$ 2.43 ∗	55.78 $\pm$ 2.84 ∗
PGD-N2L [ 18 ]	61.41 $\pm$ 2.26 ∗	74.25 $\pm$ 1.94 ∗

TABLE I: Subjective evaluation results on the EMALG dataset in terms of Lombard similarity (LMUSHRA) and speech quality (QMUSHRA). Scores are reported as mean

\pm

95% confidence intervals. ∗ indicates a statistically significant difference (

p<0.05

) from ProLombard using a paired

t

-test. Bold indicates the best result among converted systems.

	Lombard Similarity						Intelligibility				Speaker				Quality
Model	F0RMSE $\downarrow$		$\alpha$ MAE $\downarrow$		AutoPCP $\uparrow$		WRR $\uparrow$		SIIB $\uparrow$		SECS $\uparrow$		EER $\downarrow$		UTMOS $\uparrow$
	CN	EN	CN	EN	CN	EN	CN	EN	CN	EN	CN	EN	CN	EN	CN	EN
Normal	72.99	54.03	5.96	4.74	3.58	3.80	29.53	16.63	28.53	23.67	0.68	0.75	1.90	0.04	3.40	3.97
Lombard	0.00	0.00	0.00	0.00	4.75	4.50	35.26	26.36	33.74	27.68	1.00	1.00	4.59	0.00	2.70	4.00
CycleGAN	51.74	73.03 †	2.92	2.06	3.27	3.06	27.52	13.03	29.81	28.63	0.66	0.57	2.50	1.90	2.29	2.52
StarGAN	62.57	58.41 †	2.74	3.44	3.42	3.44	29.24	11.48	29.99	25.53	0.65	0.67	1.75	0.38	2.64	2.86

TABLE II: Overall comparison with baseline systems on EMALG (CN) and Lombard Grid (EN) datasets. Metrics are grouped into Lombard similarity, intelligibility, speaker preservation, and quality. † indicates the average F0 or

\alpha

ratio mean is higher than that of the reference Lombard speech. Bold indicates the best result, and underline indicates the second-best. ProLombard achieves consistently strong performance across all evaluation dimensions.

Fig. 4: Intelligibility evaluation across different SNR levels under stationary (SSN) and dynamic (Babble) noise conditions in two datasets (EMALG and Lombard Grid). ProLombard achieves improved SIIB and WRR in most conditions across both datasets, demonstrating the robustness and effectiveness of the proposed multi-scale modeling approach.

Fig. 5: Log-STFT spectrograms on EMALG. ProLombard is closest to the reference (L80), showing more coherent and clear harmonic structures across 0–8 kHz, while baseline methods exhibit high-frequency artifacts and blurred harmonic patterns.

Model	FLOPs $\downarrow$ (G)	Params $\downarrow$ (M)
CycleGAN	4.11	37.90
StarGAN	3.90	5.81
DurFlex-Lomb	265.26	152.78
PGD-N2L	42.73	30.99
ProLombard	42.84	32.79

TABLE III: Model efficiency comparison in terms of computational cost (FLOPs) and model size (parameters). ProLombard achieves comparable efficiency to the backbone model (PGD-N2L) while providing significantly improved performance. Bold indicates the best result, and underline indicates the second-best.

Speaker Encoder	Lombard Similarity			Intelligibility		Speaker
Speaker Encoder	F0RMSE $\downarrow$	$\alpha$ MAE $\downarrow$	AutoPCP $\uparrow$	WRR $\uparrow$	SIIB $\uparrow$	SECS $\uparrow$	EER $\downarrow$
ASE	50.48	1.06	3.88	41.31	33.08	0.64	8.88
ECAPA [ 18 ]	55.04	2.37	3.79	39.37	32.51	0.60	7.62
Trainable Enc [ 21 ]	52.41	1.31	3.81	39.66	32.23	0.56	15.78
ECAPA + GRL	51.41	1.43	3.85	41.19	33.42	0.62	13.22
ECAPA + MI [ 22 ]	55.00	1.91	3.80	38.49	31.26	0.66	7.00

TABLE IV: Ablation study of speaker encoding strategies. ASE (Aligned Speaker Encoder) achieves the best trade-off across metrics between Lombard similarity, intelligibility, and speaker preservation on EMALG dataset. Bold and underline indicate the best and second-best results, respectively.

Configuration	D-U	$\textbf{De-Lomb}_{\textbf{P}}$	$\textbf{En-Lomb}_{\textbf{P}}$	Lombard Similarity			Intelligibility		Speaker
				F0RMSE $\downarrow$	$\alpha$ MAE $\downarrow$	AutoPCP $\uparrow$	WRR $\uparrow$	SIIB $\uparrow$	SECS $\uparrow$	EER $\downarrow$
Full Model	✓	✓	✓	50.48	1.06	3.88	41.31	33.08	0.64	8.88
Frame-level Only				50.71	1.91	3.89	39.09	32.07	0.63	7.59
+ D-U Only	✓			53.63	2.29	3.80	40.20	32.52	0.63	6.72
+ De-Lomb ${}_{\textbf{P}}$	✓	✓		52.49	1.87	3.83	38.19	31.46	0.63	7.63
+ En-Lomb ${}_{\textbf{P}}$	✓		✓	53.81	1.93	3.81	40.48	32.05	0.60	8.16

TABLE V: Ablation study of phoneme-aware modeling components, including down–up sampling (D-U), phoneme-level disentanglement (

\text{De-Lomb}_{\text{P}}

), and phoneme-level injection (

\text{En-Lomb}_{\text{P}}

). The full model (top row) achieves the best overall performance, validating the effectiveness of phoneme-aware Lombard modeling on EMALG dataset. Bold and underline indicate the best and second-best results, respectively.

Segmentation	Aggregation	Lombard Similarity			Intelligibility		Speaker
		F0RMSE $\downarrow$	$\alpha$ MAE $\downarrow$	AutoPCP $\uparrow$	WRR $\uparrow$	SIIB $\uparrow$	SECS $\uparrow$	EER $\downarrow$
VQ	Median Frame	50.48	1.06	3.88	41.31	33.08	0.64	8.88
	Pooling	48.20	1.80	3.67	20.24	35.46	0.60	9.19
	Centroid	51.69	1.83	3.81	37.53	31.86	0.62	7.88
HuBERT	Median Frame	51.83	1.82	3.87	41.86	32.19	0.63	7.66
	Pooling	50.07	1.75	3.85	40.14	33.00	0.63	6.90

TABLE VI: Ablation study of segmentation and aggregation methods. The combination of VQ-based segmentation and median-frame-based aggregation achieves the best trade-off across Lombard similarity, intelligibility, and speaker preservation on EMALG dataset. Bold and underline indicate the best and second-best results, respectively.

Voice Conversion

Voice conversion changes a speaker's vocal identity or delivery while preserving the spoken content.

See all 35 papers

Jan 19, 2026

Zero-Shot Lombard Speech Synthesis with Controllable Style Embeddings

The Lombard effect plays a key role in natural communication, particularly in noisy environments or when addressing hearing-impaired listeners. We present a controllable text-to-speech (TTS) system capable of synthesizing Lombard-like speech in a zero-shot manner without requiring Lombard-specific training data. Our approach extends F5-TTS with a learned style embedding representation and analyzes the resulting latent space using principal component analysis (PCA) to identify directions associated with Lombard-related attributes. By manipulating these directions, we obtain interpretable control over vocal effort and articulation and generate speech at different Lombard levels. Experimental results show that the proposed method preserves speaker identity and naturalness, improves intelligibility under noisy conditions, and generalizes to previously unseen speakers. These findings demonstrate that style-embedding manipulation provides an effective and scalable framework for controllable zero-shot Lombard speech synthesis.

Seymanur Akti, Alexander Waibel

KIT Campus Transfer GmbH (KCT) · Karlsruhe Institute of Technology (KIT) · Carnegie Mellon University (CMU)

TTS Synthesis Zero-Shot TTS Controllable Speech Generation Speech Language Models

Ask PDF

Oct 14, 2026

Noise-Aware Lombard Speech Synthesis via Flow-GRPO for Intelligibility Optimization

Voice-based conversational agents are increasingly deployed in real-world settings, where environmental noise often degrades the interactive experience. Therefore, robust acoustic adaptability is essential to maintain natural and intelligible human-computer communication. However, most existing systems remain acoustically insensitive and fail to emulate the human Lombard effect, in which speakers involuntarily adjust their vocal production to preserve intelligibility in the presence of noise. As a result, their communicative reliability degrades under adverse acoustic conditions. To remain intelligible in noisy environments, conversational agents require a mechanism to emulate noise-induced adaptive behavior during synthesis. We propose NIL-TTS, a noise-induced Lombard TTS framework. We integrate a feature-wise linear modulation (FiLM) mechanism into a flow-matching backbone for dynamic vocal adaptation. A two-stage training paradigm is utilized: initial supervised fine-tuning on paired Lombard corpora, followed by Flow-GRPO post-training to optimize for intelligibility. Evaluations confirm that NIL-TTS autonomously adjusts its output in response to varying noise conditions, exhibiting key correlates of the human Lombard effect. Driven by these adaptive modulations, the synthesized speech achieves significant intelligibility improvements that scale proportionally with noise intensity. Ultimately, this work presents a flow-matching and reinforcement learning-based advancement within the adaptive Lombard speech synthesis paradigm for challenging real-world scenarios.

Zhaoyang Wang, Dingding Yao, Chengzhong Wang +1

Key Laboratory of Speech Acoustics and Content Understanding, Institute of Acoustics, Chinese Academy of Sciences, Beijing 100190, China, and also with University of Chinese Academy of Sciences, Beijing 100049, China.

TTS Synthesis

Ask PDF

Jun 22, 2026

Synthesizing the Lombard Effect: Multi-Level Control of Speech Clarity and Vocal Effort in TTS

Humans tend to speak louder and clearer in challenging environments, such as noisy conditions or when addressing hearingimpaired listeners, which is called Lombard effect. To simulate this behavior in speech synthesis systems, we introduce a flow-matching based text-to-speech (TTS) model trained with vocal effort and articulation pseudo-labels. The proposed model achieves continuous and disentangled control of vocal effort and articulation, while also enabling word-level emphasis for clarifying specific segments of an utterance. Experimental results show that these control mechanisms effectively improve clarityrelated acoustic features. Furthermore, speech-in-noise experiments demonstrate that our model successfully simulates the intelligibility gains of human clear speech in noisy conditions.

Seymanur Akti, Alexander Waibel

Karlsruhe Institute of Technology (KIT), Germany · KIT Campus Transfer (KCT), Germany · Carnegie Mellon University (CMU), USA

TTS Synthesis Speech Prosody Controllable Speech Generation

Ask PDF

Abstract

Voice Conversion

Explore similar work

Zero-Shot Lombard Speech Synthesis with Controllable Style Embeddings

Noise-Aware Lombard Speech Synthesis via Flow-GRPO for Intelligibility Optimization

Synthesizing the Lombard Effect: Multi-Level Control of Speech Clarity and Vocal Effort in TTS