ProLombard: Structured Multi-Scale Modeling for Normal-to-Lombard Speech Conversion
Authors: Hongyang Chen, Xinmeng Xu, Youqiang Zheng, Xingyu Liu, Yuhong Yang, Zhongyuan Wang, Weiping Tu, Song Lin
Organizations: National Engineering Research Center for Multimedia Software (NERCMS), School of Computer Science, Wuhan University, Wuhan 430072, China · Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, Wuhan 430072, China · Guangdong OPPO Mobile Telecommunications Corp., China
Normal-to-Lombard (N2L) speech conversion aims to improve speech intelligibility in noisy environments by transforming normal speech into Lombard-style speech while preserving linguistic content, speaker identity, and speech quality. Despite recent progress, existing methods typically model the Lombard effect at the utterance level or the frame level, overlooking its hierarchical nature and its entanglement with both speaker identity and phoneme-level content. This limitation leads to Lombard leakage in speaker representations and incomplete separation between Lombard characteristics and linguistic content. In this work, we propose ProLombard, a structured multi-scale N2L framework that explicitly models the Lombard effect across utterance-, phoneme-, and frame-level representations. To address Lombard-speaker entanglement, we introduce an aligned speaker encoder (ASE) that suppresses Lombard leakage by aligning Lombard-speech speaker embeddings with their normal-speech counterparts. To achieve more complete Lombard-content disentanglement, we develop a phoneme-aware disentanglement and injection mechanism that extends conventional frame-level modeling to the phoneme level. Furthermore, we design a vector quantization (VQ)-median module that provides robust phoneme-level representations through VQ-based segmentation and median-frame-based aggregation. Extensive experiments on Mandarin and English Lombard datasets demonstrate that the proposed approach consistently improves speech intelligibility, Lombard similarity, and perceptual quality over baselines while maintaining speaker identity. These results highlight the importance of structured multi-scale modeling for effective N2L speech conversion.
Figures & tables
Fig. 1: Overview of ProLombard. The model is organized into three temporal scales: utterance level (top), frame level (middle), and phoneme level (bottom). The aligned speaker encoder achieves Lombard-speaker disentanglement and extracts Lombard-invariant speaker embeddings gs at the utterance level. The VQ-median module converts frame-level content features Zcf into phoneme-level features Zcp . Phoneme-level De-Lomb block (De-Lomb P ) and frame-level De-Lomb block (De-Lomb F ) remove Lombard-related information from content features, while phoneme-level En-Lomb block (En-Lomb P ) and frame-level En-Lomb block (En-Lomb F ) reintroduce Lombard characteristics conditioned on the style embedding g . This structured design enables effective modeling of the Lombard effect across temporal scales.
Fig. 2: Aligned Speaker Encoder. Speaker alignment loss and the Lombard remover module align the speaker embedding of input speech x with the parallel normal speech xN (an utterance spoken by the same speaker with identical content but in a normal style) to remove Lombard-related information.
Fig. 3: Structure of the De-Lomb and En-Lomb blocks. The De-Lomb and En-Lomb blocks form a complementary pair for disentanglement and injection. The phoneme-level (De-Lomb P , En-Lomb P ) and frame-level (De-Lomb F , En-Lomb F ) blocks share the same architecture, differing only in their input features and training targets.
Model
LMUSHRA ↑
QMUSHRA ↑
Normal
46.54 ± 2.68 ∗
83.64 ± 1.74 ∗
Lombard
93.31 ± 0.86 ∗
89.60 ± 1.40 ∗
CycleGAN [ 14 ]
69.37 ± 2.40 ∗
61.55 ± 2.49 ∗
StarGAN [ 15 ]
59.86 ± 2.42 ∗
69.94 ± 2.18 ∗
DurFlex-Lomb [ 21 ]
58.27 ± 2.43 ∗
55.78 ± 2.84 ∗
PGD-N2L [ 18 ]
61.41 ± 2.26 ∗
74.25 ± 1.94 ∗
TABLE I: Subjective evaluation results on the EMALG dataset in terms of Lombard similarity (LMUSHRA) and speech quality (QMUSHRA). Scores are reported as mean ± 95% confidence intervals. ∗ indicates a statistically significant difference ( p<0.05 ) from ProLombard using a paired t -test. Bold indicates the best result among converted systems.
Lombard Similarity
Intelligibility
Speaker
Quality
Model
F0RMSE ↓
α MAE ↓
AutoPCP ↑
WRR ↑
SIIB ↑
SECS ↑
EER ↓
UTMOS ↑
CN
EN
CN
EN
CN
EN
CN
EN
CN
EN
CN
EN
CN
EN
CN
EN
Normal
72.99
54.03
5.96
4.74
3.58
3.80
29.53
16.63
28.53
23.67
0.68
0.75
1.90
0.04
3.40
3.97
Lombard
0.00
0.00
0.00
0.00
4.75
4.50
35.26
26.36
33.74
27.68
1.00
1.00
4.59
0.00
2.70
4.00
CycleGAN
51.74
73.03 †
2.92
2.06
3.27
3.06
27.52
13.03
29.81
28.63
0.66
0.57
2.50
1.90
2.29
2.52
StarGAN
62.57
58.41 †
2.74
3.44
3.42
3.44
29.24
11.48
29.99
25.53
0.65
0.67
1.75
0.38
2.64
2.86
TABLE II: Overall comparison with baseline systems on EMALG (CN) and Lombard Grid (EN) datasets. Metrics are grouped into Lombard similarity, intelligibility, speaker preservation, and quality. † indicates the average F0 or α ratio mean is higher than that of the reference Lombard speech. Bold indicates the best result, and underline indicates the second-best. ProLombard achieves consistently strong performance across all evaluation dimensions.
Fig. 4: Intelligibility evaluation across different SNR levels under stationary (SSN) and dynamic (Babble) noise conditions in two datasets (EMALG and Lombard Grid). ProLombard achieves improved SIIB and WRR in most conditions across both datasets, demonstrating the robustness and effectiveness of the proposed multi-scale modeling approach.
Fig. 5: Log-STFT spectrograms on EMALG. ProLombard is closest to the reference (L80), showing more coherent and clear harmonic structures across 0–8 kHz, while baseline methods exhibit high-frequency artifacts and blurred harmonic patterns.
Model
FLOPs ↓ (G)
Params ↓ (M)
CycleGAN
4.11
37.90
StarGAN
3.90
5.81
DurFlex-Lomb
265.26
152.78
PGD-N2L
42.73
30.99
ProLombard
42.84
32.79
TABLE III: Model efficiency comparison in terms of computational cost (FLOPs) and model size (parameters). ProLombard achieves comparable efficiency to the backbone model (PGD-N2L) while providing significantly improved performance. Bold indicates the best result, and underline indicates the second-best.
Speaker Encoder
Lombard Similarity
Intelligibility
Speaker
F0RMSE ↓
α MAE ↓
AutoPCP ↑
WRR ↑
SIIB ↑
SECS ↑
EER ↓
ASE
50.48
1.06
3.88
41.31
33.08
0.64
8.88
ECAPA [ 18 ]
55.04
2.37
3.79
39.37
32.51
0.60
7.62
Trainable Enc [ 21 ]
52.41
1.31
3.81
39.66
32.23
0.56
15.78
ECAPA + GRL
51.41
1.43
3.85
41.19
33.42
0.62
13.22
ECAPA + MI [ 22 ]
55.00
1.91
3.80
38.49
31.26
0.66
7.00
TABLE IV: Ablation study of speaker encoding strategies. ASE (Aligned Speaker Encoder) achieves the best trade-off across metrics between Lombard similarity, intelligibility, and speaker preservation on EMALG dataset. Bold and underline indicate the best and second-best results, respectively.
Configuration
D-U
De-LombP
En-LombP
Lombard Similarity
Intelligibility
Speaker
F0RMSE ↓
α MAE ↓
AutoPCP ↑
WRR ↑
SIIB ↑
SECS ↑
EER ↓
Full Model
✓
✓
✓
50.48
1.06
3.88
41.31
33.08
0.64
8.88
Frame-level Only
50.71
1.91
3.89
39.09
32.07
0.63
7.59
+ D-U Only
✓
53.63
2.29
3.80
40.20
32.52
0.63
6.72
+ De-Lomb P
✓
✓
52.49
1.87
3.83
38.19
31.46
0.63
7.63
+ En-Lomb P
✓
✓
53.81
1.93
3.81
40.48
32.05
0.60
8.16
TABLE V: Ablation study of phoneme-aware modeling components, including down–up sampling (D-U), phoneme-level disentanglement ( De-LombP ), and phoneme-level injection ( En-LombP ). The full model (top row) achieves the best overall performance, validating the effectiveness of phoneme-aware Lombard modeling on EMALG dataset. Bold and underline indicate the best and second-best results, respectively.
Segmentation
Aggregation
Lombard Similarity
Intelligibility
Speaker
F0RMSE ↓
α MAE ↓
AutoPCP ↑
WRR ↑
SIIB ↑
SECS ↑
EER ↓
VQ
Median Frame
50.48
1.06
3.88
41.31
33.08
0.64
8.88
Pooling
48.20
1.80
3.67
20.24
35.46
0.60
9.19
Centroid
51.69
1.83
3.81
37.53
31.86
0.62
7.88
HuBERT
Median Frame
51.83
1.82
3.87
41.86
32.19
0.63
7.66
Pooling
50.07
1.75
3.85
40.14
33.00
0.63
6.90
TABLE VI: Ablation study of segmentation and aggregation methods. The combination of VQ-based segmentation and median-frame-based aggregation achieves the best trade-off across Lombard similarity, intelligibility, and speaker preservation on EMALG dataset. Bold and underline indicate the best and second-best results, respectively.
The Lombard effect plays a key role in natural communication, particularly in noisy environments or when addressing hearing-impaired listeners. We present a controllable text-to-speech (TTS) system capable of synthesizing Lombard-like speech in a zero-shot manner without requiring Lombard-specific training data. Our approach extends F5-TTS with a learned style embedding representation and analyzes the resulting latent space using principal component analysis (PCA) to identify directions associated with Lombard-related attributes. By manipulating these directions, we obtain interpretable control over vocal effort and articulation and generate speech at different Lombard levels. Experimental results show that the proposed method preserves speaker identity and naturalness, improves intelligibility under noisy conditions, and generalizes to previously unseen speakers. These findings demonstrate that style-embedding manipulation provides an effective and scalable framework for controllable zero-shot Lombard speech synthesis.
Seymanur Akti, Alexander Waibel
KIT Campus Transfer GmbH (KCT) · Karlsruhe Institute of Technology (KIT) · Carnegie Mellon University (CMU)
Voice-based conversational agents are increasingly deployed in real-world settings, where environmental noise often degrades the interactive experience. Therefore, robust acoustic adaptability is essential to maintain natural and intelligible human-computer communication. However, most existing systems remain acoustically insensitive and fail to emulate the human Lombard effect, in which speakers involuntarily adjust their vocal production to preserve intelligibility in the presence of noise. As a result, their communicative reliability degrades under adverse acoustic conditions. To remain intelligible in noisy environments, conversational agents require a mechanism to emulate noise-induced adaptive behavior during synthesis. We propose NIL-TTS, a noise-induced Lombard TTS framework. We integrate a feature-wise linear modulation (FiLM) mechanism into a flow-matching backbone for dynamic vocal adaptation. A two-stage training paradigm is utilized: initial supervised fine-tuning on paired Lombard corpora, followed by Flow-GRPO post-training to optimize for intelligibility. Evaluations confirm that NIL-TTS autonomously adjusts its output in response to varying noise conditions, exhibiting key correlates of the human Lombard effect. Driven by these adaptive modulations, the synthesized speech achieves significant intelligibility improvements that scale proportionally with noise intensity. Ultimately, this work presents a flow-matching and reinforcement learning-based advancement within the adaptive Lombard speech synthesis paradigm for challenging real-world scenarios.
Zhaoyang Wang, Dingding Yao, Chengzhong Wang +1
Key Laboratory of Speech Acoustics and Content Understanding, Institute of Acoustics, Chinese Academy of Sciences, Beijing 100190, China, and also with University of Chinese Academy of Sciences, Beijing 100049, China.
Humans tend to speak louder and clearer in challenging environments, such as noisy conditions or when addressing hearingimpaired listeners, which is called Lombard effect. To simulate this behavior in speech synthesis systems, we introduce a flow-matching based text-to-speech (TTS) model trained with vocal effort and articulation pseudo-labels. The proposed model achieves continuous and disentangled control of vocal effort and articulation, while also enabling word-level emphasis for clarifying specific segments of an utterance. Experimental results show that these control mechanisms effectively improve clarityrelated acoustic features. Furthermore, speech-in-noise experiments demonstrate that our model successfully simulates the intelligibility gains of human clear speech in noisy conditions.
Seymanur Akti, Alexander Waibel
Karlsruhe Institute of Technology (KIT), Germany · KIT Campus Transfer (KCT), Germany · Carnegie Mellon University (CMU), USA