Vocal Intensity Conversion with Continuous SPL Control via Adversarial Training

Quentin Le Tellier; Albert Rilliard; Olivier Perrotin; Marc Evrard

eess.ASOct 9, 2026

Vocal Intensity Conversion with Continuous SPL Control via Adversarial Training

Authors: Quentin Le Tellier, Albert Rilliard, Olivier Perrotin, Marc Evrard

Organizations: Universit´e Paris-Saclay, CNRS, LISN, Orsay, France · Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, Grenoble, France

Abstract

Vocal intensity conversion renders a recorded utterance at a different vocal intensity, rather than simply scaling its amplitude, while preserving its content and speaker identity. Existing systems control intensity through a reference utterance or a few discrete categories. Instead, we control the conversion with any target sound pressure level (SPL) on a continuous dB scale. A lightweight converter, trained as a conditional cycle-consistent GAN without parallel data, transforms the features of a frozen speech encoder, from which a pretrained vocoder resynthesizes the speech. We compare three representations, log-mel spectrograms, WavLM, and SpeechTokenizer, and introduce an objective protocol that measures the achieved intensity change against the target one on a calibrated corpus. With WavLM, the converter achieves a measured 83% of the target change in intensity and also shifts the fundamental frequency. A perceptual evaluation confirms that perceived vocal intensity increases with the target level on amplitude-normalized stimuli.

Figures & tables

Panel of Figure 1 — Figure 1: Modeling strategy: $\mathrm{Enc}$ and $\mathrm{Dec}$ are frozen; $C_{\theta}$ is trained. The dashed arc is the cycle: the converted frames pass back through the same converter at the source level and must return $\mathbf{z}$ . $D_{\psi}$ scores (sequence, level) pairs (Sec. 2.3 ). At inference, only the top row runs.

Encoder/decoder pair		intensity ( $P_{\phi}$ )	RMS	$f_{0}$	F1	alpha ratio	WER	UTMOS
Log-mel	converted	0.62 [0.58,0.66]	1.08 [1.00,1.18]	0.42 [0.28,0.62]	0.65 [0.59,0.72]	0.75 [0.66,0.85]	0.136 [0.095,0.176]	2.60 [2.37,2.86]
Log-mel	resynthesis	0.97 [0.96,0.99]	1.00 [1.00,1.00]	0.98 [0.94,1.00]	0.98 [0.97,0.99]	0.99 [0.99,1.00]	0.047 [0.033,0.058]	3.56 [3.40,3.74]
WavLM L6	converted	0.83 [0.78,0.89]	1.04 [0.98,1.09]	0.48 [0.32,0.68]	0.43 [0.37,0.51]	0.43 [0.37,0.47]	0.083 [0.059,0.109]	2.89 [2.60,3.17]
WavLM L6	resynthesis	1.01 [0.90,1.11]	1.03 [0.98,1.08]	0.92 [0.80,1.00]	0.95 [0.87,1.03]	0.89 [0.85,0.94]	0.067 [0.047,0.088]	3.14 [2.82,3.42]
SpeechTokenizer	converted	0.58 [0.51,0.66]	0.73 [0.67,0.81]	0.13 [0.07,0.22]	0.42 [0.36,0.48]	0.63 [0.57,0.71]	0.155 [0.108,0.195]	3.07 [2.81,3.32]
SpeechTokenizer	resynthesis	0.90 [0.85,0.94]	0.95 [0.90,0.98]	0.94 [0.86,0.99]	0.89 [0.84,0.96]	0.97 [0.95,1.00]	0.124 [0.083,0.163]	3.14 [2.88,3.38]

Table 1: Displacement slopes per encoder/decoder pair; brackets give 95% confidence intervals over speakers. WER and UTMOS are absolute values, not slopes (0.047 and 3.65 on the original recordings). Bold marks the best converted entry per column (slope closest to one) and any tied with it under a speaker-paired test.

Figure 2: Perceived level of the conversions against their target level, both in dB SPL, from 54 listeners. Points are the fitted perceived levels, with intervals resampled across listeners, and the dashed diagonal marks a conversion heard at the target level. Original recordings measured on the same scale give the upper line.

Controllable Speech Generation

Controllable speech generation steers synthesized speech's voice, prosody, emotion, timing, pronunciation, and conversational delivery.

See all 117 papers

EmoFI-TTS: Face-Conditioned Expressive Voice Synthesis with Intensity Modulation
Oct 21, 2026Yejin Jeon, Luel Hagos Beyene, Marie Maltais +2Expressive Speech Synthesis Emotional Speech Synthesis
Mila-Québec AI Institute · McGill University · NM-AIST +3
From A to B to A: Palindromic Zero-Shot Voice Conversion with Non-Parallel Data
Jun 7, 2026Moshe Mandel, Shlomo E. ChazanZero-Shot Voice Conversion Speech Processing
Independent, Israel · OriginAI, Israel
Advancing Speaker-Based Vocal Effort Classification with WavLM and Data Augmentation in Naturalistic Non-Calibrated Speech Recordings
Jun 25, 2026Zahra Omidi, John H. L. HansenData Augmentation Audio Classification
Center for Robust Speech Systems, The University of Texas at Dallas, USA

Oct 21, 2026J/K move · Enter open · S save

EmoFI-TTS: Face-Conditioned Expressive Voice Synthesis with Intensity Modulation

Yejin Jeon, Luel Hagos Beyene, Marie Maltais, Hyounghun Kim+1

Mila-Québec AI Institute · McGill University · NM-AIST+3

Expressive Speech Synthesis Emotional Speech Synthesis

Open paper AskPDF arXiv

Controlling emotional intensity remains challenging in expressive text-to-speech synthesis, particularly when emotion must be inferred without explicit labels. We propose EmoFI-TTS, a face-conditioned framework that derives emotion category and intensity from a single facial image while extracting speaker characteristics from reference audio. Modality-specific representations are fused to condition a pretrained speech synthesizer, with cross-speaker training encouraging separation of affect and vocal identity. An intensity-conditioned refinement module enhances the generated mel-spectrogram, while interpolation of learned intensity embeddings enables continuous control without additional training. On speaker-disjoint MEAD evaluation, EmoFI-TTS achieves on average 64.4% and 75.2% objective and subjective emotion intensity accuracies, while retaining low word error rate and high naturalness.

Abstract

Controllable Speech Generation

Explore similar work

EmoFI-TTS: Face-Conditioned Expressive Voice Synthesis with Intensity Modulation

From A to B to A: Palindromic Zero-Shot Voice Conversion with Non-Parallel Data

Advancing Speaker-Based Vocal Effort Classification with WavLM and Data Augmentation in Naturalistic Non-Calibrated Speech Recordings