Organizations: Kyushu University, Japan · Donghua University, China · National Institute of Informatics, Japan · Independent Researcher · Tencent Hunyuan, China · The University of Tokyo, Japan
Recent diffusion- and conditional flow matching (CFM)-based emotional voice conversion (EVC) systems achieve strong conversion quality. However, they typically rely on multiple sampling steps, because the conditioning specifies which emotion to impose while its acoustic realization is left to the sampler. In this paper, we propose Prime-EVC, a one-step expressive EVC architecture based on emotion-primed flow matching. Our key idea is to reallocate the emotional conversion between the conditioning and the sampler: a FiLM-fused speaker-emotion style, an emotional variance adaptor, and direct emotion and spectro-temporal supervision assemble an emotion-rich conditioning upstream, rendering the residual conversion left to the CFM decoder attainable within one Euler step. Experiments on the ESD corpus demonstrate that single-step Prime-EVC surpasses five representative EVC systems on every quality metric and exceeds 4- and 100-step DurFlex-EVC in subjective naturalness and emotion while running 3.4--50 times faster than both.
Figures & tables
Method
nMOS ↑
eMOS ↑
UTMOS ↑
CER ↓
WER ↓
EECS ↑
SECS ↑
RTF ↓
GT
4.02 ±0.03
4.06 ±0.02
3.65
2.15
5.69
0.96
0.84
-
GT (vocoded)
3.96 ±0.05
4.01 ±0.03
3.42
2.39
5.85
0.95
0.84
-
StarGAN-EVC
2.05 ±0.06
2.11 ±0.04
1.39
49.77
72.38
0.37
0.57
0.005
Seq2seq-EVC
2.72 ±0.03
2.69 ±0.05
1.48
22.19
37.62
0.39
0.61
0.233
Emovox
3.06 ±0.05
3.17 ±0.03
2.11
18.69
34.18
0.49
0.69
0.211
Textless-EVC
3.43 ±0.06
3.39 ±0.04
2.84
15.17
26.73
0.65
0.66
0.434
Table 1: Overall comparison on the ESD corpus. Best among the conversion systems in bold .
Model
WER ↓
UTMOS ↑
EECS ↑
SECS ↑
Prime-EVC
9.61
3.37
0.87
0.80
w/o FiLM
9.86
3.34
0.83
0.77
w/o intensity
9.55
3.35
0.82
0.80
w/o classifier
9.52
3.41
0.81
0.81
w/o STFT
9.17
3.23
0.85
0.80
w/o EVA
8.25
2.89
0.46
0.78
Table 2: Ablation of Prime-EVC. Above the rule, each row removes one conditioning component from the full model. Below the rule are the conditioning alone and alternative decoder objectives.
NFE
UTMOS ↑
WER(%) ↓
EECS ↑
SECS ↑
RTF ↓
Prime-EVC
1
3.37
9.61
0.87
0.80
0.038
4
3.35
9.70
0.88
0.80
0.045
16
3.37
9.64
0.87
0.80
0.078
Global-cond. CFM
1
2.80
8.07
0.37
0.77
0.036
Table 3: Effect of NFE for Prime-EVC and the Global-cond. CFM , in which the global style is the only emotion pathway.
Flow-matching approaches to voice conversion (VC) have gained attention owing to their high speech quality and strong speaker similarity. Among them, one-step models such as MeanVoiceFlow are particularly attractive because they enable efficient inference; however, their reliance on a computationally intensive content encoder remains a bottleneck. We therefore propose MeanVoiceFlow2, a framework that jointly optimizes a flow-based conversion module and a computationally efficient content encoder. The model is trained through conversion distillation using MeanVoiceFlow and the reconstruction of real data. We further incorporate diffusion-GAN training with sample mixing and teacher-guided conditioning augmentation to enhance realism and disentanglement. Experiments on zero-shot VC showed that MeanVoiceFlow2 achieved higher perceptual quality and approximately 9× faster inference than MeanVoiceFlow while maintaining comparable speaker similarity. Audio samples are available at https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/meanvoiceflow2/.
Speech Emotion Conversion (SEC) aims to transform the emotion of a source utterance into a target emotion while preserving content and speaker identity. SEC on in-the-wild data is challenging due to the non-parallel nature of training data and complex real-world acoustics. Existing fixed-duration approaches either struggle to shift the emotion effectively (high quality, low conversion) or degrade speech naturalness (low quality, high conversion). We propose TargetSEC, an embedding-driven latent diffusion framework that generates emotion-focused style embeddings conditioned on speaker identity and continuous emotion. Unlike methods that diffuse over spectrograms, TargetSEC operates in a compact latent space. Experiments on the MSP-Podcast dataset show that TargetSEC outperforms current non-duration baselines and matches or exceeds the duration-prediction baseline in conversion MSE and naturalness without explicit temporal modeling. Demo and code available at.
Constantin Alexander Auga
Hasso Plattner Institute / University of Potsdam, Potsdam, Germany
In voice conversion (VC) applications, diffusion and flow-matching models have exhibited exceptional speech quality and speaker similarity performances. However, they are limited by slow conversion owing to their iterative inference. Consequently, we propose MeanVoiceFlow, a novel one-step nonparallel VC model based on mean flows, which can be trained from scratch without requiring pretraining or distillation. Unlike conventional flow matching that uses instantaneous velocity, mean flows employ average velocity to more accurately compute the time integral along the inference path in a single step. However, training the average velocity requires its derivative to compute the target velocity, which can cause instability. Therefore, we introduce a structural margin reconstruction loss as a zero-input constraint, which moderately regularizes the input-output behavior of the model without harmful statistical averaging. Furthermore, we propose conditional diffused-input training in which a mixture of noise and source data is used as input to the model during both training and inference. This enables the model to effectively leverage source information while maintaining consistency between training and inference. Experimental results validate the effectiveness of these techniques and demonstrate that MeanVoiceFlow achieves performance comparable to that of previous multi-step and distillation-based models, even when trained from scratch. Audio samples are available at https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/meanvoiceflow/.