In voice conversion (VC) applications, diffusion and flow-matching models have exhibited exceptional speech quality and speaker similarity performances. However, they are limited by slow conversion owing to their iterative inference. Consequently, we propose MeanVoiceFlow, a novel one-step nonparallel VC model based on mean flows, which can be trained from scratch without requiring pretraining or distillation. Unlike conventional flow matching that uses instantaneous velocity, mean flows employ average velocity to more accurately compute the time integral along the inference path in a single step. However, training the average velocity requires its derivative to compute the target velocity, which can cause instability. Therefore, we introduce a structural margin reconstruction loss as a zero-input constraint, which moderately regularizes the input-output behavior of the model without harmful statistical averaging. Furthermore, we propose conditional diffused-input training in which a mixture of noise and source data is used as input to the model during both training and inference. This enables the model to effectively leverage source information while maintaining consistency between training and inference. Experimental results validate the effectiveness of these techniques and demonstrate that MeanVoiceFlow achieves performance comparable to that of previous multi-step and distillation-based models, even when trained from scratch. Audio samples are available at https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/meanvoiceflow/.
Figures & tables
Figure 1: Comparison of (a) instantaneous velocity used in conventional flow matching and (b) average velocity used in mean flows. (a) Instantaneous velocity v(zt,t) (blue arrow) represents the tangent direction of the path for a single time step t . (b) Average velocity u(zt,r,t) (orange arrow) aligns with the displacement between two time steps r and t . In MeanVoiceFlow , a zero-input constraint is imposed on u(ϵˉ,0,1) (green arrow), the average velocity for a zero-input sample ϵˉ=0 , using a structural margin reconstruction loss to moderately guide learning.
Figure 2: Comparison of input types during inference and training. Previous studies (e.g., [ 34 ] ) use input type (c) during training and (b) during inference, causing a training–inference mismatch. In contrast, the proposed method uses (d) during training and (b) during inference, effectively eliminating this mismatch.
Metric
Margin
Input
pMOS s ↑
pMOS n ↑
pMOS v ↑
CER ↓
SECS ↑
( A )
–
–
–
3.72
3.73
4.00
1.3
0.882
( B )
L1
–
Zero
3.81
3.65
3.96
1.2
0.882
( C )
L2
–
Zero
3.80
3.66
3.98
1.2
0.882
( D )
SSIM
–
Zero
3.79
3.77
4.05
1.3
0.882
( E )
SSIM
✓
Zero
3.90
3.77
4.08
1.2
0.883
( F )
SSIM
✓
All
3.89
3.76
4.03
1.3
0.883
Table 1: Analysis of zero-input constraint. Performance changes were analyzed while varying metric , margin , and input type in the constraint. ( E ) denotes the proposed configuration.
Figure 3: Analysis of conditional diffused-input training. Conditional diffused-input training (pink line) enhances both robustness to the mixing ratio t′ and peak performance.
Model
NFE ↓
nMOS ↑
sMOS ↑
pMOS s ↑
pMOS n ↑
pMOS v ↑
CER ↓
SECS ↑
Ground truth
–
4.26 ± 0.10 ∗
3.62 ± 0.08 ∗
4.14
3.75
4.05
0.1
0.940
DiffVC
30
3.59 ± 0.11 ∗
2.55 ± 0.13 ∗
3.76
3.75
4.03
5.4
0.880
VoiceGrad-DM
1
2.77 ± 0.09 ∗
2.48 ± 0.13 ∗
3.72
3.68
3.99
1.4
0.884
VoiceGrad-DM
30
3.79 ± 0.11
2.70 ± 0.13 ∗
3.88
3.77
4.02
1.4
0.885
VoiceGrad-FM
1
3.14 ± 0.11 ∗
2.60 ± 0.13 ∗
3.81
3.69
4.01
1.1
0.885
VoiceGrad-FM
30
3.79 ± 0.10
2.92 ± 0.12
3.88
3.79
4.05
1.1
0.885
Table 2: Comparisons with prior models. † indicates models that require additional modules (a pretrained teacher and a pretrained feature extractor for the discriminator) and extra training procedures (distillation and adversarial training). For nMOS and sMOS , ∗ indicates a statistically significant difference from MeanVoiceFlow based on the Mann–Whitney U test ( p<0.05 ). Values following ± represent 95% confidence intervals.
Model
NFE ↓
pMOS s ↑
pMOS n ↑
pMOS v ↑
CER ↓
SECS ↑
VoiceGrad-DM
1
3.20
3.32
3.26
1.1
0.873
VoiceGrad-DM
30
3.81
3.75
3.58
1.2
0.865
VoiceGrad-FM
1
3.22
3.38
3.28
1.1
0.875
VoiceGrad-FM
30
3.77
3.77
3.38
1.3
0.866
MeanVoiceFlow
1
3.93
3.70
3.70
1.1
0.879
Table 3: Versatility analysis on LibriTTS. Models trained under similar conditions (i.e., from scratch ) were compared.
Flow-matching approaches to voice conversion (VC) have gained attention owing to their high speech quality and strong speaker similarity. Among them, one-step models such as MeanVoiceFlow are particularly attractive because they enable efficient inference; however, their reliance on a computationally intensive content encoder remains a bottleneck. We therefore propose MeanVoiceFlow2, a framework that jointly optimizes a flow-based conversion module and a computationally efficient content encoder. The model is trained through conversion distillation using MeanVoiceFlow and the reconstruction of real data. We further incorporate diffusion-GAN training with sample mixing and teacher-guided conditioning augmentation to enhance realism and disentanglement. Experiments on zero-shot VC showed that MeanVoiceFlow2 achieved higher perceptual quality and approximately 9× faster inference than MeanVoiceFlow while maintaining comparable speaker similarity. Audio samples are available at https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/meanvoiceflow2/.
Streaming zero-shot voice conversion (VC) has become increasingly popular due to its potential for real-time applications. The recently proposed MeanVC achieves lightweight streaming zero-shot VC, but it has several limitations: its chunk-wise autoregressive denoising doubles the effective training sequence length, conversion quality degrades under small-chunk settings, and its timbre encoder directly relies on reference mel-spectrograms, making it sensitive to reference audio quality. To address these limitations we propose MeanVC 2. We introduce future-receptive chunking (FRC), which explicitly schedules past and future receptive fields across diffusion transformer decoder layers and removes clean-chunk teacher forcing. By incorporating bounded future context, FRC enables stable conversion with a 40 ms chunk size. We further introduce a universal timbre token encoder, which constructs a timbre representation from a global speaker embedding and retrieves fine-grained timbre cues via cross-attention, improving robustness to low-quality references and enhancing zero-shot speaker similarity. Experimental results show that MeanVC 2 significantly outperforms MeanVC, while reducing latency from 211 ms to 110 ms. Audio samples are publicly available. The source code will be publicly released.
Guobin Ma, Yuxuan Xia, Yuepeng Jiang +6
The University of New South Wales, Australia · WeNet Open Source Community, China
We present a voice conversion (VC) framework that utilizes K-Nearest Neighbors (KNN) retrieval over WavLM representations to align non-parallel source and target speech, constructing synthetic training pairs for supervised learning. The retrieved segments serve as synthetic inputs, while real target audio provides ground-truth outputs, forming a synthetic-to-real training paradigm that naturally supports multilingual data without requiring parallel corpora or explicit alignment. To ensure consistent target-speaker identity, we incorporate a speaker loss derived from a pretrained speaker verification model. Experiments across multiple languages demonstrate that the proposed approach achieves high naturalness and strong speaker similarity, outperforming competitive VC baselines, despite being trained exclusively on English data. Samples can be accessed at: https://palindromic-vc.github.io.