MeanVoiceFlow2: Joint Optimization of Mean Flow and Content Encoder for Fast One-Step Zero-Shot Voice Conversion

Takuhiro Kaneko; Hirokazu Kameoka; Kou Tanaka; Yuto Kondo

cs.SDSep 30, 2026

MeanVoiceFlow2: Joint Optimization of Mean Flow and Content Encoder for Fast One-Step Zero-Shot Voice Conversion

Authors: Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, Yuto Kondo

Organizations: NTT, Inc., Japan

Abstract

Flow-matching approaches to voice conversion (VC) have gained attention owing to their high speech quality and strong speaker similarity. Among them, one-step models such as MeanVoiceFlow are particularly attractive because they enable efficient inference; however, their reliance on a computationally intensive content encoder remains a bottleneck. We therefore propose MeanVoiceFlow2, a framework that jointly optimizes a flow-based conversion module and a computationally efficient content encoder. The model is trained through conversion distillation using MeanVoiceFlow and the reconstruction of real data. We further incorporate diffusion-GAN training with sample mixing and teacher-guided conditioning augmentation to enhance realism and disentanglement. Experiments on zero-shot VC showed that MeanVoiceFlow2 achieved higher perceptual quality and approximately $9\times$ faster inference than MeanVoiceFlow while maintaining comparable speaker similarity. Audio samples are available at https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/meanvoiceflow2/.

Figures & tables

Figure 1: Overview of MeanVoiceFlow2 . The student jointly learns a computationally efficient content encoder

c_{\phi}

and an average velocity network

u_{\phi}

through (a) conversion distillation and (b) real-data reconstruction. We further incorporate diffusion-GAN training with sample mixing using a discriminator

\mathcal{D}_{\psi}

to promote realism, and teacher-guided conditioning augmentation based on

x_{\theta}^{\mathrm{aug}}

to promote disentanglement.

	Conv	Rec	UT $\uparrow$	DNSP $\uparrow$	DNS $\uparrow$	CER $\downarrow$	SECS $\uparrow$
(a)	$\checkmark$		4.00	2.94	3.80	1.8	0.885
(b)		$\checkmark$	3.60	2.40	3.77	0.1	0.641
(c)	$\checkmark$	$\checkmark$	4.04	2.97	3.80	1.5	0.885

Table 1: Analysis of joint conversion and reconstruction. Conv and Rec indicate the use of conversion distillation and real-data reconstruction, respectively.

	GAN	Diffuse	Mix	UT $\uparrow$	DNSP $\uparrow$	DNS $\uparrow$	CER $\downarrow$	SECS $\uparrow$
(a)	None	–	–	4.04	2.79	3.75	2.2	0.882
(b)	Proposed			4.01	2.87	3.79	1.9	0.883
(c)	Proposed	$\checkmark$		4.04	2.86	3.78	1.5	0.882
(d)	Proposed		$\checkmark$	3.93	2.90	3.80	1.9	0.882
(e)	Proposed	$\checkmark$	$\checkmark$	4.04	2.97	3.80	1.5	0.885
(f)	WD [ 55 ]	–	–	4.04	2.89	3.79	1.8	0.884

Table 2: Analysis of adversarial training. Diffuse and Mix indicate the use of diffusion-GAN training and sample mixing, respectively.

		UT $\uparrow$	DNSP $\uparrow$	DNS $\uparrow$	CER $\downarrow$	SECS $\uparrow$
(a)	w/o CondAug	4.04	2.97	3.80	1.5	0.885
(b)	w/ CondAug	4.05	2.99	3.81	1.2	0.887
(c)	Direct Distill	4.05	2.94	3.80	1.9	0.884

Table 3: Analysis of conditioning augmentation (CondAug). Direct Distill adds an explicit

\ell_{1}

loss between the student and teacher content representations to the w/o CondAug objective.

(a)	GT	4.26 $\pm$ .09 ∗	3.64 $\pm$ .06 ∗	4.15	2.89	3.75	0.1	0.940	–
		nMOS $\uparrow$	sMOS $\uparrow$	UT $\uparrow$	DNSP $\uparrow$	DNS $\uparrow$	CER $\downarrow$	SECS $\uparrow$	RTF $\downarrow$
(b)	DiffVC	3.43 $\pm$ .11 ∗	2.24 $\pm$ .10 ∗	3.76	2.64	3.75	5.4	0.880	0.19
(c)	MVF	3.76 $\pm$ .09 ∗	2.74 $\pm$ .11	3.98	2.85	3.78	1.2	0.886	0.0072
(d)	MVF2	3.93 $\pm$ .10	2.70 $\pm$ .11	4.05	2.99	3.81	1.2	0.887	0.00084
(e)	FVG2	3.72 $\pm$ .10 ∗	2.63 $\pm$ .10	4.03	2.79	3.82	1.2	0.890	0.00084

Table 4: Comparison with previous models in terms of subjective metrics (nMOS and sMOS with 95% confidence intervals), objective metrics, and RTF. ∗ indicates a statistically significant difference from MVF2 on the Mann–Whitney U test (

p<0.05

		UT $\uparrow$	DNSP $\uparrow$	DNS $\uparrow$	CER $\downarrow$	SECS $\uparrow$	RTF $\downarrow$
(a)	MVF	3.93	3.01	3.70	1.1	0.879	0.0089
(b)	MVF2	4.05	3.10	3.71	1.1	0.880	0.0010

Table 5: Comparison on LibriTTS.

Explore similar work

MeanVC 2: Robust Low-Latency Streaming Zero-Shot Voice Conversion
Jun 8, 2026Guobin Ma, Yuxuan Xia, Yuepeng Jiang +6Voice Conversion Streaming
The University of New South Wales, Australia · WeNet Open Source Community, China
Zero-VC: Zero-Lookahead Streaming Voice Conversion via Speaker Anonymization
Jun 18, 2026Yudong Li, Zihao Fang, Junwen Qiu +4Speaker Prosody
The Chinese University of Hong Kong, Shenzhen · Shenzhen Transsion Holdings Co., Ltd. · Shenzhen Loop Area Institute +1
From A to B to A: Palindromic Zero-Shot Voice Conversion with Non-Parallel Data
Jun 7, 2026Moshe Mandel, Shlomo E. ChazanVoice Conversion Wav2Vec
Independent, Israel · OriginAI, Israel

Jun 8, 2026, eess.ASJ/K move · Enter open · S save

MeanVC 2: Robust Low-Latency Streaming Zero-Shot Voice Conversion

Guobin Ma, Yuxuan Xia, Yuepeng Jiang, Dake Guo+5

The University of New South Wales, Australia · WeNet Open Source Community, China

Voice Conversion Streaming Zero-Shot Autoregressive Diffusion

Open paper AskPDF arXiv

Streaming zero-shot voice conversion (VC) has become increasingly popular due to its potential for real-time applications. The recently proposed MeanVC achieves lightweight streaming zero-shot VC, but it has several limitations: its chunk-wise autoregressive denoising doubles the effective training sequence length, conversion quality degrades under small-chunk settings, and its timbre encoder directly relies on reference mel-spectrograms, making it sensitive to reference audio quality. To address these limitations we propose MeanVC 2. We introduce future-receptive chunking (FRC), which explicitly schedules past and future receptive fields across diffusion transformer decoder layers and removes clean-chunk teacher forcing. By incorporating bounded future context, FRC enables stable conversion with a 40 ms chunk size. We further introduce a universal timbre token encoder, which constructs a timbre representation from a global speaker embedding and retrieves fine-grained timbre cues via cross-attention, improving robustness to low-quality references and enhancing zero-shot speaker similarity. Experimental results show that MeanVC 2 significantly outperforms MeanVC, while reducing latency from 211 ms to 110 ms. Audio samples are publicly available. The source code will be publicly released.

Abstract

Explore similar work

MeanVC 2: Robust Low-Latency Streaming Zero-Shot Voice Conversion

Zero-VC: Zero-Lookahead Streaming Voice Conversion via Speaker Anonymization

From A to B to A: Palindromic Zero-Shot Voice Conversion with Non-Parallel Data