X-VC: Zero-shot Streaming Voice Conversion in Codec Space

Qixi Zheng; Yuxiang Zhao; Tianrui Wang; Wenxi Chen; Kele Xu; Yikang Li; Qinyuan Cheng; Xipeng Qiu

eess.ASApr 14, 2026

X-VC: Zero-shot Streaming Voice Conversion in Codec Space

Authors: Qixi Zheng, Yuxiang Zhao, Tianrui Wang, Wenxi Chen, Kele Xu, Yikang Li, Qinyuan Cheng, Xipeng Qiu,

Organizations: Shanghai Jiao Tong University, Shanghai, China · Tianjin University, Tianjin, China · Shanghai Jiao Tong University, Shanghai Innovation Institute, Shanghai, China · State Key Laboratory of Complex & Critical Software Environment, Changsha, China ·

Abstract

Zero-shot voice conversion (VC) aims to convert a source utterance into the voice of an unseen target speaker while preserving its linguistic content. Although recent systems have improved conversion quality, building zero-shot VC systems for interactive scenarios remains challenging because high-fidelity speaker transfer and low-latency streaming inference are difficult to achieve simultaneously. In this work, we present X-VC, a zero-shot streaming VC system that performs one-step conversion in the latent space of a pretrained neural codec. X-VC uses a dual-conditioning acoustic converter that jointly models source codec latents and frame-level acoustic conditions derived from target reference speech, while injecting utterance-level target speaker information through adaptive normalization. To reduce the mismatch between training and inference, we train the model with generated paired data and a role-assignment strategy that combines standard, reconstruction, and reversed modes. For streaming inference, we further adopt a chunkwise inference scheme with overlap smoothing that is aligned with the segment-based training paradigm of the codec. Experiments on Seed-TTS-Eval show that X-VC achieves the best streaming WER in both English and Chinese, strong speaker similarity in same-language and cross-lingual settings, and substantially lower offline real-time factor than the compared baselines. These results suggest that codec-space one-step conversion is a practical approach for building high-quality low-latency zero-shot VC systems. Our audio samples, code and checkpoints are released at https://github.com/Jerrister/X-VC.

Figures & tables

Figure 1. Overall framework of X-VC. A pretrained speech encoder maps the input waveform into latent representations, which are then transformed by the acoustic converter conditioned on target-side reference speech and finally decoded into waveform. The figure presents a pipeline consisting of multiple modules. On the left, a speech encoder includes a semantic encoder with vector quantization and an acoustic encoder with vector quantization, both marked as frozen. Their outputs are combined and passed through an adapter and a prenet module to form codec latent representations. These representations are then processed by an acoustic converter module. A speaker encoder extracts a global speaker embedding from reference audio and provides conditioning information. The converted latent representations are finally fed into an acoustic decoder to synthesize waveform audio. Colored dots represent latent sequences at different stages. Snowflake icons indicate frozen components.

Figure 2. Architecture of the acoustic converter. Source codec latents and the mel sequence extracted from $\mathbf{x}^{\mathrm{tgt}}_{\mathrm{cond}}$ are jointly updated through shared attention, while the utterance-level speaker embedding modulates each block through adaptive normalization. The figure illustrates a stacked transformer-like architecture with N repeated blocks. Each block contains two parallel processing streams corresponding to different modalities. Both streams include LayerNorm followed by scale-and-shift modulation, linear projections, multi-head self-attention, and feed-forward networks with residual connections. The attention module receives queries, keys, and values from both streams, with concatenation points indicated by “C”. Global conditioning parameters (gamma, beta, and others) are generated from a speaker embedding through an MLP and injected via scale-and-shift operations. The lower part shows separate input embeddings with positional encoding for acoustic tokens and mel-spectrogram features, as well as a speaker encoder that produces the global conditioning vector.

Figure 3. Training data construction and role assignment. A pretrained VC model generates paired utterances from real speech, which are then used to form source-target pairs under standard, reconstruction, and reversed modes. The diagram shows the training data construction process. Real utterances x0 and x1 are first used to generate converted samples x0 to 1 and x1 to 0 with a pretrained voice conversion model. One real utterance and one generated utterance with matched content are then passed to a role assignment stage. Three role configurations are illustrated: standard, where the generated utterance is used as the source and the real utterance is used as the target; reconstruction, where both source and target are taken from the same real utterance; and reversed, where the real utterance is used as the source and the generated utterance is used as the target.

Figure 4. Chunkwise streaming inference of X-VC. Each step uses a fixed processing window composed of history, current, overlap, and future regions. Only the current region is emitted as valid output, while the front part of the current chunk is blended with the overlap retained from the previous step to ensure continuity. The diagram shows a streaming processing timeline for input and output audio. The input audio is segmented into multiple regions labeled history context, current segment, overlap, and future context. These segments are fed into a chunkwise voice conversion module. The module processes the current segment while utilizing past and limited future information. The output is generated in chunks corresponding to the current segments, with adjacent outputs overlapping in time. The overlapping regions are merged using smoothing to produce a continuous output waveform.

Model	#Param.		Seed-TTS test-en			Seed-TTS test-zh			Latency (ms) $\downarrow$
	Total	Converter	WER (%) $\downarrow$	SIM $\uparrow$	UTMOS $\uparrow$	WER (%) $\downarrow$	SIM $\uparrow$	UTMOS $\uparrow$	$T_{\text{model}}$	$T_{\text{compute}}$
Ground Truth	-	-	1.96	-	-	1.33	-	-	-	-
Seed-VC ${}_{\textit{tiny}}$	262M	27M	3.31	0.40	2.97	3.36	0.60	2.46	380	120.75
MeanVC	47M	14M	-	-	-	4.89	0.72	2.22	250	32.12
X-VC (Ours)	539M	44M	3.14	0.62	3.07	2.65	0.72	2.35	240	58.17

Table 1. Streaming zero-shot voice conversion results on Seed-TTS-Eval. Best values are shown in bold.

\uparrow

and

\downarrow

indicate higher or lower values are better. Because the test set is non-parallel, only GT WER is reported, using the source utterances as content references; GT SIM and GT UTMOS are omitted as they are not meaningful in this setting.

Model	Seed-TTS test-en	Seed-TTS test-zh
Seed-VC	3.56 $\pm$ 0.12	3.38 $\pm$ 0.17
MeanVC	-	3.44 $\pm$ 0.15
X-VC (Ours)	3.98 $\pm$ 0.10	3.89 $\pm$ 0.13

Table 2. Subjective speaker similarity (SMOS

\uparrow

) with 95% confidence intervals computed from the t-test.

Model	#Param.		Seed-TTS test-en			Seed-TTS test-zh			RTF $\downarrow$
	Total	Converter	WER (%) $\downarrow$	SIM $\uparrow$	UTMOS $\uparrow$	WER (%) $\downarrow$	SIM $\uparrow$	UTMOS $\uparrow$
Ground Truth	-	-	1.96	-	-	1.33	-	-	-
Seed-VC ${}_{\textit{small}}$	296M	98M	2.57	0.56	3.37	2.52	0.73	2.73	0.161
Seed-VC ${}_{\textit{tiny}}$	262M	27M	2.24	0.41	3.67	1.79	0.60	3.08	0.069
MeanVC	47M	14M	-	-	-	3.89	0.73	2.81	0.094
X-VC (Ours)	539M	44M	2.83	0.63	3.31	1.99	0.73	2.69	0.014

Table 3. Offline zero-shot voice conversion results on Seed-TTS-Eval.

Model	English-to-Chinese		Chinese-to-English
	WER (%) $\downarrow$	SIM $\uparrow$	WER (%) $\downarrow$	SIM $\uparrow$
Ground Truth	1.91	-	1.14	-
Seed-VC ${}_{\textit{small}}$	2.82	0.52	2.36	0.42
Seed-VC ${}_{\textit{tiny}}$	2.23	0.34	1.70	0.29
MeanVC	-	-	3.89	0.42
X-VC (Ours)	2.67	0.52	2.15	0.49

Table 4. Offline cross-lingual results on Seed-TTS-Eval.

Model	WER (%) $\downarrow$	SIM $\uparrow$
X-VC	2.02	0.72
Condition Modeling
w/o updating frame-level condition $\mathbf{c}$	2.15	0.66
w/o utterance-level condition $\mathbf{g}$	2.20	0.61
Data Construction
standard only ( $p_{\text{std}}=1.0$ )	2.31	0.72

Table 5. Offline ablation results on Seed-TTS test-zh . All models are trained for 300k steps for fair comparison.

Zero-Shot Voice Conversion

See all 10 papers

MeanVC 2: Robust Low-Latency Streaming Zero-Shot Voice Conversion
Jun 8, 2026Guobin Ma, Yuxuan Xia, Yuepeng Jiang +6Zero-Shot Voice Conversion Diffusion Transformer
The University of New South Wales, Australia · WeNet Open Source Community, China
Zero-VC: Zero-Lookahead Streaming Voice Conversion via Speaker Anonymization
Jun 18, 2026Yudong Li, Zihao Fang, Junwen Qiu +4Speech Processing Speaker Anonymization
The Chinese University of Hong Kong, Shenzhen · Shenzhen Transsion Holdings Co., Ltd. · Shenzhen Loop Area Institute +1
VOSSA: Voiceprint Optimization for Streaming Speech Architectures
Sep 30, 2026Mu-Ruei Tseng, Waris Quamer, Ghady Nasrallah +1Audio Representation Learning Speech Processing
Department of Computer Science & Engineering, Texas A&M University, College Station, US

Jun 8, 2026J/K move · Enter open · S save

MeanVC 2: Robust Low-Latency Streaming Zero-Shot Voice Conversion

Guobin Ma, Yuxuan Xia, Yuepeng Jiang, Dake Guo+5

The University of New South Wales, Australia · WeNet Open Source Community, China

Zero-Shot Voice Conversion Diffusion Transformer Audio Diffusion Models Speech Processing Voice Conversion

Open paper AskPDF arXiv

Streaming zero-shot voice conversion (VC) has become increasingly popular due to its potential for real-time applications. The recently proposed MeanVC achieves lightweight streaming zero-shot VC, but it has several limitations: its chunk-wise autoregressive denoising doubles the effective training sequence length, conversion quality degrades under small-chunk settings, and its timbre encoder directly relies on reference mel-spectrograms, making it sensitive to reference audio quality. To address these limitations we propose MeanVC 2. We introduce future-receptive chunking (FRC), which explicitly schedules past and future receptive fields across diffusion transformer decoder layers and removes clean-chunk teacher forcing. By incorporating bounded future context, FRC enables stable conversion with a 40 ms chunk size. We further introduce a universal timbre token encoder, which constructs a timbre representation from a global speaker embedding and retrieves fine-grained timbre cues via cross-attention, improving robustness to low-quality references and enhancing zero-shot speaker similarity. Experimental results show that MeanVC 2 significantly outperforms MeanVC, while reducing latency from 211 ms to 110 ms. Audio samples are publicly available. The source code will be publicly released.

Abstract

Zero-Shot Voice Conversion

Explore similar work

MeanVC 2: Robust Low-Latency Streaming Zero-Shot Voice Conversion

Zero-VC: Zero-Lookahead Streaming Voice Conversion via Speaker Anonymization

VOSSA: Voiceprint Optimization for Streaming Speech Architectures