Organizations: Shanghai Jiao Tong University, Shanghai, China · Tianjin University, Tianjin, China · Shanghai Jiao Tong University, Shanghai Innovation Institute, Shanghai, China · State Key Laboratory of Complex & Critical Software Environment, Changsha, China · Shanghai Innovation Institute, Shanghai, China · Fudan University, Shanghai Innovation Institute, Shanghai, China
Zero-shot voice conversion (VC) aims to convert a source utterance into the voice of an unseen target speaker while preserving its linguistic content. Although recent systems have improved conversion quality, building zero-shot VC systems for interactive scenarios remains challenging because high-fidelity speaker transfer and low-latency streaming inference are difficult to achieve simultaneously. In this work, we present X-VC, a zero-shot streaming VC system that performs one-step conversion in the latent space of a pretrained neural codec. X-VC uses a dual-conditioning acoustic converter that jointly models source codec latents and frame-level acoustic conditions derived from target reference speech, while injecting utterance-level target speaker information through adaptive normalization. To reduce the mismatch between training and inference, we train the model with generated paired data and a role-assignment strategy that combines standard, reconstruction, and reversed modes. For streaming inference, we further adopt a chunkwise inference scheme with overlap smoothing that is aligned with the segment-based training paradigm of the codec. Experiments on Seed-TTS-Eval show that X-VC achieves the best streaming WER in both English and Chinese, strong speaker similarity in same-language and cross-lingual settings, and substantially lower offline real-time factor than the compared baselines. These results suggest that codec-space one-step conversion is a practical approach for building high-quality low-latency zero-shot VC systems. Our audio samples, code and checkpoints are released at https://github.com/Jerrister/X-VC.
Figures & tables
Figure 1. Overall framework of X-VC. A pretrained speech encoder maps the input waveform into latent representations, which are then transformed by the acoustic converter conditioned on target-side reference speech and finally decoded into waveform. The figure presents a pipeline consisting of multiple modules. On the left, a speech encoder includes a semantic encoder with vector quantization and an acoustic encoder with vector quantization, both marked as frozen. Their outputs are combined and passed through an adapter and a prenet module to form codec latent representations. These representations are then processed by an acoustic converter module. A speaker encoder extracts a global speaker embedding from reference audio and provides conditioning information. The converted latent representations are finally fed into an acoustic decoder to synthesize waveform audio. Colored dots represent latent sequences at different stages. Snowflake icons indicate frozen components.
Figure 2. Architecture of the acoustic converter. Source codec latents and the mel sequence extracted from xcondtgt are jointly updated through shared attention, while the utterance-level speaker embedding modulates each block through adaptive normalization. The figure illustrates a stacked transformer-like architecture with N repeated blocks. Each block contains two parallel processing streams corresponding to different modalities. Both streams include LayerNorm followed by scale-and-shift modulation, linear projections, multi-head self-attention, and feed-forward networks with residual connections. The attention module receives queries, keys, and values from both streams, with concatenation points indicated by “C”. Global conditioning parameters (gamma, beta, and others) are generated from a speaker embedding through an MLP and injected via scale-and-shift operations. The lower part shows separate input embeddings with positional encoding for acoustic tokens and mel-spectrogram features, as well as a speaker encoder that produces the global conditioning vector.
Figure 3. Training data construction and role assignment. A pretrained VC model generates paired utterances from real speech, which are then used to form source-target pairs under standard, reconstruction, and reversed modes. The diagram shows the training data construction process. Real utterances x0 and x1 are first used to generate converted samples x0 to 1 and x1 to 0 with a pretrained voice conversion model. One real utterance and one generated utterance with matched content are then passed to a role assignment stage. Three role configurations are illustrated: standard, where the generated utterance is used as the source and the real utterance is used as the target; reconstruction, where both source and target are taken from the same real utterance; and reversed, where the real utterance is used as the source and the generated utterance is used as the target.
Figure 4. Chunkwise streaming inference of X-VC. Each step uses a fixed processing window composed of history, current, overlap, and future regions. Only the current region is emitted as valid output, while the front part of the current chunk is blended with the overlap retained from the previous step to ensure continuity. The diagram shows a streaming processing timeline for input and output audio. The input audio is segmented into multiple regions labeled history context, current segment, overlap, and future context. These segments are fed into a chunkwise voice conversion module. The module processes the current segment while utilizing past and limited future information. The output is generated in chunks corresponding to the current segments, with adjacent outputs overlapping in time. The overlapping regions are merged using smoothing to produce a continuous output waveform.
Model
#Param.
Seed-TTS test-en
Seed-TTS test-zh
Latency (ms) ↓
Total
Converter
WER (%) ↓
SIM ↑
UTMOS ↑
WER (%) ↓
SIM ↑
UTMOS ↑
Tmodel
Tcompute
Ground Truth
-
-
1.96
-
-
1.33
-
-
-
-
Seed-VC tiny
262M
27M
3.31
0.40
2.97
3.36
0.60
2.46
380
120.75
MeanVC
47M
14M
-
-
-
4.89
0.72
2.22
250
32.12
X-VC (Ours)
539M
44M
3.14
0.62
3.07
2.65
0.72
2.35
240
58.17
Table 1. Streaming zero-shot voice conversion results on Seed-TTS-Eval. Best values are shown in bold. ↑ and ↓ indicate higher or lower values are better. Because the test set is non-parallel, only GT WER is reported, using the source utterances as content references; GT SIM and GT UTMOS are omitted as they are not meaningful in this setting.
Model
Seed-TTS test-en
Seed-TTS test-zh
Seed-VC
3.56 ± 0.12
3.38 ± 0.17
MeanVC
-
3.44 ± 0.15
X-VC (Ours)
3.98 ± 0.10
3.89 ± 0.13
Table 2. Subjective speaker similarity (SMOS ↑ ) with 95% confidence intervals computed from the t-test.
Model
#Param.
Seed-TTS test-en
Seed-TTS test-zh
RTF ↓
Total
Converter
WER (%) ↓
SIM ↑
UTMOS ↑
WER (%) ↓
SIM ↑
UTMOS ↑
Ground Truth
-
-
1.96
-
-
1.33
-
-
-
Seed-VC small
296M
98M
2.57
0.56
3.37
2.52
0.73
2.73
0.161
Seed-VC tiny
262M
27M
2.24
0.41
3.67
1.79
0.60
3.08
0.069
MeanVC
47M
14M
-
-
-
3.89
0.73
2.81
0.094
X-VC (Ours)
539M
44M
2.83
0.63
3.31
1.99
0.73
2.69
0.014
Table 3. Offline zero-shot voice conversion results on Seed-TTS-Eval.
Model
English-to-Chinese
Chinese-to-English
WER (%) ↓
SIM ↑
WER (%) ↓
SIM ↑
Ground Truth
1.91
-
1.14
-
Seed-VC small
2.82
0.52
2.36
0.42
Seed-VC tiny
2.23
0.34
1.70
0.29
MeanVC
-
-
3.89
0.42
X-VC (Ours)
2.67
0.52
2.15
0.49
Table 4. Offline cross-lingual results on Seed-TTS-Eval.
Model
WER (%) ↓
SIM ↑
X-VC
2.02
0.72
Condition Modeling
w/o updating frame-level condition c
2.15
0.66
w/o utterance-level condition g
2.20
0.61
Data Construction
standard only ( pstd=1.0 )
2.31
0.72
Table 5. Offline ablation results on Seed-TTS test-zh . All models are trained for 300k steps for fair comparison.
Streaming zero-shot voice conversion (VC) has become increasingly popular due to its potential for real-time applications. The recently proposed MeanVC achieves lightweight streaming zero-shot VC, but it has several limitations: its chunk-wise autoregressive denoising doubles the effective training sequence length, conversion quality degrades under small-chunk settings, and its timbre encoder directly relies on reference mel-spectrograms, making it sensitive to reference audio quality. To address these limitations we propose MeanVC 2. We introduce future-receptive chunking (FRC), which explicitly schedules past and future receptive fields across diffusion transformer decoder layers and removes clean-chunk teacher forcing. By incorporating bounded future context, FRC enables stable conversion with a 40 ms chunk size. We further introduce a universal timbre token encoder, which constructs a timbre representation from a global speaker embedding and retrieves fine-grained timbre cues via cross-attention, improving robustness to low-quality references and enhancing zero-shot speaker similarity. Experimental results show that MeanVC 2 significantly outperforms MeanVC, while reducing latency from 211 ms to 110 ms. Audio samples are publicly available. The source code will be publicly released.
Guobin Ma, Yuxuan Xia, Yuepeng Jiang +6
The University of New South Wales, Australia · WeNet Open Source Community, China
Streaming zero-shot voice conversion struggles to disentangle timbre from linguistic content without degrading utility or inflating latency. Current methods rely on information bottleneck (IB) or speaker perturbation. While IB filters out timbre, it discards prosody, forcing models to explicitly inject features like fundamental frequency. This often requires buffering future frames, creating algorithmic lookahead latency. On the other hand, existing perturbation methods largely overlook the crucial trade-off between timbre leakage and utility preservation. Recognizing this neglected trade-off, we find that the inherent objective of Speaker Anonymization (SA) aligns well with balancing these factors. Thus, we introduce SA as a novel perturbation mechanism to explicitly mitigate timbre leakage while retaining prosodic utility. Crucially, SA's robust representations significantly alleviate the generator's reliance on future context, enabling our strictly causal, zero-lookahead network. Audio samples are available at https://amphionteam.github.io/Zero-VC-demo/.
Yudong Li, Zihao Fang, Junwen Qiu +4
The Chinese University of Hong Kong, Shenzhen · Shenzhen Transsion Holdings Co., Ltd. · Shenzhen Loop Area Institute +1
Real-time voice conversion (VC) systems commonly rely on pretrained speaker embeddings from automatic speaker verification (ASV) models. While effective for speaker discrimination, these embeddings are trained to remain stable across phonetic and prosodic variations within-speaker, which may conflict with frame-level acoustic generation in streaming constraints. To address this issue, we propose VOSSA (Voiceprint Optimization for Streaming Speech Architectures), a speaker representation framework that extracts speaker information from intermediate content encoder layers and aggregates using attentive statistics pooling. The embedding is trained jointly with VC objectives, removing the need for a separate speaker encoder. Across six datasets, VOSSA improves F0 dynamics and vowel-discriminative acoustic cues while maintaining comparable NISQA-MOS, WER, and speaker similarity. Perceptual tests further indicate improvements in naturalness, speaker similarity, intelligibility, and vibrancy.
Mu-Ruei Tseng, Waris Quamer, Ghady Nasrallah +1
Department of Computer Science & Engineering, Texas A&M University, College Station, US