StyleStream: Real-Time Zero-Shot Voice Style Conversion

Yisi Liu; Nicholas Lee; Gopala Anumanchipalli

cs.SDFeb 23, 2026

StyleStream: Real-Time Zero-Shot Voice Style Conversion

Authors: Yisi Liu, Nicholas Lee, Gopala Anumanchipalli

Organizations: UC Berkeley, USA

Abstract

Voice style conversion aims to transform an input utterance to match a target speaker's timbre, accent, and emotion, with a central challenge being the disentanglement of linguistic content from style. While prior work has explored this problem, conversion quality remains limited, and real-time voice style conversion has not been addressed. We propose StyleStream, the first streamable zero-shot voice style conversion system that achieves state-of-the-art performance. StyleStream consists of two components: a Destylizer, which removes style attributes while preserving linguistic content, and a Stylizer, a diffusion transformer (DiT) that reintroduces target style conditioned on reference speech. Robust content-style disentanglement is enforced through text supervision and a highly constrained information bottleneck. This design enables a fully non-autoregressive architecture, achieving real-time voice style conversion with an end-to-end latency of 1 second. Samples and real-time demo: https://berkeley-speech-group.github.io/StyleStream/.

Figures & tables

Figure 1: System overview of StyleStream. The Destylizer extracts content features disentangled from style, and the Stylizer generates speech that preserves the source linguistic content while adopting the target timbre, accent, and emotion.

Figure 2: Destylizer architecture. The Destylizer, as part of the ASR encoder, is trained with a sequence-to-sequence ASR loss. The continuous representations immediately before the FSQ module are taken as content features.

Figure 3: Stylizer architecture. The Stylizer contains a style encoder and a diffusion transformer.

Model	WER ( $\%$ ) $\downarrow$	S-SIM $\uparrow$	A-SIM $\uparrow$	E-SIM $\uparrow$	NMOS $\uparrow$	A-SMOS $\uparrow$	E-SMOS $\uparrow$	S-SMOS $\uparrow$
Ground Truth	3.8	—	—	—	3.71 $\pm$ .11	—	—	—
FACodec [ 10 ]	15.5	0.763	0.408	0.668	2.55 $\pm$ .13	2.02 $\pm$ .13	2.76 $\pm$ .14	2.98 $\pm$ .14
CosyVoice 2.0 [ 7 ]	9.5	0.794	0.450	0.655	3.47 $\pm$ .11	2.26 $\pm$ .13	2.58 $\pm$ .15	3.25 $\pm$ .14
SeedVC v2 6	21.7	0.766	0.549	0.688	2.65 $\pm$ .12	3.34 $\pm$ .12	3.21 $\pm$ .13	3.11 $\pm$ .13
Vevo [ 11 ]	17.5	0.818	0.596	0.712	3.38 $\pm$ .12	3.93 $\pm$ .11	3.49 $\pm$ .13	3.76 $\pm$ .13
Vevo 1.5 [ 52 ]	20.5	0.798	0.554	0.683	3.10 $\pm$ .12	3.15 $\pm$ .14	3.15 $\pm$ .14	3.34 $\pm$ .14

Table 1: Results for zero-shot voice style conversion. The best results are shown in bold , and the second best results are underlined .

Chunk size (ms)	WER ( $\%$ ) $\downarrow$	S-SIM $\uparrow$	A-SIM $\uparrow$	E-SIM $\uparrow$	UTMOS $\uparrow$
200	19.8	0.797	0.586	0.704	2.33 $\pm$ .02
400	15.0	0.806	0.597	0.720	2.69 $\pm$ .03
600	13.6	0.811	0.602	0.734	2.85 $\pm$ .03
800	12.7	0.816	0.607	0.745	2.94 $\pm$ .03
1000	12.6	0.820	0.613	0.751	3.00 $\pm$ .03

Table 2: Chunk size vs. quality tradeoff

Chunk size (ms)	RTX 4060 (s)	RTX A6000 (s)
100	0.537 $\pm$ .053	0.427 $\pm$ .013
200	0.574 $\pm$ .059	0.429 $\pm$ .009
300	0.615 $\pm$ .057	0.432 $\pm$ .010
400	0.647 $\pm$ .090	0.441 $\pm$ .008
500	0.653 $\pm$ .051	0.431 $\pm$ .009
600	0.668 $\pm$ .048	0.429 $\pm$ .006

Table 3: Chunk size vs. processing time on RTX 4060 and RTX A6000. Entries show mean

\pm

standard deviation over runs.

Content Features	WER ( $\%$ ) $\downarrow$	S-SIM $\uparrow$	A-SIM $\uparrow$	E-SIM $\uparrow$	UTMOS $\uparrow$
HuBERT-Large-ASR 18th	12.5	0.731	0.389	0.627	3.28 $\pm$ .020
Vevo Continuous	12.6	0.767	0.420	0.660	2.92 $\pm$ .029
Vevo Indices	25.6	0.757	0.564	0.657	2.65 $\pm$ .036
Destylizer (offline)	10.7	0.837	0.626	0.733	3.22 $\pm$ .029

Table 4: Stylizer performance trained on different content features.

Content Features	Accent Acc. $\downarrow$	Emotion Acc. $\downarrow$	Speaker Acc. $\downarrow$
HuBERT-Large-ASR 18th	78.00%	85.60%	86.60%
Vevo Continuous	68.93%	78.73%	64.76%
Vevo Indices	55.60%	53.40%	23.00%
CosyVoice 2.0 Indices	50.60%	56.20%	25.00%
Destylizer (offline)	43.50%	47.60%	3.50%
Destylizer (streaming)	33.80%	37.90%	3.01%

Table 5: Style classification accuracy across different content features. “Acc.” stands for accuracy.

Model	WER ( $\%$ ) $\downarrow$	S-SIM $\uparrow$	A-SIM $\uparrow$	E-SIM $\uparrow$	UTMOS $\uparrow$
StyleStream (offline)	10.7	0.837	0.626	0.733	3.22 $\pm$ .029
w/o Style Emb	15.3	0.775	0.509	0.653	3.47 $\pm$ .024
w/ FSQ Indices	123.5	0.829	0.573	0.717	2.41 $\pm$ .024

Table 6: Ablation studies on the use of style encoder and the choice of Destylizer content features.

FSQ Levels	WER ( $\%$ ) $\downarrow$	S-SIM $\uparrow$	A-SIM $\uparrow$	E-SIM $\uparrow$	UTMOS $\uparrow$
[7,5,5,5,5]	13.5	0.745	0.439	0.638	3.58 $\pm$ .022
[5,5,3,3]	14.9	0.822	0.600	0.725	3.17 $\pm$ .031
[5,3,3]*	10.7	0.837	0.626	0.733	3.22 $\pm$ .029
[3,3,3]	19.0	0.825	0.618	0.732	3.13 $\pm$ .032
[5,3]	101.4	0.834	0.582	0.714	2.70 $\pm$ .029

Table 7: Ablation on Destylizer FSQ bottleneck size. “*” denotes the architecture used in all main experiments.

Training data	WER (%) $\downarrow$	S-SIM $\uparrow$	A-SIM $\uparrow$	E-SIM $\uparrow$
LMG (1.3k h)	10.7	0.837	0.626	0.733
Emilia (50k h)	10.3	0.816	0.594	0.687

Table 8: Ablation on the Destylizer training data configurations.

Model	WER (%) $\downarrow$	S-SIM $\uparrow$	A-SIM $\uparrow$	E-SIM $\uparrow$
Target 5s (offline)	9.2	0.852	0.640	0.827
Target 5s (streaming)	15.3	0.856	0.635	0.803
Target 2s (offline)	12.1	0.824	0.594	0.747
Target 2s (streaming)	19.2	0.833	0.598	0.728

Table 9: Ablation on target utterance lengths.

Zero-Shot Voice Conversion

See all 10 papers

MeanVC 2: Robust Low-Latency Streaming Zero-Shot Voice Conversion
Jun 8, 2026Guobin Ma, Yuxuan Xia, Yuepeng Jiang +6Zero-Shot Voice Conversion Diffusion Transformer
The University of New South Wales, Australia · WeNet Open Source Community, China
Zero-VC: Zero-Lookahead Streaming Voice Conversion via Speaker Anonymization
Jun 18, 2026Yudong Li, Zihao Fang, Junwen Qiu +4Speech Processing Speaker Anonymization
The Chinese University of Hong Kong, Shenzhen · Shenzhen Transsion Holdings Co., Ltd. · Shenzhen Loop Area Institute +1
MeanVoiceFlow2: Joint Optimization of Mean Flow and Content Encoder for Fast One-Step Zero-Shot Voice Conversion
Sep 30, 2026Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka +1Zero-Shot Voice Conversion Flow Matching
NTT, Inc., Japan

Jun 8, 2026J/K move · Enter open · S save

MeanVC 2: Robust Low-Latency Streaming Zero-Shot Voice Conversion

Guobin Ma, Yuxuan Xia, Yuepeng Jiang, Dake Guo+5

The University of New South Wales, Australia · WeNet Open Source Community, China

Zero-Shot Voice Conversion Diffusion Transformer Audio Diffusion Models Speech Processing Voice Conversion

Open paper AskPDF arXiv

Streaming zero-shot voice conversion (VC) has become increasingly popular due to its potential for real-time applications. The recently proposed MeanVC achieves lightweight streaming zero-shot VC, but it has several limitations: its chunk-wise autoregressive denoising doubles the effective training sequence length, conversion quality degrades under small-chunk settings, and its timbre encoder directly relies on reference mel-spectrograms, making it sensitive to reference audio quality. To address these limitations we propose MeanVC 2. We introduce future-receptive chunking (FRC), which explicitly schedules past and future receptive fields across diffusion transformer decoder layers and removes clean-chunk teacher forcing. By incorporating bounded future context, FRC enables stable conversion with a 40 ms chunk size. We further introduce a universal timbre token encoder, which constructs a timbre representation from a global speaker embedding and retrieves fine-grained timbre cues via cross-attention, improving robustness to low-quality references and enhancing zero-shot speaker similarity. Experimental results show that MeanVC 2 significantly outperforms MeanVC, while reducing latency from 211 ms to 110 ms. Audio samples are publicly available. The source code will be publicly released.

Abstract

Zero-Shot Voice Conversion

Explore similar work

MeanVC 2: Robust Low-Latency Streaming Zero-Shot Voice Conversion

Zero-VC: Zero-Lookahead Streaming Voice Conversion via Speaker Anonymization

MeanVoiceFlow2: Joint Optimization of Mean Flow and Content Encoder for Fast One-Step Zero-Shot Voice Conversion