VOSSA: Voiceprint Optimization for Streaming Speech Architectures

Mu-Ruei Tseng; Waris Quamer; Ghady Nasrallah; Ricardo Gutierrez-Osuna

eess.ASSep 30, 2026

VOSSA: Voiceprint Optimization for Streaming Speech Architectures

Authors: Mu-Ruei Tseng, Waris Quamer, Ghady Nasrallah, Ricardo Gutierrez-Osuna

Organizations: Department of Computer Science & Engineering, Texas A&M University, College Station, US

Abstract

Real-time voice conversion (VC) systems commonly rely on pretrained speaker embeddings from automatic speaker verification (ASV) models. While effective for speaker discrimination, these embeddings are trained to remain stable across phonetic and prosodic variations within-speaker, which may conflict with frame-level acoustic generation in streaming constraints. To address this issue, we propose VOSSA (Voiceprint Optimization for Streaming Speech Architectures), a speaker representation framework that extracts speaker information from intermediate content encoder layers and aggregates using attentive statistics pooling. The embedding is trained jointly with VC objectives, removing the need for a separate speaker encoder. Across six datasets, VOSSA improves F0 dynamics and vowel-discriminative acoustic cues while maintaining comparable NISQA-MOS, WER, and speaker similarity. Perceptual tests further indicate improvements in naturalness, speaker similarity, intelligibility, and vibrancy.

Figures & tables

Figure 1: Training workflow for the TVTSyn backbone . (a) content encoder trained against HuBERT k-means pseudo-labels, and (b) decoder conditioned on speaker embedding trained with self-supervision and discriminator objectives. (c) Overview of the training protocol in VOSSA . The self-reconstruction path (bottom, purple) uses segments from the same LibriTTS speaker to provide fully supervised training, while the non-parallel VC path (top, orange) uses VoxCeleb targets to condition conversion. (d) Speaker embedding extraction from the frozen content encoder. We collect features from the last CNN layer and every other layer of the MHSA stack, concatenate them to form $\mathbf{H}\in\mathbb{R}^{T\times Ld}$ , and apply attentive statistics pooling and an MLP to obtain a global speaker embedding.

Models	Ground truth	slt24 [ 1 ]	DarkStream [ 2 ]	GenVC-s [ 6 ]	TVTSyn [ 3 ]	VOSSA (Ours)
Standard VC metrics and harmonics
NISQA-MOS $(\uparrow)$	$4.06\pm 0.84$	$3.46\pm 0.80$	$3.12\pm 0.81$	$3.04\pm 0.83$	$\mathbf{3.52\pm 0.81}$	$3.48\pm 0.91$
WER $(\downarrow)$	$0.07\pm 0.16$	$0.18\pm 0.25$	$0.25\pm 0.27$	$0.20\pm 0.23$	$0.17\pm 0.23$	$\mathbf{0.17\pm 0.23}$
$\mathrm{Sim}_{\text{src}}^{\text{syn}}$ $(\downarrow)$	–	$0.21\pm 0.11$	$\mathbf{0.06\pm 0.11}$	–	$0.10\pm 0.10$	$0.11\pm 0.31$
$\mathrm{Sim}_{\text{trg}}^{\text{syn}}$ $(\uparrow)$	–	$0.46\pm 0.11$	$0.54\pm 0.12$	–	$0.59\pm 0.11$	$\mathbf{0.86\pm 0.19}$
HNR $(\uparrow)$	–	$9.52\pm 2.90$	$7.93\pm 3.69$	$8.67\pm 3.04$	$\mathbf{9.90\pm 2.75}$	$9.72\pm 2.80$

Table 1: Evaluation of VOSSA against SOTA streaming VC baselines.

\mathrm{Sim}_{\text{src}}^{\text{syn}}

can be interpreted as anonymization strength (lower is better), whereas

\mathrm{Sim}_{\text{trg}}^{\text{syn}}

represents VC strength (higher is better). Best results are bolded ; second-best are underlined .

Figure 2: F1 distribution by vowel height for target speaker id00061 (Voxceleb). GT: original speech.

Metric	TVTSyn	VOSSA (Ours)
Speaker similarity ( $\uparrow$ , %)	46	54
Intelligibility ( $\uparrow$ , %)	44	56
Vibrancy ( $\uparrow$ , %)	48	52

Table 2: Perceptual listening test results.

Explore similar work

Zero-VC: Zero-Lookahead Streaming Voice Conversion via Speaker Anonymization
Jun 18, 2026Yudong Li, Zihao Fang, Junwen Qiu +4Speaker Prosody
The Chinese University of Hong Kong, Shenzhen · Shenzhen Transsion Holdings Co., Ltd. · Shenzhen Loop Area Institute +1
MeanVC 2: Robust Low-Latency Streaming Zero-Shot Voice Conversion
Jun 8, 2026Guobin Ma, Yuxuan Xia, Yuepeng Jiang +6Voice Conversion Streaming
The University of New South Wales, Australia · WeNet Open Source Community, China
MeanVoiceFlow2: Joint Optimization of Mean Flow and Content Encoder for Fast One-Step Zero-Shot Voice Conversion
Sep 30, 2026Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka +1Voice Conversion Speaker Similarity
NTT, Inc., Japan

Jun 18, 2026, cs.SDJ/K move · Enter open · S save

Zero-VC: Zero-Lookahead Streaming Voice Conversion via Speaker Anonymization

Yudong Li, Zihao Fang, Junwen Qiu, Ruihai Jing+3

The Chinese University of Hong Kong, Shenzhen · Shenzhen Transsion Holdings Co., Ltd. · Shenzhen Loop Area Institute+1

Speaker Prosody Anonymization Streaming Lookahead

Open paper AskPDF arXiv

Streaming zero-shot voice conversion struggles to disentangle timbre from linguistic content without degrading utility or inflating latency. Current methods rely on information bottleneck (IB) or speaker perturbation. While IB filters out timbre, it discards prosody, forcing models to explicitly inject features like fundamental frequency. This often requires buffering future frames, creating algorithmic lookahead latency. On the other hand, existing perturbation methods largely overlook the crucial trade-off between timbre leakage and utility preservation. Recognizing this neglected trade-off, we find that the inherent objective of Speaker Anonymization (SA) aligns well with balancing these factors. Thus, we introduce SA as a novel perturbation mechanism to explicitly mitigate timbre leakage while retaining prosodic utility. Crucially, SA's robust representations significantly alleviate the generator's reliance on future context, enabling our strictly causal, zero-lookahead network. Audio samples are available at https://amphionteam.github.io/Zero-VC-demo/.

Abstract

Explore similar work

Zero-VC: Zero-Lookahead Streaming Voice Conversion via Speaker Anonymization

MeanVC 2: Robust Low-Latency Streaming Zero-Shot Voice Conversion

MeanVoiceFlow2: Joint Optimization of Mean Flow and Content Encoder for Fast One-Step Zero-Shot Voice Conversion