On the Distillation Loss Functions of Speech VAE for Unified Reconstruction, Understanding, and Generation

Changhao Cheng; Wei Wang; Wangyou Zhang; Dongya Jia; Jian Wu; Zhuo Chen; Yanmin Qian

cs.SDApr 14, 2026

On the Distillation Loss Functions of Speech VAE for Unified Reconstruction, Understanding, and Generation

Authors: Changhao Cheng, Wei Wang, Wangyou Zhang, Dongya Jia, Jian Wu, Zhuo Chen, Yanmin Qian

Organizations: School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China · Auditory Cognition and Computational Acoustics Lab, MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · ByteDance Seed, China

Abstract

Continuous speech representations based on Variational Autoencoders (VAEs) have emerged as a promising alternative to traditional spectrogram or discrete token based features for speech generation and reconstruction. Recent research has tried to enrich the structural information in VAE latent representations by aligning with self-supervised learning (SSL) features, aiming for better generation performance. However, it remains unclear whether the widely-used alignment approach based on time-axis distillation is optimal when considering more tasks. To address this problem, this paper systematically explores different alignment approaches and analyzes their impact on the performances over three axes: reconstruction, understanding, and generation. We investigate various design choices in the distillation loss. Extensive experiments show that the joint-marginal alignment approach with adaptive weighting can achieve the best overall performance while allowing for a controllable balance.

Figures & tables

Figure 1: T-axis Aligned Semantic VAE (TAS-VAE) distills semantic knowledge from speech foundation models via Eq. 2 alignment loss, achieving TTS performance comparable to mel spectrograms with minor reconstruction degradation. Its latent representations still underperform on downstream speech understanding. Baseline: Mel+Vocos [ 16 ] (reconstruction), Fbank (understanding), Mel+F5-TTS [ 15 ] (generation).

Figure 2: The design space of distillation loss functions for speech VAEs. For mathematical formulations, see Eqs. 2 to 6 .

Method	Recon.			Under.									Gene.			Overall Score $\uparrow$
	Recon.			ER	PR	ASR	KS	SID	ASV	SD	IC		Gene.
	PESQ $\uparrow$	STOI $\uparrow$	$x_{r}\uparrow$	Acc $\uparrow$	PER $\downarrow$	WER $\downarrow$	Acc $\uparrow$	Acc $\uparrow$	EER $\downarrow$	DER $\downarrow$	Acc $\uparrow$	$x_{u}\uparrow$	WER $\downarrow$	SIM $\uparrow$	$x_{g}\uparrow$
Vanilla VAE	4.12	0.985	0.905	36.87	89.40	53.48	29.80	7.74	14.64	17.11	5.98	0.382	2.72	0.58	0.776	0.645
Semantic-VAE [ 14 ]	3.97	0.981	0.888	45.99	85.71	41.75	43.49	15.97	12.41	15.21	9.25	0.449	2.01	0.67	0.825	0.690
EnCodec [ 3 ]	2.77	0.938	0.746	50.41	80.75	28.90	48.46	18.08	12.05	14.10	10.04	0.489	4.71	0.56	0.756	0.651
Baseline (Mel/Fbank)	3.60	0.978	0.849	35.39	82.01	20.86	8.63	8.5E-4	9.56	10.05	9.10	0.413	2.23	0.61	0.794	0.653

Table 1: Performance comparison of different speech representation methods. For methods, ⋆ denotes the use of adaptive weighting in Section 2.2 . The metrics for the eight understanding tasks and the WER metric in the generation task are all percentages.

Panel of Figure 3 — Figure 3: The adaptive weighting of the two distillation losses for JMAS-VAE as a function of training steps. The vertical axis adopts a logarithmic coordinate system.

Panel of Figure 4 — Figure 4: Reconstruction, understanding, generation, and overall performance as well as marginal loss distances for different margin parameter combinations. Mosaic patterns indicate that the VAE training has diverged.

Distance	Recon.	Under.	Gene.			Overall
Distance	Recon.	Under.	1-WER	SIM	Overall	Overall
$\mathcal{L}_{\text{mcos}}$	0.723	-0.615	-0.552	0.701	0.694	0.133
$\mathcal{L}_{\text{mdss}}$	-0.585	0.284	0.391	-0.384	-0.378	-0.267

Table 2: Pearson Correlation Coefficients (PCCs) between distance metrics calculated on LibriSpeech-test-clean and task performance scores.

Variational Autoencoders

Variational autoencoders are probabilistic encoder–decoder models for learning latent representations, generating data, and estimating uncertainty.

See all 208 papers

STAR-VAE: Structured Topology-Aware Regularization for Audio Reconstruction and Generation
Jun 22, 2026Huadai Liu, Wen Wang, Kaicheng Luo +3Variational Autoencoders Text-to-Audio Generation
Hong Kong University of Science and Technology · Tongyi Fun Team, Alibaba Group.
Taming Audio VAEs via Target-KL Regularization
May 16, 2026Prem Seetharaman, Rithesh KumarVariational Autoencoders Text-to-Audio Generation
Adobe Research, San Francisco, CA
SPEAR-Gen: Generation-Aware Pre-training for Unified Speech Representations
Sep 28, 2026Xiaoyu Yang, Arthur Hinsvark, Antonios Alexos +3Representation Learning Self-Supervised Speech Representation Learning
Department of Engineering, University of Cambridge, Cambridge, UK · Meta Superintelligence Labs, Menlo Park, USA

Jun 22, 2026J/K move · Enter open · S save

STAR-VAE: Structured Topology-Aware Regularization for Audio Reconstruction and Generation

Huadai Liu, Wen Wang, Kaicheng Luo, Qian Chen+2

Hong Kong University of Science and Technology · Tongyi Fun Team, Alibaba Group.

Variational Autoencoders Text-to-Audio Generation Audio Representation Learning

Open paper AskPDF arXiv

Continuous Variational Autoencoders (VAEs) serve as the fundamental continuous tokenizer for modern neural audio generation systems, enabling high-fidelity reconstruction while providing a compact, smooth latent space for downstream generative priors. However, continuous VAEs face a fundamental conflict among compression rate, reconstruction fidelity, and latent space topology, which we formalize as the Rate-Distortion-Regularity Trilemma. This trilemma stems from a topological mismatch: the isotropic Gaussian prior in standard VAEs imposes a flat latent geometry that fails to accommodate audio's hierarchical nature, where low-frequency components are structured and compressible while high-frequency components are stochastic and incompressible, leading to disordered information packing in which crucial semantic features are interleaved with high-entropy noise. To address this challenge, we propose Structured Topology-Aware Regularization (STAR), a general training strategy that reshapes latent space geometry by imposing a growth-based constraint field, routing structural and textural information into channel subspaces with matching capacities. STAR is applicable to any VAE architecture and effectively resolves the trilemma, as demonstrated in CNN-based VAEs. We further present STAR-VAE, which combines STAR with a hybrid CNN-Mamba architecture for local feature extraction and linear-complexity global context modeling, and STAR-Gen, an LLM-based Flow Matching framework that leverages STAR-VAE's structured latent space for high-fidelity generation without vector quantization artifacts. Experiments across diverse audio domains show that STAR-VAE achieves state-of-the-art reconstruction fidelity and enhanced semantic information preservation, while the structured latent space improves both traditional diffusion models and STAR-Gen for text-to-audio generation.

Abstract

Variational Autoencoders

Explore similar work

STAR-VAE: Structured Topology-Aware Regularization for Audio Reconstruction and Generation

Taming Audio VAEs via Target-KL Regularization

SPEAR-Gen: Generation-Aware Pre-training for Unified Speech Representations