SPEAR-Gen: Generation-Aware Pre-training for Unified Speech Representations

Xiaoyu Yang; Arthur Hinsvark; Antonios Alexos; Osama Hanna; Philip C. Woodland; Yiting Lu

eess.ASSep 28, 2026

SPEAR-Gen: Generation-Aware Pre-training for Unified Speech Representations

Authors: Xiaoyu Yang, Arthur Hinsvark, Antonios Alexos, Osama Hanna, Philip C. Woodland, Yiting Lu

Organizations: Department of Engineering, University of Cambridge, Cambridge, UK · Meta Superintelligence Labs, Menlo Park, USA

Abstract

Speech understanding and generation place different demands on speech representations, and existing models are typically optimised towards one capability or the other. To reduce this gap, we introduce SPEAR-Gen, a speech representation model that learns a single representation for both capabilities. Task-aligned feature aggregation consolidates complementary linguistic and paralinguistic information across a frozen encoder into discrete targets for masked prediction, while a coarse-to-fine objective combines log-Mel reconstruction with residual flow matching to preserve spectral structure and fine-grained acoustic variation. Experiments on SUPERB and speech resynthesis show that SPEAR-Gen maintains strong understanding performance while substantially improving resynthesis quality and speaker preservation. These results demonstrate that a single speech representation can effectively support both understanding and generation.

Figures & tables

		# Encoder Params	SUPERB: linguistic				SUPERB: paralinguistic / speaker
Method	PT data	# Encoder Params	PR $\downarrow$	ASR $\downarrow$	KWS $\uparrow$	IC $\uparrow$	ER $\uparrow$	SID $\uparrow$	SV $\downarrow$	SD $\downarrow$
HuBERT Base [ 9 ]	LS-960	95M	6.74	6.97	96.23	96.36	62.44	61.45	6.18	7.19
WavLM Base [ 2 ]	LS-960	95M	5.59	6.42	96.36	96.57	61.35	47.04	8.01	6.41
SPEAR † [ 23 ]	LS-960	95M	4.09	3.90	96.85	98.37	65.78	55.47	6.80	3.23
WavCube [ 20 ]	LS-960	350M	9.91	9.36	97.42	90.41	63.47	42.36	5.86	8.14
SPEAR-Gen Base	LS-960	95M	4.53	3.71	97.47	98.73	69.48	70.23	5.08	3.26

Table 2: Final-layer results on SUPERB. Upper block: understanding-oriented models; lower block: unified understanding–generation models. PT data denotes representation pre-training data. Note: SPEAR-Gen uses external supervision only for TAFA target construction. SPEAR † : reproduced version of SPEAR on LS-960. Best in bold and second-best underlined.

Method	PESQ $\uparrow$	ViSQOL $\uparrow$	UTMOS $\uparrow$	STOI $\uparrow$	WER $\downarrow$	SIM $\uparrow$
WavLM Base [ 2 ]	1.16	1.68	3.37	0.79	2.50	0.473
SPEAR † [ 23 ]	1.20	1.70	3.38	0.81	2.61	0.467
WavCube [ 20 ]	2.93	3.92	3.72	0.95	2.26	0.900
SPEAR-Gen Base	2.94	4.12	3.69	0.95	2.22	0.892
SPEAR-Gen Large	2.98	4.19	3.78	0.96	2.19	0.910

Table 3: Speech resynthesis results on LibriSpeech test-clean.

		SUPERB			Resynthesis
MTP target	Gen.	ASR $\downarrow$	ER $\uparrow$	SV $\downarrow$	ViSQOL $\uparrow$	WER $\downarrow$	SIM $\uparrow$
Single-layer	$\times$	3.90	65.78	6.80	1.70	2.61	0.467
TAFA	$\times$	3.83	68.32	6.15	1.86	2.54	0.555
None	$\checkmark$	20.72	53.83	14.42	4.29	2.23	0.909
Single-layer	$\checkmark$	3.94	66.70	6.42	4.07	2.24	0.866
LayerAvg	$\checkmark$	4.35	66.91	6.02	4.06	2.24	0.873

Table 4: Ablation of MTP target and generative modelling (Gen.). MTP target specifies the representation from which MVQ prediction targets are constructed. None denotes training without MTP. LayerAvg denotes uniform layer averaging.

	#Decoder Params	SUPERB			Resynthesis
Gen. Obj.	#Decoder Params	ASR $\downarrow$	ER $\uparrow$	SV $\downarrow$	PESQ $\uparrow$	UTMOS $\uparrow$	WER $\downarrow$	SIM $\uparrow$
Mel $\ell_{1}$	15M	3.79	68.42	5.41	2.76	3.55	2.25	0.883
Mel $\ell_{1}$	30M	3.70	69.28	5.46	2.73	3.61	2.22	0.883
Full-mel FM	30M	3.72	68.92	5.59	2.61	3.58	2.26	0.875
SPEAR-Gen	30M	3.71	69.48	5.08	2.94	3.69	2.22	0.892

Table 5: Ablation of the generative objectives. #Params denotes the total number of parameters in decoders.

Explore similar work

WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
May 7, 2026Guanrou Yang, Tian Tan, Qian Chen +12Flow-Matching Text-To-Speech
Shanghai Jiao Tong University · Independent Researcher · Shanghai Innovation Institute +4
SPARCLE: SPeaker-aware Aligned Representations via Contrastive Language Embeddings
May 1, 2026Priyam Mazumdar, Yurii Halychanskyi, Steven Guo +2Discrete Speech Representations Grapheme-To-Phoneme
University of Illinois Urbana-Champaign, USA · National Center for Supercomputing Applications, USA
OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL
Jun 29, 2026Karl El Hajal, Mathew Magimai. -DossSelf-Supervised Speech Models Self-Supervised Learning
Idiap Research Institute, Switzerland · EPFL, Switzerland

May 7, 2026, eess.ASJ/K move · Enter open · S save

WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling

Guanrou Yang, Tian Tan, Qian Chen, Zhikang Niu+11

Shanghai Jiao Tong University · Independent Researcher · Shanghai Innovation Institute+4

Flow-Matching Text-To-Speech

Open paper AskPDF arXiv

Integrating speech understanding and generation is a pivotal step toward building unified speech models. However, the different representations required for these two tasks currently pose significant compatibility challenges. Typically, semantics-oriented features are learned from self-supervised learning (SSL), and acoustic-oriented features from reconstruction. Such fragmented representations hinder the realization of truly unified speech systems. We present WavCube, a compact continuous latent derived from an SSL speech encoder that simultaneously supports speech understanding, reconstruction, and generation. WavCube employs a two-stage training scheme. Stage 1 trains a semantic bottleneck to filter off-manifold redundancy that makes raw SSL features intractable for diffusion. Stage 2 injects fine-grained acoustic details via end-to-end reconstruction, while a semantic anchoring loss ensures the representation remains grounded within its original semantic manifold. Comprehensive experiments show that WavCube closely approaches WavLM performance on SUPERB despite an 8x dimensional compression, attains reconstruction quality on par with existing acoustic representations, delivers state-of-the-art zero-shot TTS performance with markedly faster training convergence, and excels in speech enhancement, separation, and voice conversion tasks on the SUPERB-SG benchmark. Systematic ablations reveal that WavCube's two-stage recipe resolves two intrinsic flaws of SSL features for generative modeling, paving the way for future unified speech systems. Codes and checkpoints are available at https://github.com/yanghaha0908/WavCube.

Abstract

Explore similar work

WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling

SPARCLE: SPeaker-aware Aligned Representations via Contrastive Language Embeddings

OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL