ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining

Anuj Diwan; Eunsol Choi; David Harwath

eess.ASMar 30, 2026

ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining

Authors: Anuj Diwan, Eunsol Choi, David Harwath

Organizations: Department of Computer Science, The University of Texas at Austin, USA · Computer Science and Data Science, New York University, USA

Abstract

We introduce ParaSpeechCLAP, a family of dual-encoder models that map speech and text style captions into a shared embedding space, supporting rich intrinsic (speaker-level) and situational (utterance-level) descriptors, such as pitch, texture, and emotion, beyond the narrow set handled by existing models. We train separate Intrinsic and Situational models alongside a unified Combined model, finding that specialized models are stronger on individual style dimensions while the unified model excels on compositional evaluation. We further show that ParaSpeechCLAP-Intrinsic benefits from an additional classification loss and class-balanced training. We demonstrate performance on style caption retrieval, speech attribute classification, and usability as inference-time reward models for style-prompted TTS. ParaSpeechCLAP models outperform baselines on most metrics across all three applications. Our models and code are released at https://github.com/ajd12342/paraspeechclap .

Figures & tables

Figure 1: An overview of the ParaSpeechCLAP dual-encoder training methodology. (Left) All ParaSpeechCLAP models are trained with an InfoNCE contrastive loss to align speech and text embeddings. (Right) For ParaSpeechCLAP-Intrinsic, this is supplemented with a classification loss, where audio embeddings are aligned with text embeddings generated from templates filled with class labels.

Dual-Encoder Baselines
	Situational					Intrinsic					Combined
Model	R@1	R@10	MedR $\downarrow$	UAR	F1	R@1	R@10	MedR $\downarrow$	UAR	F1	R@1	R@10	MedR $\downarrow$
Random Projection	$0.14$	$2.65$	$170$	$6.56$	$2.66$	$1.45$	$17.66$	$28$	$23.64$	$16.21$	$0.00$	$0.49$	$725$
ParaCLAP [ 1 ]	$0.41$	$3.98$	$106$	$11.56$	$3.67$	$1.95$	$26.32$	$19$	$29.16$	$20.65$	$0.34$	$1.67$	$462$
ParaCLAP-PSC	$15.64$	$72.55$	$5$	$23.91$	$23.57$	$11.49$	$60.69$	$8$	$40.27$	$37.27$	$5.58$	$32.12$	$23$
Classifier Baseline

Table 1: Performance on situational, intrinsic, and combined evaluation datasets. We report retrieval metrics (Recall@k, Median Rank) and classification metrics (Unweighted Average Recall, Macro F1). For all metrics except median rank, higher is better. PSCLAP refers to ParaSpeechCLAP. Our ParaSpeechCLAP models outperform all baselines on most metrics. The unified ParaSpeechCLAP-Combined model is weaker than the specialized variants on specialized evaluation datasets and stronger on the combined evaluation dataset.

	Style Consistency			Quality	Intell.
Model	CMOS	Int TR	Sit TR	NMOS	WER $\mathbf{\downarrow}$
GT	$4.04\scriptstyle{\pm 0.07}$	$73.1\%$	$82.3\%$	$4.42\scriptstyle{\pm 0.06}$	$8.04$
Vanilla	$3.61\scriptstyle{\pm 0.07}$	$57.9\%$	$69.2\%$	$\mathbf{3.34}\scriptstyle{\pm 0.09}$	$8.14$
w/ PSCLAP	$\mathbf{3.70}\scriptstyle{\pm 0.07}$	$\mathbf{62.4\%}$	$\mathbf{74.3\%}$	$\mathbf{3.35}\scriptstyle{\pm 0.09}$	$\mathbf{7.56}$

Table 2: Evaluation results comparing TTS inference without (Vanilla) and with ParaSpeechCLAP-guided best-of-N selection. We report style consistency (CMOS, Intrinsic Rich Tag Recall, and Situational Rich Tag Recall), speech quality (NMOS), and intelligibility (WER). PSCLAP refers to ParaSpeechCLAP. Using ParaSpeechCLAP models results in improved style consistency without affecting quality or intelligibility.

Ablations
	Intrinsic
Model	R@1	R@10	MedR $\downarrow$	UAR	F1
PSCLAP-Intrinsic	$\mathbf{18.62}$	$\mathbf{65.23}$	$\mathbf{6}$	$\mathbf{46.58}$	$\mathbf{38.27}$
w/o New Encoders	$11.77$	$63.63$	$7$	$41.44$	$33.18$
w/o Multitask	$13.76$	$63.17$	$7$	$33.11$	$30.00$
w/o Class-Balancing	$13.94$	$63.92$	$\mathbf{6}$	$40.60$	$31.87$

Table 3: Ablation study for PSCLAP-Intrinsic. We compare our final model against versions with components removed.

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 303 papers

Beyond Speech Captions: Speech-Rewarded Style Planning for Conversational Text-to-Speech
Oct 8, 2026Shiao Zhu, Lianbo Liu, Sizhen Lyu +3TTS Synthesis Controllable Speech Generation
Institute of Science Tokyo, Tokyo, Japan · Independent Researcher, Tokyo, Japan
A Benchmark for Controllable Speaking-Style Captioning in Audio-Language Models
Oct 15, 2026Shreeram Suresh Chandra, Ashwyn Sharma, Liang Lu +4Audio Captioning Audio-Language Model Evaluation
Center for Language and Speech Processing, Johns Hopkins University, USA · Otter AI · The University of Texas at Dallas, USA +1
How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speech
Jun 18, 2026Nityanand Mathur, Hamees Sayed, Wasim Madha +4TTS Synthesis Audio Diffusion Models

Oct 8, 2026J/K move · Enter open · S save

Beyond Speech Captions: Speech-Rewarded Style Planning for Conversational Text-to-Speech

Shiao Zhu, Lianbo Liu, Sizhen Lyu, Yuzhe Wang+2

Institute of Science Tokyo, Tokyo, Japan · Independent Researcher, Tokyo, Japan

TTS Synthesis Controllable Speech Generation

Open paper AskPDF arXiv

Natural-language style descriptions provide an interpretable interface between large language models (LLMs) and controllable text-to-speech (TTS). However, using descriptions as pseudo-labels compresses target acoustics into text, and descriptive fidelity need not imply effective control of a particular synthesizer. We empirically show that speech-text alignment only weakly predicts downstream acoustic similarity among candidate instructions for the same utterance. We therefore propose Speech-Rewarded Style Planning (SRSP), which trains a text-based style planner through a frozen downstream TTS model. Given dialogue history and response text, the planner generates candidate instructions and is optimized with group-relative policy optimization (GRPO), using the teacher-forced likelihood of target speech tokens as the reward. On an English subset of the ISCSLP 2026 CoT-TTS corpus, SRSP achieves higher speech-style and emotion similarity to target speech and lower mel-cepstral distortion than the Base LLM and target-audio-informed captioning baselines. LLM-based expressive speech evaluation further shows gains over all baselines in contextual appropriateness and reference consistency.

Abstract

Audio-Language Models

Explore similar work

Beyond Speech Captions: Speech-Rewarded Style Planning for Conversational Text-to-Speech

A Benchmark for Controllable Speaking-Style Captioning in Audio-Language Models

How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speech