Holistic Parallel Supervision for Expressive Speech to Speech Translation

Hao Zhang; Ismail Rasim Ulgen; Bismarck Bamfo Odoom; Rui Liu; Zihan Zhang; Berrak Sisman; Philipp Koehn

cs.CLNov 4, 2026

Holistic Parallel Supervision for Expressive Speech to Speech Translation

Authors: Hao Zhang, Ismail Rasim Ulgen, Bismarck Bamfo Odoom, Rui Liu, Zihan Zhang, Berrak Sisman, Philipp Koehn

Organizations: Center for Language and Speech Processing, Johns Hopkins University, MD, USA

Abstract

Expressiveness is central to speech-to-speech translation (S2ST), yet existing datasets rarely align source and target speech jointly in content, prosody, timing, and speaker identity. Consequently, current systems often rely on indirect conditioning or transfer mechanisms. To obtain this supervision, we introduce a training-data construction strategy that combines professional dubbing, which provides human-performed correspondence in content, prosody, and timing, with multilingual voice conversion to reduce speaker mismatch. We evaluate this supervision using existing expressive S2ST systems and DirectS2ST, an end-to-end method developed in this work that predicts the complete target-speech representation directly from source speech. Experiments on English-German and English-Spanish show that holistic parallel supervision improves expressive realization across all evaluated systems. DirectS2ST exhibits the largest gains in content, prosody, timing, speaker similarity, and human-rated expressiveness. To our knowledge, this work provides the first systematic investigation of holistic parallel supervision for expressive S2ST.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the holistic parallel supervision: (a) VC-Dub Construction; (b) Parallel Supervision; (c) DirectS2ST.

Pair	Dataset	#Pairs	Src. Hrs	Tgt. Hrs	Total Hrs	Avg. Dur. (s)
En-De	CVSS-T	127,250	128.19	122.16	250.35	3.63 $\|$ 3.46
En-De	VC-Dub	131,399	127.40	135.82	263.22	3.49 $\|$ 3.72
En-Es	CVSS-T	78,963	79.76	72.77	152.53	3.64 $\|$ 3.32
En-Es	VC-Dub	79,200	84.61	86.52	171.13	3.85 $\|$ 3.93

Table 1: Statistics of the speech-to-speech training data. Average duration is reported as source

|

target.

Dir.	Dataset	BLASER $\uparrow$	A.PCP $\uparrow$	Rate $\uparrow$	Pause $\uparrow$	Vsim $\uparrow$	DNSMOS Pro $\uparrow$
En $\downarrow$ De	CVSS-T	3.405	2.505	0.365	0.524	0.208	3.160 $\|$ 2.797
	Raw-Dub	3.634	2.759	0.416	0.682	0.060	3.900 $\|$ 3.684
	VC-Dub	3.611	3.000	0.435	0.698	0.406	4.104 $\|$ 3.684
	w/ Shuffled	2.285	2.218	-0.009	0.394	0.025	4.104 $\|$ 3.684
En $\downarrow$ Es	CVSS-T	3.400	2.537	0.372	0.550	0.194	3.173 $\|$ 2.780
	Raw-Dub	3.632	2.920	0.417	0.676	0.046	3.913 $\|$ 3.714

Table 2: Pair-level correspondence and speech-quality results for CVSS-T, Raw-Dub , and VC-Dub . Shuffled VC-Dub pairs provide a negative control. DNSMOS Pro is reported as source (En)

|

target (De/Es).

Dir.	Model	Train	ASR-BLEU $\uparrow$	BLASER $\uparrow$	A.PCP $\uparrow$	Rate $\uparrow$	Pause $\uparrow$	Vsim $\uparrow$	NAT $\uparrow$
En $\downarrow$ De	TransVIP	CVSS-T	12.25	3.397	2.609	0.470	0.489	0.126	3.257
	TransVIP	VC-Dub	13.84	3.399	2.743 †	0.600 †	0.549 †	0.169 †	3.472 †
	DirectS2ST	CVSS-T	6.01	3.234	2.162	0.334	0.687	0.114	3.339
	DirectS2ST	VC-Dub	10.52 †	3.373 †	2.905 †,‡	0.589 †	0.752 †,‡	0.217 †,‡	3.450 †
En $\downarrow$ Es	TransVIP	CVSS-T	15.53	3.354	2.664	0.601	0.494	0.138	3.092
	TransVIP	VC-Dub	18.96 †	3.433 †	2.803 †	0.671 †	0.562 †	0.158 †	3.407 †

Table 3: Expressive S2ST results on mExpresso in both translation directions. † denotes significance between training sets within the same model, and ‡ against the strongest baseline in the same language direction. Significance is determined by two-sided paired randomization tests with 10,000 permutations and Holm–Bonferroni correction (

p_{\mathrm{adj}}<0.05

Dir.	Train	ASR-BLEU $\uparrow$	BLASER $\uparrow$	A.PCP $\uparrow$	Rate $\uparrow$	Pause $\uparrow$	Vsim $\uparrow$	NAT $\uparrow$
En $\rightarrow$ De	VC-Dub	10.52	3.373	2.905 †	0.589 †	0.752	0.217 †	3.450
En $\rightarrow$ De	Raw-Dub	9.37	3.339	2.712	0.455	0.741	0.065	3.430
En $\rightarrow$ Es	VC-Dub	14.22	3.259	3.091 †	0.669 †	0.733	0.205 †	3.466
En $\rightarrow$ Es	Raw-Dub	14.53	3.266	2.824	0.555	0.730	0.058	3.470

Table 4: VC-Dub ablation of DirectS2ST on mExpresso.

Model	Training Data	MOS $\uparrow$	95% CI	ICC(A,1)	ICC(A,9)
TransVIP	CVSS-T	1.82	[1.62, 2.01]	0.445	0.878
TransVIP	VC-Dub	2.24 †	[2.03, 2.44]
Dub-S2ST	CVSS-T	2.88	[2.63, 3.12]
Dub-S2ST	VC-Dub	3.59 †	[3.34, 3.84]
DirectS2ST	CVSS-T	2.13	[1.93, 2.33]
DirectS2ST	VC-Dub	3.42 †	[3.20, 3.64]

Table 5: Expressiveness MOS with 95% confidence intervals and inter-rater reliability (ICC). Significance is assessed using crossed linear mixed-effects models with system configurations as a fixed effect and listener and utterance as random effects, followed by Holm–Bonferroni correction.

{}^{\dagger}p_{\mathrm{adj}}<0.05

Speech Translation

Speech translation converts spoken language into text or speech in another language, including real-time systems that balance fidelity and latency.

See all 62 papers

STEB: A Speech-to-Speech Translation Expressiveness Benchmark for Evaluating Beyond Translation Fidelity
Jun 24, 2026Sitong Cheng, Weizhen Bian, Songjun Cao +9Speech Translation Speech Translation Evaluation
Hong Kong University of Science and Technology · Tencent Youtu Lab · Shenzhen International Graduate School, Tsinghua University
Learning Timeline-Preserving Speech-to-Speech Translation
Oct 31, 2026Zhisheng Zheng, David HarwathSpeech Translation
The University of Texas at Austin
PACT: A Pragmatic Prompting Framework for Expressive and Voice-Preserving Speech-to-Speech Translation
Oct 13, 2026Yuchen Song, Satoshi NakamuraSpeech Translation
The Chinese University of Hong Kong, Shenzhen, China · Shenzhen Loop Area Institute, China

Jun 24, 2026J/K move · Enter open · S save

STEB: A Speech-to-Speech Translation Expressiveness Benchmark for Evaluating Beyond Translation Fidelity

Sitong Cheng, Weizhen Bian, Songjun Cao, Jin Li+8

Hong Kong University of Science and Technology · Tencent Youtu Lab · Shenzhen International Graduate School, Tsinghua University

Speech Translation Speech Translation Evaluation

Open paper AskPDF arXiv

Speech-to-speech translation (S2ST) should preserve not only lexical meaning, but also expressive attributes: emotion, scenario style (e.g., news reporting vs. dramatic dialogue), and nonverbal vocalizations (NVs). Moreover, collecting cross-lingual target speech that is both translation-faithful and expressively aligned with the source is difficult at scale, making reference-based evaluation impractical. We introduce STEB (Speech-to-Speech Translation Expressiveness Benchmark), a 32.6-hour Chinese--English benchmark that evaluates both standard dimensions (translation fidelity, speaker similarity, duration alignment) and expressiveness dimensions (emotion, scenario style, NV preservation). For expressiveness evaluation, STEB uses a caption-then-summarize framework that converts speech into structured expressive attributes and compares source and hypothesis attributes with an LLM judge. Human validation shows statistically significant correlations with listener judgments across all expressive dimensions. We evaluate six S2ST systems covering cascaded systems, end-to-end models, and speech large language models. Many systems, especially cascaded ones, achieve strong translation fidelity, but they still struggle with emotion preservation (best: 3.82/5) and NV preservation (best: 2.31/5). These results reveal a gap between semantic transfer and expressive transfer, identifying expressiveness preservation as an open challenge for S2ST. Audio samples are available at https://cmots.github.io/steb.github.io/.

Abstract

Speech Translation

Explore similar work

STEB: A Speech-to-Speech Translation Expressiveness Benchmark for Evaluating Beyond Translation Fidelity

Learning Timeline-Preserving Speech-to-Speech Translation

PACT: A Pragmatic Prompting Framework for Expressive and Voice-Preserving Speech-to-Speech Translation