Is Prosody Lost in Translation? Fine-Grained Cross-Lingual Prosody Similarity Across Languages

Haopeng Xie; Ismail Rasim Ulgen; Sofia Son; Berrak Sisman; Philipp Koehn

cs.SDAug 28, 2026

Is Prosody Lost in Translation? Fine-Grained Cross-Lingual Prosody Similarity Across Languages

Authors: Haopeng Xie, Ismail Rasim Ulgen, Sofia Son, Berrak Sisman, Philipp Koehn

Organizations: Center for Language and Speech Processing (CLSP), Johns Hopkins University Baltimore, MD, USA

Abstract

Prosody plays an important role in speech translation, conveying information such as emphasis, emotion, and intent beyond lexical content. However, despite recent progress in expressive speech-to-speech translation (S2ST), little is known about how prosodic patterns are similar/different across languages. Understanding these cross-lingual similarities and differences is crucial for effectively incorporating prosody into expressive S2ST systems. In this work, we present the first fine-grained cross-lingual analysis of prosody using multilingual dubbing data across English-German, English-Spanish, and English-French language pairs. We analyze the similarity of pitch, energy, and temporal feature patterns between source and target speech and investigate the linguistic and alignment-related factors affecting this similarity. Our analysis reveals inherent cross-lingual correlations in prosodic structure between certain languages. The findings provide important insights into the transferability of prosody across languages and offer empirical guidance for future expressive speech-to-speech translation systems.

Figures & tables

Figure 1: Overview of the cross-lingual prosody analysis pipeline.

Stage	Metric	Threshold
Audio quality	SI-SDR	$\geq$ 7.5
Audio quality	STOI	$\geq$ 0.91
Audio quality	PESQ	$\geq$ 1.25
Diarization	Speakers	$\leq$ 1
Semantic filtering	SONAR cosine	$\geq$ 0.63

Table 1: Filtering thresholds used for audio quality, speaker diarization, and semantic consistency.

Stage	DE-EN	EN-ES	EN-FR
Raw pairs	100,000	100,000	100,000
After SQUIM	41,610	42,974	42,045
After diarization	28,041	28,990	29,448
After SONAR	20,964	22,598	20,342
After pitch scoring	16,335	16,939	15,957

Table 2: Number of utterance pairs remaining after each filtering stage. The final row excludes utterance pairs for which valid non-constant pitch contours could not be detected in either the source or target speech.

Pair	Utts.	Src. hrs	Tgt. hrs	Total hrs
DE–EN	16,335	18.8	17.9	36.7
EN–ES	16,939	18.8	19.1	37.9
EN–FR	15,957	17.6	18.1	35.7

Table 3: Statistics of the final analysis corpus. Hours are computed over utterances retained for prosodic correlation scoring.

Figure 2: Example of the cross-lingual prosody alignment pipeline. Frame-level pitch contours are extracted from source and target utterances using word timestamps and bilingual word alignments. Word-level prosodic segments are grouped into semantically corresponding alignment clusters and reordered prior to correlation analysis.

Figure 3: Distribution of utterance-level zero-lag Spearman correlations for pitch and energy across aligned source–target utterance pairs.

Feature	Pair	Mean $r$	95% CI	Shuffled
Pitch	DE–EN	0.229	[0.223, 0.236]	0.007
	EN–ES	0.241	[0.235, 0.247]	0.005
	EN–FR	0.246	[0.239, 0.253]	0.010
Energy	DE–EN	0.183	[0.179, 0.187]	0.043
	EN–ES	0.174	[0.170, 0.178]	0.030
	EN–FR	0.176	[0.172, 0.179]	0.026

Table 4: Mean utterance-level Spearman correlation between aligned source and target prosodic contours. Confidence intervals are computed using episode-clustered bootstrap resampling. The shuffled baseline randomizes contours within each utterance. All aligned correlations significantly exceed the shuffled baseline (paired Wilcoxon,

p<10^{-80}

Feature	Class	DE–EN	EN–ES	EN–FR
Pitch	Nouns	+0.0228	+0.0208	+0.0203
	Verbs	+0.0034	+0.0156	+0.0045
	Modifiers	+0.0083	+0.0057	+0.0055
	Function	-0.0065	+0.0108	+0.0132
Energy	Nouns	+0.0355	+0.0396	+0.0402
	Verbs	-0.0041	-0.0064	-0.0128

Table 5: Leave-one-group-out POS class analysis. Positive values indicate that the POS class has a positive effect on cross-lingual prosodic correspondence.

Figure 4: Leave-one-group-out POS class analysis across language pairs and prosodic dimensions. Bars denote the change in utterance-level Spearman correlation after removing all prosodic frames associated with a POS category. Positive values indicate that the lexical class has a positive effect on cross-lingual prosodic correspondence. Error bars denote 95% confidence intervals on the mean effect, computed as

1.96\cdot(\sigma/\sqrt{n})

, where

\sigma

is the standard deviation of per-utterance values.

Pair	Feature	Corr.
DE–EN	Phonemes	0.885
EN–ES	Phonemes	0.875
EN–FR	Phonemes	0.871
DE–EN	Duration	0.891
EN–ES	Duration	0.890
EN–FR	Duration	0.884

Table 6: Utterance-level correlations between source and target temporal features (total number of phonemes, duration) for each language pair.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Symm.	AER $\downarrow$	P	R	F
GD	37.4%	66.7%	58.8%	0.625
GDFA	37.7%	65.2%	59.6%	0.623
Inter.	39.6%	82.2%	47.6%	0.603
GDF	40.0%	58.4%	61.7%	0.600
Union	40.8%	56.3%	62.6%	0.593

Appendix

Table 7: FastAlign symmetrization heuristics evaluated on the RWTH German–English gold alignment Koehn (2005) ; Vilar et al. (2006) benchmark. GD: grow-diag; GDFA: grow-diag-final-and; GDF: grow-diag-final; Inter.: intersection. P, R, and F denote precision, recall, and F-measure. Bold denotes the highest precision. Intersection symmetrization is used in the prosody pipeline due to its substantially higher precision.

Feature	Comparison	Mean diff.	95% CI	Cliff’s $\delta$	$p$ (Holm)
Pitch	DE–EN vs. EN–ES	$-0.0114$	[ $-0.020$ , $-0.003$ ]	$-0.020$	$3\times 10^{-3}$
Pitch	DE–EN vs. EN–FR	$-0.0165$	[ $-0.026$ , $-0.007$ ]	$-0.029$	$2\times 10^{-5}$
Pitch	EN–ES vs. EN–FR	$-0.0051$	[ $-0.014$ , $+0.004$ ]	$-0.009$	$0.16$
Energy	DE–EN vs. EN–ES	$+0.0092$	[ $+0.004$ , $+0.015$ ]	$+0.031$	$3\times 10^{-6}$
Energy	DE–EN vs. EN–FR	$+0.0074$	[ $+0.002$ , $+0.013$ ]	$+0.023$	$7\times 10^{-4}$
Energy	EN–ES vs. EN–FR	$-0.0019$	[ $-0.008$ , $+0.003$ ]	$-0.008$	$0.18$

Appendix

Table 8: Pairwise differences in mean prosodic correlation between language pairs. Confidence intervals are obtained using episode-clustered bootstrap resampling.

p

-values are Holm-corrected.

Pair	Pitch	Energy	Diff.	95% CI	P $>$ E
DE–EN	0.229	0.183	$+0.046$	[0.040, 0.052]	57%
EN–ES	0.241	0.174	$+0.067$	[0.061, 0.074]	60%
EN–FR	0.246	0.176	$+0.070$	[0.063, 0.077]	60%

Appendix

Table 9: Paired comparison of pitch and energy correlations within the same utterances. Confidence intervals are obtained using episode-clustered bootstrap resampling of the mean paired difference. The P

>

E column reports the percentage of utterances for which pitch correlation exceeds energy correlation. Paired Wilcoxon signed-rank tests give

p<10^{-80}

for all three pairs.

POS group	DE–EN	EN–ES	EN–FR
Nouns	39,953	42,961	39,782
Verbs	55,572	58,861	56,067
Modifiers	25,575	25,774	24,009
Function	100,125	106,768	100,662
Other	4,420	4,696	4,224
Total	225,645	239,060	224,744

Appendix

Table 10: English word counts per POS group in the scored utterances.

Feature	Group	Pair	$n_{\text{total}}$	$n_{\text{group}}$	$r_{\text{full}}$	$r_{\text{without}}$	$\Delta r_{\text{mean}}$	$\Delta r_{\text{std}}$
Pitch	Nouns	DE–EN	16,335	16,170	0.2295	0.2063	+0.0228	0.2099
	Nouns	EN–ES	16,939	16,753	0.2409	0.2206	+0.0208	0.2273
	Nouns	EN–FR	15,957	15,815	0.2460	0.2259	+0.0203	0.2132
	Verbs	DE–EN	16,335	16,236	0.2295	0.2262	+0.0034	0.1983
	Verbs	EN–ES	16,939	16,756	0.2409	0.2261	+0.0156	0.2187
	Verbs	EN–FR	15,957	15,915	0.2460	0.2416	+0.0045	0.1751

Appendix

Table 11: Full leave-one-group-out POS-group ablation statistics across language pairs and prosodic dimensions.

n_{\text{group}}

denotes the number of utterances containing at least one aligned word from the corresponding POS group. Positive

\Delta r

values indicate that removing the group decreases overall utterance-level correlation, suggesting that the group has a positive effect on cross-lingual prosodic correspondence.

Feature	Group	DE–EN 95% CI	EN–ES 95% CI	EN–FR 95% CI
Pitch	Nouns	[+0.0193, +0.0266]	[+0.0180, +0.0253]	[+0.0166, +0.0235]
Pitch	Verbs	[-0.0003, +0.0049]	[+0.0119, +0.0181]	[+0.0023, +0.0074]
Pitch	Modifiers	[+0.0062, +0.0105]	[+0.0034, +0.0080]	[+0.0031, +0.0079]
Pitch	Function	[-0.0098, -0.0018]	[+0.0076, +0.0149]	[+0.0100, +0.0185]
Energy	Nouns	[+0.0331, +0.0370]	[+0.0378, +0.0420]	[+0.0377, +0.0421]
Energy	Verbs	[-0.0004, +0.0031]	[-0.0085, -0.0042]	[-0.0080, -0.0044]

Appendix

Table 12: Episode-clustered bootstrap 95% confidence intervals for the leave-one-group-out POS-group ablation effects. An interval excluding zero indicates a statistically detectable ablation effect.

Feature	Pair	Noun–Function	95% CI	$p$	Result
Pitch	DE–EN	+0.0285	[+0.022, +0.035]	$2\times 10^{-26}$	Significant
Pitch	EN–ES	+0.0104	[+0.005, +0.016]	$2\times 10^{-4}$	Significant
Pitch	EN–FR	+0.0056	[-0.001, +0.012]	$2\times 10^{-3}$	Marginal
Energy	DE–EN	+0.0771	[+0.073, +0.081]	$<10^{-300}$	Significant
Energy	EN–ES	+0.0598	[+0.056, +0.064]	$3\times 10^{-226}$	Significant
Energy	EN–FR	+0.0706	[+0.066, +0.075]	$2\times 10^{-261}$	Significant

Appendix

Table 13: Paired comparison of noun and function-word POS-group ablation effects. Confidence intervals are obtained using episode-clustered bootstrap resampling;

p

-values are from paired Wilcoxon signed-rank tests.

Pair	10	15	20	30	40	Spread
EN–ES	0.2496	0.2498	0.2495	0.2496	0.2498	0.0002
DE–EN	0.2310	0.2310	0.2315	0.2318	0.2318	0.0008
EN–FR	0.2448	0.2444	0.2451	0.2450	0.2450	0.0007

Appendix

Table 14: Robustness of mean per-utterance Spearman correlation to different frames-per-word values (columns) using interpolated F0 contours. Each value is computed on a random subset of 3,000 utterance pairs per language pair.

Variant	10	15	20	30	40
Interpolated	0.2317	0.2316	0.2315	0.2315	0.2318
Raw	0.1675	0.1668	0.1671	0.1670	0.1666

Appendix

Table 15: Comparison of interpolated and raw F0 correlations for EN–ES across frames-per-word values (columns). The maximum spread across settings is 0.0003 for interpolated and 0.0009 for raw contours. Results are evaluated on a subset of 3,000 utterance pairs.

Explore similar work

Evaluating and Preserving Lexical Stress in English-to-Chinese Speech-to-Speech Translation
Jun 13, 2026Yuchen Song, Xi Chen, Mingze Li +1Speech Translation Speech Translation Evaluation
The Chinese University of Hong Kong, Shenzhen, China · Shenzhen Loop Area Institute, China
Do Speech Emphasis Models Generalize across Languages and Emotions?
Jun 26, 2026Megan Wei, Deepali Aneja, Jiaqi Su +3Speech Prosody Cross-Lingual Transfer
Adobe Research, USA · Brown University, USA
Using Prosody to Predict Syntactic Structure
Aug 31, 2026Junghyun Min, Alex Warstadt, Tamar I. Regev +2Speech Prosody
Georgetown University · UC San Diego · MIT +1

Jun 13, 2026J/K move · Enter open · S save

Evaluating and Preserving Lexical Stress in English-to-Chinese Speech-to-Speech Translation

Yuchen Song, Xi Chen, Mingze Li, Satoshi Nakamura

The Chinese University of Hong Kong, Shenzhen, China · Shenzhen Loop Area Institute, China

Speech Translation Speech Translation Evaluation

Open paper AskPDF arXiv

Speech-to-speech translation (S2ST) systems have achieved impressive progress in semantic accuracy and speech naturalness. However, the cross-lingual transfer of lexical stress, a vital cue for emphasis and speaker intent, remains heavily underexplored, compounded by a lack of reliable automatic evaluation metrics for tonal languages like Chinese. We investigate English-to-Chinese S2ST stress transfer by constructing a stress-annotated Chinese dataset and an XLS-R-based Mandarin stress detector. Integrating this with the English EmphAssess system, we propose a novel objective metric for cross-lingual stress evaluation. Furthermore, we fine-tune CosyVoice3 to build a stress-aware S2ST system. Experiments demonstrate that our proposed S2ST architecture significantly outperforms existing systems in stress translation capability while maintaining competitive translation quality. Furthermore, our evaluation metric exhibits a strong correlation with human subjective judgments.

Abstract

Explore similar work

Evaluating and Preserving Lexical Stress in English-to-Chinese Speech-to-Speech Translation

Do Speech Emphasis Models Generalize across Languages and Emotions?

Using Prosody to Predict Syntactic Structure