Prosody plays an important role in speech translation, conveying information such as emphasis, emotion, and intent beyond lexical content. However, despite recent progress in expressive speech-to-speech translation (S2ST), little is known about how prosodic patterns are similar/different across languages. Understanding these cross-lingual similarities and differences is crucial for effectively incorporating prosody into expressive S2ST systems. In this work, we present the first fine-grained cross-lingual analysis of prosody using multilingual dubbing data across English-German, English-Spanish, and English-French language pairs. We analyze the similarity of pitch, energy, and temporal feature patterns between source and target speech and investigate the linguistic and alignment-related factors affecting this similarity. Our analysis reveals inherent cross-lingual correlations in prosodic structure between certain languages. The findings provide important insights into the transferability of prosody across languages and offer empirical guidance for future expressive speech-to-speech translation systems.
Figures & tables
Figure 1: Overview of the cross-lingual prosody analysis pipeline.
Stage
Metric
Threshold
Audio quality
SI-SDR
≥ 7.5
Audio quality
STOI
≥ 0.91
Audio quality
PESQ
≥ 1.25
Diarization
Speakers
≤ 1
Semantic filtering
SONAR cosine
≥ 0.63
Table 1: Filtering thresholds used for audio quality, speaker diarization, and semantic consistency.
Stage
DE-EN
EN-ES
EN-FR
Raw pairs
100,000
100,000
100,000
After SQUIM
41,610
42,974
42,045
After diarization
28,041
28,990
29,448
After SONAR
20,964
22,598
20,342
After pitch scoring
16,335
16,939
15,957
Table 2: Number of utterance pairs remaining after each filtering stage. The final row excludes utterance pairs for which valid non-constant pitch contours could not be detected in either the source or target speech.
Pair
Utts.
Src. hrs
Tgt. hrs
Total hrs
DE–EN
16,335
18.8
17.9
36.7
EN–ES
16,939
18.8
19.1
37.9
EN–FR
15,957
17.6
18.1
35.7
Table 3: Statistics of the final analysis corpus. Hours are computed over utterances retained for prosodic correlation scoring.
Figure 2: Example of the cross-lingual prosody alignment pipeline. Frame-level pitch contours are extracted from source and target utterances using word timestamps and bilingual word alignments. Word-level prosodic segments are grouped into semantically corresponding alignment clusters and reordered prior to correlation analysis.
Figure 3: Distribution of utterance-level zero-lag Spearman correlations for pitch and energy across aligned source–target utterance pairs.
Feature
Pair
Mean r
95% CI
Shuffled
Pitch
DE–EN
0.229
[0.223, 0.236]
0.007
EN–ES
0.241
[0.235, 0.247]
0.005
EN–FR
0.246
[0.239, 0.253]
0.010
Energy
DE–EN
0.183
[0.179, 0.187]
0.043
EN–ES
0.174
[0.170, 0.178]
0.030
EN–FR
0.176
[0.172, 0.179]
0.026
Table 4: Mean utterance-level Spearman correlation between aligned source and target prosodic contours. Confidence intervals are computed using episode-clustered bootstrap resampling. The shuffled baseline randomizes contours within each utterance. All aligned correlations significantly exceed the shuffled baseline (paired Wilcoxon, p<10−80 ).
Feature
Class
DE–EN
EN–ES
EN–FR
Pitch
Nouns
+0.0228
+0.0208
+0.0203
Verbs
+0.0034
+0.0156
+0.0045
Modifiers
+0.0083
+0.0057
+0.0055
Function
-0.0065
+0.0108
+0.0132
Energy
Nouns
+0.0355
+0.0396
+0.0402
Verbs
-0.0041
-0.0064
-0.0128
Table 5: Leave-one-group-out POS class analysis. Positive values indicate that the POS class has a positive effect on cross-lingual prosodic correspondence.
Figure 4: Leave-one-group-out POS class analysis across language pairs and prosodic dimensions. Bars denote the change in utterance-level Spearman correlation after removing all prosodic frames associated with a POS category. Positive values indicate that the lexical class has a positive effect on cross-lingual prosodic correspondence. Error bars denote 95% confidence intervals on the mean effect, computed as 1.96⋅(σ/n) , where σ is the standard deviation of per-utterance values.
Pair
Feature
Corr.
DE–EN
Phonemes
0.885
EN–ES
Phonemes
0.875
EN–FR
Phonemes
0.871
DE–EN
Duration
0.891
EN–ES
Duration
0.890
EN–FR
Duration
0.884
Table 6: Utterance-level correlations between source and target temporal features (total number of phonemes, duration) for each language pair.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Symm.
AER ↓
P
R
F
GD
37.4%
66.7%
58.8%
0.625
GDFA
37.7%
65.2%
59.6%
0.623
Inter.
39.6%
82.2%
47.6%
0.603
GDF
40.0%
58.4%
61.7%
0.600
Union
40.8%
56.3%
62.6%
0.593
Appendix
Table 7: FastAlign symmetrization heuristics evaluated on the RWTH German–English gold alignment Koehn (2005) ; Vilar et al. (2006) benchmark. GD: grow-diag; GDFA: grow-diag-final-and; GDF: grow-diag-final; Inter.: intersection. P, R, and F denote precision, recall, and F-measure. Bold denotes the highest precision. Intersection symmetrization is used in the prosody pipeline due to its substantially higher precision.
Feature
Comparison
Mean diff.
95% CI
Cliff’s δ
p (Holm)
Pitch
DE–EN vs. EN–ES
−0.0114
[ −0.020 , −0.003 ]
−0.020
3×10−3
Pitch
DE–EN vs. EN–FR
−0.0165
[ −0.026 , −0.007 ]
−0.029
2×10−5
Pitch
EN–ES vs. EN–FR
−0.0051
[ −0.014 , +0.004 ]
−0.009
0.16
Energy
DE–EN vs. EN–ES
+0.0092
[ +0.004 , +0.015 ]
+0.031
3×10−6
Energy
DE–EN vs. EN–FR
+0.0074
[ +0.002 , +0.013 ]
+0.023
7×10−4
Energy
EN–ES vs. EN–FR
−0.0019
[ −0.008 , +0.003 ]
−0.008
0.18
Appendix
Table 8: Pairwise differences in mean prosodic correlation between language pairs. Confidence intervals are obtained using episode-clustered bootstrap resampling. p -values are Holm-corrected.
Pair
Pitch
Energy
Diff.
95% CI
P > E
DE–EN
0.229
0.183
+0.046
[0.040, 0.052]
57%
EN–ES
0.241
0.174
+0.067
[0.061, 0.074]
60%
EN–FR
0.246
0.176
+0.070
[0.063, 0.077]
60%
Appendix
Table 9: Paired comparison of pitch and energy correlations within the same utterances. Confidence intervals are obtained using episode-clustered bootstrap resampling of the mean paired difference. The P > E column reports the percentage of utterances for which pitch correlation exceeds energy correlation. Paired Wilcoxon signed-rank tests give p<10−80 for all three pairs.
POS group
DE–EN
EN–ES
EN–FR
Nouns
39,953
42,961
39,782
Verbs
55,572
58,861
56,067
Modifiers
25,575
25,774
24,009
Function
100,125
106,768
100,662
Other
4,420
4,696
4,224
Total
225,645
239,060
224,744
Appendix
Table 10: English word counts per POS group in the scored utterances.
Feature
Group
Pair
ntotal
ngroup
rfull
rwithout
Δrmean
Δrmedian
Δrstd
Pitch
Nouns
DE–EN
16,335
16,170
0.2295
0.2063
+0.0228
0.0000
0.2099
Nouns
EN–ES
16,939
16,753
0.2409
0.2206
+0.0208
0.0000
0.2273
Nouns
EN–FR
15,957
15,815
0.2460
0.2259
+0.0203
0.0000
0.2132
Verbs
DE–EN
16,335
16,236
0.2295
0.2262
+0.0034
0.0000
0.1983
Verbs
EN–ES
16,939
16,756
0.2409
0.2261
+0.0156
0.0000
0.2187
Verbs
EN–FR
15,957
15,915
0.2460
0.2416
+0.0045
0.0000
0.1751
Appendix
Table 11: Full leave-one-group-out POS-group ablation statistics across language pairs and prosodic dimensions. ngroup denotes the number of utterances containing at least one aligned word from the corresponding POS group. Positive Δr values indicate that removing the group decreases overall utterance-level correlation, suggesting that the group has a positive effect on cross-lingual prosodic correspondence.
Feature
Group
DE–EN 95% CI
EN–ES 95% CI
EN–FR 95% CI
Pitch
Nouns
[+0.0193, +0.0266]
[+0.0180, +0.0253]
[+0.0166, +0.0235]
Pitch
Verbs
[-0.0003, +0.0049]
[+0.0119, +0.0181]
[+0.0023, +0.0074]
Pitch
Modifiers
[+0.0062, +0.0105]
[+0.0034, +0.0080]
[+0.0031, +0.0079]
Pitch
Function
[-0.0098, -0.0018]
[+0.0076, +0.0149]
[+0.0100, +0.0185]
Energy
Nouns
[+0.0331, +0.0370]
[+0.0378, +0.0420]
[+0.0377, +0.0421]
Energy
Verbs
[-0.0004, +0.0031]
[-0.0085, -0.0042]
[-0.0080, -0.0044]
Appendix
Table 12: Episode-clustered bootstrap 95% confidence intervals for the leave-one-group-out POS-group ablation effects. An interval excluding zero indicates a statistically detectable ablation effect.
Feature
Pair
Noun–Function
95% CI
p
Result
Pitch
DE–EN
+0.0285
[+0.022, +0.035]
2×10−26
Significant
Pitch
EN–ES
+0.0104
[+0.005, +0.016]
2×10−4
Significant
Pitch
EN–FR
+0.0056
[-0.001, +0.012]
2×10−3
Marginal
Energy
DE–EN
+0.0771
[+0.073, +0.081]
<10−300
Significant
Energy
EN–ES
+0.0598
[+0.056, +0.064]
3×10−226
Significant
Energy
EN–FR
+0.0706
[+0.066, +0.075]
2×10−261
Significant
Appendix
Table 13: Paired comparison of noun and function-word POS-group ablation effects. Confidence intervals are obtained using episode-clustered bootstrap resampling; p -values are from paired Wilcoxon signed-rank tests.
Pair
10
15
20
30
40
Spread
EN–ES
0.2496
0.2498
0.2495
0.2496
0.2498
0.0002
DE–EN
0.2310
0.2310
0.2315
0.2318
0.2318
0.0008
EN–FR
0.2448
0.2444
0.2451
0.2450
0.2450
0.0007
Appendix
Table 14: Robustness of mean per-utterance Spearman correlation to different frames-per-word values (columns) using interpolated F0 contours. Each value is computed on a random subset of 3,000 utterance pairs per language pair.
Variant
10
15
20
30
40
Interpolated
0.2317
0.2316
0.2315
0.2315
0.2318
Raw
0.1675
0.1668
0.1671
0.1670
0.1666
Appendix
Table 15: Comparison of interpolated and raw F0 correlations for EN–ES across frames-per-word values (columns). The maximum spread across settings is 0.0003 for interpolated and 0.0009 for raw contours. Results are evaluated on a subset of 3,000 utterance pairs.
Speech-to-speech translation (S2ST) systems have achieved impressive progress in semantic accuracy and speech naturalness. However, the cross-lingual transfer of lexical stress, a vital cue for emphasis and speaker intent, remains heavily underexplored, compounded by a lack of reliable automatic evaluation metrics for tonal languages like Chinese. We investigate English-to-Chinese S2ST stress transfer by constructing a stress-annotated Chinese dataset and an XLS-R-based Mandarin stress detector. Integrating this with the English EmphAssess system, we propose a novel objective metric for cross-lingual stress evaluation. Furthermore, we fine-tune CosyVoice3 to build a stress-aware S2ST system. Experiments demonstrate that our proposed S2ST architecture significantly outperforms existing systems in stress translation capability while maintaining competitive translation quality. Furthermore, our evaluation metric exhibits a strong correlation with human subjective judgments.
Yuchen Song, Xi Chen, Mingze Li +1
The Chinese University of Hong Kong, Shenzhen, China · Shenzhen Loop Area Institute, China
Prosodic emphasis varies across languages, emotions, and speaking styles, yet existing emphasis detection models are largely trained and evaluated on monolingual neutral read speech. We introduce MMEE (Multilingual Multi-Emotion Emphasis), a corpus of 10,000 professionally recorded expressive utterances (14.13 hours) across 7 languages and 34 emotion/style categories, with three-level perceptual labels (10 annotations per sample). We benchmark two state-of-the-art architectures under monolingual, cross-lingual, multilingual, cross-emotion, cross-dataset, and data-scale settings. Monolingual models show limited zero-shot transfer, degrading across typologically distant languages, while multilingual training substantially improves robustness. Models transfer robustly between high- and low-arousal emotions; bidirectional transfer between synthetic and perceptual benchmarks suggests shared prosodic structure; and performance stays robust even at smaller training scales.
While it is well-established that prosody carries crucial cues for syntactic structure, the degree and nature of correspondence between these two domains remains contested. We investigate the syntax-prosody interface through an information-theoretic lens, quantifying the interaction between prosodic features and syntactic representations as their mutual information. We provide a general-purpose framework for estimating this quantity over large speech-text corpora using multimodal language models. Our framework is structure-agnostic and modular, insofar as it can be used to measure the contributions of individual prosodic features or components of structure. We evaluate the syntax-prosody relationship for two features (word duration and inter-word pauses) across two domains--read audiobooks and spontaneous conversations--both in English. Our results demonstrate that prosody contains measurable syntactic information, with prosodic features reducing syntactic uncertainty in spontaneous conversations by up to 10.2%. Our findings offer new empirical support for several theoretical accounts of the syntax-prosody interface.