Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representations

Haitong Sun; Stephen McIntosh; Kwanghee Choi; Eunjung Yeo; Daisuke Saito; Nobuaki Minematsu

cs.CLApr 2, 2026

Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representations

Authors: Haitong Sun, Stephen McIntosh, Kwanghee Choi, Eunjung Yeo, Daisuke Saito, Nobuaki Minematsu

Organizations: The University of Tokyo, Japan · University of Texas at Austin, USA

Abstract

Speech representations from self-supervised speech models (S3Ms) are known to be sensitive to phonemic contrasts, but their sensitivity to prosodic contrasts has not been directly measured. The ABX discrimination task has been used to measure phonemic contrast in S3M representations via minimal pairs. We introduce prosodic ABX, an extension of this framework to evaluate prosodic contrast with only a handful of examples and no explicit labels. Also, we build and release a dataset of English and Japanese minimal pairs and use it along with a Mandarin dataset to evaluate contrast in English stress, Japanese pitch accent, and Mandarin tone. Finally, we show that model and layer rankings are often preserved across several experimental conditions, making it practical for low-resource settings.

Figures & tables

Language	Source	#Pairs	#Spks/Voices	Duration
English	Recording	15	10	4.6
	G-TTS	15	4	1.8
	Kokoro	15	4	1.0
Japanese	Recording	23	10	5.3
	G-TTS	23	4	4.2
Mandarin	MCAE [ 27 ]	2310	6	100.2

Table 1 : Overview of the prosodic minimal pair datasets. Duration (in minutes) includes the minimal pairs only.

	$r_{m}$	Regret m (%)	$\rho_{\textrm{model}}$ [95% CI]
Japanese (G)	0.93	0.04	0.81	$[\phantom{-}0.53,$	$0.93]$
Mandarin (G)	0.93	0.07	0.52	$[\phantom{-}0.10,$	$0.89]$
English (G)	0.50	0.76	0.35	$[-0.15,$	$0.79]$
English (K)	0.85	0.47	0.61	$[\phantom{-}0.25,$	$0.82]$

Table 2 : TTS proxy quality for layer and model selection. Subscript

m

denotes a median across models. Regret is the increase in error rate on natural speech when the best-performing layer is estimated using synthesized speech.

\rho_{\textrm{model}}

is the Spearman rank correlation of best-layer error rates across models. G = G-TTS; K = Kokoro.

Explore similar work

Do Speech Emphasis Models Generalize across Languages and Emotions?
Jun 26, 2026Megan Wei, Deepali Aneja, Jiaqi Su +3Multilingual Automatic Speech Recognition Emotion Recognition
Adobe Research, USA · Brown University, USA
ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge
Jun 23, 2026Jisu Jeon, Seungyeon Jwa, Joosung Lee +6Paralinguistic Cues Large Audio Language Models
Hongik University, South Korea · Seoul National University, South Korea · NAVER Cloud, South Korea +1
PRiSM: Benchmarking Phone Realization in Speech Models
Jan 20, 2026Shikhar Bharadwaj, Chin-Jou Li, Yoonjae Kim +13Grapheme-To-Phoneme Dial
1CMU · 2GIST · 3UT Austin +4

Jun 26, 2026, cs.CLJ/K move · Enter open · S save

Do Speech Emphasis Models Generalize across Languages and Emotions?

Megan Wei, Deepali Aneja, Jiaqi Su, Yunyun Wang+2

Adobe Research, USA · Brown University, USA

Multilingual Automatic Speech Recognition Emotion Recognition Prosody Emotion Expressivity

Open paper AskPDF arXiv

Prosodic emphasis varies across languages, emotions, and speaking styles, yet existing emphasis detection models are largely trained and evaluated on monolingual neutral read speech. We introduce MMEE (Multilingual Multi-Emotion Emphasis), a corpus of 10,000 professionally recorded expressive utterances (14.13 hours) across 7 languages and 34 emotion/style categories, with three-level perceptual labels (10 annotations per sample). We benchmark two state-of-the-art architectures under monolingual, cross-lingual, multilingual, cross-emotion, cross-dataset, and data-scale settings. Monolingual models show limited zero-shot transfer, degrading across typologically distant languages, while multilingual training substantially improves robustness. Models transfer robustly between high- and low-arousal emotions; bidirectional transfer between synthetic and perceptual benchmarks suggests shared prosodic structure; and performance stays robust even at smaller training scales.

Abstract

Explore similar work

Do Speech Emphasis Models Generalize across Languages and Emotions?

ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge

PRiSM: Benchmarking Phone Realization in Speech Models