Phoneme- and Word-Level Metrics Using Self-Supervised Speech Representations for Forced Alignment Evaluation

V. S. D. S. Mahesh Akavarapu; Michael Daniel; Gerhard Jäger

cs.CLAug 28, 2026

Phoneme- and Word-Level Metrics Using Self-Supervised Speech Representations for Forced Alignment Evaluation

Authors: V. S. D. S. Mahesh Akavarapu, Michael Daniel, Gerhard Jäger

Organizations: University of Tübingen · University of Jena

Abstract

Forced alignment evaluation typically requires manually annotated timestamps, limiting large-scale and multilingual analysis. We introduce two corpus-level metrics based on self-supervised (SSL) speech representations for reference-free forced alignment evaluation: Phoneme-Cluster Mutual Information (PCMI) and Word Acoustic Consistency Score (WACS). PCMI measures agreement between aligned phoneme labels and clusters induced from SSL-speech representations, while WACS measures consistency of repeated word realizations using dynamic time warping similarity between word representation sequences. Using both random and systematic perturbations, we show that PCMI and WACS degrade consistently under alignment perturbations. We further analyze the metrics across multiple alignment systems on 85 languages from FLEURS, validate them against manually annotated alignments from 45 languages in DoReCo, and evaluate them on two phonologically complex low-resource languages. The metrics effectively separate high- and low-quality alignments and correlate strongly with timestamp-based alignment quality measures. Our results demonstrate that SSL-speech representations enable scalable, reference-free forced alignment evaluation. The metrics are available as an open-source Python package at https://github.com/mahesh-ak/forced-aligner-metrics.

Figures & tables

Emb.	Alignment model	#Langs	PCMI $\uparrow$	WACS $\uparrow$
MMS	Baseline	85	0.09 $\pm$ 0.02	0.02 $\pm$ 0.01
	MFA	82	0.25 $\pm$ 0.11	0.14 $\pm$ 0.08
	MMS-300m-IPA*	85	0.24 $\pm$ 0.03	0.19 $\pm$ 0.02
	Wav2Vec2-IPA*	85	0.24 $\pm$ 0.03	0.19 $\pm$ 0.03
	Qwen3-FA	10	-	0.21 $\pm$ 0.02
XLSR	Baseline	85	0.09 $\pm$ 0.02	0.02 $\pm$ 0.01

Table 1: Multilingual evaluation of forced alignment using PCMI and WACS across 85 languages from FLEURS dataset. Emb. denotes representation model used. *Trained in this work.

Panel of Figure 1 — Figure 1: Behavior of PCMI and WACS under random and systematic perturbations of Buckeye alignments using MMS-300M and MFCC representations. Curves show mean scores over 5 random perturbation samples, with shaded regions denoting standard deviation.

Panel of Figure 2 — Figure 2: Layer-wise behavior of PCMI and WACS under increasing perturbation severity using MMS (left) and XLSR (right) representations. Middle transformer layers provide the strongest separation between clean and perturbed alignments, while later layers exhibit reduced sensitivity.

FLEURS		DoReCo
Family	#	Family	#	Family	#
Indo-European	41	Austronesian	7	Mayan	1
Niger-Congo	9	Sino-Tibetan	4	Mixe-Zoque	1
Afroasiatic	7	Indo-European	3	Nilo-Saharan	1
Austronesian	5	Niger-Congo	3	Pama-Nyungan	1
Turkic	5	Afroasiatic	2	Pano-Tacanan	1

Table 2: Language family distributions in the multilingual evaluation datasets — FLEURS (85 languages) and DoReCo (45 languages).

Figure 3: Distribution of PCMI (left) and WACS (right) across multilingual alignments on FLEURS. The top row uses MMS representations while the bottom row uses XLSR representations. MFA exhibits a clear bimodal structure corresponding to successful and failed alignments, while CTC-based aligners produce substantially more stable distributions across languages. * Trained in this work.

Embedding	Metric Pair	Pearson’s r	p-value
MMS	PCMI vs AAS	-0.7769	$<0.001^{***}$
MMS	WACS vs AAS	-0.6257	$<0.001^{***}$
XLSR	PCMI vs AAS	-0.7761	$<0.001^{***}$
XLSR	WACS vs AAS	-0.6686	$<0.001^{***}$

Table 3: Pearson correlation between reference-free metrics (PCMI, WACS) and alignment quality measured by AAS on the 45-language DoReCo evaluation set. Lower AAS indicates better alignment quality.

Aligner	Embedding	PCMI	WACS
MMS-300M-IPA*	MMS	-0.161 / -0.137	-0.294 / -0.308
MMS-300M-IPA*	XLSR	-0.092 / -0.050	-0.355 / -0.363
Wav2Vec2-IPA*	MMS	-0.008 / -0.073	-0.027 / -0.111
Wav2Vec2-IPA*	XLSR	-0.001 / -0.108	-0.152 / -0.212
MMS-300M-DORECO*	MMS	-0.200 / -0.225	-0.259 / -0.237
MMS-300M-DORECO*	XLSR	-0.225 / -0.214	-0.261 / -0.230

Table 4: Correlation between ASR error rates (WER, CER) and alignment metrics (PCMI, WACS). Values are Pearson/Spearman correlations. * trained in this work

Language	Aligner	AAS $\downarrow$	MMS		XLSR
Language	Aligner	AAS $\downarrow$	PCMI $\uparrow$	WACS $\uparrow$	PCMI $\uparrow$	WACS $\uparrow$
Archi	Gold	0.0	–	0.225	–	0.220
	CTC-IPA-Archi	88.5	0.306	0.203	0.307	0.194
	CTC-IPA-Archi - SIL	46.3	0.319	0.215	0.330	0.215
	Baseline	85.8	0.204	0.112	0.203	0.102
Rutul	Gold	0.0	–	0.182	–	0.182

Table 5: Evaluation against manually annotated alignments for Archi and Rutul. Lower AAS is better, while higher PCMI and WACS are better.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Behavior of PCMI under varying numbers of K-means clusters and increasing perturbation severity using MMS (left) and XLSR (right) representations. The metric resolution, measured as the gap between clean and severely perturbed alignments, remains stable over cluster counts $25$ – $100$ .

Figure 5: Distribution of PCMI (top row), WACS (middle) and AAS (bottom) across multilingual alignments on DoReCo corpus. The left column uses MMS representations while the right uses XLSR representations. * Trained in this work.

Language	ISO-1	ISO-3	Family	Subfamily	Division	Train	Dev	Test	G2P
Afrikaans	af	afr	Indo-European	Germanic	West	1032	198	264	epitran
Amharic	am	amh	Afroasiatic	Semitic	South	3163	223	516	epitran
Arabic	ar	ara	Afroasiatic	Semitic	Central	2104	295	428	epitran
Asturian	-	ast	Indo-European	Romance	West	2511	398	946	xpf
Azerbaijani	az	aze	Turkic	Oghuz	-	2665	400	923	epitran
Belarusian	be	bel	Indo-European	Slavic	East	2433	408	967	xpf

Appendix

Table 6: Languages from FLEURS ( Conneau et al., 2023 ) used in multilingual evaluation.

Language	ISO-3	Family	Subfamily	Division	Train	Dev	Test	Source
Anal	anm	Sino-Tibetan	Tibeto-Burman	Kuki-Chin	1588	227	453	Ozerov (2024)
Yali (Apahapsili)	na	Trans-New Guinea	Dani	-	1057	151	302	Riesberg (2024)
Arapaho	arp	Algic	Algonquian	-	782	112	223	Cowell (2024)
Baïnounk Gubëeher	bab	Niger-Congo	West-Atlantic	-	1315	188	376	Cobbinah (2024)
Beja	bej	Afroasiatic	Cushitic	North	3172	453	906	Vanhove (2024)
Bora	boa	Boran	-	-	733	105	209	Seifart (2024a)

Appendix

Table 7: DoReCo ( Seifart et al., 2024 ) languages used for multilingual validation.

Language	N	WER $\downarrow$		CER $\downarrow$
Language		MMS-IPA*	W2V2-IPA*	MMS-IPA*	W2V2-IPA*
Afrikaans	264	0.576	0.852	0.190	0.323
Amharic	516	0.638	0.866	0.124	0.208
Arabic	428	0.526	0.927	0.141	0.308
Asturian	946	0.370	0.694	0.086	0.168
Azerbaijani	923	0.509	0.839	0.113	0.223

Appendix

Table 8: Language-wise multilingual ASR performance on FLEURS.

N

denotes sizes of test splits. MMS-IPA* and W2V2-IPA* are same as MMS-300M-IPA* and Wav2Vec2-IPA* respectively. * Trained in this work.

Language	N	WER $\downarrow$	CER $\downarrow$
Anal	453	0.979	0.454
Arapaho	223	1.007	0.630
Asimjeeg Datooga	325	0.989	0.356
Baïnounk Gubëeher	376	0.957	0.366
Beja	906	0.975	0.322
Bora	209	0.998	0.399

Appendix

Table 9: Language-wise ASR performance on DoReCo. Scores are reported for MMS-300M-DORECO*.

N

denotes sizes of test splits

Language	MMS				XLSR
Language	Base	MFA	MMS-IPA*	W2V2-IPA*	Base	MFA	MMS-IPA*	W2V2-IPA*
Afrikaans	0.093	0.112	0.221	0.226	0.101	0.105	0.218	0.215
Amharic	0.103	0.122	0.258	0.251	0.108	0.120	0.249	0.234
Arabic	0.114	0.316	0.254	0.254	0.107	0.281	0.244	0.232
Asturian	0.091	0.117	0.223	0.219	0.096	0.106	0.199	0.203
Azerbaijani	0.071	0.124	0.228	0.243	0.076	0.126	0.227	0.220

Appendix

Table 10: Language-wise multilingual PCMI scores on FLEURS. Higher values are better. MMS-IPA* and W2V2-IPA* are same as MMS-300M-IPA* and Wav2Vec2-IPA* respectively. * Trained in this work.

Language	MMS					XLSR
Language	Base	MFA	MMS-IPA*	W2V2-IPA*	Qwen3-FA	Base	MFA	MMS-IPA*	W2V2-IPA*	Qwen3-FA
Afrikaans	0.008	0.041	0.181	0.173	-	0.008	0.025	0.161	0.159	-
Amharic	0.056	0.044	0.219	0.225	-	0.053	0.044	0.198	0.198	-
Arabic	0.041	0.208	0.207	0.206	-	0.034	0.185	0.177	0.178	-
Asturian	0.032	0.052	0.186	0.182	-	0.023	0.053	0.172	0.172	-
Azerbaijani	0.026	0.031	0.220	0.220	-	0.032	0.033	0.195	0.192	-

Appendix

Table 11: Language-wise multilingual WACS scores on FLEURS. Higher values are better. MMS-IPA* and W2V2-IPA* are same as MMS-300M-IPA* and Wav2Vec2-IPA* respectively. * Trained in this work.

Language	MMS				XLSR
Language	Base	MMS-DORECO*	MMS-DORECO*-SIL	Gold	Base	MMS-DORECO*	MMS-DORECO*-SIL	Gold
Anal	0.133	0.209	0.206	0.330	0.122	0.197	0.195	0.316
Yali (Apahapsili)	0.127	0.196	0.193	0.263	0.125	0.184	0.184	0.248
Arapaho	0.075	0.192	0.210	0.287	0.072	0.193	0.204	0.292
Baïnounk Gubëeher	0.097	0.185	0.194	0.301	0.098	0.182	0.193	0.308
Beja	0.228	0.284	0.279	0.386	0.217	0.270	0.278	0.363

Appendix

Table 12: Language-wise multilingual PCMI scores on DoReCo. Higher values are better. * Trained in this work.

Language	MMS				XLSR
Language	Base	MMS-DORECO*	MMS-DORECO*-SIL	Gold	Base	MMS-DORECO*	MMS-DORECO*-SIL	Gold
Anal	0.042	0.091	0.097	0.104	0.046	0.096	0.095	0.105
Yali (Apahapsili)	0.049	0.084	0.085	0.083	0.053	0.095	0.101	0.111
Arapaho	0.071	0.100	0.116	0.136	0.072	0.115	0.123	0.151
Baïnounk Gubëeher	0.037	0.092	0.102	0.114	0.038	0.093	0.097	0.114
Beja	0.084	0.104	0.092	0.111	0.083	0.118	0.119	0.127

Appendix

Table 13: Language-wise multilingual WACS scores on DoReCo. Higher values are better. * Trained in this work.

Language	Base	MMS-DORECO*	MMS-DORECO*-SIL
Anal	143.4	81.8	79.4
Yali (Apahapsili)	91.6	63.6	62.8
Arapaho	529.8	298.2	250.1
Baïnounk Gubëeher	172.3	101.9	92.4
Beja	57.7	52.8	53.1
Bora	426.1	178.0	171.2

Appendix

Table 14: Language-wise multilingual AAS scores on DoReCo. Lower values are better. * Trained in this work.

Aligner	Metric	MMS				XLSR
Aligner		Mean	Std	Min	Max	Mean	Std	Min	Max
Baseline (FLEURS)	PCMI	0.091	0.017	0.048	0.148	0.093	0.016	0.053	0.148
	Mean cluster purity	0.135	0.022	0.083	0.183	0.138	0.026	0.081	0.194
	Std cluster purity	0.140	0.041	0.036	0.226	0.146	0.049	0.047	0.270
	Cluster entropy	3.834	0.024	3.750	3.875	3.853	0.014	3.811	3.887
	Norm. cluster entropy	0.980	0.006	0.959	0.991	0.985	0.004	0.974	0.994

Appendix

Table 15: Multilingual statistics across aligners and embedding models. * Trained in this work.

Reference-Free Evaluation

Reference-free evaluation measures AI outputs or models without relying on gold answers, using source-grounded checks, learned judges, or intrinsic signals.

See all 25 papers

Multilingual Word-Level Forced Alignment with Self-Supervised Representations and Learned Dynamic Programming
Jun 9, 2026Roy Weber, Meidan Zehavi, Rotem Rousso +1Dynamic Programming
Faculty of Electrical and Computer Engineering, Technion – Israel Institute of Technology, Haifa, 3200003, Israel
FuseAlign: Forced Alignment in the Wild
Sep 27, 2026Mithilesh Vaidya, Stephen Bailey, Sumukh Badam +2Forced Alignment Audio-Text Alignment
Descript, Inc.
Phonetic forced alignment for low-resource language varieties: Model training and evaluation on Chengdu Mandarin
Jul 23, 2026Zhiheng Qian, Aini Li, Hai Hu +1Low-Resource Language Processing
Shanghai Jiao Tong University · City University of Hong Kong · The Hong Kong Polytechnic University +1

Jun 9, 2026J/K move · Enter open · S save

Multilingual Word-Level Forced Alignment with Self-Supervised Representations and Learned Dynamic Programming

Roy Weber, Meidan Zehavi, Rotem Rousso, Joseph Keshet

Faculty of Electrical and Computer Engineering, Technion – Israel Institute of Technology, Haifa, 3200003, Israel

Dynamic Programming

Open paper AskPDF arXiv

We present a method for accurate multilingual word-level forced alignment, consisting of an alignment encoder and a learned alignment decoder. The encoder integrates two representations: one from the Massively Multilingual Speech (MMS) model and another from a self-supervised phoneme boundary detector (UnSupSeg). It learns to fuse them and to estimate word-boundary probabilities over long temporal contexts. The alignment decoder is a learned dynamic programming that combines encoder outputs with segmental features over the MMS and UnSupSeg representations to infer final word boundaries. Trained iteratively on TIMIT and Buckeye, the proposed approach outperforms Montreal Forced Aligner (MFA) and MMS-based alignment on both datasets. On unseen languages (Dutch, German, and Hebrew), the proposed model achieves performance consistently better than or on par with existing alignment approaches, indicating its potential to scale to 1100+ languages supported by MMS without further training.

Abstract

Reference-Free Evaluation

Explore similar work

Multilingual Word-Level Forced Alignment with Self-Supervised Representations and Learned Dynamic Programming

FuseAlign: Forced Alignment in the Wild

Phonetic forced alignment for low-resource language varieties: Model training and evaluation on Chengdu Mandarin