Multi-Task Learning for Non-Canonical Phoneme Recognition via Articulatory Feature Decomposition

Sophia Riaz; Haoze Zheng; Amos Roche; Miyu Zhang; Anamika Ragu; Salvatore Penachio; Kaustav Mukherjee; Aneesh Jonelagadda

cs.SDAug 23, 2026

Multi-Task Learning for Non-Canonical Phoneme Recognition via Articulatory Feature Decomposition

Authors: Sophia Riaz, Haoze Zheng, Amos Roche, Miyu Zhang, Anamika Ragu, Salvatore Penachio, Kaustav Mukherjee, Aneesh Jonelagadda

Organizations: Kaliber AI, San Mateo CA

Abstract

Pathological and more broadly non-canonical speech present significant challenges for automatic phoneme recognition due to systematic deviations from canonical pronunciation and limited availability of labeled clinical speech data. Existing phoneme recognition systems are typically trained on canonical speech and treat phonemes as atomic categorical labels, limiting their ability to detect structured articulatory errors common in speech disorders and accents. In this work, we introduce a linguistically structured approach to non-canonical phoneme recognition that decomposes phoneme prediction into articulatory feature dimensions such as manner, place, and voicing. We implement this formulation using a hierarchical multi-task learning architecture in which task-specific articulatory feature heads learn feature-level representations that are subsequently integrated through a cross-attention-based fusion module to produce phoneme predictions. To address the scarcity and noise of pathological speech labels, we combine this framework with semi-supervised learning via Momentum Pseudo-Labeling (MPL) and propose a cascaded training strategy that progressively introduces articulatory feature tasks while employing staged unfreezing of a pretrained speech encoder. Experiments on L2-ARCTIC, used as a proxy for pathological speech variation, show that the proposed approach achieves substantial improvements in phoneme recognition performance compared to strong baseline architectures, while yielding interpretable error patterns aligned with phonological feature structure. These results suggest that articulatory feature supervision is a promising strategy for robust and interpretable phoneme recognition in non-canonical speech, and motivate future validation on clinically diagnosed pathological speech datasets.

Figures & tables

Figure 1: Comparison of the IPA consonant inventory and the derived ARPAbet articulatory inventory used in this work. The ARPAbet inventory was constructed from the L2-ARCTIC ARPAbet–IPA phoneme correspondence and organized according to IPA articulatory feature categories.

Figure 2: Hierarchical MTL architecture. (Stage 1) Input audio is first passed into a frozen WavLM backbone, where acoustic and phoneme representations are learned as two separate weighted sums of transformer layers. (Stage 2) Acoustic features are passed into separate articulatory feature estimation heads for consonant, vowel, and type representations. (Stage 3) These, along with the phoneme WavLM features are passed into a cross-attention layer, whose output is residually fused with the phoneme features. (Stage 4) This is then passed through a bidirectional LSTM to capture temporal dependencies before the final phoneme classification head.

Fine-Tuned Baselines (WavLM-base-plus)
Model	PER $\downarrow$	Prec	Rec	F1
Canonical Baselines
Wav2Vec2Phoneme	22.76	0.311	0.368	0.337
WavLM-base-plus	28.02	0.281	0.477	0.353
Wav2Vec2Phoneme	14.81	0.540	0.479	0.507
WavLM	14.71	0.545	0.508	0.526

Table 1: Model performance on L2-ARCTIC (Phoneme Error Rate,

\%

)

Model	Error Type	Before	With MPL	$\Delta$ (%)
Baseline	Substitutions	3055	3006	-1.60
	Deletions	673	705	4.76
	Insertions	555	495	-10.81
	Cross-class	123	117	-4.88
HMTL	Substitutions	2860	2793	-2.34
	Deletions	457	534	16.85

Table 2: Changes in phoneme error types after applying MPL.

Figure 3: Breakdown of within-domain substitution errors by articulatory feature dimensions for the best baseline model.

Category	Metric	$\Delta$ (%)	$\Delta$ Domain Share (%)
Type	Cross-class	$+41.0$	-
Articulatory	Place	$-3.24$	$+1.01$
	Manner	$-3.86$	$+0.66$
	Voicing	$-10.14$	$-1.67$
	Height	$-5.65$	$-0.06$
	Backness	$-5.83$	$-0.07$

Table 3: Changes in recognition errors and articulatory feature substitutions between the best baseline (WavLM + Aug + MPL) and the best proposed model (HMTL + CA + Aug + MPL).

\Delta

Domain Share is computed within consonant or vowel substitution classes and denotes normalized shift in error distribution.

Panel of Figure 4 — Figure 4: Leave-one-out attention head ablation analysis of phoneme error rate (PER). (a) Aggregate increase in PER when ablating heads associated with each phonological feature, showing the relative contribution of different task heads. (b) Heatmap of PER changes for individual head removals (x-axis) across articulatory feature error types (y-axis), where color encodes the change in error relative to the baseline model.

Configuration	PER (%)
All augmentations	13.49
$-$ Noise	13.76
$-$ VTLP	13.80
$-$ Phase perturbation	13.83
$-$ Prosody (Time-stretch)	13.87
$-$ Pitch shift	13.94

Table 4: Leave-one-out ablation of audio augmentations.

ARPA	IPA	L2-Arctic Substitute	Example	Comments
-		HH	am”ma:m	Arabic pharyngeal
-		SH		Mandarin retroflex fricative
-	k h	K HH	k h a:na:	Hindi aspiration
-	J	Y	kami	Korean nasal
-	B	B	aBa	Spanish bilabial fricative
Q		-	ba	Vietnamese glottal

Table 5: Language-Specific missing or ambiguous mappings between ARPAbet, IPA, and L2-ARCTIC annotations.

Index	ARPA	L2-ARCTIC IPA	Comments
5	AW	aU	diphthong
7	AY	aI	diphthong
9	CH	tS	affricate
13	ER		rhotic allophone of 3
14	EY	eI	diphthong
20	JH	dZ	affricate

Table 6: Diphthongs and IPA symbols with diacritics

ARPA	IPA	L2-Arctic Substitute	Example	Comments
DX		T	bottle /”bA l ” /	approximation
EL	l ”	AH L	bottle /”bA l ” /	closure
EM	m ”	AH M	rhythm /”rID m ” /	closure
EN	n ”	AH N	button /”bVt n ” /	closure
NX	ñ	N	winner /”wIñ@r/	nasalized
WH		W	why /aI/	approximation

Table 7: Phonetic distinctions collapsed by the L2-ARCTIC ARPAbet inventory.

Metric	Best Baseline	Best HMTL	$\Delta\%$
PER ( $\%$ )	14.46	13.47	$-6.8\%$
Substitutions (within-class)	2889	2732	$-5.4\%$
Deletions	705	520	$-26.2\%$
Insertions	495	501	$+1.2\%$
Cross-class subs.	117	165	$+41.0\%$

Table 8: Change in errors (Best baseline to best model).

Model	Substitutions	Deletions	Insertions	Cross-class
Wav2Vec2Phoneme	3143	523	642	122
WavLM-base-plus	3003	776	499	121
WavLM + Aug	3055	673	555	123
WavLM + MPL	3067	657	504	123
WavLM + Aug. + MPL	3006	705	495	117
Parallel MTL	2978	381	745	207

Table 9: Phoneme error types by model.

Model	Cross-class	Place	Manner	Voicing	Height	Backness	Roundedness
Wav2Vec2Phoneme	122	716	603	810	1105	431	286
WavLM-base-plus	121	652	549	769	1094	441	275
WavLM + Aug	123	665	565	794	1087	444	290
WavLM + MPL	123	640	540	830	1074	436	301
WavLM + Aug + MPL	117	679	570	789	1044	429	275
Parallel MTL	207	715	627	786	1067	445	298

Table 10: Phoneme feature errors by model.

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 484 papers

Jun 15, 2026

ArtNet: A JEPA-Like Articulatory Predictive Framework for Robust Zero-Shot Phoneme Recognition

Zero-shot cross-lingual phoneme recognition is often hindered by the fragility of direct acoustic-to-symbol mapping, which is susceptible to language-specific variations. Echoing joint-embedding predictive architecture (JEPA) work in vision, we propose ArtNet, a framework that explores a structured feature prediction task based on articulatory features to enhance acoustic robustness. Specifically, ArtNet integrates an articulatory predictor, designed to extract universal articulatory representations from self-supervised learning (SSL) features, with a variational information bottleneck (VIB) to suppress language-specific variations. Experiments on seven unseen languages demonstrate that ArtNet, particularly when synergized with the proposed vector-space inventory alignment (VSIA) strategy, significantly outperforms competitive baselines, achieving a 20.56% relative reduction in phoneme error rate (PER) and 7.01% in phoneme feature error rate (PFER).

Zeqian Hu, Fuliang Weng, Shu Shang +1

Fudan University, China · Pedawise, Shanghai, China

Zero-Shot Learning Joint-Embedding Predictive Architecture Speech Processing Self-Supervised Speech Representation Learning Variational Information Bottleneck Cross-Lingual Transfer

Ask PDF

Jul 26, 2026

Improving Zero-Shot Phonetic Classification through Language-Agnostic Articulatory Features

Recent Phonetic Foundation Models (PFMs) for Speech-to-IPA transcription rely on Grapheme-to-Phoneme (G2P) labels, but the phoneme labels are not necessarily phonetically faithful. To investigate this issue, we evaluate zero-shot phonetic classification on Chinese aspiration and Japanese moraic nasals. A PFM trained on G2P-labeled data excluding these two languages yields poor accuracy on both tasks, showing that multilingual coverage with discrete IPA tokens is not sufficient for unseen settings. To overcome this limitation, we propose a classification method based on continuous Articulatory Feature (AF) vectors extracted from each frame. This AF-based approach outperforms discrete token-based methods, particularly for rare phones. We further show that it is crucial to adopt the optimal temporal aggregation of AF vectors for the target distinction: single-frame classification is best for aspiration, while segmental classification substantially improves nasal classification.

Ryo Magoshi, Jaeyoung Lee, Shinsuke Sakai +1

Graduate School of Informatics, Kyoto University, Japan · NTT, Inc., Japan

Zero-Shot Learning Speech Foundation Models

Ask PDF

Aug 10, 2026

Structured Phonological Representations for Audio-Articulatory rtMRI Speech Classification

Real-time MRI makes it possible to observe vocal-tract articulation during speech, but mapping these articulatory patterns to phonetic and phonological categories remains challenging. We investigate whether PhonoQ, an audio-based model trained to recognize structured phonological features, provides useful information for audio--articulatory modeling. Specifically, we extract representations from PhonoQ's Conformer module, whose training is shaped by supervision for manner, place, voicing, and vowel features. Using articulatory contours with synchronized audio-derived features, we compare WavLM-large and HuBERT-large baselines with models that incorporate PhonoQ-derived representations. Across unseen-speech and unseen-subject settings, these features improve macro-F1 for phonological targets including manner, place, voicing, vowel height, and vowel backness, and also improve fine-grained 39-phoneme classification. In a contour-only inference setting, audio-derived teacher supervision yields modest but consistent gains over contour-only training, indicating that phonological information from synchronized audio can be partially transferred to articulatory models. Finally, posterior analyses show interpretable surface-sensitive patterns consistent with flapping-like /t/ realizations, /t/-/r/ retraction or affrication, and nasal place assimilation.

Abner Hernandez, Tomás Arias Vergara, Daiqi Liu +2

Pattern Recognition Lab Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) Erlangen, Germany · Department of Electronic Engineering Universidad de Antioquia Medell´ın, Colombia

Speech Processing

Ask PDF

Abstract

Automatic Speech Recognition

Explore similar work

ArtNet: A JEPA-Like Articulatory Predictive Framework for Robust Zero-Shot Phoneme Recognition

Improving Zero-Shot Phonetic Classification through Language-Agnostic Articulatory Features

Structured Phonological Representations for Audio-Articulatory rtMRI Speech Classification