Explicit prosodic cues may help automatic speech recognition (ASR) of spontaneous speech, but auxiliary representations typically require additional trainable components, making it unclear whether gains come from the auxiliary information or the fusion mechanism. We address this using a frozen HuBERT backbone and a 64-dimensional representation trained to predict log F0, voicing, Delta log F0, log energy, and spectral tilt. We compare a frozen-backbone recognizer (Baseline), trainable fusion with zero auxiliary input (Null), and the same fusion supplied with the learned representation (Learned). Across Buckeye, Switchboard, and AMI IHM, Null reduces WER by 0.71-1.45 points over Baseline, whereas Learned differs from Null by +0.07, -0.09, and +0.00 points, with no significant differences. However, removing or mismatching the representation at inference increases Learned WER. Thus, Learned depends on the representation yet shows no measurable incremental WER benefit over the parameter-matched control.
Figures & tables
Figure 1: Two-phase system. Phase 1 trains a 64-D representation using five acoustic-prosodic targets. In Phase 2, frozen HuBERT and the frozen Phase 1 encoder produce representations independently. Twelve post-hoc fusion modules modify h1,…,h12 ; h0 bypasses fusion, and modified states are not fed back into HuBERT. Baseline omits fusion, Null uses the same modules with p=0 , and Learned uses the prosody-trained p .
Corpus
Baseline
Null
Learned
Learned − Null
Null − Baseline
Buckeye
37.27 ± 0.10
35.82 ± 0.13
35.89 ± 0.11
+0.07 [ − 0.22, +0.34]
− 1.45 [ − 1.93, − 0.94]
Switchboard
31.95 ± 0.09
31.24 ± 0.04
31.15 ± 0.07
− 0.09 [ − 0.21, +0.03]
− 0.71 [ − 0.85, − 0.58]
AMI IHM
39.11 ± 0.22
38.23 ± 0.22
38.23 ± 0.20
+0.00 [ − 0.27, +0.28]
− 0.89 [ − 1.24, − 0.54]
Table 1: Test WER (%, mean ± SD over three training seeds) with 95% hierarchical bootstrap CIs for absolute WER differences. Negative values favor the first condition; bold indicates the lowest unrounded mean WER.
Corpus
Time shuffle
Utt. subst.
Zero
Buckeye
+0.17
+0.42
+4.85
Switchboard
+0.30
+1.06
+16.93
AMI IHM
+0.35
+1.01
+12.77
Table 2: WER increase (percentage points) after intervening on the auxiliary representation of the trained Learned model. Positive values indicate degradation relative to the unmodified representation.
Figure 2: Layerwise relative residual magnitude ρℓ , averaged over three seeds. Larger values indicate greater modification of frozen HuBERT states.
Self-supervised speech foundation models are now used in a wide array of downstream applications, including traditional speech recognition and as the basis for tokens in speech-aware language models. Attempts to understand their usefulness have largely focused on probing their representations' ability to discriminate phonemes and words. However, discriminative ability for words need not imply specialized representation of words per se. Good discrimination of words may be explained by good encoding of word form (phonemes) rather than form-independent word representations encoding identity or syntactic/semantic properties. By partialling out phoneme information using residualization, we show that, in later layers, HuBERT and wav2vec 2.0 do in general learn representations which encode words with reasonable fidelity independently of local phonetic content. We show that this simple approach to disentanglement can enhance higher-order linguistic information in word discovery tasks.
Speech representations from self-supervised speech models (S3Ms) are known to be sensitive to phonemic contrasts, but their sensitivity to prosodic contrasts has not been directly measured. The ABX discrimination task has been used to measure phonemic contrast in S3M representations via minimal pairs. We introduce prosodic ABX, an extension of this framework to evaluate prosodic contrast with only a handful of examples and no explicit labels. Also, we build and release a dataset of English and Japanese minimal pairs and use it along with a Mandarin dataset to evaluate contrast in English stress, Japanese pitch accent, and Mandarin tone. Finally, we show that model and layer rankings are often preserved across several experimental conditions, making it practical for low-resource settings.
Haitong Sun, Stephen McIntosh, Kwanghee Choi +3
The University of Tokyo, Japan · University of Texas at Austin, USA
This paper introduces MauBERT, a multilingual extension of HuBERT that leverages articulatory features for robust cross-lingual phonetic representation learning. We continue HuBERT pre-training with supervision based on a phonetic-to-articulatory feature mapping in 55 languages. Our models learn from multilingual data to predict articulatory features or phones, resulting in language-independent representations that capture multilingual phonetic properties. Through comprehensive ABX discriminability testing, we show MauBERT models produce more context-invariant representations than state-of-the-art multilingual self-supervised learning models. Additionally, the models effectively adapt to unseen languages and casual speech with minimal self-supervised fine-tuning (10 hours of speech). This establishes an effective approach for instilling linguistic inductive biases in self-supervised speech models.
ENS, PSL Research University, EHESS, CNRS, Paris, France · Meta AI Research, France · Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, Grenoble, France