eess.ASSep 29, 2026

Does a prosody-trained representation help beyond trainable fusion? A parameter-matched study with frozen HuBERT

Authors: Ki Woong Moon, Daniel Brenner

Organizations: Department of Linguistics, University of Arizona, Tucson, AZ 85721 · Speak, 360 Spear St., 4F, San Francisco, CA 94105

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Explicit prosodic cues may help automatic speech recognition (ASR) of spontaneous speech, but auxiliary representations typically require additional trainable components, making it unclear whether gains come from the auxiliary information or the fusion mechanism. We address this using a frozen HuBERT backbone and a 64-dimensional representation trained to predict log F0, voicing, Delta log F0, log energy, and spectral tilt. We compare a frozen-backbone recognizer (Baseline), trainable fusion with zero auxiliary input (Null), and the same fusion supplied with the learned representation (Learned). Across Buckeye, Switchboard, and AMI IHM, Null reduces WER by 0.71-1.45 points over Baseline, whereas Learned differs from Null by +0.07, -0.09, and +0.00 points, with no significant differences. However, removing or mismatching the representation at inference increases Learned WER. Thus, Learned depends on the representation yet shows no measurable incremental WER benefit over the parameter-matched control.

Figures & tables

Figure 1: Two-phase system. Phase 1 trains a 64-D representation using five acoustic-prosodic targets. In Phase 2, frozen HuBERT and the frozen Phase 1 encoder produce representations independently. Twelve post-hoc fusion modules modify h1,…,h12h_{1},\ldots,h_{12} ; h0h_{0} bypasses fusion, and modified states are not fed back into HuBERT. Baseline omits fusion, Null uses the same modules with p=0p=0 , and Learned uses the prosody-trained pp .
CorpusBaselineNullLearnedLearned −- NullNull −- Baseline
Buckeye37.27 ±\pm 0.1035.82 ±\pm 0.1335.89 ±\pm 0.11+0.07 [ −- 0.22, +0.34]−- 1.45 [ −- 1.93, −- 0.94]
Switchboard31.95 ±\pm 0.0931.24 ±\pm 0.0431.15 ±\pm 0.07−- 0.09 [ −- 0.21, +0.03]−- 0.71 [ −- 0.85, −- 0.58]
AMI IHM39.11 ±\pm 0.2238.23 ±\pm 0.2238.23 ±\pm 0.20+0.00 [ −- 0.27, +0.28]−- 0.89 [ −- 1.24, −- 0.54]
Table 1: Test WER (%, mean ±\pm SD over three training seeds) with 95% hierarchical bootstrap CIs for absolute WER differences. Negative values favor the first condition; bold indicates the lowest unrounded mean WER.
CorpusTime shuffleUtt. subst.Zero
Buckeye+0.17+0.42+4.85
Switchboard+0.30+1.06+16.93
AMI IHM+0.35+1.01+12.77
Table 2: WER increase (percentage points) after intervening on the auxiliary representation of the trained Learned model. Positive values indicate degradation relative to the unmodified representation.
Figure 2: Layerwise relative residual magnitude ρℓ\rho_{\ell} , averaged over three seeds. Larger values indicate greater modification of frozen HuBERT states.
More in this topic

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 480 papers

Explore similar work

CardsList

  1. Do speech foundation models really learn words?

    Sep 9, 2026Robin Huo, Ewan DunbarRepresentation LearningSpeech Foundation Models

    University of Toronto, Canada

  2. Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representations

    Apr 2, 2026Haitong Sun, Stephen McIntosh, Kwanghee Choi +3Speech ProsodySelf-Supervised Speech Representation Learning

    The University of Tokyo, Japan · University of Texas at Austin, USA

  3. MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery

    Dec 22, 2025Angelo Ortiz Tandazo, Manel Khentout, Youssef Benchekroun +2Speech ProcessingSelf-Supervised Speech Representation Learning

    ENS, PSL Research University, EHESS, CNRS, Paris, France · Meta AI Research, France · Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, Grenoble, France