Deep Speech Representations Track Word Timing along a Low-Dimensional Circular Trajectory

Burin Naowarat; Hao Tang; Sharon Goldwater

cs.CLOct 12, 2026

Deep Speech Representations Track Word Timing along a Low-Dimensional Circular Trajectory

Authors: Burin Naowarat, Hao Tang, Sharon Goldwater

Organizations: The Centre for Speech Technology Research, School of Informatics, University of Edinburgh

Abstract

Speech timing is fundamental for speech models to recognize words accurately, but how word-timing information is represented in speech models remains underexplored. We study this question using the framework of representational geometry, which investigates whether and how concepts are represented in low-dimensional subspaces of the representation space. Prior work in cognitive neuroscience has shown that when HuBERT feature sequences are averaged across words of the same duration, they form near-circular trajectories from word onset to word offset in a two-dimensional "temporal subspace." Here we show that this circular structure of word timing is a general property of deep speech representations, by demonstrating its emergence in nine additional speech models covering both supervised and self-supervised. We then move beyond averaged word segments and show that in HuBERT, the circular trajectories are consistent enough at the level of individual words to perform word segmentation. Our approach, Rotary Segmentation (RoSeg), achieves 64.3% boundary F-score on Librispeech by detecting word boundaries when cycles complete in the temporal subspace. Although supervision is needed to identify a good subspace, nevertheless this result demonstrates much stronger implicit word knowledge in HuBERT than has previously been shown.

Figures & tables

Panel of Figure 1 — Fig. 1 : Circular word trajectories track word timing in HuBERT representations. (a) The two-dimensional projections of averaged word segments onto a temporal subspace reveal circular trajectories. Each trajectory (left) shows the average sequence of HuBERT (layer 9) features in word segments of a particular duration (colours). For for all durations, the average features at each timestep (crosses) follow a nearly complete rotation from word onset (circles) to word offset (pluses). The average features for each word duration rotate with roughly constant angular velocity, resulting in a near-linear relationship between the elapsed % of the word and the rotation angle (right). (b) Frames from individual words (colours) also follow roughly circular trajectories through the temporal subspace (left), so we can segment words by predicting boundaries (triangles) at minima of $a_{t}$ , the angular distance relative to utterance onset (right; true boundaries shown as dashed lines).

Model	Architecture	Training paradigm	Prediction target	PE	Dataset	HuggingFace name
HuBERT [ 18 ]	Non-causal Trans	Masked prediction	Cluster ID	ConvPE	LS 960	facebook/hubert-base-ls960
MelHuBERT [ 31 ]	Non-causal Trans	Masked prediction	Cluster ID	ConvPE	LS 360	-
Wav2Vec2 [ 32 ]	Non-causal Trans	Masked prediction	Online codebook	ConvPE	LS 960	facebook/wav2vec2-base
WavLM [ 33 ]	Non-causal Trans	Masked prediction + denoising	Cluster ID	ConvPE	LS 960	microsoft/wavlm-base
DinoSR [ 34 ]	Non-causal Trans	Masked prediction + distillation	Online cluster ID	ConvPE	LS 960	-
SpidR [ 35 ]	Non-causal Trans	Masked prediction + distillation	Online cluster ID	ConvPE	LS 960	-

TABLE I : Models used in our first study. We abbreviate Trans(formers), Conv(olutional), positional encoding (PE), and LibriSpeech (LS). All models have a frame rate of 50 Hz except Mimi and Vibevoice, whose frame rates vary across layers. For Mimi, we used the 25 Hz continuous features from the Transformer layer. For Vibevoice, we used layer 5, the only one with a similar frame rate to other models (60 Hz).

Panel of Figure 3 — Fig. 2 : Word trajectories in the top two dimensions of the CPCA temporal subspace from different model representations, visualized as in Fig 1 . Rotational dynamics appear in all trained models (first three columns) but only one of the untrained models (last column). For each model, we visualize layer (L) with the best $R^{2}$ scores. Models in the first two columns are non-causal; those in the third column are causal.

Panel of Figure 4 — Fig. 3 : $R^{2}$ and $\mathrm{completion}\%$ scores for all layers, where layer 0 indicates the inputs to the Transformer layers. The big markers indicate the layer with the best scores. We exclude Vibevoice-acoustic and -semantic as we study only layer 5.

Fig. 4 : Average word trajectories in the top two dimensions of the temporal subspace estimated by LEACE from HuBERT L9 features. The visualization is on the whitened feature space. Unlike in CPCA (Fig 1 ), words are not binned by duration when estimating the subspace, only for visualization.

Fig. 5 : Examples for RoSeg with temporal subspaces estimated by CPCA (blue) and LEACE (orange), overlaid on spectrogram. The oscillating lines depict angular distance between the features at each timestep and the first frame of the utterance. RoSeg predicts word boundaries at troughs that fall below an absolute threshold (triangles), and filters out other troughs (pluses). True word boundaries are shown as dashed lines.

dev-clean
Model	Method	F1 ( $\uparrow$ )	OS ( $\rightarrow\!\!0$ )	Tok F1 ( $\uparrow$ )	R-val ( $\uparrow$ )
-	Gold syllables	85.5	33.9	65.9	71.1
SSL HB	Dissim. [ 1 ]	43.9	-3.3	17.0	52.6
SSL HB	RoSeg-CPCA	49.1	-12.9	20.5	57.6
	$-$ abs threshold	48.4	-4.8	19.9	56.6
SSL HB	RoSeg-LEACE	66.0	-4.6	42.3	71.2

TABLE II : Word segmentation performance of RoSeg on Librispeech dev and test sets (in %), using pre-trained (SSL) and untrained (Unt.) HuBERT representations. We compare CPCA vs. LEACE for subspace estimation, and show ablations without absolute thresholding on the dev set. Results for the unsupervised baseline (Dissim.) were obtained using the code at https://github.com/s-malan/prom-wordseg .

Panel of Figure 8 — Fig. 6 : Word segmentation performance for words with different numbers of syllables (#syllable) on the Librispeech dev-clean set, using SSL HuBERT L9 features. RoSeg degrades slower than gold syllables as #syllable increases, especially on OS and R-value, which account for the number of predicted boundaries. Bars show sample counts (right-hand y-axis).

Panel of Figure 9 — Fig. 7 : Trajectories of the two multisyllable words from Fig. 5 through the temporal subspaces estimated from HuBERT L9 by CPCA (left) or LEACE (right). Trajectories are color-coded by syllable identity, with the within-syllable position indicated by a dark-to-light color gradient.

Neural Representation Geometry

Neural representation geometry studies how learned features are arranged and how that structure relates to model behavior and performance.

See all 256 papers

Sep 9, 2026

Do speech foundation models really learn words?

Self-supervised speech foundation models are now used in a wide array of downstream applications, including traditional speech recognition and as the basis for tokens in speech-aware language models. Attempts to understand their usefulness have largely focused on probing their representations' ability to discriminate phonemes and words. However, discriminative ability for words need not imply specialized representation of words per se. Good discrimination of words may be explained by good encoding of word form (phonemes) rather than form-independent word representations encoding identity or syntactic/semantic properties. By partialling out phoneme information using residualization, we show that, in later layers, HuBERT and wav2vec 2.0 do in general learn representations which encode words with reasonable fidelity independently of local phonetic content. We show that this simple approach to disentanglement can enhance higher-order linguistic information in word discovery tasks.

Robin Huo, Ewan Dunbar

University of Toronto, Canada

Representation Learning Speech Foundation Models Representation Disentanglement Self-Supervised Speech Representation Learning

Ask PDF

May 2, 2026

A framework for analyzing concept representations in neural models

Understanding how neural models represent human-interpretable concepts is challenging. Prior work has explored linear concept subspaces from diverse perspectives, such as probing and concept erasure. We introduce a unified framework to study these subspaces along two axes: containment, which tests if a concept is fully represented in a subspace but not outside it, and disentanglement, which tests for isolation from other concepts. In experiments on both text and speech models, we first highlight that concept subspaces may not be uniquely determined, and discuss the implications for concept subspace analysis. Then, we compare properties of concept subspaces estimated using five estimators, proposed in different communities. We find that (1) the choice of estimator impacts the containment and disentanglement properties; (2) the state-of-the-art concept erasure method, LEACE, performs well on both testing axes, but still struggles to generalize to unseen data; and (3) in HuBERT speech representations, phone information is both contained and disentangled from speaker information, while speaker information is hard to contain in a compact subspace, despite being disentangled from phones.

Burin Naowarat, Hao Tang, Sharon Goldwater

The Centre for Speech Technology Research School of Informatics, University of Edinburgh, United Kingdom

Representation Disentanglement Neural Network Interpretability Concept Erasure

Ask PDF

Sep 29, 2026

Does a prosody-trained representation help beyond trainable fusion? A parameter-matched study with frozen HuBERT

Explicit prosodic cues may help automatic speech recognition (ASR) of spontaneous speech, but auxiliary representations typically require additional trainable components, making it unclear whether gains come from the auxiliary information or the fusion mechanism. We address this using a frozen HuBERT backbone and a 64-dimensional representation trained to predict log F0, voicing, Delta log F0, log energy, and spectral tilt. We compare a frozen-backbone recognizer (Baseline), trainable fusion with zero auxiliary input (Null), and the same fusion supplied with the learned representation (Learned). Across Buckeye, Switchboard, and AMI IHM, Null reduces WER by 0.71-1.45 points over Baseline, whereas Learned differs from Null by +0.07, -0.09, and +0.00 points, with no significant differences. However, removing or mismatching the representation at inference increases Learned WER. Thus, Learned depends on the representation yet shows no measurable incremental WER benefit over the parameter-matched control.

Ki Woong Moon, Daniel Brenner

Department of Linguistics, University of Arizona, Tucson, AZ 85721 · Speak, 360 Spear St., 4F, San Francisco, CA 94105

Automatic Speech Recognition

Ask PDF

Abstract

Neural Representation Geometry

Explore similar work

Do speech foundation models really learn words?

A framework for analyzing concept representations in neural models

Does a prosody-trained representation help beyond trainable fusion? A parameter-matched study with frozen HuBERT