Beyond Deep Learning: Speech Segmentation and Phone Classification with Neural Assemblies

Trevor Adelson; Vidhyasaharan Sethu; Ting Dang

eess.ASMar 11, 2026

Beyond Deep Learning: Speech Segmentation and Phone Classification with Neural Assemblies

Authors: Trevor Adelson, Vidhyasaharan Sethu, Ting Dang

Organizations: The University of Melbourne, Australia · The University of New South Wales, Australia

Abstract

Deep learning dominates speech processing but relies on massive datasets, global backpropagation-guided weight updates, and produces entangled representations. Assembly Calculus (AC), which models sparse neuronal assemblies via Hebbian plasticity and winner-take-all competition, offers a biologically grounded alternative, yet prior work focused on discrete symbolic inputs. We introduce an AC-based speech processing framework that operates directly on continuous speech by combining three key contributions:(i) neural encoding that converts speech into assembly-compatible spike patterns using probabilistic mel binarisation and population-coded MFCCs; (ii) a multi-area architecture organising assemblies across hierarchical timescales and classes; and (iii) cross-area update schemes for downstream tasks. Applied to two core tasks of boundary detection and segment classification, our framework detects phone (F1=0.69) and word (F1=0.61) boundaries without any weight training, and achieves 47.5% and 45.1% accuracy on phone and command recognition. These results show that AC-based dynamical systems are a viable alternative to deep learning for speech processing.

Figures & tables

Figure 1: Fundamental Project operation of Assembly Calculus. Binary input vectors are projected into a neural area where the $k$ -cap operation selects the top- $k$ most activated neurons to form a sparse assembly. Recurrent plasticity strengthens connections between co-active neurons, causing assemblies to stabilise over repeated presentations of the same input.

Panel of Figure 2 — Figure 2: Inside a per-class RecurrentArea $c$ over a period of $n$ frames. $C$ independent areas (one per class) process the same input in parallel. Each neuron receives feedforward drive from the current input $\mathbf{x}^{(t)}$ plus recurrent drive from the previous assembly $\mathbf{a}_{c}^{(t{-}1)}$ ; the top- $k$ neurons fire ( $k$ -cap). Plasticity strengthens co-active feedforward and recurrent edges, causing each area to learn class-specific spectro-temporal trajectories. The area with learned dynamics that best match the input (highest resonance score $R_{c}$ ) determines the predicted class.

Panel of Figure 3 — Figure 3: Conceptual overview. Speech is processed by two separate AC pipelines. Top: binarised mel frames drive a frozen-weight refractory hierarchy; the change signal $c^{(t)}$ marks boundaries ( $\beta{=}0$ ). Bottom: population-coded MFCCs drive per-class RecurrentAreas; resonance scoring $R_{c}$ identifies the class ( $\beta{>}0$ ). Columns mark the three contributions: (i) neural encoding, (ii) area architecture, (iii) task read-out.

Figure 4: Probabilistic mel binarisation applied to a single phone segment (/eh/, 133 ms). Left: continuous mel spectrogram normalised to $[0,1]$ . Right: binary spike pattern obtained by treating each mel bin value as a Bernoulli firing probability. Brighter spectral regions produce denser activations.

Figure 5: Population-coded MFCC binarisation pipeline. Top row: mel spectrogram of a single phone (/eh/) is transformed into 13 MFCC coefficients, encoded by Gaussian tuning curves into a continuous population code, and thresholded to produce a sparse binary vector. Bottom row: detail of the Gaussian tuning curves for one coefficient (C1), showing how a single MFCC value activates overlapping neurons, and the resulting continuous and binary activations.

Panel of Figure 6 — Figure 6: Architecture overview. (a) Boundary detection: probabilistically binarised mel frames are processed by two cascaded refractory areas; the assembly change signal $c^{(t)}$ marks phoneme and word boundaries without any learned weights. (b) Classification: population-coded MFCC frames are fed in parallel to $C$ independent per-class RecurrentAreas trained with Hebbian plasticity; the resonance score $R_{c}$ (Eq. 7 ) determines the predicted class $\hat{c}=\arg\max_{c}R_{c}$ .

Figure 7: Assembly change signal $c^{(t)}$ (Eq. 5 ) for a single TIMIT utterance. Top: Level 1 (phone boundaries). Bottom: Level 2 (word boundaries). Green vertical lines mark ground-truth boundaries; red dashed lines mark detected peaks. Phone/word labels are shown below each axis.

Figure 8: Trajectory-based resonance scoring. Each of $C$ per-class RecurrentAreas (a) has learned a characteristic trajectory through assembly space (solid, coloured). The same test input (dashed) is presented to all areas in parallel. The inset (b) shows the frame-by-frame operation: feedforward drive from the current input combines with recurrent drive from the previous assembly, and the top- $k$ neurons fire. The predicted class is the label of the area with the learned trajectory that most closely matches the input, as measured by the resonance score $R_{c}$ .

	Precision	Recall	F1
Level 1 (phone)	0.67	0.74	0.69
Level 2 (word)	0.51	0.80	0.61
L1-direct (word baseline)	0.27	0.94	0.42

Table 1: Boundary detection results (frozen weights,

\beta{=}0

) on 20 TIMIT test utterances, averaged over 10 random seeds. Prominence threshold is oracle-selected per utterance (Section 4.5 ).

	Phone	Word
	(TIMIT)	(Speech Cmds)
Classes $C$	39	10
Chance accuracy	2.6%	10.0%
Test samples	4,328	2,000
MFCCs $M$	11	19
Pop. neurons $N_{\mathrm{pop}}$	14	9

Table 2: Classification results with per-class RecurrentAreas trained using ABS plasticity (Eq. 2 ) and scored by trajectory resonance (Eq. 7 ).

Figure 9: Row-normalised confusion matrix for phone classification on TIMIT (39 classes). Confusions follow phonetic similarity: fricatives ( s , sh , th , z ), low vowels ( aa , ah , aw ), and nasals ( m , n , ng ) form visible off-diagonal clusters.

Speech Processing

Speech processing develops computational methods for recognizing, synthesizing, analyzing, and transforming spoken audio.

See all 412 papers

Phone Segmentation and Recognition through Phonological Activation Mapping
Jul 10, 2026Shikhar Bharadwaj, Kwanghee Choi, Stephen McIntosh +8Speech Processing Automatic Speech Recognition
CMU, USA · UT Austin, USA · UTokyo, Japan +1
ASSET: Adaptive Speech Segmentation via Reinforcement Learning
Oct 9, 2026Cihan Xiao, Yuan Tseng, Matthew Wiesner +4Audio Token Compression Speech Processing
Johns Hopkins University, USA · The University of Texas at Austin, USA · HLTCOE, Johns Hopkins University, USA +1
Bio-Inspired Auditory Processing: Coupling Parallel Resonate-and-Fire Dynamics with Ternary Recurrent Networks for Speech Recognition
Oct 21, 2026Pavlos Poulos, Katelyn Sadorf, Themos Stafylakis +1Automatic Speech Recognition Spiking Neural Networks
Department of Informatics, Athens University of Economics and Business, Greece · Archimedes AI, Athena Research Center, Greece · California Institute of Technology, USA +1

Jul 10, 2026J/K move · Enter open · S save

Phone Segmentation and Recognition through Phonological Activation Mapping

Shikhar Bharadwaj, Kwanghee Choi, Stephen McIntosh, Chin-Jou Li+7

CMU, USA · UT Austin, USA · UTokyo, Japan+1

Speech Processing Automatic Speech Recognition

Open paper AskPDF arXiv

Phone segmentation and recognition are inherently related tasks, yet modern approaches typically model them separately. We argue that phonetic structure is already latent in the representations of self-supervised speech models (S3Ms), so one only needs to steer them to solve both tasks. We leverage S3M-based Phonological Activation Mapping (SPAM), which maps each S3M representation frame to a vector of phonological feature activations, such as voicing and nasality. On top of SPAM, we introduce two simple but effective lightweight, gradient-descent-free prediction heads: a recognition head and a segmentation head. Our method requires less than a minute of phonetic transcriptions, and the recognition head generalizes to phones unseen during training. Across a diverse range of datasets, our approach attains strong segmentation performance.

Abstract

Speech Processing

Explore similar work

Phone Segmentation and Recognition through Phonological Activation Mapping

ASSET: Adaptive Speech Segmentation via Reinforcement Learning

Bio-Inspired Auditory Processing: Coupling Parallel Resonate-and-Fire Dynamics with Ternary Recurrent Networks for Speech Recognition