Deep learning dominates speech processing but relies on massive datasets, global backpropagation-guided weight updates, and produces entangled representations. Assembly Calculus (AC), which models sparse neuronal assemblies via Hebbian plasticity and winner-take-all competition, offers a biologically grounded alternative, yet prior work focused on discrete symbolic inputs. We introduce an AC-based speech processing framework that operates directly on continuous speech by combining three key contributions:(i) neural encoding that converts speech into assembly-compatible spike patterns using probabilistic mel binarisation and population-coded MFCCs; (ii) a multi-area architecture organising assemblies across hierarchical timescales and classes; and (iii) cross-area update schemes for downstream tasks. Applied to two core tasks of boundary detection and segment classification, our framework detects phone (F1=0.69) and word (F1=0.61) boundaries without any weight training, and achieves 47.5% and 45.1% accuracy on phone and command recognition. These results show that AC-based dynamical systems are a viable alternative to deep learning for speech processing.
Figures & tables
Figure 1: Fundamental Project operation of Assembly Calculus. Binary input vectors are projected into a neural area where the k -cap operation selects the top- k most activated neurons to form a sparse assembly. Recurrent plasticity strengthens connections between co-active neurons, causing assemblies to stabilise over repeated presentations of the same input.
Figure 2: Inside a per-class RecurrentArea c over a period of n frames. C independent areas (one per class) process the same input in parallel. Each neuron receives feedforward drive from the current input x(t) plus recurrent drive from the previous assembly ac(t−1) ; the top- k neurons fire ( k -cap). Plasticity strengthens co-active feedforward and recurrent edges, causing each area to learn class-specific spectro-temporal trajectories. The area with learned dynamics that best match the input (highest resonance score Rc ) determines the predicted class.
Figure 3: Conceptual overview. Speech is processed by two separate AC pipelines. Top: binarised mel frames drive a frozen-weight refractory hierarchy; the change signal c(t) marks boundaries ( β=0 ). Bottom: population-coded MFCCs drive per-class RecurrentAreas; resonance scoring Rc identifies the class ( β>0 ). Columns mark the three contributions: (i) neural encoding, (ii) area architecture, (iii) task read-out.
Figure 4: Probabilistic mel binarisation applied to a single phone segment (/eh/, 133 ms). Left: continuous mel spectrogram normalised to [0,1] . Right: binary spike pattern obtained by treating each mel bin value as a Bernoulli firing probability. Brighter spectral regions produce denser activations.
Figure 5: Population-coded MFCC binarisation pipeline. Top row: mel spectrogram of a single phone (/eh/) is transformed into 13 MFCC coefficients, encoded by Gaussian tuning curves into a continuous population code, and thresholded to produce a sparse binary vector. Bottom row: detail of the Gaussian tuning curves for one coefficient (C1), showing how a single MFCC value activates overlapping neurons, and the resulting continuous and binary activations.
Figure 6: Architecture overview. (a) Boundary detection: probabilistically binarised mel frames are processed by two cascaded refractory areas; the assembly change signal c(t) marks phoneme and word boundaries without any learned weights. (b) Classification: population-coded MFCC frames are fed in parallel to C independent per-class RecurrentAreas trained with Hebbian plasticity; the resonance score Rc (Eq. 7 ) determines the predicted class c^=argmaxcRc .
Figure 7: Assembly change signal c(t) (Eq. 5 ) for a single TIMIT utterance. Top: Level 1 (phone boundaries). Bottom: Level 2 (word boundaries). Green vertical lines mark ground-truth boundaries; red dashed lines mark detected peaks. Phone/word labels are shown below each axis.
Figure 8: Trajectory-based resonance scoring. Each of C per-class RecurrentAreas (a) has learned a characteristic trajectory through assembly space (solid, coloured). The same test input (dashed) is presented to all areas in parallel. The inset (b) shows the frame-by-frame operation: feedforward drive from the current input combines with recurrent drive from the previous assembly, and the top- k neurons fire. The predicted class is the label of the area with the learned trajectory that most closely matches the input, as measured by the resonance score Rc .
Precision
Recall
F1
Level 1 (phone)
0.67
0.74
0.69
Level 2 (word)
0.51
0.80
0.61
L1-direct (word baseline)
0.27
0.94
0.42
Table 1: Boundary detection results (frozen weights, β=0 ) on 20 TIMIT test utterances, averaged over 10 random seeds. Prominence threshold is oracle-selected per utterance (Section 4.5 ).
Phone
Word
(TIMIT)
(Speech Cmds)
Classes C
39
10
Chance accuracy
2.6%
10.0%
Test samples
4,328
2,000
MFCCs M
11
19
Pop. neurons Npop
14
9
Table 2: Classification results with per-class RecurrentAreas trained using ABS plasticity (Eq. 2 ) and scored by trajectory resonance (Eq. 7 ).
Figure 9: Row-normalised confusion matrix for phone classification on TIMIT (39 classes). Confusions follow phonetic similarity: fricatives ( s , sh , th , z ), low vowels ( aa , ah , aw ), and nasals ( m , n , ng ) form visible off-diagonal clusters.
Phone segmentation and recognition are inherently related tasks, yet modern approaches typically model them separately. We argue that phonetic structure is already latent in the representations of self-supervised speech models (S3Ms), so one only needs to steer them to solve both tasks. We leverage S3M-based Phonological Activation Mapping (SPAM), which maps each S3M representation frame to a vector of phonological feature activations, such as voicing and nasality. On top of SPAM, we introduce two simple but effective lightweight, gradient-descent-free prediction heads: a recognition head and a segmentation head. Our method requires less than a minute of phonetic transcriptions, and the recognition head generalizes to phones unseen during training. Across a diverse range of datasets, our approach attains strong segmentation performance.
Shikhar Bharadwaj, Kwanghee Choi, Stephen McIntosh +8
Speech large language models (LLMs) are often built using pretrained acoustic encoders as an audio interface to an LLM decoder. However, LLMs scale poorly with input sequence length, while acoustic encoders tend to produce substantially longer sequences than typical textual inputs. This motivates learning to compress encoder outputs while preserving task-oriented information. To this end, we introduce ASSET, a reinforcement learning (RL)-based framework that learns to adaptively compress encoder outputs by first partitioning them into variable-duration segments and then pooling each segment into a continuous audio token. We use group-relative policy optimization (GRPO) to train a segmenter with task performance feedback and a token-rate penalty. In automatic speech recognition (ASR) experiments, ASSET achieves performance comparable to a transcript-assisted character-boundary oracle with approximately 25% fewer audio tokens on English LibriSpeech, while improving on a phone-boundary oracle at a similar token rate on Mandarin AISHELL. Exploratory experiments on non-ASR tasks further illustrate how learned segmentation changes when the framework is adapted to tasks beyond transcription.
Cihan Xiao, Yuan Tseng, Matthew Wiesner +4
Johns Hopkins University, USA · The University of Texas at Austin, USA · HLTCOE, Johns Hopkins University, USA +1
Spiking Neural Networks (SNNs) promise energy-efficient inference through sparse, event-driven computation, yet remain constrained on continuous, context-dependent tasks such as automatic speech recognition (ASR). A key bottleneck is the Leaky Integrate-and-Fire (LIF) neuron, whose membrane decay erases information across pauses and phonetic boundaries while it must both extract features and retain long-range context. Inspired by cortical separation of nonlinear filtering and somatic integration, we introduce the Spiking Ternary GRU (stGRU): a ternary-weight gated recurrent unit performs temporal filtering, while downstream Resonate-and-Fire (RF) neurons generate the binary spike stream driving the next block's event pathway. This makes binary event transmission a native training constraint rather than a separate deployment-time activation quantization step; together with ternary weights, learned projections avoid dense floating-point matrix MACs. Trained end-to-end on LibriSpeech-960, Small, Medium, and Large stGRUs reach 8.09%, 7.04%, and 6.44±0.17% WER on test-clean without an external language model. To our knowledge, these are the first matrix-MAC-free spiking networks without a learned ANN sequence decoder to achieve single-digit WER on LibriSpeech. The architecture also achieves 10.88% test CER on Aishell-1 (9.28% on dev) when trained from scratch. Internally, trained networks develop a depth-graded hierarchy: shallow layers act as fast acoustic filters, while deep layers form slower semantic integrators whose latent representations segregate into part-of-speech manifolds and word-level spike events. Deep-layer firing rates follow lognormal distributions with a sparsity gradient, mirroring hierarchical processing observed in human cortex in vivo. Our biologically grounded architecture combines competitive ASR with favorable modeled compute and memory cost.
Department of Informatics, Athens University of Economics and Business, Greece · Archimedes AI, Athena Research Center, Greece · California Institute of Technology, USA +1