Multirate State Space Models for End-to-End Processing of Pulse Density Modulated Speech Signals

Ludovic Boulanger; Sean U. N. Wood

cs.SDAug 28, 2026

Multirate State Space Models for End-to-End Processing of Pulse Density Modulated Speech Signals

Authors: Ludovic Boulanger, Sean U. N. Wood

Organizations: Department of Electrical and Computer Engineering at the University of Sherbrooke, Qu´ebec, Canada

Abstract

Deep neural networks (DNNs) based on state-space models (SSMs) are increasingly applied to speech processing, but typically operate on pulse-code-modulated (PCM) audio. This constrains deployment on low-power, always-on edge devices, which commonly use single-bit pulse-density-modulated (PDM) micro-electromechanical (MEMS) microphones for their noise robustness, low cost, and variable sampling rates that enable low-power operation. In fact, converting PDM to PCM requires low-pass filtering and decimation, imposing costly overhead on resource-constrained hardware. While prior works have attempted to process PDM signals directly, they require long training times and generalize poorly across sampling rates. In this paper, we show that the SSM has two key properties that remediate these issues: its continuous-time parametrization allows it to produce a consistent representation of the input audio signal, regardless of the modulation strategy and sampling rate, and its long-term memory enables this representation to be aggressively downsampled without needing any anti-aliasing operations. We then propose a novel end-to-end PDM speech processing architecture that uses an SSM to encode the input audio signal into a modulation- and sampling-rate-invariant latent representation. We show that our proposed architecture achieves robust speech classification and enhancement gains at low-power sampling-rates (512 kHz) and similar performance to state-of-the-art algorithms operating on PCM data when tested on standard PDM sampling-rates of 2 MHz. Moreover, we show that the SSM's output can be downsampled by more than 65,000 times, thus significantly reducing the number of processing timesteps in downstream layers.

Figures & tables

Fig. 1: Comparison of a PCM encoded signal (top) and the same PDM encoded signal (middle and bottom). The orange overlay on the PDM data represents the low-pass-filtered PDM and demonstrates that the PCM baseband is preserved in the low-frequency range of the PDM. The low-pass was created using a 128-tap FIR filter with a cutoff frequency of $7.5\text{\,}\mathrm{kHz}$ . The bottom plot shows a $200\text{\,}\mathrm{\SIUnitSymbolMicro s}$ zoom of the PDM encoded signal.

Fig. 2: Comparison of the PDM and PCM power spectral densities (PSD) for a 1-second speech segment. The black dotted line represents the PCM Nyquist frequency of $8\text{\,}\mathrm{kHz}$ , i.e., the maximum frequency present in the signal. The pink line represents the PCM PSD.

Fig. 3: Demonstration of the basis functions and the memory windows of the FouT, LegT, and LegS SSMs. The SSMs all have a state dimension $H$ of 1024 and a discretization step $\Delta$ of 0.001 ( $16\text{\,}\mathrm{kHz}$ sampling rate). The FouT and the LegT SSMs have a memory window size $\theta$ of $250\text{\,}\mathrm{ms}$ (4000 samples at $16\text{\,}\mathrm{kHz}$ ). a) The first 4 basis functions used by FouT, LegT, and LegS. b) The reconstruction capabilities of the three initializations. The memory window length is denoted by the green overlay, and the original and reconstructed signals are represented by the blue and orange lines, respectively. The original signal is a $16\text{\,}\mathrm{kHz}$ speech signal lowpassed using a 1024-tap FIR filter with a cutoff frequency of $100\text{\,}\mathrm{Hz}$ for visualization purposes.

Fig. 4: Our proposed SSM encoder layer. The SSM is responsible for creating a modulation- and sampling-rate-invariant representation of the input signal. To do so, it encodes the input data as a sequence of SSM coefficient vectors, then downsamples them from the variable-OSR PDM rate to a fixed rate by retaining one coefficient vector every $K=OSR$ time steps.

Fig. 5: The frequency spectra of the FouT (a) and LegT (b) basis-functions for an SSM with $\theta=$ $2\text{\,}\mathrm{ms}$ , $H=32$ . The black dotted line represents the $16\text{\,}\mathrm{kHz}$ PCM Nyquist frequency. For each SSM, the frequency spectrum was obtained by sampling each basis function at 1,000,000 uniformly distributed points over its domain and performing a 2,000,000-point DT Fourier transform.

Fig. 6: Comparison of the magnitude of the FouT coefficients SSM for a PCM and PDM input with $OSR=64$ . The FouT SSM used a state dimension $H$ of 128 and a memory window size $\theta$ of $8\text{\,}\mathrm{ms}$ . The discretization step was set to $\Delta=1/16,000$ and $\Delta=1/1,024,000$ for the PCM and PDM inputs respectively. a) The SSM coefficients sampled at the PCM rate, b) The SSM coefficients sampled at the PDM rate, and c) the SSM coefficients sampled at the PDM rate, then downsampled to the PCM rate by keeping only each 64th coefficient vector.

Fig. 7: Our proposed PDM processing architecture. The SSM encoder layer is responsible for creating a modulation- and sampling-rate-invariant representation of the input signal. This representation can then be sent to a downstream DNN, which can be either a classifier for the keyword spotting task or a speech enhancement DNN. In the speech enhancement task, the predicted coefficients are used to reconstruct a clean time-domain PCM waveform.

Fig. 8: Effect of $H$ and $\theta$ on the classification results on the Google Speech Commands test set encoded using a single-bit second-order $\Sigma\Delta$ modulator with an OSR of 128 ( $2\text{\,}\mathrm{MHz}$ ). For both SSM families, the decimation factor $K_{PDM}$ was fixed at a value of 128, the window size $\theta$ was varied from $2\text{\,}\mathrm{-}$ $32\text{\,}\mathrm{ms}$ , and the state dimension $H$ was varied from 32 to 512.

Fig. 9: Effect of $K_{PDM}$ and $\bar{\theta}_{PDM}$ on the classification results on the Google Speech Commands test set encoded using a single-bit second-order $\Sigma\Delta$ modulator with an OSR of 128 ( $2\text{\,}\mathrm{MHz}$ ). For both SSM families, the decimation factor was varied from 128-65536, and the window size $\bar{\theta}_{PDM}$ was varied from 4,096-65,536 ( $2\text{\,}\mathrm{-}$ $32\text{\,}\mathrm{ms}$ ). The state dimension $H$ was fixed at 128.

Fig. 10: The performance of the LegT SSM with $\theta=$ $2\text{\,}\mathrm{ms}$ and $H=32$ on the Google Speech Commands test set encoded using a single-bit second-order $\Sigma\Delta$ modulator with OSRs varying from 8-128 ( $128\text{\,}\mathrm{kHz}$ - $2\text{\,}\mathrm{MHz}$ ). The horizontal dotted line indicates the test-time accuracy of 93.53%, which was obtained on the PCM data.

Fig. 11: Effect of $\theta$ and $H$ on speech enhancement performance on the VoiceBank+DEMAND test set. For both SSM families, the decimation factor $K_{PDM}$ was fixed at 128, the window size $\theta$ was varied from $2\text{\,}\mathrm{ms}$ to $32\text{\,}\mathrm{ms}$ , and the state dimension H was varied from 32 to 512. The number at the top represents the PESQ score for a given combination, and the number at the bottom represents the STOI score. a) The results obtained on the test set encoded using a single-bit second-order $\Sigma\Delta$ modulator with an OSR of 128 ( $2\text{\,}\mathrm{MHz}$ ). b) The results obtained on test set encoded using $16\text{\,}\mathrm{kHz}$ PCM.

Fig. 12: The frequency spectra of the basis vectors for both the FouT (left) and LegT (right) SSMs with $H=32$ and using window lengths of $2\text{\,}\mathrm{ms}$ (top), $8\text{\,}\mathrm{ms}$ (middle), and $32\text{\,}\mathrm{ms}$ (bottom). The black dotted line represents the PCM Nyquist frequency of $8\text{\,}\mathrm{kHz}$ .

Fig. 13: Effect of the overlap and the memory window length $\bar{\theta}_{PDM}$ on speech enhancement performance on the VoiceBank+DEMAND test set encoded using a single-bit second-order $\Sigma\Delta$ modulator with an OSR of 128 ( $2\text{\,}\mathrm{MHz}$ ). For both SSM families, the overlap was varied from 0-87.5% and $\bar{\theta}_{PDM}$ was varied from 4,096 to 65,536.

Fig. 14: The performance of the FouT SSM with a $2\text{\,}\mathrm{ms}$ window size, a state dimension of 32, and an overlap of 75% on the VoiceBank+DEMAND test set encoded using a single-bit second-order $\Sigma\Delta$ modulator with a OSRs varying from 8-128 ( $128\text{\,}\mathrm{kHz}$ - $2\text{\,}\mathrm{MHz}$ ). The black dotted lines represent the PESQ/STOI scores of 2.0 and 0.92 obtained by comparing the noisy speech versus the clean speech on the same test set.

Speech Processing

Speech processing develops computational methods for recognizing, synthesizing, analyzing, and transforming spoken audio.

See all 413 papers

WaveSSM: Multiscale State-Space Models for Non-stationary Signal Attention
Feb 25, 2026Ruben Solozabal, Velibor Bojkovic, Hilal Alquabeh +3State Space Models Multiscale Time-Series Modeling
MBZUAI · RIKEN AIP
DSSMs: State Space Models with Explicit Memory via Delay Differential Equations
Jul 11, 2026Yixiao Qian, Song Chen, Jiaxu Liu +2Dynamical Systems Long-Context Modeling
College of Control Science and Engineering, Zhejiang University · Department of Mathematics, National University of Singapore · School of Mathematical Sciences, Zhejiang University
Beyond Model Size: Redesigning LiSenNet for embedded speech enhancement
Sep 24, 2026Clément Laroche, Rasmus Kongsgaard OlssonEfficient Neural Network Inference Speech Enhancement
GN A/S, Audio Research, Lauptrupbjerg 7, 2750, Denmark

Feb 25, 2026J/K move · Enter open · S save

WaveSSM: Multiscale State-Space Models for Non-stationary Signal Attention

Ruben Solozabal, Velibor Bojkovic, Hilal Alquabeh, Klea Ziu+2

MBZUAI · RIKEN AIP

State Space Models Multiscale Time-Series Modeling Non-Stationary Time Series Wavelet Methods

Open paper AskPDF arXiv

State-space models (SSMs) have emerged as a powerful foundation for long-range sequence modeling, with the HiPPO framework showing that continuous-time projection operators can be used to derive stable, memory-efficient dynamical systems that encode the past history of the input signal. However, existing projection-based SSMs often rely on polynomial bases with global temporal support, whose inductive biases are poorly matched to signals exhibiting localized or transient structure. In this work, we introduce WaveSSM, a collection of SSMs constructed over wavelet frames. Our key observation is that wavelet frames yield a localized support on the temporal dimension, useful for tasks requiring precise localization. Empirically, we show that on equal conditions, WaveSSM outperforms orthogonal counterparts as S4 on real-world datasets with transient dynamics, including physiological signals on the PTB-XL dataset and raw audio on Speech Commands.

Abstract

Speech Processing

Explore similar work

WaveSSM: Multiscale State-Space Models for Non-stationary Signal Attention

DSSMs: State Space Models with Explicit Memory via Delay Differential Equations

Beyond Model Size: Redesigning LiSenNet for embedded speech enhancement