Multirate State Space Models for End-to-End Processing of Pulse Density Modulated Speech Signals

Ludovic Boulanger; Sean U. N. Wood

cs.SDAug 28, 2026

Multirate State Space Models for End-to-End Processing of Pulse Density Modulated Speech Signals

Authors: Ludovic Boulanger, Sean U. N. Wood

Organizations: Department of Electrical and Computer Engineering at the University of Sherbrooke, Qu´ebec, Canada

Abstract

Deep neural networks (DNNs) based on state-space models (SSMs) are increasingly applied to speech processing, but typically operate on pulse-code-modulated (PCM) audio. This constrains deployment on low-power, always-on edge devices, which commonly use single-bit pulse-density-modulated (PDM) micro-electromechanical (MEMS) microphones for their noise robustness, low cost, and variable sampling rates that enable low-power operation. In fact, converting PDM to PCM requires low-pass filtering and decimation, imposing costly overhead on resource-constrained hardware. While prior works have attempted to process PDM signals directly, they require long training times and generalize poorly across sampling rates. In this paper, we show that the SSM has two key properties that remediate these issues: its continuous-time parametrization allows it to produce a consistent representation of the input audio signal, regardless of the modulation strategy and sampling rate, and its long-term memory enables this representation to be aggressively downsampled without needing any anti-aliasing operations. We then propose a novel end-to-end PDM speech processing architecture that uses an SSM to encode the input audio signal into a modulation- and sampling-rate-invariant latent representation. We show that our proposed architecture achieves robust speech classification and enhancement gains at low-power sampling-rates (512 kHz) and similar performance to state-of-the-art algorithms operating on PCM data when tested on standard PDM sampling-rates of 2 MHz. Moreover, we show that the SSM's output can be downsampled by more than 65,000 times, thus significantly reducing the number of processing timesteps in downstream layers.

Figures & tables

Fig. 1: Comparison of a PCM encoded signal (top) and the same PDM encoded signal (middle and bottom). The orange overlay on the PDM data represents the low-pass-filtered PDM and demonstrates that the PCM baseband is preserved in the low-frequency range of the PDM. The low-pass was created using a 128-tap FIR filter with a cutoff frequency of $7.5\text{\,}\mathrm{kHz}$ . The bottom plot shows a $200\text{\,}\mathrm{\SIUnitSymbolMicro s}$ zoom of the PDM encoded signal.

Fig. 2: Comparison of the PDM and PCM power spectral densities (PSD) for a 1-second speech segment. The black dotted line represents the PCM Nyquist frequency of $8\text{\,}\mathrm{kHz}$ , i.e., the maximum frequency present in the signal. The pink line represents the PCM PSD.

Fig. 3: Demonstration of the basis functions and the memory windows of the FouT, LegT, and LegS SSMs. The SSMs all have a state dimension $H$ of 1024 and a discretization step $\Delta$ of 0.001 ( $16\text{\,}\mathrm{kHz}$ sampling rate). The FouT and the LegT SSMs have a memory window size $\theta$ of $250\text{\,}\mathrm{ms}$ (4000 samples at $16\text{\,}\mathrm{kHz}$ ). a) The first 4 basis functions used by FouT, LegT, and LegS. b) The reconstruction capabilities of the three initializations. The memory window length is denoted by the green overlay, and the original and reconstructed signals are represented by the blue and orange lines, respectively. The original signal is a $16\text{\,}\mathrm{kHz}$ speech signal lowpassed using a 1024-tap FIR filter with a cutoff frequency of $100\text{\,}\mathrm{Hz}$ for visualization purposes.

Fig. 4: Our proposed SSM encoder layer. The SSM is responsible for creating a modulation- and sampling-rate-invariant representation of the input signal. To do so, it encodes the input data as a sequence of SSM coefficient vectors, then downsamples them from the variable-OSR PDM rate to a fixed rate by retaining one coefficient vector every $K=OSR$ time steps.

Fig. 5: The frequency spectra of the FouT (a) and LegT (b) basis-functions for an SSM with $\theta=$ $2\text{\,}\mathrm{ms}$ , $H=32$ . The black dotted line represents the $16\text{\,}\mathrm{kHz}$ PCM Nyquist frequency. For each SSM, the frequency spectrum was obtained by sampling each basis function at 1,000,000 uniformly distributed points over its domain and performing a 2,000,000-point DT Fourier transform.

Fig. 6: Comparison of the magnitude of the FouT coefficients SSM for a PCM and PDM input with $OSR=64$ . The FouT SSM used a state dimension $H$ of 128 and a memory window size $\theta$ of $8\text{\,}\mathrm{ms}$ . The discretization step was set to $\Delta=1/16,000$ and $\Delta=1/1,024,000$ for the PCM and PDM inputs respectively. a) The SSM coefficients sampled at the PCM rate, b) The SSM coefficients sampled at the PDM rate, and c) the SSM coefficients sampled at the PDM rate, then downsampled to the PCM rate by keeping only each 64th coefficient vector.

Fig. 7: Our proposed PDM processing architecture. The SSM encoder layer is responsible for creating a modulation- and sampling-rate-invariant representation of the input signal. This representation can then be sent to a downstream DNN, which can be either a classifier for the keyword spotting task or a speech enhancement DNN. In the speech enhancement task, the predicted coefficients are used to reconstruct a clean time-domain PCM waveform.

Fig. 8: Effect of $H$ and $\theta$ on the classification results on the Google Speech Commands test set encoded using a single-bit second-order $\Sigma\Delta$ modulator with an OSR of 128 ( $2\text{\,}\mathrm{MHz}$ ). For both SSM families, the decimation factor $K_{PDM}$ was fixed at a value of 128, the window size $\theta$ was varied from $2\text{\,}\mathrm{-}$ $32\text{\,}\mathrm{ms}$ , and the state dimension $H$ was varied from 32 to 512.

Fig. 9: Effect of $K_{PDM}$ and $\bar{\theta}_{PDM}$ on the classification results on the Google Speech Commands test set encoded using a single-bit second-order $\Sigma\Delta$ modulator with an OSR of 128 ( $2\text{\,}\mathrm{MHz}$ ). For both SSM families, the decimation factor was varied from 128-65536, and the window size $\bar{\theta}_{PDM}$ was varied from 4,096-65,536 ( $2\text{\,}\mathrm{-}$ $32\text{\,}\mathrm{ms}$ ). The state dimension $H$ was fixed at 128.

Fig. 10: The performance of the LegT SSM with $\theta=$ $2\text{\,}\mathrm{ms}$ and $H=32$ on the Google Speech Commands test set encoded using a single-bit second-order $\Sigma\Delta$ modulator with OSRs varying from 8-128 ( $128\text{\,}\mathrm{kHz}$ - $2\text{\,}\mathrm{MHz}$ ). The horizontal dotted line indicates the test-time accuracy of 93.53%, which was obtained on the PCM data.

Fig. 11: Effect of $\theta$ and $H$ on speech enhancement performance on the VoiceBank+DEMAND test set. For both SSM families, the decimation factor $K_{PDM}$ was fixed at 128, the window size $\theta$ was varied from $2\text{\,}\mathrm{ms}$ to $32\text{\,}\mathrm{ms}$ , and the state dimension H was varied from 32 to 512. The number at the top represents the PESQ score for a given combination, and the number at the bottom represents the STOI score. a) The results obtained on the test set encoded using a single-bit second-order $\Sigma\Delta$ modulator with an OSR of 128 ( $2\text{\,}\mathrm{MHz}$ ). b) The results obtained on test set encoded using $16\text{\,}\mathrm{kHz}$ PCM.

Fig. 12: The frequency spectra of the basis vectors for both the FouT (left) and LegT (right) SSMs with $H=32$ and using window lengths of $2\text{\,}\mathrm{ms}$ (top), $8\text{\,}\mathrm{ms}$ (middle), and $32\text{\,}\mathrm{ms}$ (bottom). The black dotted line represents the PCM Nyquist frequency of $8\text{\,}\mathrm{kHz}$ .

Fig. 13: Effect of the overlap and the memory window length $\bar{\theta}_{PDM}$ on speech enhancement performance on the VoiceBank+DEMAND test set encoded using a single-bit second-order $\Sigma\Delta$ modulator with an OSR of 128 ( $2\text{\,}\mathrm{MHz}$ ). For both SSM families, the overlap was varied from 0-87.5% and $\bar{\theta}_{PDM}$ was varied from 4,096 to 65,536.

Fig. 14: The performance of the FouT SSM with a $2\text{\,}\mathrm{ms}$ window size, a state dimension of 32, and an overlap of 75% on the VoiceBank+DEMAND test set encoded using a single-bit second-order $\Sigma\Delta$ modulator with a OSRs varying from 8-128 ( $128\text{\,}\mathrm{kHz}$ - $2\text{\,}\mathrm{MHz}$ ). The black dotted lines represent the PESQ/STOI scores of 2.0 and 0.92 obtained by comparing the noisy speech versus the clean speech on the same test set.

Speech Processing

Speech processing develops computational methods for recognizing, synthesizing, analyzing, and transforming spoken audio.

See all 413 papers

Feb 25, 2026

WaveSSM: Multiscale State-Space Models for Non-stationary Signal Attention

State-space models (SSMs) have emerged as a powerful foundation for long-range sequence modeling, with the HiPPO framework showing that continuous-time projection operators can be used to derive stable, memory-efficient dynamical systems that encode the past history of the input signal. However, existing projection-based SSMs often rely on polynomial bases with global temporal support, whose inductive biases are poorly matched to signals exhibiting localized or transient structure. In this work, we introduce WaveSSM, a collection of SSMs constructed over wavelet frames. Our key observation is that wavelet frames yield a localized support on the temporal dimension, useful for tasks requiring precise localization. Empirically, we show that on equal conditions, WaveSSM outperforms orthogonal counterparts as S4 on real-world datasets with transient dynamics, including physiological signals on the PTB-XL dataset and raw audio on Speech Commands.

Ruben Solozabal, Velibor Bojkovic, Hilal Alquabeh +3

MBZUAI · RIKEN AIP

State Space Models Multiscale Time-Series Modeling Non-Stationary Time Series Wavelet Methods

Ask PDF

Jul 11, 2026

DSSMs: State Space Models with Explicit Memory via Delay Differential Equations

State Space Models (SSMs) have emerged as a powerful paradigm for efficient long-sequence modeling, offering parallel training and fast linear-time recurrent inference. However, like other recurrent architectures, SSMs must compress an unbounded history into a fixed-size state, which limits context retention and makes precise retrieval over long-range context inherently difficult. To overcome this limitation, we propose Delay State Space Models (DSSMs), a delay differential equation (DDE)-inspired extension of diagonal SSMs that augments discrete SSM recurrences with explicit delayed-state feedback. Making explicit delayed feedback practical requires new stability parameterization, history management, and FFT-training tools. We address these challenges with a practical discretization and parameterization grounded in a simple delay-independent stability condition. To bypass direct time-domain kernel construction, we derive the DSSM transfer function and compute kernels in the frequency domain, using a kernel contour shift to suppress aliasing and recover accurate FFT training. Empirically, DSSMs substantially improve targeted delayed-retrieval tasks while outperforming S4D on most standard sequence metrics and remaining close on the others.

Yixiao Qian, Song Chen, Jiaxu Liu +2

College of Control Science and Engineering, Zhejiang University · Department of Mathematics, National University of Singapore · School of Mathematical Sciences, Zhejiang University

Dynamical Systems Long-Context Modeling State Space Models

Ask PDF

Sep 24, 2026

Beyond Model Size: Redesigning LiSenNet for embedded speech enhancement

Deploying real-time speech enhancement on resource-constrained devices requires meeting strict latency, memory, and energy constraints. Microcontroller NPUs can accelerate neural inference under these constraints, but only through a restricted set of operators in static, integer-quantized graphs. Recent speech-enhancement networks have reduced parameter counts and MACs to levels nominally suitable for microcontrollers, but their operators and execution patterns often remain incompatible with restricted NPUs. We address this gap by redesigning LiSenNet, a 37k parameter sub-band dual-path model, for the STM32N6570-DK Neural-ART accelerator. We replace its recurrent bottleneck with convolutional frequency and temporal mixers, reformulate unsupported operations as static int8-compatible primitives, and use bounded decoder activations to preserve quality after quantization. On VoiceBank-DEMAND, the final NPU-compatible model matches or exceeds the recurrent LiSenNet baseline, reaching PESQ 3.08 versus 3.01 in FP32 and 3.01 versus 2.93 in int8. Deployed on a microcontroller, it processes each 16 ms input hop in 4.83 ms, corresponding to a real-time factor of 0.30. Stateless receptive-field recomputation is an order of magnitude slower at the same frame rate despite higher accelerator utilization. These results show that parameter count and operator compatibility, quantization range, and persistent streaming state must be co-designed to achieve efficient real-time speech enhancement on restricted NPUs.

Clément Laroche, Rasmus Kongsgaard Olsson

GN A/S, Audio Research, Lauptrupbjerg 7, 2750, Denmark

Efficient Neural Network Inference Speech Enhancement Neural Network Quantization

Ask PDF

Abstract

Speech Processing

Explore similar work

WaveSSM: Multiscale State-Space Models for Non-stationary Signal Attention

DSSMs: State Space Models with Explicit Memory via Delay Differential Equations

Beyond Model Size: Redesigning LiSenNet for embedded speech enhancement