HiPPO Zoo: Explicit Memory Mechanisms for Interpretable State Space Models
Authors: Jack Goffinet, Casey Hanks, David E. Carlson
Organizations: Department of Computer Science, Duke University, Durham NC, USA · Departments of Civil and Environmental Engineering; Biostatistics and Bioinformatics; and Electrical and Computer Engineering, Duke University, Durham NC, USA
Representing the past in a compressed, efficient, and informative manner is a central problem for systems trained on sequential data. The HiPPO framework, originally proposed by Gu & Dao et al., provides a principled approach to sequential compression by projecting signals onto orthogonal polynomial (OP) bases via structured linear ordinary differential equations. Subsequent works have embedded these dynamics in state space models (SSMs), where HiPPO structure serves as an initialization. Nonlinear successors of these SSM methods such as Mamba are state-of-the-art for many tasks with long-range dependencies, but the mechanisms by which they represent and prioritize history remain largely implicit. In this work, we revisit the HiPPO framework with the goal of making these mechanisms explicit. We show how polynomial representations of history can be extended to support capabilities of modern SSMs such as adaptive memory allocation and associative memory, while retaining direct interpretability in the OP basis. We introduce a unified framework comprising five such extensions, which we collectively refer to as a "HiPPO zoo." Each extension exposes a specific modeling capability through an explicit, interpretable modification of the HiPPO framework. The resulting models adapt their memory online and train in streaming settings with efficient updates. We illustrate the behaviors and modeling advantages of these extensions through a range of synthetic sequence modeling tasks, demonstrating that capabilities typically associated with modern SSMs can be realized through explicit, interpretable polynomial memory structures.
Figures & tables
Figure 1: A: Online learning of a continuous time-invariant system. The ground truth system has a nonzero second-order Volterra kernel (middle). Online learning of a linear HiPPO system is insufficient to learn the kernel. Using a second-order (quadratic) Volterra HiPPO system, the correct kernel can be inferred (right), with faster convergence than an unstructured MLP readout (left, mean ± SEM across five independent runs). B: The time warping interpretation of Salience HiPPO. Salience HiPPO is equivalent to a standard HiPPO system under an invertible time warp φ . Consequently, the static weighting functions in warped time (left; green, red, and yellow) correspond to dynamic weighting functions in real time (bottom; green, red, and yellow).
Figure 2: Salience HiPPO applied to a selective copying task. Top: Input sequence with informative tokens (colors) and uninformative tokens (white). Gray tokens indicate the recall phase, where informative tokens must be reproduced in order. Second: Induced history measures allocate weight to informative timepoints while downweighting uninformative timepoints. Third: Learned salience signal g(t) decreases during uninformative inputs. Bottom: Output prediction functionals show peaks at corresponding informative timepoints, showing strong performance on the task.
Model
Params
SC Acc.
AR Acc.
HiPPO variants ( ∼ 25k params, single-layer)
Vanilla
25k
27.6%
22.8%
Vanilla + MLP
25k
27.7%
20.4%
Salience
25k
100.0%
27.1%
Assoc. Memory
25k
65.7%
100.0%
Baselines (single-layer, parameter-matched)
Table 1: Performance on synthetic long-range memory tasks: selective copying ( SC ) and associative recall ( AR ). See Appendix C.6 for experimental details.
Figure 3: Associative Memory HiPPO uses OP associative memory to solve an associative recall task. Top: Task schematic. Middle: The system learns to read and write from consistent locations. Bottom: OP memory before and after a write operation.
Figure 4: Multiscale HiPPO provides stable and parsimonious explicit representations over a continuum of timescales. Top: a single Multiscale HiPPO system (red) remembers the past at many timescales, while vanilla HiPPO systems (blue) remember single timescales. Bottom: When tasked with reproducing 16 Legendre polynomial coefficients to summarize the past at a range of timescales, Multiscale HiPPO consistently outputs coefficients of equal or better quality than single-timescale HiPPO systems (with timescales 101,…,104 , light to dark blue).
Figure 5: A : Forecasting HiPPO schematic. B : Forecasting HiPPO with a reduced-rank linear forecasting map reveals different “predictive memories” under different forecasting horizons. C: Objective-dependent history geometries in the top eigenfunctions of Q , the predictive history metric, show broader features for the long-horizon objective.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
ω(τ)
Pn(τ)
bn
Ank
Leg-T
U[0,1]
Ln(τ)
2n+1
⎩⎨⎧−bnbk(−1)n−k,−bnbk,0,n>kn=kn<k
Leg-S
exp(−τ)
Ln(1−e−τ)
2n+1
⎩⎨⎧0,−bnbk2n+1n+1,−bnbk,n<kn=kn>k
Appendix
Table 2: Orthogonal Polynomial HiPPO systems used in this paper. Leg-T corresponds to the truncated Legendre HiPPO system with uniform weighting on a finite window, while Leg-S corresponds to the exponentially weighted (sliding) Legendre HiPPO system.
Figure 6: Reproducing kernel K(xkey,xquery) of the Legendre OP basis with nassoc=32 memory coefficients, evaluated over the address space x∈[0,1] . Diagonal entries equal 1 by definition ( κ(x,x)=1 ). Learned write addresses (markers) are spaced to minimize off-diagonal kernel values, resulting in near-zero interference between stored associations.
Setting
Model
# Layers
Params
Test Acc.
HiPPO Zoo baseline
Baseline
Assoc. Memory HiPPO
1
25k
100.0%
Transformer variants
Baseline
Transformer
1
27k
32.9%
Baseline + PE
Transformer
1
27k
33.2%
Capacity 10 ×
Transformer
1
270k
32.4%
Appendix
Table 3: Follow-up associative recall (AR) experiments investigating the effects of model scale, dataset size, optimization budget, and positional embeddings (PEs). Baseline settings correspond to the original AR experiments. “Capacity 10 × ” increases model width to approximately 10 × the original parameter count, “Data 10 × ” increases the number of training sequences by 10 × , and “Steps 2 × ” doubles the number of optimization steps.
Figure 7: The log-timescale Multiscale HiPPO variant consistently outperforms the basic and Jeffreys-style variants on the variable-horizon reconstruction experiment from Section 3.4 . Single-scale HiPPO systems with timescales 101,…,104 are shown in light to dark blue for comparison. Shaded regions denote ± SEM for 64 independently sampled trajectories of length T=3×104 .
Model
Params
Test PPL
Baselines (single-layer)
LSTM
25k
5.46
S4D
25k
5.75
Mamba
26k
5.54
Baselines (two-layer)
LSTM
42k
4.75
Appendix
Table 4: WikiText-2-raw character-level language modeling results.
HiPPO gives recurrent states memory semantics as coefficients of online polynomial projections, but in fixed channel coordinates. Modern selective SSMs, by contrast, rely on token-dependent control and channel interaction. We introduce SHiPPO (Sylvester HiPPO), a transported projection-memory prior that lifts HiPPO coefficient memories into a moving channel frame. For any fixed or realized right-transport path, SHiPPO transports the approximation family and channel metric together; conditional on that path, the state is ordinary HiPPO in a tied moving frame and follows Sylvester coefficient dynamics, preserving the left online-memory operator while adding right-action transport. For selective-SSM execution, we derive a restricted group-local realization with controller-compatible right actions, exponential-adjusted updates, exact block-affine scan, and recurrent decoding. We also give a simultaneous-reducibility criterion identifying when right transports collapse to static mixing plus independent scalar or blockwise banks. Controlled diagnostics show that larger current-token write rank improves ordinary prediction error but cannot recover order-sensitive changes to already-written memory; transported-memory variants recover this signal, which disappears when the transport pathway is removed. A finite-field associative-recall diagnostic with interleaved bindings, operations, and queries provides complementary autoregressive evidence while leaving the preferred right-action realization open. Taken together, these results support SHiPPO as a mechanistically grounded transported-memory prior, with evidence focused on memory mechanisms rather than broad sequence-modeling dominance.
Tomoya Mizuguchi, Bum Jun Kim
Kyoto University, Japan · The University of Tokyo, Japan
State Space Models (SSMs) compress sequence history into a bounded recurrent state, making the resulting memory law a central architectural choice for long-context performance. Most modern SSMs rely on ODE-based dynamics that lead to exponential forgetting, limiting their ability to retain information over broad temporal ranges. We introduce FRAC, a selective SSM architecture derived from fractional dynamics that replaces this exponential decay with power-law long memory. To make fractional dynamics practical, FRAC approximates the heavy-tailed target kernel with a finite-state, log-spaced sum of exponential modes. This construction turns fractional memory into an efficient recurrent module with parallel training and prefill, while retaining bounded-state autoregressive decoding. Extensive experiments, including 1.3B-parameter language modeling, demonstrate that FRAC consistently improves long-context performance over state-of-the-art SSM baselines while staying competitive on short-context. These results show that fractional dynamics provide a practical and effective prior for long-context SSMs.
Ivan Kobyzev, Abbas Ghaddar, Ali Nasiri-Sarvi +2
Huawei Noah’s Ark Lab, Montreal Research Center, Canada
State-space models (SSMs) have emerged as a powerful foundation for long-range sequence modeling, with the HiPPO framework showing that continuous-time projection operators can be used to derive stable, memory-efficient dynamical systems that encode the past history of the input signal. However, existing projection-based SSMs often rely on polynomial bases with global temporal support, whose inductive biases are poorly matched to signals exhibiting localized or transient structure. In this work, we introduce WaveSSM, a collection of SSMs constructed over wavelet frames. Our key observation is that wavelet frames yield a localized support on the temporal dimension, useful for tasks requiring precise localization. Empirically, we show that on equal conditions, WaveSSM outperforms orthogonal counterparts as S4 on real-world datasets with transient dynamics, including physiological signals on the PTB-XL dataset and raw audio on Speech Commands.