HiPPO Zoo: Explicit Memory Mechanisms for Interpretable State Space Models

Jack Goffinet; Casey Hanks; David E. Carlson

cs.LGFeb 24, 2026

HiPPO Zoo: Explicit Memory Mechanisms for Interpretable State Space Models

Authors: Jack Goffinet, Casey Hanks, David E. Carlson

Organizations: Department of Computer Science, Duke University, Durham NC, USA · Departments of Civil and Environmental Engineering; Biostatistics and Bioinformatics; and Electrical and Computer Engineering, Duke University, Durham NC, USA

Abstract

Representing the past in a compressed, efficient, and informative manner is a central problem for systems trained on sequential data. The HiPPO framework, originally proposed by Gu & Dao et al., provides a principled approach to sequential compression by projecting signals onto orthogonal polynomial (OP) bases via structured linear ordinary differential equations. Subsequent works have embedded these dynamics in state space models (SSMs), where HiPPO structure serves as an initialization. Nonlinear successors of these SSM methods such as Mamba are state-of-the-art for many tasks with long-range dependencies, but the mechanisms by which they represent and prioritize history remain largely implicit. In this work, we revisit the HiPPO framework with the goal of making these mechanisms explicit. We show how polynomial representations of history can be extended to support capabilities of modern SSMs such as adaptive memory allocation and associative memory, while retaining direct interpretability in the OP basis. We introduce a unified framework comprising five such extensions, which we collectively refer to as a "HiPPO zoo." Each extension exposes a specific modeling capability through an explicit, interpretable modification of the HiPPO framework. The resulting models adapt their memory online and train in streaming settings with efficient updates. We illustrate the behaviors and modeling advantages of these extensions through a range of synthetic sequence modeling tasks, demonstrating that capabilities typically associated with modern SSMs can be realized through explicit, interpretable polynomial memory structures.

Figures & tables

Figure 1: A: Online learning of a continuous time-invariant system. The ground truth system has a nonzero second-order Volterra kernel (middle). Online learning of a linear HiPPO system is insufficient to learn the kernel. Using a second-order (quadratic) Volterra HiPPO system, the correct kernel can be inferred (right), with faster convergence than an unstructured MLP readout (left, mean $\pm$ SEM across five independent runs). B: The time warping interpretation of Salience HiPPO. Salience HiPPO is equivalent to a standard HiPPO system under an invertible time warp $\varphi$ . Consequently, the static weighting functions in warped time (left; green, red, and yellow) correspond to dynamic weighting functions in real time (bottom; green, red, and yellow).

Figure 2: Salience HiPPO applied to a selective copying task. Top: Input sequence with informative tokens (colors) and uninformative tokens (white). Gray tokens indicate the recall phase, where informative tokens must be reproduced in order. Second: Induced history measures allocate weight to informative timepoints while downweighting uninformative timepoints. Third: Learned salience signal

g(t)

decreases during uninformative inputs. Bottom: Output prediction functionals show peaks at corresponding informative timepoints, showing strong performance on the task.

HiPPO variants ( $\sim$ 25k params, single-layer)
Model	Params	SC Acc.	AR Acc.
Vanilla	25k	27.6%	22.8%
Vanilla + MLP	25k	27.7%	20.4%
Salience	25k	100.0%	27.1%
Assoc. Memory	25k	65.7%	100.0%
Baselines (single-layer, parameter-matched)

Table 1: Performance on synthetic long-range memory tasks: selective copying ( SC ) and associative recall ( AR ). See Appendix C.6 for experimental details.

Figure 3: Associative Memory HiPPO uses OP associative memory to solve an associative recall task. Top: Task schematic. Middle: The system learns to read and write from consistent locations. Bottom: OP memory before and after a write operation.

Figure 4: Multiscale HiPPO provides stable and parsimonious explicit representations over a continuum of timescales. Top: a single Multiscale HiPPO system (red) remembers the past at many timescales, while vanilla HiPPO systems (blue) remember single timescales. Bottom: When tasked with reproducing 16 Legendre polynomial coefficients to summarize the past at a range of timescales, Multiscale HiPPO consistently outputs coefficients of equal or better quality than single-timescale HiPPO systems (with timescales

10^{1},\dots,10^{4}

, light to dark blue).

Figure 5: A : Forecasting HiPPO schematic. B : Forecasting HiPPO with a reduced-rank linear forecasting map reveals different “predictive memories” under different forecasting horizons. C: Objective-dependent history geometries in the top eigenfunctions of

Q

, the predictive history metric, show broader features for the long-horizon objective.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	$\omega(\tau)$	$P_{n}(\tau)$	$b_{n}$	$A_{nk}$
Leg-T	$\mathcal{U}[0,1]$	$L_{n}(\tau)$	$\sqrt{2n+1}$	$\begin{cases}-b_{n}b_{k}(-1)^{n-k},&n>k\\ -b_{n}b_{k},&n=k\\ 0,&n<k\end{cases}$
Leg-S	$\exp(-\tau)$	$L_{n}(1-e^{-\tau})$	$\sqrt{2n+1}$	$\begin{cases}0,&n<k\\ -b_{n}b_{k}\dfrac{n+1}{2n+1},&n=k\\ -b_{n}b_{k},&n>k\end{cases}$

Appendix

Table 2: Orthogonal Polynomial HiPPO systems used in this paper. Leg-T corresponds to the truncated Legendre HiPPO system with uniform weighting on a finite window, while Leg-S corresponds to the exponentially weighted (sliding) Legendre HiPPO system.

Figure 6: Reproducing kernel $K(x_{key},x_{query})$ of the Legendre OP basis with $n_{assoc}=32$ memory coefficients, evaluated over the address space $x\in[0,1]$ . Diagonal entries equal 1 by definition ( $\kappa(x,x)=1$ ). Learned write addresses (markers) are spaced to minimize off-diagonal kernel values, resulting in near-zero interference between stored associations.

Setting	Model	# Layers	Params	Test Acc.
HiPPO Zoo baseline
Baseline	Assoc. Memory HiPPO	1	25k	100.0%
Transformer variants
Baseline	Transformer	1	27k	32.9%
Baseline + PE	Transformer	1	27k	33.2%
Capacity 10 $\times$	Transformer	1	270k	32.4%

Appendix

Table 3: Follow-up associative recall (AR) experiments investigating the effects of model scale, dataset size, optimization budget, and positional embeddings (PEs). Baseline settings correspond to the original AR experiments. “Capacity 10

\times

” increases model width to approximately 10

\times

the original parameter count, “Data 10

\times

” increases the number of training sequences by 10

\times

, and “Steps 2

\times

” doubles the number of optimization steps.

Figure 7: The log-timescale Multiscale HiPPO variant consistently outperforms the basic and Jeffreys-style variants on the variable-horizon reconstruction experiment from Section 3.4 . Single-scale HiPPO systems with timescales $10^{1},\dots,10^{4}$ are shown in light to dark blue for comparison. Shaded regions denote $\pm$ SEM for 64 independently sampled trajectories of length $T=3\times 10^{4}$ .

Baselines (single-layer)
Model	Params	Test PPL
LSTM	25k	5.46
S4D	25k	5.75
Mamba	26k	5.54
Baselines (two-layer)
LSTM	42k	4.75

Appendix

Table 4: WikiText-2-raw character-level language modeling results.

Sequence Modeling

Sequence modeling studies how to represent, predict, classify, and generate ordered data such as text, time series, events, and biological sequences.

See all 249 papers

SHiPPO: Recurrent Memory with Transported Polynomial Projections
Jul 3, 2026Tomoya Mizuguchi, Bum Jun KimSelective SSMs Recurrent Neural Networks
Kyoto University, Japan · The University of Tokyo, Japan
Fractional State Space Transition for Long Sequence Modeling
Sep 28, 2026Ivan Kobyzev, Abbas Ghaddar, Ali Nasiri-Sarvi +2State Space Language Models State Space Models
Huawei Noah’s Ark Lab, Montreal Research Center, Canada
WaveSSM: Multiscale State-Space Models for Non-stationary Signal Attention
Feb 25, 2026Ruben Solozabal, Velibor Bojkovic, Hilal Alquabeh +3State Space Models Multiscale Time-Series Modeling
MBZUAI · RIKEN AIP

Jul 3, 2026J/K move · Enter open · S save

SHiPPO: Recurrent Memory with Transported Polynomial Projections

Tomoya Mizuguchi, Bum Jun Kim

Kyoto University, Japan · The University of Tokyo, Japan

Selective SSMs Recurrent Neural Networks

Open paper AskPDF arXiv

HiPPO gives recurrent states memory semantics as coefficients of online polynomial projections, but in fixed channel coordinates. Modern selective SSMs, by contrast, rely on token-dependent control and channel interaction. We introduce SHiPPO (Sylvester HiPPO), a transported projection-memory prior that lifts HiPPO coefficient memories into a moving channel frame. For any fixed or realized right-transport path, SHiPPO transports the approximation family and channel metric together; conditional on that path, the state is ordinary HiPPO in a tied moving frame and follows Sylvester coefficient dynamics, preserving the left online-memory operator while adding right-action transport. For selective-SSM execution, we derive a restricted group-local realization with controller-compatible right actions, exponential-adjusted updates, exact block-affine scan, and recurrent decoding. We also give a simultaneous-reducibility criterion identifying when right transports collapse to static mixing plus independent scalar or blockwise banks. Controlled diagnostics show that larger current-token write rank improves ordinary prediction error but cannot recover order-sensitive changes to already-written memory; transported-memory variants recover this signal, which disappears when the transport pathway is removed. A finite-field associative-recall diagnostic with interleaved bindings, operations, and queries provides complementary autoregressive evidence while leaving the preferred right-action realization open. Taken together, these results support SHiPPO as a mechanistically grounded transported-memory prior, with evidence focused on memory mechanisms rather than broad sequence-modeling dominance.

Abstract

Sequence Modeling

Explore similar work

SHiPPO: Recurrent Memory with Transported Polynomial Projections

Fractional State Space Transition for Long Sequence Modeling

WaveSSM: Multiscale State-Space Models for Non-stationary Signal Attention