Generative sequence modeling for infinite memory processes via predictive states

Michael Wieck-Sosa; Cosma Rohilla Shalizi

stat.MLSep 29, 2026

Generative sequence modeling for infinite memory processes via predictive states

Authors: Michael Wieck-Sosa, Cosma Rohilla Shalizi

Organizations: Department of Statistics & Data Science Carnegie Mellon University

Abstract

We consider estimating the one-step-ahead conditional distribution of a multivariate stochastic process. Many existing approaches rely on assumptions such as finite-range memory, sparsity, or additivity, which can be poorly suited to processes with long-range nonlinear interactions. However, without such structural assumptions, nonparametric estimation is challenging due to the curse of dimensionality. To address this challenge, we introduce a new estimation approach based on the predictive states of a process, possibly with infinite-range memory. We show that our estimator achieves fast convergence rates when the past history can be compressed into a low-dimensional statistic that is sufficient for predicting the future. Specifically, we show that the statistical complexity of the estimation problem is determined by the intrinsic dimension of the predictive state space. We establish guarantees for an instantiation of our method based on deep neural network estimators, and we support these theoretical results with experiments.

Figures & tables

Figure 1:

L^{2}

distance by sample size

n

. We fit each estimator on the first

n

samples, we calculate

(L_{\mathrm{eval}}^{-1}\sum_{t=n}^{n+L_{\mathrm{eval}}-1}\int_{B_{\mathrm{trunc}}}(\hat{f}_{t}(x)-f_{t}(x))^{2}dx)^{1/2}

, where

L_{\mathrm{eval}}=500

f_{t}

is the true conditional density, and

\hat{f}_{t}\in\{\hat{f}_{t}^{\mathrm{Ours}},\hat{f}_{t}^{\mathrm{MDN}}\}

is the fitted conditional density, then we average this distance over 100 replications. Our method performs comparably to the MDN with a growing lag window.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Illustration of the RNN architecture.

Figure 3: Relationships among classes of sets and functions (Figure 3 in Amir et al. (2023) ).

Explore similar work

Recursive Entropic Variational Inference for Nonlinear State-Space Models
Nov 19, 2025Hany Abdulsamad, Ángel F. García-Fernández, Simo SärkkäVariational Inference State Space Models
Amsterdam Machine Learning Lab University of Amsterdam, Netherlands · Information Processing and Telecommunications Center Universidad Politécnica de Madrid, Spain · ELLIS Institute Finland Department of Electrical Engineering and Automation Aalto University, Finland
The Simulacrum: Decision-Theoretic Pretraining for Near-Optimal Time-Series Forecasting and Inference
Jun 26, 2026Pablo Montero-Manso, Marcel ScharthTime Series Forecasting Finite-Sample
University of Sydney Business School, Australia
Scalable estimation of VARMA models
Aug 6, 2026Daniel Paulin, Victor ElviraMaximum Likelihood Autocorrelations
College of Computing & Data Science Nanyang Technological University Singapore · School of Mathematics University of Edinburgh Edinburgh, UK

Nov 19, 2025, cs.LGJ/K move · Enter open · S save

Recursive Entropic Variational Inference for Nonlinear State-Space Models

Hany Abdulsamad, Ángel F. García-Fernández, Simo Särkkä

Amsterdam Machine Learning Lab University of Amsterdam, Netherlands · Information Processing and Telecommunications Center Universidad Politécnica de Madrid, Spain · ELLIS Institute Finland Department of Electrical Engineering and Automation Aalto University, Finland

Variational Inference State Space Models State Estimation Lagrangian Methods Backward Passes Trust Region

Open paper AskPDF arXiv

We present a class of algorithms for state estimation in nonlinear, non-Gaussian state-space models. Our approach is based on a variational Lagrangian formulation that casts Bayesian inference as a sequence of entropic trust-region updates subject to dynamic consistency constraints. This framework gives rise to a family of forward-backward algorithms whose structure is determined by the chosen factorization of the variational posterior. By focusing on Gauss--Markov approximations, we derive recursive schemes with favorable computational complexity. For general nonlinear, non-Gaussian models, we close the recursions using generalized statistical linear regression and Fourier--Hermite moment matching.

Abstract

Explore similar work

Recursive Entropic Variational Inference for Nonlinear State-Space Models

The Simulacrum: Decision-Theoretic Pretraining for Near-Optimal Time-Series Forecasting and Inference

Scalable estimation of VARMA models