ST-GRU PredRNN-V2: A GRU-Based Reformulation of PredRNN-V2 for Video Prediction

Augusta van Haren; Hanna Hoogen; Luca Pattavina; Quincy Rommen

cs.CVOct 13, 2026

ST-GRU PredRNN-V2: A GRU-Based Reformulation of PredRNN-V2 for Video Prediction

Authors: Augusta van Haren, Hanna Hoogen, Luca Pattavina, Quincy Rommen

Organizations: Radboud University

Abstract

Video prediction requires models to capture complex spatiotemporal dependencies and to retain relevant information over long sequences. PredRNN and its successor PredRNN-V2 address this challenge with spatiotemporal Long Short-Term Memory (ST-LSTM) units that maintain two separate memory states, at the cost of a relatively complex recurrent building block. In this work, we investigate whether the simpler gating of Gated Recurrent Units (GRUs) can be transferred to spatiotemporal predictive learning. To this end, we develop a spatiotemporal GRU (ST-GRU) unit as a GRU-based alternative to the ST-LSTM. The ST-GRU replaces the ST-LSTM's three gates per memory state with two GRU-style gates (update and reset), and merges the temporal memory state into the hidden state, which reduces the number of parameters by about 5%. We incorporate the ST-GRU into PredRNN-V2 and evaluate the resulting model on the Moving MNIST video prediction benchmark, following the original experimental protocol. Averaged over ten predicted frames, the ST-GRU-based PredRNN-V2 achieves an MSE of 55.1, an SSIM of 0.883, an LPIPS of 0.065 and a PSNR of 19.53dB. It does not surpass the original ST-LSTM-based PredRNN-V2 on MSE, SSIM or PSNR, but it achieves a lower LPIPS and performs on par with the earlier ST-LSTM-based PredRNN, while using a simpler recurrent cell. These results suggest that ST-GRUs are a promising building block for spatiotemporal predictive learning and motivate further work on their design and integration.

Figures & tables

Figure 1: Frame-wise metrics (MSE, SSIM, LPIPS, PSNR) of the ST-GRU-based PredRNN-V2 on the Moving MNIST test set over 80,000 training iterations. The frame number indicates the number of time steps between the predicted frame and the last observed frame.

Model	MSE ( $\downarrow$ )	SSIM ( $\uparrow$ )	LPIPS ( $\downarrow$ )	PSNR ( $\uparrow$ )
ConvLSTM (RNN) ( Shi et al., 2015 )	103.3	0.707	0.156	–
ST-LSTM (PredRNN) ( Wang et al., 2017 )	56.8	0.867	0.107	–
ST-LSTM (PredRNN-V2) ( Wang et al., 2021 )	48.4	0.891	0.071	20.05
ConvGRU (RNN) ( Ballas et al., 2016 )	57.8	0.839	0.091	–
ST-GRU (PredRNN) ( Dong et al., 2025 )	53.4	0.893	0.057	–
ST-GRU (PredRNN-V2), ours	55.1	0.883	0.065	19.53

Table 1: Model performance on the Moving MNIST test set. All metrics are averaged over ten predicted frames. The best result for each metric is shown in bold, and our model in teal . Results for the LSTM-based baselines are taken from Wang et al. (2021) and for the GRU-based baselines from Dong et al. (2025) . PSNR (in dB) was computed only for the two PredRNN-V2 variants.

Figure 2: Frame-wise MSE, SSIM, LPIPS and PSNR on the Moving MNIST test set. All models were trained and evaluated following the protocol described in Section 2.5 . Results for the LSTM-based models are taken from Wang et al. (2021) , and those for the GRU-based models from Dong et al. (2025) . PSNR for the ST-LSTM-based PredRNN-V2 was computed with the official trained model ( Wu, 2022 ) . All results for the ST-GRU-based PredRNN-V2 are from our own evaluation.

Figure 3: Predictions of the ST-GRU-based PredRNN-V2 on five Moving MNIST test sequences, showing the input sequence, the ground-truth future sequence and the predicted future sequence. Both global and local similarity to the ground truth decline for frames further into the future.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Side-by-side mathematical comparison of the ST-LSTM and ST-GRU building blocks.

Panel of Figure 5 — Figure 5: Side-by-side comparison of the ST-LSTM-based and ST-GRU-based PredRNN(-V2) architectures, with the flows of information shown in different colours. The ST-LSTM-based PredRNN(-V2) has two memory states: the temporal memory $\mathcal{C}_{t}^{l}$ flows horizontally from the previous time step to the current one ( $t$ ), whereas the spatiotemporal memory $\mathcal{M}_{t}^{l}$ zigzags through the layers ( $l$ ) of PredRNN(-V2), capturing short-term dynamics. Both memory states are used to update the hidden state $\mathcal{H}_{t}^{l}$ . The ST-GRU-based PredRNN(-V2), in contrast, integrates the horizontal memory flow into the hidden state $\mathcal{H}_{t}^{l}$ , while the spatiotemporal memory is still carried by $\mathcal{M}_{t}^{l}$ .

Panel of Figure 6 — Figure 6: Side-by-side comparison of the ST-LSTM and ST-GRU building blocks, with the flows of information shown in different colours. The ST-LSTM has two memory states: the temporal memory $\mathcal{C}_{t}^{l}$ across time steps ( $t$ ), and the spatiotemporal memory $\mathcal{M}_{t}^{l}$ , which captures short-term dynamics across layers ( $l$ ). Each memory state is modulated by three gates, $i_{t}$ , $f_{t}$ and $g_{t}$ (respectively $i^{\prime}_{t}$ , $f^{\prime}_{t}$ and $g^{\prime}_{t}$ ), and both contribute to the hidden state $\mathcal{H}_{t}^{l}$ via the output gate $o_{t}$ . The ST-GRU, in contrast, integrates the horizontal memory flow into the hidden state $\mathcal{H}_{t}^{l}$ , while the spatiotemporal memory is still carried by $\mathcal{M}_{t}^{l}$ . Each of these states is modulated by two gates, $z_{t}$ and $r_{t}$ (respectively $z^{\prime}_{t}$ and $r^{\prime}_{t}$ ), and they are combined via the output gate $o_{t}$ . The reduction in both gates and memory states is what makes the ST-GRU simpler than the ST-LSTM.

Explore similar work

Feb 26, 2026

Towards Long-Form Spatio-Temporal Video Grounding

In real scenarios, videos can span several minutes or even hours. However, existing research on spatio-temporal video grounding (STVG), given a textual query, mainly focuses on localizing targets in short videos of tens of seconds, typically less than one minute, which limits real-world applications. In this paper, we explore Long-Form STVG (LF-STVG), which aims to locate targets in long-term videos. Compared with short videos, long-term videos contain much longer temporal spans and more irrelevant information, making it difficult for existing STVG methods that process all frames at once. To address this challenge, we propose an AutoRegressive Transformer architecture for LF-STVG, termed ART-STVG. Unlike conventional STVG methods that require the entire video sequence to make predictions at once, ART-STVG treats the video as streaming input and processes frames sequentially, enabling efficient handling of long videos. To model spatio-temporal context, we design spatial and temporal memory banks and apply them to the decoders. Since memories from different moments are not always relevant to the current frame, we introduce simple yet effective memory selection strategies to provide more relevant information to the decoders, significantly improving performance. Furthermore, instead of parallel spatial and temporal localization, we propose a cascaded spatio-temporal design that connects the spatial decoder to the temporal decoder, allowing fine-grained spatial cues to assist complex temporal localization in long videos. Experiments on newly extended LF-STVG datasets show that ART-STVG significantly outperforms state-of-the-art methods, while achieving competitive performance on conventional short-form STVG. Our code is at: https://github.com/HengLan/ART-STVG.

Xin Gu, Bing Fan, Jiali Yao +5

University of Chinese Academy of Sciences · 2UNT · Institute of Software Chinese Academy of Sciences +2

Temporal Video Grounding Long-Context Modeling Vision-Language Grounding

Ask PDF

Feb 24, 2026

PFGNet: A Fully Convolutional Frequency-Guided Peripheral Gating Network for Efficient Spatiotemporal Predictive Learning

Spatiotemporal predictive learning (STPL) aims to forecast future frames from past observations and is essential across a wide range of applications. Compared with recurrent or hybrid architectures, pure convolutional models offer superior efficiency and full parallelism, yet their fixed receptive fields limit their ability to adaptively capture spatially varying motion patterns. Inspired by biological center-surround organization and frequency-selective signal processing, we propose PFGNet, a fully convolutional framework that dynamically modulates receptive fields through pixel-wise frequency-guided gating. The core Peripheral Frequency Gating (PFG) block extracts localized spectral cues and adaptively fuses multi-scale large-kernel peripheral responses with learnable center suppression, effectively forming spatially adaptive band-pass filters. To maintain efficiency, all large kernels are decomposed into separable 1D convolutions (

1 \times k

followed by

k \times 1

), reducing per-channel computational cost from

O(k^2)

O(2k)

. PFGNet enables structure-aware spatiotemporal modeling without recurrence or attention. Experiments on Moving MNIST, TaxiBJ, Human3.6M, and KTH show that PFGNet delivers SOTA or near-SOTA forecasting performance with substantially fewer parameters and FLOPs. Our code is available at https://github.com/fhjdqaq/PFGNet.

Xinyong Cai, Changbin Sun, Yong Wang +2

College of Computer Science, Sichuan University · Department of Data and Systems Engineering, The University of Hong Kong

Spatiotemporal Forecasting Video Prediction Convolutional Neural Networks Dynamic Convolution Frequency-Domain Feature Learning Lightweight CNNs

Ask PDF

May 21, 2026

Enhancing Visual Token Representations for Video Large Language Models via Training-Free Spatial-Temporal Pooling and Gridding

Recent advances in Multimodal Large Language Models (MLLMs) have significantly advanced video understanding tasks, yet challenges remain in efficiently compressing visual tokens while preserving spatiotemporal interactions. Existing methods, such as LLaVA family, utilize simplistic pooling or interpolation techniques that overlook the intricate dynamics of visual tokens. To bridge this gap, we propose ST-GridPool, a novel training-free visual token enhancement method designed specifically for Video LLMs. Our approach integrates Pyramid Temporal Gridding (PTG), which captures multi-grained spatiotemporal interactions through hierarchical temporal gridding, and Norm-based Spatial Pooling (NSP), which preserves high-information visual regions by leveraging the correlation between token norms and semantic richness. Extensive experiments on various benchmarks demonstrate that ST-GridPool consistently enhances performance of Video LLMs without requiring costly retraining. Our method offers an efficient and plug-and-play solution for improving visual token representations. Our code is available in https://github.com/bingjunluo/ST-GridPool.

Bingjun Luo, Tony Wang, Hanqi Chen +1

1Tsinghua University · 2Shenzhen University · 3Xidian University

Video-Language Models Multimodal Token Compression Visual Token Compression Temporal Video Understanding

Ask PDF