ST-GRU PredRNN-V2: A GRU-Based Reformulation of PredRNN-V2 for Video Prediction

Augusta van Haren; Hanna Hoogen; Luca Pattavina; Quincy Rommen

cs.CVOct 13, 2026

ST-GRU PredRNN-V2: A GRU-Based Reformulation of PredRNN-V2 for Video Prediction

Authors: Augusta van Haren, Hanna Hoogen, Luca Pattavina, Quincy Rommen

Organizations: Radboud University

Abstract

Video prediction requires models to capture complex spatiotemporal dependencies and to retain relevant information over long sequences. PredRNN and its successor PredRNN-V2 address this challenge with spatiotemporal Long Short-Term Memory (ST-LSTM) units that maintain two separate memory states, at the cost of a relatively complex recurrent building block. In this work, we investigate whether the simpler gating of Gated Recurrent Units (GRUs) can be transferred to spatiotemporal predictive learning. To this end, we develop a spatiotemporal GRU (ST-GRU) unit as a GRU-based alternative to the ST-LSTM. The ST-GRU replaces the ST-LSTM's three gates per memory state with two GRU-style gates (update and reset), and merges the temporal memory state into the hidden state, which reduces the number of parameters by about 5%. We incorporate the ST-GRU into PredRNN-V2 and evaluate the resulting model on the Moving MNIST video prediction benchmark, following the original experimental protocol. Averaged over ten predicted frames, the ST-GRU-based PredRNN-V2 achieves an MSE of 55.1, an SSIM of 0.883, an LPIPS of 0.065 and a PSNR of 19.53dB. It does not surpass the original ST-LSTM-based PredRNN-V2 on MSE, SSIM or PSNR, but it achieves a lower LPIPS and performs on par with the earlier ST-LSTM-based PredRNN, while using a simpler recurrent cell. These results suggest that ST-GRUs are a promising building block for spatiotemporal predictive learning and motivate further work on their design and integration.

Figures & tables

Figure 1: Frame-wise metrics (MSE, SSIM, LPIPS, PSNR) of the ST-GRU-based PredRNN-V2 on the Moving MNIST test set over 80,000 training iterations. The frame number indicates the number of time steps between the predicted frame and the last observed frame.

Model	MSE ( $\downarrow$ )	SSIM ( $\uparrow$ )	LPIPS ( $\downarrow$ )	PSNR ( $\uparrow$ )
ConvLSTM (RNN) ( Shi et al., 2015 )	103.3	0.707	0.156	–
ST-LSTM (PredRNN) ( Wang et al., 2017 )	56.8	0.867	0.107	–
ST-LSTM (PredRNN-V2) ( Wang et al., 2021 )	48.4	0.891	0.071	20.05
ConvGRU (RNN) ( Ballas et al., 2016 )	57.8	0.839	0.091	–
ST-GRU (PredRNN) ( Dong et al., 2025 )	53.4	0.893	0.057	–
ST-GRU (PredRNN-V2), ours	55.1	0.883	0.065	19.53

Table 1: Model performance on the Moving MNIST test set. All metrics are averaged over ten predicted frames. The best result for each metric is shown in bold, and our model in teal . Results for the LSTM-based baselines are taken from Wang et al. (2021) and for the GRU-based baselines from Dong et al. (2025) . PSNR (in dB) was computed only for the two PredRNN-V2 variants.

Figure 2: Frame-wise MSE, SSIM, LPIPS and PSNR on the Moving MNIST test set. All models were trained and evaluated following the protocol described in Section 2.5 . Results for the LSTM-based models are taken from Wang et al. (2021) , and those for the GRU-based models from Dong et al. (2025) . PSNR for the ST-LSTM-based PredRNN-V2 was computed with the official trained model ( Wu, 2022 ) . All results for the ST-GRU-based PredRNN-V2 are from our own evaluation.

Figure 3: Predictions of the ST-GRU-based PredRNN-V2 on five Moving MNIST test sequences, showing the input sequence, the ground-truth future sequence and the predicted future sequence. Both global and local similarity to the ground truth decline for frames further into the future.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Side-by-side mathematical comparison of the ST-LSTM and ST-GRU building blocks.

Panel of Figure 5 — Figure 5: Side-by-side comparison of the ST-LSTM-based and ST-GRU-based PredRNN(-V2) architectures, with the flows of information shown in different colours. The ST-LSTM-based PredRNN(-V2) has two memory states: the temporal memory $\mathcal{C}_{t}^{l}$ flows horizontally from the previous time step to the current one ( $t$ ), whereas the spatiotemporal memory $\mathcal{M}_{t}^{l}$ zigzags through the layers ( $l$ ) of PredRNN(-V2), capturing short-term dynamics. Both memory states are used to update the hidden state $\mathcal{H}_{t}^{l}$ . The ST-GRU-based PredRNN(-V2), in contrast, integrates the horizontal memory flow into the hidden state $\mathcal{H}_{t}^{l}$ , while the spatiotemporal memory is still carried by $\mathcal{M}_{t}^{l}$ .

Panel of Figure 6 — Figure 6: Side-by-side comparison of the ST-LSTM and ST-GRU building blocks, with the flows of information shown in different colours. The ST-LSTM has two memory states: the temporal memory $\mathcal{C}_{t}^{l}$ across time steps ( $t$ ), and the spatiotemporal memory $\mathcal{M}_{t}^{l}$ , which captures short-term dynamics across layers ( $l$ ). Each memory state is modulated by three gates, $i_{t}$ , $f_{t}$ and $g_{t}$ (respectively $i^{\prime}_{t}$ , $f^{\prime}_{t}$ and $g^{\prime}_{t}$ ), and both contribute to the hidden state $\mathcal{H}_{t}^{l}$ via the output gate $o_{t}$ . The ST-GRU, in contrast, integrates the horizontal memory flow into the hidden state $\mathcal{H}_{t}^{l}$ , while the spatiotemporal memory is still carried by $\mathcal{M}_{t}^{l}$ . Each of these states is modulated by two gates, $z_{t}$ and $r_{t}$ (respectively $z^{\prime}_{t}$ and $r^{\prime}_{t}$ ), and they are combined via the output gate $o_{t}$ . The reduction in both gates and memory states is what makes the ST-GRU simpler than the ST-LSTM.

Explore similar work

Towards Long-Form Spatio-Temporal Video Grounding
Feb 26, 2026Xin Gu, Bing Fan, Jiali Yao +5Temporal Video Grounding Long-Context Modeling
University of Chinese Academy of Sciences · 2UNT · Institute of Software Chinese Academy of Sciences +2
PFGNet: A Fully Convolutional Frequency-Guided Peripheral Gating Network for Efficient Spatiotemporal Predictive Learning
Feb 24, 2026Xinyong Cai, Changbin Sun, Yong Wang +2Spatiotemporal Forecasting Video Prediction
College of Computer Science, Sichuan University · Department of Data and Systems Engineering, The University of Hong Kong
Enhancing Visual Token Representations for Video Large Language Models via Training-Free Spatial-Temporal Pooling and Gridding
May 21, 2026Bingjun Luo, Tony Wang, Hanqi Chen +1Video-Language Models Multimodal Token Compression
1Tsinghua University · 2Shenzhen University · 3Xidian University

Feb 26, 2026J/K move · Enter open · S save

Towards Long-Form Spatio-Temporal Video Grounding

Xin Gu, Bing Fan, Jiali Yao, Zhipeng Zhang+4

University of Chinese Academy of Sciences · 2UNT · Institute of Software Chinese Academy of Sciences+2

Temporal Video Grounding Long-Context Modeling Vision-Language Grounding

Open paper AskPDF arXiv

In real scenarios, videos can span several minutes or even hours. However, existing research on spatio-temporal video grounding (STVG), given a textual query, mainly focuses on localizing targets in short videos of tens of seconds, typically less than one minute, which limits real-world applications. In this paper, we explore Long-Form STVG (LF-STVG), which aims to locate targets in long-term videos. Compared with short videos, long-term videos contain much longer temporal spans and more irrelevant information, making it difficult for existing STVG methods that process all frames at once. To address this challenge, we propose an AutoRegressive Transformer architecture for LF-STVG, termed ART-STVG. Unlike conventional STVG methods that require the entire video sequence to make predictions at once, ART-STVG treats the video as streaming input and processes frames sequentially, enabling efficient handling of long videos. To model spatio-temporal context, we design spatial and temporal memory banks and apply them to the decoders. Since memories from different moments are not always relevant to the current frame, we introduce simple yet effective memory selection strategies to provide more relevant information to the decoders, significantly improving performance. Furthermore, instead of parallel spatial and temporal localization, we propose a cascaded spatio-temporal design that connects the spatial decoder to the temporal decoder, allowing fine-grained spatial cues to assist complex temporal localization in long videos. Experiments on newly extended LF-STVG datasets show that ART-STVG significantly outperforms state-of-the-art methods, while achieving competitive performance on conventional short-form STVG. Our code is at: https://github.com/HengLan/ART-STVG.

Abstract

Explore similar work

Towards Long-Form Spatio-Temporal Video Grounding

PFGNet: A Fully Convolutional Frequency-Guided Peripheral Gating Network for Efficient Spatiotemporal Predictive Learning

Enhancing Visual Token Representations for Video Large Language Models via Training-Free Spatial-Temporal Pooling and Gridding