PFGNet: A Fully Convolutional Frequency-Guided Peripheral Gating Network for Efficient Spatiotemporal Predictive Learning

Xinyong Cai; Changbin Sun; Yong Wang; Hongyu Yang; Yuankai Wu

cs.CVFeb 24, 2026

PFGNet: A Fully Convolutional Frequency-Guided Peripheral Gating Network for Efficient Spatiotemporal Predictive Learning

Authors: Xinyong Cai, Changbin Sun, Yong Wang, Hongyu Yang, Yuankai Wu

Organizations: College of Computer Science, Sichuan University · Department of Data and Systems Engineering, The University of Hong Kong

Abstract

Spatiotemporal predictive learning (STPL) aims to forecast future frames from past observations and is essential across a wide range of applications. Compared with recurrent or hybrid architectures, pure convolutional models offer superior efficiency and full parallelism, yet their fixed receptive fields limit their ability to adaptively capture spatially varying motion patterns. Inspired by biological center-surround organization and frequency-selective signal processing, we propose PFGNet, a fully convolutional framework that dynamically modulates receptive fields through pixel-wise frequency-guided gating. The core Peripheral Frequency Gating (PFG) block extracts localized spectral cues and adaptively fuses multi-scale large-kernel peripheral responses with learnable center suppression, effectively forming spatially adaptive band-pass filters. To maintain efficiency, all large kernels are decomposed into separable 1D convolutions ( $1 \times k$ followed by $k \times 1$ ), reducing per-channel computational cost from $O(k^2)$ to $O(2k)$ . PFGNet enables structure-aware spatiotemporal modeling without recurrence or attention. Experiments on Moving MNIST, TaxiBJ, Human3.6M, and KTH show that PFGNet delivers SOTA or near-SOTA forecasting performance with substantially fewer parameters and FLOPs. Our code is available at https://github.com/fhjdqaq/PFGNet.

Figures & tables

Figure 1: Performance-efficiency trade-off on TaxiBJ. Bubble size denotes FLOPs. PFGNet achieves SOTA MSE with minimal cost.

Figure 2: Overall architecture and core modules of PFGNet. The model follows a SimVP-style encoder–translator–decoder pipeline. The input sequence $\{\mathbf{I}_{t}\}_{t=1}^{T_{\mathrm{in}}}$ is encoded into latent features, temporally packed, and processed by a MSInit followed by $N_{t}$ PFG blocks.

Figure 3: Frequency-Guided Peripheral Gating in the PFG block. Three local spectral cues (gradient magnitude, Laplacian, local variance) are extracted via fixed depthwise filters, channel-averaged, concatenated into a 3-channel frequency map, and passed through a $1\!\times\!1$ conv to produce per-pixel gate logits. Softmax over the $K$ scales yields selection weights $\boldsymbol{\alpha}_{k}$ (the argmax visualization is only used to illustrate the preferred scale).

Dataset	Resolution	$T_{\mathrm{in}}$	$T_{\mathrm{out}}$	$N_{s}$	$N_{t}$	DropPath	Epochs	Learning Rate	Batch Size	GPU
Moving MNIST	$(64,64,1)$	10	10	4	8	0	2000	$1\mathrm{e}{-3}$	16	$1\times$ RTX 4090
TaxiBJ	$(32,32,2)$	4	4	2	8	0.1	50	$2\mathrm{e}{-3}$	16	$1\times$ RTX 3090
KTH	$(128,128,1)$	10	20/40	2	6	0.1	100	$2\mathrm{e}{-4}/1\mathrm{e}{-4}$	2/1	$4\times$ RTX 4090
Human3.6M	$(256,256,3)$	4	4	4	8	0.1	50	$1.5\mathrm{e}{-3}$	2	$5\times$ RTX 3090

Table 1: Experimental setup.

T_{\mathrm{in}}

and

T_{\mathrm{out}}

denote the numbers of input and predicted frames, respectively.

Method	MSE $\downarrow$	SSIM $\uparrow$
Recurrent-based Methods
ConvLSTM [ 47 ]	103.3	0.707
DFN [ 22 ]	89.0	0.726
FRNN [ 43 ]	69.7	0.813
VPN [ 25 ]	64.1	0.870
PredRNN [ 61 ]	56.8	0.867

Table 2: Quantitative comparison on the Moving MNIST dataset.

Figure 4: Qualitative results of PFGNet on Moving MNIST.

Method	Params	FLOPs	MSE $\downarrow$	MAE $\downarrow$	SSIM $\uparrow$
Recurrent-based Methods
ConvLSTM [ 47 ]	15.0M	20.7G	0.3358	15.32	0.9836
PredNet [ 38 ]	12.5M	0.9G	0.3516	15.91	0.9828
PredRNN [ 61 ]	23.7M	42.4G	0.3194	15.31	0.9838
PredRNN++ [ 59 ]	38.4M	63.0G	0.3348	15.37	0.9834
E3D-LSTM [ 60 ]	51.0M	98.2G	0.3421	14.98	0.9842

Table 3: Quantitative comparison on the TaxiBJ dataset.

Figure 5: Qualitative results of PFGNet on TaxiBJ.

Method	KTH (10 $\rightarrow$ 20)		KTH (10 $\rightarrow$ 40)
Method	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$
Recurrent-based Methods
ConvLSTM [ 47 ]	0.712	23.58	0.639	22.85
SAVP [ 28 ]	0.746	25.38	0.701	23.97
FRNN [ 43 ]	0.771	26.12	0.678	23.77
DFN [ 22 ]	0.794	27.26	0.652	23.01

Table 4: Quantitative comparison on the KTH dataset.

Figure 6: Qualitative results of PFGNet on KTH.

Method	Params	FLOPs	MSE $\downarrow$	MAE $\downarrow$	SSIM $\uparrow$
Recurrent-based Methods
ConvLSTM [ 47 ]	15.5M	347.0G	125.5	1566.7	0.9813
E3D-LSTM [ 60 ]	60.9M	542.0G	143.3	1442.5	0.9803
PredNet [ 38 ]	12.5M	13.7G	261.9	1625.3	0.9786
PhyDNet [ 16 ]	4.2M	19.1G	125.7	1614.7	0.9804
MAU [ 4 ]	20.2M	105.0G	127.3	1577.0	0.9812

Table 5: Quantitative comparison on the Human3.6M dataset.

Figure 7: Qualitative results of PFGNet on Human3.6M.

		model1	model2	model3	PFGNet
Archi	MSInit		✓	✓	✓
Fusion	mean		✓
Fusion	softmax	✓		✓	✓
Center	$3{\times}3$	✓	✓		✓
Center	$5{\times}5$			✓
TaxiBJ	MSE $\downarrow$	0.3119	0.3033	0.2995	0.2881

Table 6: Ablation on macro structure. We analyze the roles of MSInit and different fusion strategies under a unified kernel setting (

\mathcal{K}=\{9,15,31\}

Panel of Figure 8 — Figure 8: Ablation results on TaxiBJ.

		model1	model2	model3	model4	PFGNet
$\beta$	$-1$	✓
	$1$		✓
	$0$			✓
	learnable				✓	✓
Gate	tanh	✓	✓	✓		✓
Gate	sigmoid				✓

Table 7: Ablation on detail mechanisms under full configuration (MSInit + softmax + all operators). We compare fixed vs. learnable

\beta

, and activation functions used for the suppression gate.

Figure 9: Frequency response of the PFG block. The large-kernel response

H_{L_{k}}

decays slowly, while the small-kernel response

H_{S}

decays rapidly. Their difference

H_{k}=H_{L_{k}}-\beta_{k}H_{S}

(with

\beta_{k}{=}0.75

) is positive within the green region, forming a ring-shaped band-pass filter as stated in the main text.

Panel of Figure 10 — Figure 10: Visualization of $\tanh(\beta)$ on Moving MNIST and TaxiBJ datasets. Each pair shows (a,c) the branch-wise $\tanh(\beta)$ distributions and (b,d) the smoothed channel-wise count curves, where the horizontal axis denotes $\tanh(\beta)$ values and the vertical axis indicates the number of channels.

Method	Params	FLOPs	MSE $\downarrow$	MAE $\downarrow$	SSIM $\uparrow$
Recurrent-based Methods
ConvLSTM-S [ 47 ]	15.0M	56.8G	28.87	113.20	0.8793
ConvLSTM-L [ 47 ]	33.8M	127.0G	25.51	104.85	0.8928
PredNet [ 38 ]	12.5M	8.6G	185.94	318.30	0.6713
PhyDNet [ 16 ]	3.1M	15.3G	34.75	125.66	0.8567
PredRNN [ 61 ]	23.8M	116.0G	22.01	91.74	0.9091

Table 8: Quantitative comparison on the Moving FMNIST dataset.

Figure 11: Qualitative results of PFGNet on Moving FMNIST.

Figure 12: Ablation on asymmetric convolution (

n\times k+k\times n

Dataset	$N_{t}$	Params	FLOPs	MSE $\downarrow$	MAE $\downarrow$	SSIM $\uparrow$
Moving MNIST (100 epochs)	2	11.0M	7.4G	34.70	93.93	0.9201
	4	21.1M	10.0G	29.74	83.38	0.9326
	6	31.2M	12.6G	28.33	80.14	0.9363
	8	41.3M	15.2G	27.61	78.54	0.9381
	10	51.3M	17.7G	27.21	77.57	0.9391
TaxiBJ	2	0.53M	0.21G	0.3669	15.88	0.9826

Table 9: Ablation on the number of PFG blocks

N_{t}

on Moving MNIST and TaxiBJ.

Dataset	$N_{t}$	Params	FLOPs	SSIM $\uparrow$	PSNR $\uparrow$
KTH 10 $\rightarrow$ 20	2	10.8M	57.1G	0.908	33.88
	4	20.8M	98.5G	0.910	34.01
	6	30.9M	0.14T	0.911	34.10
	8	41.0M	0.18T	0.909	34.10
	10	51.1M	0.22T	0.912	34.24

Table 10: Ablation on the number of PFG blocks

N_{t}

on KTH (10

\rightarrow

20).

Figure 13: Qualitative visualizations on TaxiBJ.

Figure 14: Mechanistic visualization of the learned PFGNet.

Explore similar work

WaveSFNet: A Wavelet-Based Codec and Spatial--Frequency Dual-Domain Gating Network for Spatiotemporal Prediction
Mar 24, 2026Xinyong Cai, Runming Xie, Hu Chen +1Spatiotemporal Forecasting Convolutional Neural Networks
College of Software Engineering, Sichuan University · College of Computer Science, Sichuan University · Sichuan University National Key Laboratory of Fundamental Science on Synthetic Vision
MP3: Multi-Period Pattern Pre-training for Spatio-Temporal Forecasting
Jun 11, 2026Lilan Peng, Yandi Liu, Qingren Yao +2Spatiotemporal Forecasting Multiscale Time-Series Modeling
School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu 611756, China · Eindhoven University of Technology, Eindhoven 5612 AZ, Netherlands
ST-GRU PredRNN-V2: A GRU-Based Reformulation of PredRNN-V2 for Video Prediction
Oct 13, 2026Augusta van Haren, Hanna Hoogen, Luca Pattavina +1Video Prediction Recurrent Neural Networks
Radboud University

Mar 24, 2026J/K move · Enter open · S save

WaveSFNet: A Wavelet-Based Codec and Spatial--Frequency Dual-Domain Gating Network for Spatiotemporal Prediction

Xinyong Cai, Runming Xie, Hu Chen, Yuankai Wu

College of Software Engineering, Sichuan University · College of Computer Science, Sichuan University · Sichuan University National Key Laboratory of Fundamental Science on Synthetic Vision

Spatiotemporal Forecasting Convolutional Neural Networks Frequency-Domain Feature Learning

Open paper AskPDF arXiv

Spatiotemporal predictive learning aims to forecast future frames from historical observations in an unsupervised manner, and is critical to a wide range of applications. The key challenge is to model long-range dynamics while preserving high-frequency details for sharp multi-step predictions. Existing efficient recurrent-free frameworks typically rely on strided convolutions or pooling for sampling, which tends to discard textures and boundaries, while purely spatial operators often struggle to balance local interactions with global propagation. To address these issues, we propose WaveSFNet, an efficient framework that unifies a wavelet-based codec with a spatial--frequency dual-domain gated spatiotemporal translator. The wavelet-based codec preserves high-frequency subband cues during downsampling and reconstruction. Meanwhile, the translator first injects adjacent-frame differences to explicitly enhance dynamic information, and then performs dual-domain gated fusion between large-kernel spatial local modeling and frequency-domain global modulation, together with gated channel interaction for cross-channel feature exchange. Extensive experiments demonstrate that WaveSFNet achieves competitive prediction accuracy on Moving MNIST, TaxiBJ, and WeatherBench, while maintaining low computational complexity. Our code is available at https://github.com/fhjdqaq/WaveSFNet.

Abstract

Explore similar work

WaveSFNet: A Wavelet-Based Codec and Spatial--Frequency Dual-Domain Gating Network for Spatiotemporal Prediction

MP3: Multi-Period Pattern Pre-training for Spatio-Temporal Forecasting

ST-GRU PredRNN-V2: A GRU-Based Reformulation of PredRNN-V2 for Video Prediction