WaveSFNet: A Wavelet-Based Codec and Spatial--Frequency Dual-Domain Gating Network for Spatiotemporal Prediction

Xinyong Cai; Runming Xie; Hu Chen; Yuankai Wu

cs.CVMar 24, 2026

WaveSFNet: A Wavelet-Based Codec and Spatial--Frequency Dual-Domain Gating Network for Spatiotemporal Prediction

Authors: Xinyong Cai, Runming Xie, Hu Chen, Yuankai Wu

Organizations: College of Software Engineering, Sichuan University · College of Computer Science, Sichuan University · Sichuan University National Key Laboratory of Fundamental Science on Synthetic Vision

Abstract

Spatiotemporal predictive learning aims to forecast future frames from historical observations in an unsupervised manner, and is critical to a wide range of applications. The key challenge is to model long-range dynamics while preserving high-frequency details for sharp multi-step predictions. Existing efficient recurrent-free frameworks typically rely on strided convolutions or pooling for sampling, which tends to discard textures and boundaries, while purely spatial operators often struggle to balance local interactions with global propagation. To address these issues, we propose WaveSFNet, an efficient framework that unifies a wavelet-based codec with a spatial--frequency dual-domain gated spatiotemporal translator. The wavelet-based codec preserves high-frequency subband cues during downsampling and reconstruction. Meanwhile, the translator first injects adjacent-frame differences to explicitly enhance dynamic information, and then performs dual-domain gated fusion between large-kernel spatial local modeling and frequency-domain global modulation, together with gated channel interaction for cross-channel feature exchange. Extensive experiments demonstrate that WaveSFNet achieves competitive prediction accuracy on Moving MNIST, TaxiBJ, and WeatherBench, while maintaining low computational complexity. Our code is available at https://github.com/fhjdqaq/WaveSFNet.

Figures & tables

Fig. 1: Performance comparison on the TaxiBJ dataset. Bubble size denotes FLOPs. WaveSFNet achieves the lowest MSE with reduced complexity.

Fig. 2: Overall architecture and core modules of WaveSFNet. WaveSFNet follows an encoder–translator–decoder design. A wavelet-based multi-scale encoder extracts latent features from input frames. A TDI Block injects adjacent-frame differences, and $N_{t}$ stacked ST Blocks apply spatial–frequency dual-domain gating after packing time into channels. A wavelet-symmetric decoder reconstructs predictions.

Dataset		$N_{s}$	$N_{t}$	$C_{s}$	$C_{t}$	$(C,H,W)$	Batch Size	$T_{\mathrm{in}}$	$T_{\mathrm{out}}$	LR	Epochs
Moving MNIST		2	8	64	720	$(1,64,64)$	16	10	10	$9.2\times 10^{-4}$	2000
TaxiBJ		1	8	32	192	$(2,32,32)$	16	4	4	$1.5\times 10^{-3}$	50
WeatherBench	T2M	1	8	32	264	$(1,32,64)$	16	12	12	$2\times 10^{-3}$	50
	TCC	1	8	32	264	$(1,32,64)$	16	12	12	$4\times 10^{-3}$	50
	UV10	1	8	32	264	$(2,32,64)$	16	12	12	$2\times 10^{-3}$	50
	R	1	8	32	264	$(1,32,64)$	16	12	12	$3\times 10^{-3}$	50

TABLE I: Detailed experimental configurations for different datasets.

C_{s}

denotes the base channel width of the wavelet codec, and

C_{t}

denotes the time-packed channel width used by ST Blocks.

Category	Method	MSE $\downarrow$	SSIM $\uparrow$
Recurrent-based	ConvLSTM [ 1 ]	103.3	0.707
	FRNN [ 37 ]	69.7	0.813
	PredRNN [ 8 ]	56.8	0.867
	MIM [ 19 ]	44.2	0.910
	MAU [ 38 ]	27.6	0.937
	PhyDNet [ 20 ]	24.4	0.947

TABLE II: Quantitative comparison of different methods on Moving MNIST.

Fig. 3: Qualitative visualizations of WaveSFNet on TaxiBJ.

Category	Method	Params (M)	FLOPs (G)	MSE $\downarrow$	MAE $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
Recurrent-based	ConvLSTM [ 1 ]	15.0	20.7	0.3358	15.32	0.9836	39.45
	PredRNN [ 8 ]	23.7	42.4	0.3194	15.31	0.9838	39.51
	PredRNN++ [ 9 ]	38.4	63.0	0.3348	15.37	0.9834	39.47
	E3D-LSTM [ 40 ]	51.0	98.19	0.3421	14.98	0.9842	39.64
	PhyDNet [ 20 ]	3.1	5.6	0.3622	15.53	0.9828	39.46
	MIM [ 19 ]	37.9	64.1	0.3110	14.96	0.9847	39.65

TABLE III: Quantitative comparison of different methods on TaxiBJ.

Method	Params (M)	FLOPs (G)	UV10			T2M			TCC			R
Method	Params (M)	FLOPs (G)	MSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$
ConvLSTM [ 1 ]	14.98	136	1.8976	0.9215	1.3775	1.5210	0.7949	1.2330	0.0494	0.1542	0.2223	35.1460	4.0120	5.9280
PredRNN [ 8 ]	23.57	278	1.8810	0.9068	1.3715	1.3310	0.7246	1.1540	0.0550	0.1588	0.2346	37.6110	4.0960	6.1330
PredRNN++ [ 9 ]	38.40	413	1.8727	0.9019	1.3685	1.4580	0.7676	1.2070	0.0548	0.1544	0.2341	45.9930	4.7310	6.7820
MAU [ 38 ]	5.46	39.6	1.9001	0.9194	1.3784	1.2510	0.7036	1.1190	0.0496	0.1516	0.2226	34.5290	4.0040	5.8760
SimVP [ 11 ]	14.67	8.03	1.9993	0.9510	1.4140	1.2380	0.7037	1.1130	0.0477	0.1503	0.2183	34.3550	3.9940	5.8610

TABLE IV: Quantitative comparison of different methods on WeatherBench single-variable tasks, including UV10, T2M, TCC, and R.

Fig. 4: Frequency spectrum analysis on the WeatherBench T2M dataset. The plot displays the Radially Averaged Power Spectral Density, where the x-axis represents the radial spatial frequency and the y-axis denotes the log power spectrum. The inset provides a zoomed-in view of the high-frequency region.

Method	Params (M)	FLOPs (G)	MSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$
ConvLSTM [ 1 ]	15.50	43.33	108.81	5.7439	8.1810
PredRNN [ 8 ]	24.56	88.02	104.16	5.5373	7.9553
PredRNN++ [ 9 ]	39.31	129.0	106.77	5.5821	8.0568
MIM [ 19 ]	41.71	35.77	121.95	6.2786	8.7376
MAU [ 38 ]	5.46	12.07	106.13	5.6487	7.9928
PredRNNv2 [ 10 ]	24.58	88.49	108.94	5.7747	8.1872

TABLE V: Quantitative comparison of different methods on WeatherBench multi-variable forecasting (R, T, U, and V). All metrics are averaged over the four variables.

Variant	TaxiBJ			TCC
Variant	MSE $\downarrow$	MAE $\downarrow$	SSIM $\uparrow$	MSE $\downarrow$	MAE $\downarrow$
Spatial-only	0.2949	14.76	0.9854	0.04604	0.1494
Frequency-only	0.3014	14.72	0.9850	0.04653	0.1518
w/o TDI Block	0.2962	14.76	0.9854	0.04598	0.1489
Conv codec	0.3198	14.92	0.9848	0.04609	0.1472
MLP mixer	0.2997	14.91	0.9850	0.04620	0.1476

TABLE VI: Ablation results on TaxiBJ and TCC.

Explore similar work

PFGNet: A Fully Convolutional Frequency-Guided Peripheral Gating Network for Efficient Spatiotemporal Predictive Learning
Feb 24, 2026Xinyong Cai, Changbin Sun, Yong Wang +2Spatiotemporal Forecasting Video Prediction
College of Computer Science, Sichuan University · Department of Data and Systems Engineering, The University of Hong Kong
Dimensional Balance Improves Large Scale Spatiotemporal Prediction Performance
May 11, 2026Jing Chen, Shixiang Pan, Yujie Fan +3Long-Term Time Series Forecasting Spatiotemporal Forecasting
School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, 310018, China · Key Laboratory of New Industrial Internet Control Technology, Hangzhou, 310018, China · College of Economics, China Jiliang University, Hangzhou, 314423, China
MP3: Multi-Period Pattern Pre-training for Spatio-Temporal Forecasting
Jun 11, 2026Lilan Peng, Yandi Liu, Qingren Yao +2Spatiotemporal Forecasting Multiscale Time-Series Modeling
School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu 611756, China · Eindhoven University of Technology, Eindhoven 5612 AZ, Netherlands

Feb 24, 2026J/K move · Enter open · S save

PFGNet: A Fully Convolutional Frequency-Guided Peripheral Gating Network for Efficient Spatiotemporal Predictive Learning

Xinyong Cai, Changbin Sun, Yong Wang, Hongyu Yang+1

College of Computer Science, Sichuan University · Department of Data and Systems Engineering, The University of Hong Kong

Spatiotemporal Forecasting Video Prediction Convolutional Neural Networks Dynamic Convolution Frequency-Domain Feature Learning Lightweight CNNs

Open paper AskPDF arXiv

Spatiotemporal predictive learning (STPL) aims to forecast future frames from past observations and is essential across a wide range of applications. Compared with recurrent or hybrid architectures, pure convolutional models offer superior efficiency and full parallelism, yet their fixed receptive fields limit their ability to adaptively capture spatially varying motion patterns. Inspired by biological center-surround organization and frequency-selective signal processing, we propose PFGNet, a fully convolutional framework that dynamically modulates receptive fields through pixel-wise frequency-guided gating. The core Peripheral Frequency Gating (PFG) block extracts localized spectral cues and adaptively fuses multi-scale large-kernel peripheral responses with learnable center suppression, effectively forming spatially adaptive band-pass filters. To maintain efficiency, all large kernels are decomposed into separable 1D convolutions (

1 \times k

followed by

k \times 1

), reducing per-channel computational cost from

O(k^2)

O(2k)

. PFGNet enables structure-aware spatiotemporal modeling without recurrence or attention. Experiments on Moving MNIST, TaxiBJ, Human3.6M, and KTH show that PFGNet delivers SOTA or near-SOTA forecasting performance with substantially fewer parameters and FLOPs. Our code is available at https://github.com/fhjdqaq/PFGNet.

Abstract

Explore similar work

PFGNet: A Fully Convolutional Frequency-Guided Peripheral Gating Network for Efficient Spatiotemporal Predictive Learning

Dimensional Balance Improves Large Scale Spatiotemporal Prediction Performance

MP3: Multi-Period Pattern Pre-training for Spatio-Temporal Forecasting