STS-Mixer: Spatio-Temporal-Spectral Mixer for 4D Point Cloud Video Understanding

Wenhao Li; Xueying Jiang; Gongjie Zhang; Xiaoqin Zhang; Ling Shao; Shijian Lu

cs.CVApr 13, 2026

STS-Mixer: Spatio-Temporal-Spectral Mixer for 4D Point Cloud Video Understanding

Authors: Wenhao Li, Xueying Jiang, Gongjie Zhang, Xiaoqin Zhang, Ling Shao, Shijian Lu

Organizations: Nanyang Technological University · Alibaba Group · Zhejiang University of Technology · University of Chinese Academy of Sciences

Abstract

4D point cloud videos capture rich spatial and temporal dynamics of scenes which possess unique values in various 4D understanding tasks. However, most existing methods work in the spatiotemporal domain where the underlying geometric characteristics of 4D point cloud videos are hard to capture, leading to degraded representation learning and understanding of 4D point cloud videos. We address the above challenge from a complementary spectral perspective. By transforming 4D point cloud videos into graph spectral signals, we can decompose them into multiple frequency bands each of which captures distinct geometric structures of point cloud videos. Our spectral analysis reveals that the decomposed low-frequency signals capture more coarse shapes while high-frequency signals encode more fine-grained geometry details. Building on these observations, we design Spatio-Temporal-Spectral Mixer (STS-Mixer), a unified framework that mixes spatial, temporal, and spectral representations of point cloud videos. STS-Mixer integrates multi-band delineated spectral signals with spatiotemporal information to capture rich geometries and temporal dynamics, while enabling fine-grained and holistic understanding of 4D point cloud videos. Extensive experiments show that STS-Mixer achieves superior performance consistently across multiple widely adopted benchmarks on both 3D action recognition and 4D semantic segmentation tasks. Code and models are available at https://github.com/Vegetebird/STS-Mixer.

Figures & tables

Figure 1: Graph spectral analysis of point clouds with Graph Fourier Transform (GFT) and inverse GFT (IGFT). The high-frequency bands of point clouds encode fine-grained geometric details which can be observed by the difference between the original point cloud and the one w/o high frequency. The low-frequency bands capture most energy as well as the coarse shapes which can be observed by the similarity between the original point cloud and the one w/o mid-high frequency.

Figure 2: The framework of the proposed STS-Mixer. STS-Mixer decomposes 4D point cloud videos into multi-band signals and processes them to better capture information across spatial, spectral, and temporal spaces. Each STS-Mixer block comes with Frequency-Aware Attention (FA-Attention) and Frequency-Mixing MLP (FM-MLP), the former refines the spectral representation of each individual frequency band independently and the latter facilitates information exchange across the spectral representations of different frequency bands for mutual enhancement.

Figure 3: Visualization of point clouds. The first row shows the original point cloud, and the subsequent rows show the reconstructed point clouds where high-frequency bands are progressively removed.

Method	Accuracy
MeteorNet (ICCV’19) [ 26 ]	88.50
P4Transformer (CVPR’21) [ 9 ]	90.94
PSTNet (ICLR’21) [ 10 ]	91.20
PSTNet++ (TPAMI’21) [ 11 ]	92.68
Kinet (CVPR’22) [ 48 ]	93.27
PST-Transformer (TPAMI’23) [ 12 ]	93.73

Table 1: Comparisons of 3D action recognition on the benchmark MSR-Action3D.

Method	mIoU (%)
4D MinkNet14 (CVPR’19) [ 5 ]	77.46
PointNet++ (NeurIPS’17) [ 31 ]	79.35
MeteorNet (ICCV’19) [ 26 ]	81.47
PSTNet (ICLR’21) [ 10 ]	82.24
P4Transformer (CVPR’21) [ 9 ]	83.16
PST-Transformer (TPAMI’23) [ 12 ]	83.95

Table 2: Comparisons of 4D semantic segmentation on the benchmark Synthia4D.

Method	GFT	FA-Attention	FM-MLP	Accuracy (%)
Baseline [ 12 ]				94.11
Baseline w. GFT	✓			94.70
STS-Mixer w/o GFT		✓	✓	94.50
STS-Mixer w/o FM-MLP	✓	✓		95.12
STS-Mixer w/o FA-Attention	✓		✓	95.37
STS-Mixer (Ours)	✓	✓	✓	95.85

Table 3: Ablation study on different components of the proposed STS-Mixer.

Method	Accuracy (%)
STS-Mixer w/o Mid and High Frequency	94.42
STS-Mixer w/o Low and High Frequency	94.07
STS-Mixer w/o Low and Mid Frequency	93.72
STS-Mixer w/o Low Frequency	94.34
STS-Mixer w/o Mid Frequency	94.77
STS-Mixer w/o High Frequency	95.12

Table 4: Ablation study on different frequency bands.

$(f_{l},f_{h})$	Accuracy (%)
(2, 10)	93.91
(4, 10)	95.16
(6, 10)	95.85
(8, 10)	95.33
(6, 8)	95.26
(6, 10)	95.85

Table 5: Experiments on frequency bands, where

f_{l}

and

f_{h}

denote the low-frequency threshold and high-frequency threshold, respectively.

Figure 4: Visualization of frequency band division.

Figure 5: Qualitative comparison with state-of-the-art methods (PST-Transformer [ 12 ] and Mamba4D [ 25 ] ) on Synthia4D. Wrong estimations are highlighted by red circles.

$L$	$C$	Params (M)	FLOPs (G)	Accuracy (%)
3	64	0.62	0.95	93.48
3	128	2.47	3.72	95.85
3	256	9.86	14.69	93.73
3	512	39.39	58.37	92.26
3	1024	157.42	232.70	93.42
2	128	1.68	2.51	94.36

Table 6: Experiments on the involved hyperparameters including the number of STS-Mixer blocks

L

and the number of channels

C

$K$	6	8	10	12	14	16
Accuracy (%)	93.94	94.74	95.85	93.84	93.73	93.38

Table 7: Experiments on the neighbor number

K

Explore similar work

SRENet: Spectral Re-Entry Network for Point Cloud Action Recognition
Jun 2, 2026Qiuxia Wu, Jiarui Lan, Wenxiong Kang +2Frequency-Domain Feature Learning Point Cloud Learning
School of Software Engineering, South China University of Technology, Guangzhou 511400, China · School of Automation Science and Engineering, South China University of Technology, Guangzhou 511400, China · School of Computer Science, University of Sydney, Sydney, NSW 2006, Australia +1
GATS: Gaussian Aware Temporal Scaling Transformer for Invariant 4D Spatio-Temporal Point Cloud Representation
Mar 17, 2026Jiayi Tian, Jiaze WangPoint Cloud Learning Invariant Representation Learning
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Harbin Institute of Technology, Shenzhen, Pengcheng Laboratory
Velox: Learning Representations of 4D Geometry and Appearance
May 6, 2026Anagh Malik, Dorian Chan, Xiaoming Zhao +3Representation Learning Point Cloud Learning
Apple · University of Toronto

Jun 2, 2026J/K move · Enter open · S save

SRENet: Spectral Re-Entry Network for Point Cloud Action Recognition

Qiuxia Wu, Jiarui Lan, Wenxiong Kang, Zhiyong Wang+1

School of Software Engineering, South China University of Technology, Guangzhou 511400, China · School of Automation Science and Engineering, South China University of Technology, Guangzhou 511400, China · School of Computer Science, University of Sydney, Sydney, NSW 2006, Australia+1

Frequency-Domain Feature Learning Point Cloud Learning Point Cloud Understanding

Open paper AskPDF arXiv

Recognizing human actions from point cloud sequences is critical for 3D perception driven applications such as autonomous driving and human-computer interaction. However, the irregular structure and temporal inconsistency of point clouds pose unique challenges for spatio-temporal representation learning, especially in capturing both global motion context and fine-grained temporal dynamics. We propose SRENet, a spectral-aware framework designed to explicitly learn both global context and fine-grained temporal dynamics of motion from a frequency perspective for action recognition. SRENet introduces a Spectral Decomposition Block (SDeBlock) that performs wavelet-based analysis along temporal and spatial axes, disentangling features into low- and high-frequency components with frequency-specific attention. To recover residual dynamics and re-align temporal frequency structures distorted during semantic fusion, a Spectral Re-entry Block (SReBlock) performs secondary temporal decomposition. Furthermore, a spectral-aware learning strategy is devised to enhance discriminability in both frequency subspaces via contrastive loss and a curriculum schedule that gradually shifts focus from low- to high-frequency spaces in line with coarse to detailed motion patterns. Extensive experiments on MSR-Action3D, NTU-RGBD and NTU-RGBD120 demonstrate that SRENet achieves state-of-the-art performance, validating the effectiveness of frequency modeling in point cloud-based action understanding.

Abstract

Explore similar work

SRENet: Spectral Re-Entry Network for Point Cloud Action Recognition

GATS: Gaussian Aware Temporal Scaling Transformer for Invariant 4D Spatio-Temporal Point Cloud Representation

Velox: Learning Representations of 4D Geometry and Appearance