GATS: Gaussian Aware Temporal Scaling Transformer for Invariant 4D Spatio-Temporal Point Cloud Representation
Authors: Jiayi Tian, Jiaze Wang
Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Harbin Institute of Technology, Shenzhen, Pengcheng Laboratory
Understanding 4D point cloud videos is essential for enabling intelligent agents to perceive dynamic environments. However, temporal scale bias across varying frame rates and distributional uncertainty in irregular point clouds make it highly challenging to design a unified and robust 4D backbone. Existing CNN or Transformer based methods are constrained either by limited receptive fields or by quadratic computational complexity, while neglecting these implicit distortions. To address this problem, we propose a novel dual invariant framework, termed Gaussian Aware Temporal Scaling (GATS), which explicitly resolves both distributional inconsistencies and temporal. The proposed Uncertainty Guided Gaussian Convolution (UGGC) incorporates local Gaussian statistics and uncertainty aware gating into point convolution, thereby achieving robust neighborhood aggregation under density variation, noise, and occlusion. In parallel, the Temporal Scaling Attention (TSA) introduces a learnable scaling factor to normalize temporal distances, ensuring frame partition invariance and consistent velocity estimation across different frame rates. These two modules are complementary: temporal scaling normalizes time intervals prior to Gaussian estimation, while Gaussian modeling enhances robustness to irregular distributions. Our experiments on mainstream benchmarks MSR-Action3D (+6.62% accuracy), NTU RGBD (+1.4% accuracy), and Synthia4D (+1.8% mIoU) demonstrate significant performance gains, offering a more efficient and principled paradigm for invariant 4D point cloud video understanding with superior accuracy, robustness, and scalability compared to Transformer based counterparts.
Figures & tables
Figure 1: Motivation. Right: The right panel illustrates video sequences under different frame rate partitions: with a large temporal interval, some moving objects (second row) and static objects (third column) disappear, while they remain visible with a small temporal interval. This indicates that varying frame rate partitions may cause certain velocity features to vanish. Left: The left panel illustrates that GATS can adaptively adjust the scaling distribution across different frame rate partitions, thereby effectively mitigating the relative velocity bias introduced by frame rate variations and reducing fluctuations in accuracy. Consequently, GATS achieves improvements in ACC of 6.62% and 3.83% over P4D and PST, respectively.
Figure 2: Pipeline. The overall network backbone consists of two core modules: (a) UGGC Module. After the point cloud is fed into the network, the spatial variations of xit generate cross frame representations. However, different cross frames often lead to inter frame inconsistencies. The UGGC module extracts local Gaussian features and incorporates an uncertainty aware gating mechanism to jointly model geometric and Gaussian local features of 4D point clouds, thereby enhancing the robustness of feature extraction. (b) TSA Module. Under different frame rates, the estimation of relative velocity it varies, and as the temporal dimension progresses, motion features tend to produce inter frame inconsistencies. To address this, the TSA module introduces a learnable scaling factor s to normalize temporal distances, achieving frame partition invariance and ensuring consistent relative velocity estimation across varying frame rates.
Figure 3: MSR-attention Results. Attention visualization showing focus on key regions and spatio-temporal dynamics.
Methods
Input
Frames
Accuracy (%)
Actionlet [ 44 ]
skeleton
all
88.21
MeteorNet [ 25 ]
point
12
86.53
PSTNet [ 6 ]
point
12
87.88
P4Transformer [ 5 ]
point
12
87.54
PST-Transformer [ 8 ]
point
12
88.15
GATS (Ours)
point
12
90.24
Table 1: Action recognition accuracy comparison (%) on MSR-Action3D.
Method
Input
Acc (%)
SkeleMotion [ 2 ]
skeleton
69.6
GCA-LSTM [ 20 ]
skeleton
74.4
AttentionLSTM [ 19 ]
skeleton
77.1
AGC-LSTM [ 41 ]
skeleton
89.2
AS-GCN [ 15 ]
skeleton
86.8
VA-fusion [ 55 ]
skeleton
89.4
Table 2: Action recognition accuracy (%) on NTU RGBD.
Method
Input
Frame
Track
Bldn
Road
Sdwlk
Fence
Vegittn
Pole
-
3D MinkNet14
voxel
1
-
89.39
97.68
69.43
86.52
98.11
97.26
-
4D MinkNet14
voxel
3
-
90.13
98.26
73.47
87.19
99.10
97.50
-
PointNet++
point
1
-
96.88
97.72
86.20
92.75
97.12
97.09
-
MeteorNet-m
point
2
✓
98.22
97.79
90.98
93.18
98.31
97.45
-
MeteorNet-m
point
2
×
97.65
97.83
90.03
94.06
97.41
97.79
-
MeteorNet-l
point
3
×
98.10
97.72
88.65
94.00
97.98
97.65
-
Table 3: 4D semantic segmentation results (mIoU %) on the Synthia 4D dataset.
Figure 4: 4D Qualitative Results. The rows from top to bottom correspond to the input, GT, and our predictions. Detailed comparative results are highlighted in the enlarged regions of the figure.
Model
frame
Acc(%)
(a) Full Model
24
97.56
(b) w/o UGGC
24
95.12
(c) w/o TSA
24
96.16
(d) PSTTransformer (baseline)
24
93.73
Table 4: Ablation studies on MSR-Action3D.
Method
Backbone
Frame
Acc(%)
PvNeXt
CNN
24
94.77
MAMBA4D
Mamba
24
93.38
Ours
Transformer
24
97.56
Table 5: Comparison results with advanced architecture processing more frames.
Figure 5: Analysis of accuracy with varying temporal and spatial parameters.
Figure 6: Structure of GATS. The left side shows the overall process, and the right side shows the specific process of DATS.
Methods
Input
Accuracy (%)
Vieira et al.
depth
78.20
Kläser et al.
depth
81.43
Actionlet
skeleton
88.21
PointNet++
point
61.61
MeteorNet
point
88.50
GATS(Ours)
point
97.56
Table 6: Action recognition accuracy comparison on a standard benchmark.
Recent 4D Gaussian Splatting (4DGS) methods achieve impressive dynamic scene reconstruction but often rely on piecewise linear velocity approximations and short temporal windows. This disjointed modeling leads to severe temporal fragmentation, forcing primitives to be repeatedly eliminated and regenerated to track complex nonlinear dynamics. This makeshift approximation eliminates the long-term temporal identity of objects and causes an inevitable proliferation of Gaussians, hindering scalability to extended video sequences. To address this, we propose TRiGS, a novel 4D representation that utilizes unified, continuous geometric transformations. By integrating SE(3) transformations, hierarchical Bezier residuals, and learnable local anchors, TRiGS models geometrically consistent rigid motions for individual primitives. This continuous formulation preserves temporal identity and effectively mitigates unbounded memory growth. Extensive experiments demonstrate that TRiGS achieves high fidelity rendering on standard benchmarks while uniquely scaling to extended video sequences (e.g., 600 to 1200 frames) without severe memory bottlenecks, significantly outperforming prior works in temporal stability.
Suwoong Yeom, Joonsik Nam, Seunggyu Choi +7
Sogang University · POSTECH · Seoul National University +2
Point cloud video understanding is critical for robotics as it accurately encodes motion and scene interaction. We recognize that 4D datasets are far scarcer than 3D ones, which hampers the scalability of self-supervised 4D models. A promising alternative is to transfer 3D pre-trained models to 4D perception tasks. However, rigorous empirical analysis reveals two critical limitations that impede transfer capability: overfitting and the modality gap. To overcome these challenges, we develop a novel "Align then Adapt" (PointATA) paradigm that decomposes parameter-efficient transfer learning into two sequential stages. Optimal-transport theory is employed to quantify the distributional discrepancy between 3D and 4D datasets, enabling our proposed point align embedder to be trained in Stage 1 to alleviate the underlying modality gap. To mitigate overfitting, an efficient point-video adapter and a spatial-context encoder are integrated into the frozen 3D backbone to enhance temporal modeling capacity in Stage 2. Notably, with the above engineering-oriented designs, PointATA enables a pre-trained 3D model without temporal knowledge to reason about dynamic video content at a smaller parameter cost compared to previous work. Extensive experiments show that PointATA can match or even outperform strong full fine-tuning models, whilst enjoying the advantage of parameter efficiency, e.g. 97.21 % accuracy on 3D action recognition, +8.7% on 4 D action segmentation, and 84.06% on 4D semantic segmentation.
Yiding Sun, Jihua Zhu, Haozhe Cheng +4
the School of Software Engineering, Xi’an Jiaotong University, Xi’an 710048, China · the State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an 710048, China · the School of Electrical and Information Engineering, Hunan University, Changsha, 410082, China +1
4D point cloud videos capture rich spatial and temporal dynamics of scenes which possess unique values in various 4D understanding tasks. However, most existing methods work in the spatiotemporal domain where the underlying geometric characteristics of 4D point cloud videos are hard to capture, leading to degraded representation learning and understanding of 4D point cloud videos. We address the above challenge from a complementary spectral perspective. By transforming 4D point cloud videos into graph spectral signals, we can decompose them into multiple frequency bands each of which captures distinct geometric structures of point cloud videos. Our spectral analysis reveals that the decomposed low-frequency signals capture more coarse shapes while high-frequency signals encode more fine-grained geometry details. Building on these observations, we design Spatio-Temporal-Spectral Mixer (STS-Mixer), a unified framework that mixes spatial, temporal, and spectral representations of point cloud videos. STS-Mixer integrates multi-band delineated spectral signals with spatiotemporal information to capture rich geometries and temporal dynamics, while enabling fine-grained and holistic understanding of 4D point cloud videos. Extensive experiments show that STS-Mixer achieves superior performance consistently across multiple widely adopted benchmarks on both 3D action recognition and 4D semantic segmentation tasks. Code and models are available at https://github.com/Vegetebird/STS-Mixer.
Wenhao Li, Xueying Jiang, Gongjie Zhang +3
Nanyang Technological University · Alibaba Group · Zhejiang University of Technology +1