GATS: Gaussian Aware Temporal Scaling Transformer for Invariant 4D Spatio-Temporal Point Cloud Representation

Jiayi Tian; Jiaze Wang

cs.CVMar 17, 2026

GATS: Gaussian Aware Temporal Scaling Transformer for Invariant 4D Spatio-Temporal Point Cloud Representation

Authors: Jiayi Tian, Jiaze Wang

Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Harbin Institute of Technology, Shenzhen, Pengcheng Laboratory

Abstract

Understanding 4D point cloud videos is essential for enabling intelligent agents to perceive dynamic environments. However, temporal scale bias across varying frame rates and distributional uncertainty in irregular point clouds make it highly challenging to design a unified and robust 4D backbone. Existing CNN or Transformer based methods are constrained either by limited receptive fields or by quadratic computational complexity, while neglecting these implicit distortions. To address this problem, we propose a novel dual invariant framework, termed Gaussian Aware Temporal Scaling (GATS), which explicitly resolves both distributional inconsistencies and temporal. The proposed Uncertainty Guided Gaussian Convolution (UGGC) incorporates local Gaussian statistics and uncertainty aware gating into point convolution, thereby achieving robust neighborhood aggregation under density variation, noise, and occlusion. In parallel, the Temporal Scaling Attention (TSA) introduces a learnable scaling factor to normalize temporal distances, ensuring frame partition invariance and consistent velocity estimation across different frame rates. These two modules are complementary: temporal scaling normalizes time intervals prior to Gaussian estimation, while Gaussian modeling enhances robustness to irregular distributions. Our experiments on mainstream benchmarks MSR-Action3D (+6.62% accuracy), NTU RGBD (+1.4% accuracy), and Synthia4D (+1.8% mIoU) demonstrate significant performance gains, offering a more efficient and principled paradigm for invariant 4D point cloud video understanding with superior accuracy, robustness, and scalability compared to Transformer based counterparts.

Figures & tables

Figure 1: Motivation. Right: The right panel illustrates video sequences under different frame rate partitions: with a large temporal interval, some moving objects (second row) and static objects (third column) disappear, while they remain visible with a small temporal interval. This indicates that varying frame rate partitions may cause certain velocity features to vanish. Left: The left panel illustrates that GATS can adaptively adjust the scaling distribution across different frame rate partitions, thereby effectively mitigating the relative velocity bias introduced by frame rate variations and reducing fluctuations in accuracy. Consequently, GATS achieves improvements in ACC of 6.62% and 3.83% over P4D and PST, respectively.

Figure 2: Pipeline. The overall network backbone consists of two core modules: (a) UGGC Module. After the point cloud is fed into the network, the spatial variations of $x_{i}^{t}$ generate cross frame representations. However, different cross frames often lead to inter frame inconsistencies. The UGGC module extracts local Gaussian features and incorporates an uncertainty aware gating mechanism to jointly model geometric and Gaussian local features of 4D point clouds, thereby enhancing the robustness of feature extraction. (b) TSA Module. Under different frame rates, the estimation of relative velocity ${}_{i}^{t}$ varies, and as the temporal dimension progresses, motion features tend to produce inter frame inconsistencies. To address this, the TSA module introduces a learnable scaling factor $s$ to normalize temporal distances, achieving frame partition invariance and ensuring consistent relative velocity estimation across varying frame rates.

Figure 3: MSR-attention Results. Attention visualization showing focus on key regions and spatio-temporal dynamics.

Methods	Input	Frames	Accuracy (%)
Actionlet [ 44 ]	skeleton	all	88.21
MeteorNet [ 25 ]	point	12	86.53
PSTNet [ 6 ]	point	12	87.88
P4Transformer [ 5 ]	point	12	87.54
PST-Transformer [ 8 ]	point	12	88.15
GATS (Ours)	point	12	90.24

Table 1: Action recognition accuracy comparison (%) on MSR-Action3D.

Method	Input	Acc (%)
SkeleMotion [ 2 ]	skeleton	69.6
GCA-LSTM [ 20 ]	skeleton	74.4
AttentionLSTM [ 19 ]	skeleton	77.1
AGC-LSTM [ 41 ]	skeleton	89.2
AS-GCN [ 15 ]	skeleton	86.8
VA-fusion [ 55 ]	skeleton	89.4

Table 2: Action recognition accuracy (%) on NTU RGBD.

Method	Input	Frame	Track	Bldn	Road	Sdwlk	Fence	Vegittn	Pole	-
3D MinkNet14	voxel	1	-	89.39	97.68	69.43	86.52	98.11	97.26	-
4D MinkNet14	voxel	3	-	90.13	98.26	73.47	87.19	99.10	97.50	-
PointNet++	point	1	-	96.88	97.72	86.20	92.75	97.12	97.09	-
MeteorNet-m	point	2	✓	98.22	97.79	90.98	93.18	98.31	97.45	-
MeteorNet-m	point	2	$\times$	97.65	97.83	90.03	94.06	97.41	97.79	-
MeteorNet-l	point	3	$\times$	98.10	97.72	88.65	94.00	97.98	97.65	-

Table 3: 4D semantic segmentation results (mIoU %) on the Synthia 4D dataset.

Figure 4: 4D Qualitative Results. The rows from top to bottom correspond to the input, GT, and our predictions. Detailed comparative results are highlighted in the enlarged regions of the figure.

Model	frame	Acc(%)
(a) Full Model	24	97.56
(b) w/o UGGC	24	95.12
(c) w/o TSA	24	96.16
(d) PSTTransformer (baseline)	24	93.73

Table 4: Ablation studies on MSR-Action3D.

Method	Backbone	Frame	Acc(%)
PvNeXt	CNN	24	94.77
MAMBA4D	Mamba	24	93.38
Ours	Transformer	24	97.56

Table 5: Comparison results with advanced architecture processing more frames.

Figure 5: Analysis of accuracy with varying temporal and spatial parameters.

Figure 6: Structure of GATS. The left side shows the overall process, and the right side shows the specific process of DATS.

Methods	Input	Accuracy (%)
Vieira et al.	depth	78.20
Kläser et al.	depth	81.43
Actionlet	skeleton	88.21
PointNet++	point	61.61
MeteorNet	point	88.50
GATS(Ours)	point	97.56

Table 6: Action recognition accuracy comparison on a standard benchmark.

Explore similar work

TRiGS: Temporal Rigid-Body Motion for Scalable 4D Gaussian Splatting
Apr 1, 2026Suwoong Yeom, Joonsik Nam, Seunggyu Choi +74D Reconstruction Dynamic Scene Reconstruction
Sogang University · POSTECH · Seoul National University +2
Align then Adapt: Rethinking Parameter-Efficient Transfer Learning in 4D Perception
Feb 26, 2026Yiding Sun, Jihua Zhu, Haozhe Cheng +4Transfer Learning Point Cloud Learning
the School of Software Engineering, Xi’an Jiaotong University, Xi’an 710048, China · the State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an 710048, China · the School of Electrical and Information Engineering, Hunan University, Changsha, 410082, China +1
STS-Mixer: Spatio-Temporal-Spectral Mixer for 4D Point Cloud Video Understanding
Apr 13, 2026Wenhao Li, Xueying Jiang, Gongjie Zhang +3Point Cloud Learning 4D Scene Understanding
Nanyang Technological University · Alibaba Group · Zhejiang University of Technology +1

Apr 1, 2026J/K move · Enter open · S save

TRiGS: Temporal Rigid-Body Motion for Scalable 4D Gaussian Splatting

Suwoong Yeom, Joonsik Nam, Seunggyu Choi, Lucas Yunkyu Lee+6

Sogang University · POSTECH · Seoul National University+2

4D Reconstruction Dynamic Scene Reconstruction 3D Gaussian Splatting Motion Representation Learning

Open paper AskPDF arXiv

Recent 4D Gaussian Splatting (4DGS) methods achieve impressive dynamic scene reconstruction but often rely on piecewise linear velocity approximations and short temporal windows. This disjointed modeling leads to severe temporal fragmentation, forcing primitives to be repeatedly eliminated and regenerated to track complex nonlinear dynamics. This makeshift approximation eliminates the long-term temporal identity of objects and causes an inevitable proliferation of Gaussians, hindering scalability to extended video sequences. To address this, we propose TRiGS, a novel 4D representation that utilizes unified, continuous geometric transformations. By integrating

SE(3)

transformations, hierarchical Bezier residuals, and learnable local anchors, TRiGS models geometrically consistent rigid motions for individual primitives. This continuous formulation preserves temporal identity and effectively mitigates unbounded memory growth. Extensive experiments demonstrate that TRiGS achieves high fidelity rendering on standard benchmarks while uniquely scaling to extended video sequences (e.g., 600 to 1200 frames) without severe memory bottlenecks, significantly outperforming prior works in temporal stability.

Abstract

Explore similar work

TRiGS: Temporal Rigid-Body Motion for Scalable 4D Gaussian Splatting

Align then Adapt: Rethinking Parameter-Efficient Transfer Learning in 4D Perception

STS-Mixer: Spatio-Temporal-Spectral Mixer for 4D Point Cloud Video Understanding