Generating high-quality 4D objects with spatial-temporal consistency is still formidable. Existing diffusion-based methods often struggle with spatial-temporal inconsistency, as they fail to leverage outputs from all previous timesteps to guide the generation at the current timestep. Therefore, we propose a Spatial-Temporal State Propagation AutoRegressive Model (4DSTAR), which generates 4D objects maintaining temporal-spatial consistency. 4DSTAR formulates the generation problem as the prediction of tokens that represent the 4D object. It consists of two key components: (1) The dynamic spatial-temporal state propagation autoregressive model (STAR) is proposed, which achieves spatial-temporal consistent generation. Unlike standard autoregressive models, STAR divides prediction tokens into groups based on timesteps. It models long-term dependencies by propagating spatial-temporal states from previous groups and utilizes these dependencies to guide generation at the next timestep. To this end, a spatial-temporal container is proposed, which dynamically updating the effective spatial-temporal state features from all historical groups, then updated features serve as conditional features to guide the prediction of the next token group. (2) The 4D VQ-VAE is proposed, which implicitly encodes the 4D structure into discrete space and decodes the discrete tokens predicted by STAR into temporally coherent dynamic 3D Gaussians. Experiments demonstrate that 4DSTAR generates spatial-temporal consistent 4D objects, and achieves performance competitive with diffusion models.
Figures & tables
Figure 1: Diffusion-based methods, such as previous work [ 57 ] , fail to leverage outputs from all previous timesteps to guide the generation at the current timestep, which generates results with inconsistent appearance at some timesteps. Our 4DSTAR alleviates this issue by leveraging historical outputs to guide the generation at the current timestep.
Figure 2: The overall pipeline of our 4DSTAR. 4DSTAR consists of two key components: (a) 4D VQ-VAE. Given a 4D object, we first render it as a spatial-temporal matrix. Then the matrix is encoded by Encoder, and is compressed into discrete tokens. Static GS Generation decodes these tokens to static Gaussians. Meanwhile, Spatial-Temporal Offset Predictor (STOP) corrects static Gaussians into a canonical 4D space at each timestep. Finally, the model outputs dynamic 3D Gaussians. (b) Dynamic Spatial-Temporal State Propagation Autoregressive Model (STAR). The text and video conditions, which are compressed by an image tokenizer are concatenated before the start token as the context. The conditions can either expect the model to generate tokens. The SEP signals the model to begin generating tokens. Then, camera pose and timestep conditions are integrated. When the model starts to predict the next group, the historical groups are integrated into Spatial-Temporal Container (S-T Container). S-T Container updates effective spatial-temporal state features. The features serve as conditional features to guide the prediction of the next token group. Finally, STAR predicts all tokens that represent a 4D object.
Figure 3: Qualitative comparison with VQ-VAE [ 41 ] and UniTok [ 25 ] on 4D reconstruction. For VQ-VAE and UniTok, we employ them to reconstruct 2D view images. For our 4D VQ-VAE, we render results under corresponding views. Our 4D VQ-VAE can reconstruct the results with temporal coherence, while VQ-VAE and UniTok cannot reconstruct with temporal coherence.
Figure 4: Qualitative comparison with the state-of-the-art methods [ 60 , 34 , 62 , 57 ] on video-to-4D generation. For each method, we render results under two novel views at two timesteps. Our method achieves high-quality generation with spatial-temporal consistency.
Method
CLIP ↑
LPIPS ↓
FVD ↓
FID-VID ↓
VQ-VAE [ 41 ]
0.938
0.067
326.570
13.447
UniTok [ 25 ]
0.973
0.050
151.238
4.402
4D VQ-VAE (Ours)
0.973
0.048
133.372
4.175
Table 1: Evaluation and comparison of the performance on 4D object reconstruction. The best score is highlighted in bold. All the experiments of the methods are carried out using the code from their official GitHub repository.
Method
CLIP ↑
LPIPS ↓
FVD ↓
FID-VID ↓
STAG4D [ 60 ]
0.910
0.141
752.215
27.280
L4GM [ 34 ]
0.926
0.132
515.430
20.125
SV4D 2.0 [ 57 ]
0.932
0.139
497.753
19.223
GVFDiffusion [ 62 ]
0.931
0.138
528.336
19.064
4DSTAR (Ours)
0.952
0.131
464.709
15.312
Table 2: Evaluation and comparison of the performance on video-to-4D object generation. The best score is highlighted in bold. All the experiments of the methods are carried out using the code from their official GitHub repository.
Method
CLIP ↑
LPIPS ↓
FVD ↓
FID-VID ↓
w/o STOP
0.968
0.057
174.757
5.575
4D VQ-VAE (Ours)
0.973
0.048
133.372
4.175
Table 3: Ablation Experiments of 4D VQ-VAE on the test set. Each setup is based on a modification of the immediately preceding setups. The best score is highlighted in bold.
Figure 5: Ablation of 4D VQ-VAE. Our model which uses STOP, accurately recovers texture details at different timesteps.
Method
CLIP ↑
LPIPS ↓
FVD ↓
FID-VID ↓
A: L4GM w/ Explicit State Propagation
0.930
0.132
508.192
19.714
Baseline
0.939
0.148
883.946
37.066
B: w/ Temporal Average Pooling
0.942
0.146
646.558
27.016
C: w/ Learnable Token Merging
0.945
0.141
579.288
21.449
Ours: w/ S-T Container
0.952
0.131
464.709
15.312
Table 4: Ablation Experiments of STAR on the test set. Each setup is based on a modification of the immediately preceding setups. The best score is highlighted in bold.
Figure 6: Ablation of STAR. Even slight motion within input, ours produces more temporally consistent results than Baseline. More visualizations see supplementary materials.
Figure 7: Visualization of the clustering within S-T Container. The left is the original distribution of token features belonging to previous (historical) groups. The right visualizes the clustering centers after clustering. The clustering centers preserve the diversity of spatial-temporal information within token features.
Figure 8: Visualization of (a) our text-video to 4D object generation and (b) our text-image to static 3D object generation. Our approach is capable of generating a 4D object with temporal-spatial consistency and a 3D object with multi-view consistency.
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · School of Computing, National University of Singapore · School of Computing, Macquarie University
School of Management Science and Engineering, Anhui University of Finance and Economics, Bengbu, China · National University of Singapore · Wuhan University