3DSPA: A 3D Semantic Point Autoencoder for Evaluating Video Realism
Authors: Bhavik Chandna, Kelsey R. Allen
Organizations: University of California, San Diego, CA, USA · Vector Institute, Toronto, ON, Canada · University of British Columbia, Vancouver, BC, Canada
AI video generation is evolving rapidly. For video generators to be useful for applications ranging from robotics to film-making, they must consistently produce realistic videos. However, evaluating the realism of generated videos remains a largely manual process -- requiring human annotation or bespoke evaluation datasets which have restricted scope. Here we develop an automated evaluation framework for video realism which captures both semantics and coherent 3D structure and which does not require access to a reference video. Our method, 3DSPA, is a 3D spatiotemporal point autoencoder which integrates 3D point trajectories, depth cues, and DINO semantic features into a unified representation for video evaluation. 3DSPA models how objects move and what is happening in the scene, enabling robust assessments of realism, temporal consistency, and physical plausibility. Experiments show that 3DSPA reliably identifies videos which violate physical laws, is more sensitive to motion artifacts, and aligns more closely with human judgments of video quality and realism across multiple datasets. Our results demonstrate that enriching trajectory-based representations with 3D semantics offers a stronger foundation for benchmarking generative video models, and implicitly captures physical rule violations. The code and pretrained model weights will be available at https://github.com/TheProParadox/3dspa_code.
Figures & tables
Figure 1: Example 3D point tracks (projected into 2D) reconstructed by 3DSPA for an unrealistic generated video from VideoPhy-2 (video frames progress from left to right, top to bottom) ( Bansal et al., 2025 ) , depicting a man striking a wall with a hammer and clearly violating physical laws. 3DSPA assigns low realism as measured with the Average Jaccard ( AJ ) (blue: high, red: low, white: intermediate). The video ranks among the lowest in both 3DSPA ’s scores and human realism ratings, highlighting strong alignment with human judgment.
Figure 2: 3DSPA architecture overview. The encoder integrates 3D trajectories, temporal embeddings, and DINOv2 ( Oquab et al., 2023 ) semantic features into a compact latent representation using occlusion-aware attention and a Perceiver-style transformer architecture ( Jaegle et al., 2021 ) . The decoder conditions on query points to reconstruct full 3D trajectories with occlusion flags.
Method
Aria
DriveTrack
PStudio
Average
AJ↑
APD↑
OA↑
AJ↑
APD↑
OA↑
AJ↑
APD↑
OA↑
AJ↑
APD↑
OA↑
TAPIR + ZD
15.7
23.5
79.8
6.3
10.5
81.6
11.2
18.9
78.7
11.0
17.6
80.1
CoTracker + ZD
17.0
25.7
88.0
6.0
10.9
82.6
11.4
19.9
80.0
11.4
18.8
83.5
BootsTAPIR + ZD
11.8
16.3
86.7
6.4
10.9
85.3
11.6
19.6
82.6
11.6
18.9
84.9
CoTracker3 + ZD
15.6
24.1
88.6
13.3
19.6
86.8
9.0
13.6
83.9
12.6
19.1
86.4
SpatialTracker
16.7
25.7
89.3
6.9
12.4
83.7
12.3
21.6
78.5
12.0
19.9
83.8
Table 1: 3D point tracking results on the TapVid-3D minival set. 3DSPA achieves competitive 3D tracking performance across datasets and performs on par with a finetuned CoTracker3 (CoTracker3-FT+ZD), highlighting its ability to reconstruct consistent and accurate 3D tracks despite the information bottleneck.
Model / Category
Permanence
Immutability
Continuity
Solidity
Fixed
Moving
Fixed
Moving
Fixed
Moving
Fixed
Moving
GPT-4o ( Hurst et al., 2024 )
59.62
58.82
58.65
59.56
54.81
57.35
56.73
55.32
Qwen-VL 2.5 ( Bai et al., 2025 )
53.85
54.41
56.73
53.68
52.88
54.41
50.96
51.06
Gemini-1.5 Pro ( Team et al., 2024 )
55.77
55.88
56.73
56.73
54.80
54.80
56.73
56.73
Gemini-2.5 Flash ( Comanici et al., 2025 )
64.42
58.82
59.62
63.97
54.81
55.15
55.77
56.38
VideoMAEv2-g ( Wang et al., 2023 )
63.46
50.00
54.81
53.69
65.38
54.41
48.08
59.57
Table 2: Win rates (%) on IntPhys2 across physical principles. The top rows report prior models’ win rates, while the bottom rows compare 3DSPA , its ablations, and human performance. 3DSPA and 3DSPA (no 3D) achieve the strongest performance across most concept categories in detecting physically implausible events.
Figure 3: Qualitative examples of accurate reconstruction on realistic generated videos from EvalCrafter and VideoPhy-2. High-rated videos produce coherent and stable point tracks under 3DSPA (blue). The top and middle examples are from EvalCrafter ( Liu et al., 2024 ) , and the bottom example is from VideoPhy-2 ( Bansal et al., 2025 ) .
Figure 4: TRAJAN vs. 3DSPA . Qualitative comparison on videos from the EvalCrafter dataset ( Liu et al., 2024 ) . Compared to TRAJAN ( Allen et al., 2025 ) , 3DSPA produces more coherent and temporally stable point tracks, aligning more closely with human judgments of motion quality. In the top example (dog walking; human rating: 4.5/5), 3DSPA accurately captures articulated leg motion in 3D, whereas TRAJAN produces noisy and inconsistent tracks. In the bottom example (human rating: 1.67/5), the phone gradually disappears; 3DSPA correctly identifies this semantic violation, while TRAJAN fails due to smooth but semantically implausible trajectories.
Model
Spearman (PC)
Video Evaluation Models (fine-tuned)
VideoCon–Physics
0.48
VideoCon
0.13
VideoLlava
0.08
VideoScore
0.17
VIDEOPHY–2–AUTOEVAL
0.76
Table 3: Spearman rank coefficients on the VideoPhy-2 benchmark for physical commonsense (PC). Video Evaluation Models are fine-tuned vision-language models.
Model
Visual Quality
T2V
Motion Quality
Consistency
Subjective Likeness
3DSPA (no 3D, no DINO)
0.28
0.25
0.24
0.33
0.23
3DSPA (no 3D)
0.31
0.44
0.41
0.39
0.46
3DSPA (no DINO)
0.45
0.55
0.49
0.48
0.63
3DSPA (ours)
0.48
0.58
0.55
0.60
0.60
Table 4: Spearman rank coefficients between human annotations and automated AJ scores for the EvalCrafter ( Liu et al., 2024 ) dataset in the categories of visual quality, text-to-video similarity, motion quality, temporal consistency, and subjective likeness across different ablations.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Hyperparameter Value
Sinusoidal embedding (spatial + temporal + depth)
32 frequencies
Track token projection dimensionality ( C )
384
DINO feature projection dimensionality
768
Depth feature projection dimensionality
256
Compression dimensionality
96
Up-projection dimensionality
1280
Appendix
Table 5: Positional encoding and projection operator hyperparameters for 3DSPA .
Transformer Name
Attention Type
QKV Size
Layers
Heads
MLP Size
Input 3D track transformer
SA
96×8
3
8
1536
Perceiver-style transformer
CA
96×8
4
8
2048
Up-projection latent transformer (decoder)
CA
96×8
4
8
2048
Track readout transformer
CA
96×8
4
8
1536
Appendix
Table 6: Transformer architecture hyperparameters for 3DSPA . SA = self-attention, CA = cross-attention.
Figure 5: Performance comparison across models on the IntPhys2 benchmark for each of the easy , medium and hard categories.
Figure 6: Visualization of point tracks on IntPhys2 videos. For each video, we project tracked points and color them using their Average Jaccard (AJ) consistency score (red = low / unreal, blue = high / real). Real, physically consistent videos (top) show smooth, coherent tracks with high AJ scores throughout the video, while unreal videos with physical rule violations (bottom) exhibit poorly reconstructed point trajectories.
Model
Inference Time (s)
VideoCon–Physics
8.13
VideoCon
6.21
VideoLLaVA
13.52
VideoScore
4.89
VIDEOPHY–2–AUTOEVAL
10.67
TRAJAN
2.18
Appendix
Table 7: Average inference time per video for 3DSPA and TRAJAN as compared to other finetuned VLM models
Figure 7: Positive examples where 3DSPA performs well on TAP-Vid3D-Minival. Predicted 3D trajectories (green) align closely with ground truth (red) even for videos with small objects and high motion. These videos are from the DriveTrack dataset within TAP-Vid3D-Minival.
Figure 8: Failure cases of 3DSPA on TAP-Vid3D-Minival. We visualize ground-truth (red) and predicted (green) query 3D point trajectories for challenging scenes. Errors typically arise in regions with complex geometry or depth ambiguities, giving high error in depth prediction, although the motion following is good. Corresponding 2D frames (left) and reconstructed 3D trajectories (right) highlight mismatches between predicted and true motion, which mainly arise from incorrectly estimated depth.
Figure 9: Additional Visualization on Evalcrafter & Videophy-2. Low-rated motion videos lead to lower AJ scores and unstable point tracks from 3DSPA , reflecting difficulty in reconstructing unreliable motion. The top videos are from Evalcrafter ( Liu et al., 2024 ) , middle and bottom are from Videophy2 ( Bansal et al., 2025 ) .