Organizations: University of Illinois Urbana-Champaign · National Yang Ming Chiao Tung University · National Central University · Research Center for Information Technology Innovation, Academia Sinica
Current video diffusion models can generate recognizable camera-effect appearances but do not reliably follow specified physical parameters such as field of view, aperture, and readout time, partly because real dynamic videos with diverse and accurate camera-effect annotations are scarce. Physics-based simulation provides exact labels but suffers from a sim-to-real gap. Reconstructing real dynamic scenes offers an alternative, but Dynamic NeRF methods render too slowly, whereas rasterization-based Dynamic Gaussian Splatting cannot model the light transport required by these effects. We propose 4D Gaussian Ray Tracing (4D-GRT), a two-stage data-generation pipeline that reconstructs dynamic scenes as 4D Gaussians from multi-view pinhole videos and re-renders them with physically based camera models under specified parameters. On CamVFX-4D, our controlled benchmark of eight dynamic indoor scenes under four camera effects, 4D-GRT achieves up to 3.4 dB higher PSNR and 6 times faster rendering than MSTH. Finally, we demonstrate two downstream applications: fine-tuning a video generation model on our data to improve camera-effect control and extending our pipeline to monocular input. Project page: https://shigon255.github.io/4DGRT-project-page/.
Figures & tables
Figure 1: We propose 4D Gaussian Ray Tracing (4D-GRT), a novel framework for generating physically grounded, controllable camera effects in dynamic scenes. (1) Given multi-view video input, our method reconstructs a dynamic scene using 4D Gaussian Splatting (4D-GS) and differentiable ray tracing. (2) We simulate various camera effects with controllable parameters using ray tracing, generating high-quality videos with controllable camera effects. See the project page for the videos.
Figure 2: We evaluate several state-of-the-art video generation models by specifying camera-effect parameters in input text prompts. The results show that these models often fail to generate videos whose camera effects correspond to the specified physical parameters: fisheye outputs do not reflect the requested field of view, depth-of-field results remain blurry even at small aperture size, and rolling-shutter videos lack the expected skew under large readout times. Please refer to Appendix B for details.
Figure 3: The overall pipeline of 4D-GRT. Given multi-view videos, we optimize the 4D-GS representation through differentiable ray tracing. Then, given camera-effect parameters, we can utilize ray tracing to render videos with physically grounded camera effects.
Pinhole
Rolling Shutter
Depth of Field
Fisheye
Ground Truth
HexPlane [ 12 ]
MSTH [ 77 ]
4D-GRT(Ours)
Ground Truth
HexPlane [ 12 ]
MSTH [ 77 ]
4D-GRT(Ours)
Pinhole
Rolling Shutter
Figure 4: Qualitative comparison on CamVFX-4D. The artifacts are framed by red rectangles. See the project page for the videos.
Pinhole
Depth of Field
Rolling Shutter
Fisheye
Figure 5: Qualitative results of 4D-GRT on the Neural 3D Video dataset [ 46 ] . Each row shows one camera effect, and each column progresses temporally from left to right. See the project page for the videos.
Table 6
Methods
PSNR(dB) ↑
SSIM ↑
LPIPS ↓
FPS ↑
Full image
HexPlane [ 12 ]
15.6489
0.6678
0.4142
0.21
MSTH [ 77 ]
24.4222
0.8163
0.1764
9.38
4D-GRT (Ours)
24.1322
0.8162
0.1678
41.53
Within mask
HexPlane [ 12 ]
21.6869
0.7480
0.2726
–
Table 3: Quantitative comparison of fisheye rendering. The lower block reports metrics computed within the pre-defined mask.
Methods
Chunk
PSNR(dB) ↑
SSIM ↑
LPIPS ↓
FPS ↑
HexPlane [ 12 ]
N/A
21.3468
0.7723
0.3162
0.21
MSTH [ 77 ]
N/A
28.7003
0.8927
0.1190
9.35
4D-GRT (Ours)
1 row
31.6144
0.8821
0.1056
0.76
4D-GRT (Ours)
4 rows
31.6146
0.8821
0.1056
4.99
4D-GRT (Ours)
16 rows
31.6082
0.8820
0.1056
13.54
Table 4: Quantitative comparison on rolling-shutter rendering.
Figure 6: Qualitative comparison of the pretrained SVD [ 6 ] model versus our fine-tuned model. For each effect, the first, middle, and last frames of a generated sequence are shown from left to right. Red boxes highlight background areas that should be blurry because of the depth-of-field effect. Test images are sourced from the RealEstate10k [ 98 ] and EBB! [ 41 ] datasets. See the project page for the videos.
Pinhole
Depth of Field
Rolling Shutter
Fisheye
Figure 7: Qualitative results of applying our camera-effect renderer to a MoVieS [ 49 ] reconstruction of a real-world monocular video from the DAVIS dataset [ 63 ] . Each row shows one camera model or camera effect. See the project page for the videos.
Effect
Model
PSNR(dB) ↑
SSIM ↑
LPIPS ↓
Fisheye
Base SVD
12.1991
0.3568
0.5172
Fine-tuned SVD
15.4661
0.4660
0.3136
DoF
Base SVD
18.6120
0.6511
0.2618
Fine-tuned SVD
30.5652
0.9231
0.0958
Table 5: First-frame metrics of base vs. fine-tuned SVD models.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Fisheye video evaluation. Although the models produce recognizable fisheye imagery, changes in the requested field of view do not yield consistent, parameter-aligned projection changes.
Figure 9: Depth-of-field video evaluation. All models produce pronounced blur regardless of the specified aperture ( f /2.0 vs. f /16.0), revealing weak compliance with depth-of-field parameters and poor control of depth-of-field effects.
Figure 10: Rolling-shutter video evaluation. Under a rolling-shutter camera model, the fan blades should exhibit row-wise geometric skew. However, the tested models produce either non-skewed blades or generic motion-blurred blades, even when the prompt explicitly requests minimal motion blur.
Representation
PSNR ↑
SSIM ↑
LPIPS ↓
FPS ↑
Pinhole
4D-GS [ 82 ]
32.8010
0.8898
0.1018
36.56
Swift4D [ 83 ]
33.1394
0.8925
0.1697
48.75
Hybrid 3D-4DGS [ 59 ]
25.7181
0.8092
0.3576
61.60
Fisheye
4D-GS [ 82 ]
28.8927
0.8555
0.1259
41.53
Appendix
Table 6: Comparison of dynamic Gaussian representations using 3DGRT as the rendering backend. Fisheye metrics are computed within the circular image mask, and rolling shutter uses one row per chunk.
Figure 11: Qualitative comparison of the rolling-shutter effect with different numbers of rows per chunk.
Backend
PSNR ↑
SSIM ↑
LPIPS ↓
FPS ↑
Pinhole
3DGRT [ 55 ]
32.8010
0.8898
0.1018
36.56
3DGUT [ 84 ]
32.2641
0.8756
0.1433
74.37
3DGEER [ 39 ]
31.9906
0.8660
0.1400
53.36
Fisheye
3DGRT [ 55 ]
28.8927
0.8555
0.1259
41.53
Appendix
Table 7: Comparison of rendering backends using 4D-GS as the dynamic representation. Fisheye metrics are computed within the circular image mask, and rolling shutter uses one row per chunk. A dash denotes an effect not directly supported by the current backend integration.
Methods
Time ↓
Storage (MB) ↓
HexPlane [ 12 ]
12 hours
238
MSTH [ 77 ]
8 mins
889
4D-GRT (Ours)
3 hours
364.875
Appendix
Table 8: Comparison of training time and storage.
Effect
PSNR ↑
SSIM ↑
LPIPS ↓
Pinhole
24.7302
0.8842
0.0723
Fisheye
22.1009
0.8057
0.1609
Depth of field
23.4781
0.7760
0.3093
Rolling shutter
24.5100
0.8723
0.0765
Appendix
Table 9: Quantitative results of applying our camera-effect renderer to monocular 4D-GS reconstructions produced by MoVieS [ 49 ] .
This paper tackles the challenge of recovering 4D dynamic scenes from videos captured by as few as four portable cameras. Learning to model scene dynamics for temporally consistent novel-view rendering is a foundational task in computer graphics, where previous works often require dense multi-view captures using camera arrays of dozens or even hundreds of views. We propose 4C4D, a novel framework that enables high-fidelity 4D Gaussian Splatting from video captures of extremely sparse cameras. Our key insight lies that the geometric learning under sparse settings is substantially more difficult than modeling appearance. Driven by this observation, we introduce a Neural Decaying Function on Gaussian opacities for enhancing the geometric modeling capability of 4D Gaussians. This design mitigates the inherent imbalance between geometry and appearance modeling in 4DGS by encouraging the 4DGS gradients to focus more on geometric learning. Extensive experiments across sparse-view datasets with varying camera overlaps show that 4C4D achieves superior performance over prior art. Project page at: https://junshengzhou.github.io/4C4D.
Junsheng Zhou, Zhifan Yang, Liang Han +4
School of Software, Tsinghua University · Kuaishou Technology
Pretrained video diffusion models can act as renderers when the desired scene state is already specified by an animated mesh, a camera trajectory, and a reference image. This 4D generative rendering setting raises a representation question: what image-format condition lets a video backbone obey both camera motion and scene-internal animation? We propose DAR, a reference-guided renderer that extends Wan2.2 camera control from Plücker rays alone to a joint camera-plus-geometry interface. DAR projects a neural 4D G-buffer (tracking, world position, and normal) from the animated mesh and injects it through a widened control adapter while preserving the pretrained image-to-video prior. The central design choice is the pair of tracking and world position. Tracking identifies the persistent surface element that should carry appearance; world position gives its current scene-coordinate state; normal supplies local shape. Depth plus calibrated rays can recover 3D in principle, but depth is a camera-dependent chart in which camera and object motion are mixed. On the 68-case DAR-4D benchmark, LoRA DAR reaches PSNR 23.22, SSIM 0.895, and LPIPS 0.134, improving over off-the-shelf Wan2.2-Depth by 1.54 dB PSNR; a full fine-tune reaches PSNR 25.36 and SSIM 0.917. Matched ablations show that replacing world position by depth reduces PSNR by 1.26--1.55 dB at every checkpoint, supporting tracking+world-position correspondence as a practical 4D rendering condition.
Junhao Chen, Mingjin Chen, Henghaofan Zhang +9
Tsinghua University, China · The Hong Kong Polytechnic University, China · University of Electronic Science and Technology of China, China +5
This work presents SplatFlux, a feed-forward latent-to-4DGS framework for both video-based 4D reconstruction and conditional 4D generation. Our key observation is that observed videos and conditionally generated content can be represented in the same pretrained video latent space, allowing both tasks to be formulated as latent-to-4DGS decoding. To construct compact and temporally coherent 4D scenes, SplatFlux introduces a static-dynamic decoupled 4D Gaussian representation that explicitly predicts dynamic probabilities for Gaussian primitives. This representation separates static scene structures from moving regions, enabling complete 4D scene construction in a feed-forward manner through static aggregation and dynamic motion interpolation. To learn static-dynamic decomposition and dynamic Gaussian motion without costly annotations, we further develop an auxiliary distillation strategy that provides pseudo dynamic masks and optical-flow cues. Extensive experiments across 3D and 4D reconstruction and generation demonstrate that SplatFlux achieves state-of-the-art performance.