Organizations: University of Illinois Urbana-Champaign · National Yang Ming Chiao Tung University · National Central University · Research Center for Information Technology Innovation, Academia Sinica
Current video diffusion models can generate recognizable camera-effect appearances but do not reliably follow specified physical parameters such as field of view, aperture, and readout time, partly because real dynamic videos with diverse and accurate camera-effect annotations are scarce. Physics-based simulation provides exact labels but suffers from a sim-to-real gap. Reconstructing real dynamic scenes offers an alternative, but Dynamic NeRF methods render too slowly, whereas rasterization-based Dynamic Gaussian Splatting cannot model the light transport required by these effects. We propose 4D Gaussian Ray Tracing (4D-GRT), a two-stage data-generation pipeline that reconstructs dynamic scenes as 4D Gaussians from multi-view pinhole videos and re-renders them with physically based camera models under specified parameters. On CamVFX-4D, our controlled benchmark of eight dynamic indoor scenes under four camera effects, 4D-GRT achieves up to 3.4 dB higher PSNR and 6 times faster rendering than MSTH. Finally, we demonstrate two downstream applications: fine-tuning a video generation model on our data to improve camera-effect control and extending our pipeline to monocular input. Project page: https://shigon255.github.io/4DGRT-project-page/.
Figures & tables
Figure 1: We propose 4D Gaussian Ray Tracing (4D-GRT), a novel framework for generating physically grounded, controllable camera effects in dynamic scenes. (1) Given multi-view video input, our method reconstructs a dynamic scene using 4D Gaussian Splatting (4D-GS) and differentiable ray tracing. (2) We simulate various camera effects with controllable parameters using ray tracing, generating high-quality videos with controllable camera effects. See the project page for the videos.
Figure 2: We evaluate several state-of-the-art video generation models by specifying camera-effect parameters in input text prompts. The results show that these models often fail to generate videos whose camera effects correspond to the specified physical parameters: fisheye outputs do not reflect the requested field of view, depth-of-field results remain blurry even at small aperture size, and rolling-shutter videos lack the expected skew under large readout times. Please refer to Appendix B for details.
Figure 3: The overall pipeline of 4D-GRT. Given multi-view videos, we optimize the 4D-GS representation through differentiable ray tracing. Then, given camera-effect parameters, we can utilize ray tracing to render videos with physically grounded camera effects.
Pinhole
Rolling Shutter
Depth of Field
Fisheye
Ground Truth
HexPlane [ 12 ]
MSTH [ 77 ]
4D-GRT(Ours)
Ground Truth
HexPlane [ 12 ]
MSTH [ 77 ]
4D-GRT(Ours)
Pinhole
Rolling Shutter
Figure 4: Qualitative comparison on CamVFX-4D. The artifacts are framed by red rectangles. See the project page for the videos.
Pinhole
Depth of Field
Rolling Shutter
Fisheye
Figure 5: Qualitative results of 4D-GRT on the Neural 3D Video dataset [ 46 ] . Each row shows one camera effect, and each column progresses temporally from left to right. See the project page for the videos.
Table 6
Methods
PSNR(dB) ↑
SSIM ↑
LPIPS ↓
FPS ↑
Full image
HexPlane [ 12 ]
15.6489
0.6678
0.4142
0.21
MSTH [ 77 ]
24.4222
0.8163
0.1764
9.38
4D-GRT (Ours)
24.1322
0.8162
0.1678
41.53
Within mask
HexPlane [ 12 ]
21.6869
0.7480
0.2726
–
Table 3: Quantitative comparison of fisheye rendering. The lower block reports metrics computed within the pre-defined mask.
Methods
Chunk
PSNR(dB) ↑
SSIM ↑
LPIPS ↓
FPS ↑
HexPlane [ 12 ]
N/A
21.3468
0.7723
0.3162
0.21
MSTH [ 77 ]
N/A
28.7003
0.8927
0.1190
9.35
4D-GRT (Ours)
1 row
31.6144
0.8821
0.1056
0.76
4D-GRT (Ours)
4 rows
31.6146
0.8821
0.1056
4.99
4D-GRT (Ours)
16 rows
31.6082
0.8820
0.1056
13.54
Table 4: Quantitative comparison on rolling-shutter rendering.
Figure 6: Qualitative comparison of the pretrained SVD [ 6 ] model versus our fine-tuned model. For each effect, the first, middle, and last frames of a generated sequence are shown from left to right. Red boxes highlight background areas that should be blurry because of the depth-of-field effect. Test images are sourced from the RealEstate10k [ 98 ] and EBB! [ 41 ] datasets. See the project page for the videos.
Pinhole
Depth of Field
Rolling Shutter
Fisheye
Figure 7: Qualitative results of applying our camera-effect renderer to a MoVieS [ 49 ] reconstruction of a real-world monocular video from the DAVIS dataset [ 63 ] . Each row shows one camera model or camera effect. See the project page for the videos.
Effect
Model
PSNR(dB) ↑
SSIM ↑
LPIPS ↓
Fisheye
Base SVD
12.1991
0.3568
0.5172
Fine-tuned SVD
15.4661
0.4660
0.3136
DoF
Base SVD
18.6120
0.6511
0.2618
Fine-tuned SVD
30.5652
0.9231
0.0958
Table 5: First-frame metrics of base vs. fine-tuned SVD models.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Fisheye video evaluation. Although the models produce recognizable fisheye imagery, changes in the requested field of view do not yield consistent, parameter-aligned projection changes.
Figure 9: Depth-of-field video evaluation. All models produce pronounced blur regardless of the specified aperture ( f /2.0 vs. f /16.0), revealing weak compliance with depth-of-field parameters and poor control of depth-of-field effects.
Figure 10: Rolling-shutter video evaluation. Under a rolling-shutter camera model, the fan blades should exhibit row-wise geometric skew. However, the tested models produce either non-skewed blades or generic motion-blurred blades, even when the prompt explicitly requests minimal motion blur.
Representation
PSNR ↑
SSIM ↑
LPIPS ↓
FPS ↑
Pinhole
4D-GS [ 82 ]
32.8010
0.8898
0.1018
36.56
Swift4D [ 83 ]
33.1394
0.8925
0.1697
48.75
Hybrid 3D-4DGS [ 59 ]
25.7181
0.8092
0.3576
61.60
Fisheye
4D-GS [ 82 ]
28.8927
0.8555
0.1259
41.53
Appendix
Table 6: Comparison of dynamic Gaussian representations using 3DGRT as the rendering backend. Fisheye metrics are computed within the circular image mask, and rolling shutter uses one row per chunk.
Figure 11: Qualitative comparison of the rolling-shutter effect with different numbers of rows per chunk.
Backend
PSNR ↑
SSIM ↑
LPIPS ↓
FPS ↑
Pinhole
3DGRT [ 55 ]
32.8010
0.8898
0.1018
36.56
3DGUT [ 84 ]
32.2641
0.8756
0.1433
74.37
3DGEER [ 39 ]
31.9906
0.8660
0.1400
53.36
Fisheye
3DGRT [ 55 ]
28.8927
0.8555
0.1259
41.53
Appendix
Table 7: Comparison of rendering backends using 4D-GS as the dynamic representation. Fisheye metrics are computed within the circular image mask, and rolling shutter uses one row per chunk. A dash denotes an effect not directly supported by the current backend integration.
Methods
Time ↓
Storage (MB) ↓
HexPlane [ 12 ]
12 hours
238
MSTH [ 77 ]
8 mins
889
4D-GRT (Ours)
3 hours
364.875
Appendix
Table 8: Comparison of training time and storage.
Effect
PSNR ↑
SSIM ↑
LPIPS ↓
Pinhole
24.7302
0.8842
0.0723
Fisheye
22.1009
0.8057
0.1609
Depth of field
23.4781
0.7760
0.3093
Rolling shutter
24.5100
0.8723
0.0765
Appendix
Table 9: Quantitative results of applying our camera-effect renderer to monocular 4D-GS reconstructions produced by MoVieS [ 49 ] .