cs.CVSep 13, 2025

Every Camera Effect, Every Time, All at Once: 4D Gaussian Ray Tracing for Physics-based Camera Effect Data Generation

Authors: Yi-Ruei Liu, You-Zhe Xie, Yu-Hsiang Hsu, Jing-En Huang, I-Sheng Fang, Yu-Lun Liu, Jun-Cheng Chen

Organizations: University of Illinois Urbana-Champaign · National Yang Ming Chiao Tung University · National Central University · Research Center for Information Technology Innovation, Academia Sinica

Abstract

Current video diffusion models can generate recognizable camera-effect appearances but do not reliably follow specified physical parameters such as field of view, aperture, and readout time, partly because real dynamic videos with diverse and accurate camera-effect annotations are scarce. Physics-based simulation provides exact labels but suffers from a sim-to-real gap. Reconstructing real dynamic scenes offers an alternative, but Dynamic NeRF methods render too slowly, whereas rasterization-based Dynamic Gaussian Splatting cannot model the light transport required by these effects. We propose 4D Gaussian Ray Tracing (4D-GRT), a two-stage data-generation pipeline that reconstructs dynamic scenes as 4D Gaussians from multi-view pinhole videos and re-renders them with physically based camera models under specified parameters. On CamVFX-4D, our controlled benchmark of eight dynamic indoor scenes under four camera effects, 4D-GRT achieves up to 3.4 dB higher PSNR and 6 times faster rendering than MSTH. Finally, we demonstrate two downstream applications: fine-tuning a video generation model on our data to improve camera-effect control and extending our pipeline to monocular input. Project page: https://shigon255.github.io/4DGRT-project-page/.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. 4C4D: 4 Camera 4D Gaussian Splatting

    Apr 5, 2026Junsheng Zhou, Zhifan Yang, Liang Han +4Sparse-View 4D Reconstruction4D Reconstruction

  2. Video Models as Native 4D Renderers: World-Grounded Conditioning from Animated Mesh

    Jul 30, 2026Junhao Chen, Mingjin Chen, Henghaofan Zhang +9Camera-Conditioned Video GenerationVideo Diffusion Models