Organizations: Southern University of Science and Technology, Shenzhen, China · Fudan University, Shanghai, China · East China University of Science and Technology, Shanghai, China · Beijing Technology and Business University, Beijing, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
Reconstructing dynamic 3D scenes from monocular video requires inferring motion from noisy and intermittently visible observations. Long-range tracks provide valuable geometric constraints, but integrating their fragmented temporal coverage into shared, continuous motion remains challenging. We introduce RayCo4D, a ray-conditioned continuous motion model for monocular 4D reconstruction that explicitly separates visual measurements from latent 3D motion states. Each track is associated with a reference-time 3D query constrained to its camera ray, and its image and depth observations constrain shared latent motion nodes through local many-to-many associations. Each node maintains an SE(3) pose and body-frame velocity at sparse support times. A white-noise-on-acceleration process prior couples neighboring states and defines continuous-time interpolation, allowing observations across visibility intervals to jointly constrain the same latent state sequence. The inferred shared motion is transferred to Gaussian primitives through dual-quaternion blending, while reference-anchored local SE(3) residuals capture fine-scale non-rigid deformation beyond the shared scaffold. Experiments on diverse monocular dynamic scenes show that RayCo4D achieves competitive novel-view synthesis quality, improves structural and temporal consistency under rapid motion and incomplete tracks, and supports continuous-time rendering.
Figures & tables
Figure 1: RayCo4D versus prior track-driven methods. Top: MoSca [ 18 ] initializes motion support from lifted tracks; RayCo4D treats tracks as ray-conditioned measurements of shared latent nodes, without one-to-one track–node binding. Bottom: SE(3) B-splineGS [ 41 ] utilizes continuous motion with spline trajectories, whereas RayCo4D models it using sparse SE(3) pose-velocity states with a WNOA prior for continuous-time queries.
Figure 2: Overview of RayCo4D. Given a monocular video, RayCo4D constructs ray-conditioned 3D queries from 2D tracks and camera-depth estimates to constrain shared latent motion nodes. Sparse SE(3) pose–velocity states are coupled by a WNOA prior to model continuous motion. The inferred motion is propagated to Gaussians via dual-quaternion blending, while reference-anchored local residuals capture fine-grained non-rigid deformation for dynamic rendering and optimization.
DyCheck
NVIDIA
Method
mPSNR ↑
mSSIM ↑
mLPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
MoSca [ 18 ]
19.86
0.702
0.298
25.89
0.839
0.079
Shape of Motion [ 28 ]
18.63
0.674
0.321
24.89
0.732
0.113
OriGS [ 33 ]
19.97
0.707
0.307
26.53
0.850
0.068
RiGS [ 31 ]
18.89
0.689
0.354
25.24
0.793
0.098
SE(3) B-splineGS [ 41 ]
20.53
0.713
0.276
26.78
0.848
0.063
Table 1: Novel-view synthesis and component ablations. DyCheck metrics are computed within the co-visible regions, while NVIDIA metrics follow the standard held-out-view protocol. The lower block reports component ablations on both datasets; underlines mark the best baseline.
Figure 3: Qualitative comparisons on DyCheck, NVIDIA, and DAVIS. Ground-truth held-out views are shown for DyCheck and NVIDIA; DAVIS provides qualitative comparisons under rapid motion, occlusion, and non-rigid deformation.
Figure 4: Qualitative ablations on NVIDIA and DAVIS. Comparisons with the full model illustrate the effects of individual components on structural coherence and fine-scale deformation.
MoSca ( Lei et al. , 2025 )
OriGS ( Wu et al. , 2025a )
SE(3) B-splineGS ( Zhang et al. , 2026 )
RayCo4D
0.822
0.826
0.832
0.840
Table 2: Motion correspondence on DyCheck (PCK-T ↑ ). Best result is highlighted in bold.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
SE(3) B-splineGS [ 41 ]
25.76
0.835
0.081
RayCo4D
26.87
0.851
0.060
Table 3: Intermediate-time reconstruction on NVIDIA. Best result is highlighted in bold.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
Indices and time
f,i,j,g,k
Frame, track, motion-node, Gaussian, and support-time indices.
F,M,S
Numbers of video frames, motion nodes, and temporal supports.
t,τk,Δk
Query time, support time, and interval Δk=τk+1−τk .
ri,rg
Reference times of track i and Gaussian g .
tˉ
Normalized time t/max(F−1,1) for local motion curves.
Appendix
Table 4: Main notation used in RayCo4D. Symbols are grouped by their roles in the representation.
Stage
Iterations
Optimized variables
Motion warm-up
4,000
Node states and query depths
Gaussian warm-up
1,000
Gaussian geometry and appearance
Joint fitting
6,000
Scene, motion, and calibration
Final fitting
1,000
Continuous variables; fixed structure
Appendix
Table 5: Training stages. Iterations are counted separately for each row.
Parameter group
Learning rate
Reference-query depths
10−2
Node translation, rotation, and body velocity
3×10−3
Gaussian position
1.6×10−4
Gaussian rotation / scale / opacity
10−3 / 5×10−3 / 5×10−2
SH coefficients: constant / higher-order
2.5×10−3 / 1.25×10−4
Gaussian binding offsets
10−2
Appendix
Table 6: Initial learning rates. Node and query-depth entries refer to motion warm-up; their scene-fitting rates are scaled by 0.1 .
Method
Pre.
Train.
MoSca [ 18 ]
20
40
Shape of Motion [ 28 ]
35
63
OriGS [ 33 ]
30
77
RiGS [ 31 ]
37
56
SE(3) B-splineGS [ 41 ]
27
61
USPLAT4D [ 8 ]
60
20
Appendix
Table 7: Average runtime on DyCheck (min).
Figure 5: Continuous-time reconstruction on DAVIS, NVIDIA, and DyCheck. GT denotes intermediate frames excluded from image, track, and depth supervision. All renderings match the GT timestamps and camera views.
Figure 6: Challenging reconstruction examples on DyCheck, DAVIS, and NVIDIA. GT denotes held-out real frames excluded from training; RayCo4D renderings use the corresponding timestamps and camera views.
Figure 7: Qualitative reconstruction comparison on the “Breakdance-flare” scene of DAVIS [ 23 ] .
Figure 10: Visual comparison of novel-view synthesis on the “Wheel” scene of the DyCheck dataset [ 7 ] .
Figure 12: Visual comparison of novel-view synthesis on the “Paper-windmill” scene of the DyCheck dataset [ 7 ] .
Reconstructing dynamic 4D scenes remains challenging due to the presence of moving objects that corrupt camera pose estimation. Existing optimization methods alleviate this issue with additional supervision, but they are mostly computationally expensive and impractical in real-time applications. To address these limitations, we propose MoRe, a feedforward 4D reconstruction network that efficiently recovers dynamic 3D scenes from monocular videos. Built upon a strong static reconstruction backbone, MoRe employs an attention-forcing strategy to disentangle dynamic motion from static structure. To further enhance robustness, we fine-tune the model on large-scale, diverse datasets encompassing both dynamic and static scenes. Moreover, our grouped causal attention captures temporal dependencies and adapts to varying token lengths across frames, ensuring temporally coherent geometry reconstruction. Extensive experiments on multiple benchmarks demonstrate that MoRe achieves high-quality dynamic reconstructions with exceptional efficiency.
Juntong Fang, Zequn Chen, Weiqi Zhang +4
School of Software, Tsinghua University, Beijing, China · Li Auto
Reconstructing dynamic 4D scenes requires jointly estimating correspondence, geometry, object motion, and camera motion. Existing feed-forward methods typically predict dense task-specific maps or independently process source-target pairs, leading to unnecessary computation for sparse queries and limited feature reuse across different frame pairs. We present UniQuery4R, a query-conditioned framework that encodes a multi-frame clip once and selects the source view, target view, and continuous source-image coordinate only at decoding time via source-to-target cross-attention. Each query jointly predicts target correspondence, target-time 3D position, and scene flow, along with source depth, while camera parameters are estimated per view. This design allows the encoded clip to be reused across arbitrary source-target selections and supports both sparse inference and dense reconstruction through batched queries, without learned temporal embeddings tied to a fixed clip length. We further introduce a direction-magnitude parameterization of scene flow with separate supervision for moving and static points. Among the evaluated methods, UniQuery4R achieves the best macro-average results on WorldTrack for both scene-flow estimation and dynamic-point reconstruction.
Tiancheng Chen, Sheng Tang, Wenhua Jin +4
Kosmo Research · Automotive Engineering Department, Jilin University · School of Software, Tsinghua University
At the intersection of computer vision and robotic perception, 4D reconstruction of dynamic scenes connects low-level geometric sensing with high-level semantic understanding. We present Genie 4D, a framework that turns hand-held phone capture into a semantically grounded, action-controllable 4D world model. Genie 4D couples a real-time visual-inertial Gaussian splatting front-end for metric geometry with a feed-forward 4D backbone regularized by frozen DINOv3 features acting as structural priors. The semantic priors suppress identity drift during dynamic tracking, while a short conditional diffusion refiner recovers high-frequency surface detail that regression backbones smooth away. Finally, a lightweight latent-action head exposes the reconstructed 4D state to a Genie-style world model trained with a JEPA-style next-embedding objective, so that the scene can be rolled forward under user actions. On the Point Odyssey and TUM-Dynamics benchmarks, Genie 4D retains the linear time complexity O(T) of feed-forward baselines while improving 3D tracking accuracy (APD) and reconstruction completeness, and it runs interactively on a single consumer GPU (RTX 5090) from iPhone, Mac, Windows, and Linux capture clients. Genie 4D offers a practical, semantic-prior-guided path toward physically grounded world models.
Yiru Yang, Zhuojie Wu, Nishant Kumar Singh +1
University of Zurich Zurich, Switzerland · ETH Zurich Zurich, Switzerland