Organizations: Southern University of Science and Technology, Shenzhen, China · Fudan University, Shanghai, China · East China University of Science and Technology, Shanghai, China · Beijing Technology and Business University, Beijing, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
Reconstructing dynamic 3D scenes from monocular video requires inferring motion from noisy and intermittently visible observations. Long-range tracks provide valuable geometric constraints, but integrating their fragmented temporal coverage into shared, continuous motion remains challenging. We introduce RayCo4D, a ray-conditioned continuous motion model for monocular 4D reconstruction that explicitly separates visual measurements from latent 3D motion states. Each track is associated with a reference-time 3D query constrained to its camera ray, and its image and depth observations constrain shared latent motion nodes through local many-to-many associations. Each node maintains an SE(3) pose and body-frame velocity at sparse support times. A white-noise-on-acceleration process prior couples neighboring states and defines continuous-time interpolation, allowing observations across visibility intervals to jointly constrain the same latent state sequence. The inferred shared motion is transferred to Gaussian primitives through dual-quaternion blending, while reference-anchored local SE(3) residuals capture fine-scale non-rigid deformation beyond the shared scaffold. Experiments on diverse monocular dynamic scenes show that RayCo4D achieves competitive novel-view synthesis quality, improves structural and temporal consistency under rapid motion and incomplete tracks, and supports continuous-time rendering.
Figures & tables
Figure 1: RayCo4D versus prior track-driven methods. Top: MoSca [ 18 ] initializes motion support from lifted tracks; RayCo4D treats tracks as ray-conditioned measurements of shared latent nodes, without one-to-one track–node binding. Bottom: SE(3) B-splineGS [ 41 ] utilizes continuous motion with spline trajectories, whereas RayCo4D models it using sparse SE(3) pose-velocity states with a WNOA prior for continuous-time queries.
Figure 2: Overview of RayCo4D. Given a monocular video, RayCo4D constructs ray-conditioned 3D queries from 2D tracks and camera-depth estimates to constrain shared latent motion nodes. Sparse SE(3) pose–velocity states are coupled by a WNOA prior to model continuous motion. The inferred motion is propagated to Gaussians via dual-quaternion blending, while reference-anchored local residuals capture fine-grained non-rigid deformation for dynamic rendering and optimization.
DyCheck
NVIDIA
Method
mPSNR ↑
mSSIM ↑
mLPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
MoSca [ 18 ]
19.86
0.702
0.298
25.89
0.839
0.079
Shape of Motion [ 28 ]
18.63
0.674
0.321
24.89
0.732
0.113
OriGS [ 33 ]
19.97
0.707
0.307
26.53
0.850
0.068
RiGS [ 31 ]
18.89
0.689
0.354
25.24
0.793
0.098
SE(3) B-splineGS [ 41 ]
20.53
0.713
0.276
26.78
0.848
0.063
Table 1: Novel-view synthesis and component ablations. DyCheck metrics are computed within the co-visible regions, while NVIDIA metrics follow the standard held-out-view protocol. The lower block reports component ablations on both datasets; underlines mark the best baseline.
Figure 3: Qualitative comparisons on DyCheck, NVIDIA, and DAVIS. Ground-truth held-out views are shown for DyCheck and NVIDIA; DAVIS provides qualitative comparisons under rapid motion, occlusion, and non-rigid deformation.
Figure 4: Qualitative ablations on NVIDIA and DAVIS. Comparisons with the full model illustrate the effects of individual components on structural coherence and fine-scale deformation.
MoSca ( Lei et al. , 2025 )
OriGS ( Wu et al. , 2025a )
SE(3) B-splineGS ( Zhang et al. , 2026 )
RayCo4D
0.822
0.826
0.832
0.840
Table 2: Motion correspondence on DyCheck (PCK-T ↑ ). Best result is highlighted in bold.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
SE(3) B-splineGS [ 41 ]
25.76
0.835
0.081
RayCo4D
26.87
0.851
0.060
Table 3: Intermediate-time reconstruction on NVIDIA. Best result is highlighted in bold.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
Indices and time
f,i,j,g,k
Frame, track, motion-node, Gaussian, and support-time indices.
F,M,S
Numbers of video frames, motion nodes, and temporal supports.
t,τk,Δk
Query time, support time, and interval Δk=τk+1−τk .
ri,rg
Reference times of track i and Gaussian g .
tˉ
Normalized time t/max(F−1,1) for local motion curves.
Appendix
Table 4: Main notation used in RayCo4D. Symbols are grouped by their roles in the representation.
Stage
Iterations
Optimized variables
Motion warm-up
4,000
Node states and query depths
Gaussian warm-up
1,000
Gaussian geometry and appearance
Joint fitting
6,000
Scene, motion, and calibration
Final fitting
1,000
Continuous variables; fixed structure
Appendix
Table 5: Training stages. Iterations are counted separately for each row.
Parameter group
Learning rate
Reference-query depths
10−2
Node translation, rotation, and body velocity
3×10−3
Gaussian position
1.6×10−4
Gaussian rotation / scale / opacity
10−3 / 5×10−3 / 5×10−2
SH coefficients: constant / higher-order
2.5×10−3 / 1.25×10−4
Gaussian binding offsets
10−2
Appendix
Table 6: Initial learning rates. Node and query-depth entries refer to motion warm-up; their scene-fitting rates are scaled by 0.1 .
Method
Pre.
Train.
MoSca [ 18 ]
20
40
Shape of Motion [ 28 ]
35
63
OriGS [ 33 ]
30
77
RiGS [ 31 ]
37
56
SE(3) B-splineGS [ 41 ]
27
61
USPLAT4D [ 8 ]
60
20
Appendix
Table 7: Average runtime on DyCheck (min).
Figure 5: Continuous-time reconstruction on DAVIS, NVIDIA, and DyCheck. GT denotes intermediate frames excluded from image, track, and depth supervision. All renderings match the GT timestamps and camera views.
Figure 6: Challenging reconstruction examples on DyCheck, DAVIS, and NVIDIA. GT denotes held-out real frames excluded from training; RayCo4D renderings use the corresponding timestamps and camera views.
Figure 7: Qualitative reconstruction comparison on the “Breakdance-flare” scene of DAVIS [ 23 ] .
Figure 10: Visual comparison of novel-view synthesis on the “Wheel” scene of the DyCheck dataset [ 7 ] .
Figure 12: Visual comparison of novel-view synthesis on the “Paper-windmill” scene of the DyCheck dataset [ 7 ] .