Predicting scene dynamics from visual observations is challenging. Existing methods capture dynamics only within observed boundaries failing to extrapolate far beyond the training sequence. Node-RF (Neural ODE-based NeRF) overcomes this limitation by integrating Neural Ordinary Differential Equations (NODEs) with dynamic Neural Radiance Fields (NeRFs), enabling a continuous-time, spatiotemporal representation that generalizes beyond observed trajectories at constant memory cost. From visual input, Node-RF learns an implicit scene state that evolves over time via an ODE solver, propagating feature embeddings via differential calculus. A NeRF-based renderer interprets calculated embeddings to synthesize arbitrary views for long-range extrapolation. Training on multiple motion sequences with shared dynamics allows for generalization to unseen conditions. Our experiments demonstrate that Node-RF can characterize abstract system behavior without explicit model to identify critical points for future predictions.
Figures & tables
Figure 1: Node-RF Overview. Multiple observations of the red ball being dropped onto the dual ramp end in damped oscillations towards the two valleys. Node-RF learns to encodes generalized dynamics of deterministic motions from sequences such as A (pink), C (yellow) into an implicit space-time latent representation using NeRF and nODE. Embedded sequence states depicted as scene latent points zt on the embedding space are propagated through time t using an implicit neural ODE. Intermediate and future states (centre to right encodings) can be extrapolated and rendered (top, bottom) given initial frame conditions (on the left). Latent divergence ∇zt (colour coded) characterizes behaviour of the learnt system. Various initial states of dropping the ball at locations around the centre (green) propagate through time (lines) towards the steady state sinks with ∇zt<0 (right).
Figure 2: Overview of Node-RF . zcan , zt0 , zt1 are learned during warmup iterations. The Multiple Sequence Training along with NeRF (Common Loop) illustrates the Generalized Multi-Sequence Learning task. The Single Sequence Training along with NeRF (Common Loop) illustrates the Continuous Single-Sequence Dynamics task.
Table 2: Comparison on Pendulum [ 9 ] . Evaluated only on the foreground dynamic part. Interpolation and extrapolation are reported. A ✓ under TC i.e. Temporal Continuity indicates that the method models continuous temporal dynamics.
Method
3D
Oscillating Ball
Bifurcating Hill
Vid-ODE
✗
-
0.000
SimVP
✗
-
0.295
D-NeRF(c)
✓
0.0008
0.003
Ours
✓
0.3327
0.485
Table 3: Comparison on Oscillating Ball (3D) and Bifurcating Hill (2D) on IoU between GT and prediction flows. A ✓ under 3D indicates that the method can handle 3D scenes.
Figure 3: Extrapolation results on the Pendulum dataset [ 9 ] .
Figure 5: Pose error for a novel trajectory in Bifurcating Hill .
Figure 6: Dynamic Flow Masks for the predictions of a novel sequence in the Oscillating Ball scene
Figure 7: Dynamic Flow Masks for the predictions of a novel sequence in the Bifurcating Hill scene
Losses Added
SSIM
LPIPS
PSNR
IoU
LNeRF
0.630
0.4920
28.661
0.2730
LNeRF+Lp+Lv
0.661
0.4396
29.080
0.3253
LNeRF+Lp+Lv+Llipschitz
0.662
0.4364
29.091
0.3327
Table 4: Ablation study on loss term in Oscillating Ball .
# Latent Size
SSIM
LPIPS
PSNR
256
0.976
0.0318
32.29
512
0.978
0.0310
33.70
1024
0.975
0.0397
32.74
Table 5: Ablation studies of the latent size on Reconstruction set of Bouncing Balls
Figure 8: Latent Space Comparison of models w/o and w/ Lipschitz regularization. The green and red dots represent the starting and ending points of the trajectories respectively