At the intersection of computer vision and robotic perception, 4D reconstruction of dynamic scenes connects low-level geometric sensing with high-level semantic understanding. We present Genie 4D, a framework that turns hand-held phone capture into a semantically grounded, action-controllable 4D world model. Genie 4D couples a real-time visual-inertial Gaussian splatting front-end for metric geometry with a feed-forward 4D backbone regularized by frozen DINOv3 features acting as structural priors. The semantic priors suppress identity drift during dynamic tracking, while a short conditional diffusion refiner recovers high-frequency surface detail that regression backbones smooth away. Finally, a lightweight latent-action head exposes the reconstructed 4D state to a Genie-style world model trained with a JEPA-style next-embedding objective, so that the scene can be rolled forward under user actions. On the Point Odyssey and TUM-Dynamics benchmarks, Genie 4D retains the linear time complexity O(T) of feed-forward baselines while improving 3D tracking accuracy (APD) and reconstruction completeness, and it runs interactively on a single consumer GPU (RTX 5090) from iPhone, Mac, Windows, and Linux capture clients. Genie 4D offers a practical, semantic-prior-guided path toward physically grounded world models.
Figures & tables
Figure 1: Genie 4D turns a live phone stream into a semantically grounded, action-controllable 4D world model. A real-time visual–inertial Gaussian-splatting front-end (VIGS-SLAM) supplies metric geometry; a DINOv3-regularised 4D backbone with diffusion refinement supplies semantic 4D; and a latent-action head exposes the result to a world model for interactive rollout.
Figure 2: Genie 4D architecture. A frozen St4RTrack backbone (ViT encoder + dual decoder) maps an RGB video to two 3D pointmaps via a tracking head and a reconstruction head, which are lifted into 4D trajectories. In parallel, a frozen DINOv3 semantic encoder supplies patch-token priors that label the recovered geometry, yielding a semantically grounded 4D reconstruction ( train, track, tree, building, ground, object ). Snowflake icons mark frozen components; the flame icon marks the trainable module.
Table 3: Component ablation. Each module contributes, with semantic terms driving the tracking gain and the diffusion refiner driving the reconstruction gain.
Figure 3: Quantitative summary. Top: tracking accuracy and reconstruction error against baselines. Bottom-left: component ablation. Bottom-right: inference latency stays flat with sequence length, confirming O(T) scaling against the quadratic VGGT.
Figure 4: Camera-tracking accuracy of our Genie 4D front-end (ATE-RMSE ↓ ) against DROID-SLAM. Tight IMU coupling halves drift on the tested sequences.
Figure 5: Front-end signals computed across the real hand-held sequence ( video_1 ). Rows: RGB key-frames at increasing time, dense optical flow (Farneback, HSV-coloured), motion magnitude, and color-coded point trajectories. Trajectories densify on textured structure and thin out on the textureless road and sky—the regime where the semantic priors of Section 3 are designed to stabilise correspondence.
Figure 6: Real point trajectories tracked on a hand-held sequence, sparsified to the longest tracks for legibility. Colour encodes time within the window (see colour bar); the zoom-in inset over a textured façade shows correspondences held across many frames, evidencing persistent local tracking under camera motion. These 2D tracks are lifted to metric 4D using the VIGS-SLAM poses.
Figure 7: Real hand-held capture and front-end signals. Left: the live PhoneStreamer iOS client streaming RGB + IMU to the capture server (note the on-screen intrinsics fx=fy=850.2 , frame and IMU counters). Right: an RGB key-frame, dense optical flow, motion magnitude, and tracked point trajectories computed on the same sequence.
Figure 8: On-device capture and a second real sequence. Left: the PhoneStreamer iOS client mid-capture; the overlay reports the locked intrinsics, the running frame count and the buffered IMU samples, while the server address and port configure the live stream (capture on iPhone; playback/processing on Mac, Windows and Linux). Centre/right: an RGB key-frame and color-coded point trajectories from a second hand-held sequence ( video2 ), confirming the front end generalises across scenes. Per-sequence statistics are in the supplementary.
Module
Params
Latency (ms)
DINOv3 ViT-L/14 (frozen)
300 M
14.2
St4RTrack backbone (frozen)
410 M
18.6
Semantic Injection Adapter
18 M
3.1
DiffusionRefiner ( K=5 )
9 M
4.0
Latent-action world model
2 M
1.3
Genie 4D (total)
739 M
41.2
Table 4: Per-module parameter count and per-frame latency on an RTX 5090 ( 640×341 input). Only 29 M parameters are trainable; the frozen encoders are shared across the sequence.