Estimating the 3D trajectory of every pixel from a monocular video is crucial and promising for a comprehensive understanding of the 3D dynamics of videos. Recent monocular 3D tracking works demonstrate impressive performance, but are limited to either tracking sparse points on the first frame or a slow optimization-based framework for dense tracking. In this paper, we propose a feedforward model, called Track4World, enabling an efficient holistic 3D tracking of every pixel in the world-centric coordinate system. Built on the global 3D scene representation encoded by a VGGT-style ViT, Track4World applies a novel 3D correlation scheme to simultaneously estimate the pixel-wise 2D and 3D dense flow between arbitrary frame pairs. The estimated scene flow, along with the reconstructed 3D geometry, enables subsequent efficient 3D tracking of every pixel of this video. Extensive experiments on multiple benchmarks demonstrate that our approach consistently outperforms existing methods in 2D/3D flow estimation and 3D tracking, highlighting its robustness and scalability for real-world 4D reconstruction tasks.
Figures & tables
Figure 2: Overview . Given (a) the input video frames, Track4World first extracts (b) global scene representations (geometric embeddings, point clouds, and camera poses). (c) A sparse-to-dense scene flow decoder then predicts 2D-3D joint flows between arbitrary timesteps, which applies a novel 2D-to-3D correlation scheme to improve efficiency and allows 2D-3D joint supervision. (d) The pairwise flows are ultimately fused to establish holistic world-centric 3D tracking.
Figure 3: Comparison of correlation mechanisms. Prior methods rely on explicit k -nearest neighbor searches and cross-attention in 3D space, leading to high computational costs. In contrast, our proposed method anchors 3D updates directly to intermediate image-plane correlations. This design significantly improves computational efficiency and allows the 3D tracking module to be effectively boosted by abundant 2D training data.
Method
PointOdyssey [ 102 ]
ADT [ 61 ]
PStudio [ 32 ]
DriveTrack [ 3 ]
Avg.
L-16
L-50
L-16
L-50
L-16
L-50
L-16
L-50
L-16
L-50
Camera coordinate 3D tracking
SpatialTracker ∗ [ 87 ]
0.3116
0.2977
0.4962
0.4692
0.5390
0.4991
0.2529
0.2502
0.3999
0.3791
DELTA ∗ [ 59 ]
0.3529
0.3412
0.5116
0.4952
0.5922
0.5533
0.2704
0.2701
0.4317
0.4150
STV2 † [ 86 ]
0.1864
0.1785
0.2400
0.2330
0.3784
0.3690
0.1711
0.1725
0.2400
0.2383
MASt3R [ 43 ]
0.3546
0.3253
0.3368
0.3029
0.3293
0.2956
0.2767
0.2559
0.3244
0.2949
Table 2: 3D tracking estimation. We report the APD metric to evaluate 3D point tracking on the PointOdyssey [ 102 ] , ADT [ 61 ] , PStudio [ 32 ] , and DriveTrack [ 3 ] datasets. L-16 and L-50 indicate tracking within the temporal length of 16 and 50 frames, respectively. * indicates that ground-truth camera intrinsics are used. † indicates that bundle adjustment is not used. ‡ denotes methods using camera poses estimated by VGGT [ 76 ] .
Figure 4: Qualitative results on diverse in-the-wild videos.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. S1 : Effectiveness of ℓsmooth3d .
Setting
Abs Rel ↓
δ<1.25↑
w/ scale-inv
0.1060
0.8788
w/o ℓreg
0.3530
0.3547
w/o ℓlocal
0.1053
0.8790
Full (Ours)
0.0973
0.8921
Appendix
Table S1 : Ablation study on geometry encoder. The results are averaged over all datasets. Best results are highlighted in darker blue , and second best in lighter blue .
Fig. S2 : Scene flow Visualization. The deformed point maps (colored ) show that our method produces more temporally consistent geometry and motion.
Fig. S3 : Scene Flow Visualization. The deformed point maps (shown in solid colors) indicate that our method produces higher-quality geometry and motion.
Fig. S4 : Visualization of first-frame 2D dense tracking.
Fig. S5 : Visualization of first-frame 3D dense tracking.
Fig. S6 : Visualization of dense per-pixel trajectories across all frames.
Fig. S7 : Visualization of the dense trajectories across all frames in the global world coordinate system.
Dense 3D tracking from monocular video is fundamental to dynamic scene understanding. While recent 3D foundation models provide reliable per-frame geometry, recovering object motion in this geometry remains challenging and benefits from strong motion priors learned from real-world videos. Existing 3D trackers either follow iterative paradigms trained from scratch on synthetic data or fine-tune 3D reconstruction models learned from static multi-view images, both lacking real-world motion priors. Pre-trained video diffusion transformers (video DiTs) offer rich spatio-temporal priors from internet-scale videos, making them a promising foundation for 3D tracking. However, their frame-anchored formulation, which generates each frame's content, is fundamentally mismatched with reference-anchored dense 3D tracking, which must follow the same physical points from a reference frame across time. We present TrackCraft3R, the first method to repurpose a video DiT as a feed-forward dense 3D tracker. Given a monocular video and its frame-anchored reconstruction pointmap, TrackCraft3R predicts a reference-anchored tracking pointmap that follows every pixel of the first frame across time in a single forward pass, along with its visibility. We achieve this through two designs: (i) a dual-latent representation that uses per-frame geometry latents and reference-anchored track latents as dense queries, and (ii) temporal RoPE alignment, which specifies the target timestamp of each track latent. Together, these designs convert the per-frame generative paradigm of video DiTs into a reference-anchored tracking formulation with LoRA fine-tuning. TrackCraft3R achieves state-of-the-art performance on standard sparse and dense 3D tracking benchmarks, while running 1.3x faster and using 4.6x less peak memory than the strongest prior method. We further demonstrate robustness to large motions and long videos.
Dense 3D point tracking has been a prominent paradigm for modeling motion in dynamic scenes, but a point track is just a 3-DoF translation curve per pixel: it captures where pixels go, not the rotation of the underlying part, nor which pixels move together as one body. We propose MoSE3, the first feed-forward model that predicts dense SE(3) motion from monocular RGB video, producing full 6-DoF rigid transforms at every pixel in world space. Per-pixel SE(3) motion offers a richer view of how a scene moves: rotation, translation, and grouping all at once. Directly predicting SE(3) is challenging: rotations lie on a curved manifold that is ill-suited to Euclidean regression, and annotations for SE(3) are particularly difficult to acquire. To address these challenges, MoSE3 predicts per-pixel SE(3) through two jointly learned intermediates, 3D point tracks and rigidity embeddings, and recovers SE(3) by differentiably fitting transforms within each soft rigid cluster, enabling end-to-end prediction and supervision. To close the data gap, we introduce Art-Kubric, a large-scale synthetic dataset with dense SE(3) and rigidity labels for articulated objects with rich physical interactions. MoSE3 achieves state-of-the-art SE(3) estimation at pixel, part, and object levels on both rigid and articulated benchmarks, and state-of-the-art average 3D point tracking accuracy across three datasets, while showing strong generalization to real-world videos despite being trained solely on synthetic motion data.
Jiahuan Cheng, Zhiyi Li, Tian Xia +3
Harvard University · Johns Hopkins University · MIT +1
Existing point tracking models face a fundamental tradeoff: they can either track a sparse set of query points over long horizons, or track all points across only short clips. We introduce TrackEverything, a 3D point tracker that breaks this trade-off by representing videos as persistent 3D scene tracks in world coordinates. Grounded in the insight that videos are 2D projections of an underlying 3D world, TrackEverything decouples model complexity from video duration, allowing it to scale with unique physical scene geometry instead. Our approach introduces three key innovations. First, we employ a voxelization-based de-duplication mechanism at sliding-window boundaries to merge co-located tracks, preventing repeated observations of the same surface from redundantly accumulating. Second, we decompose tracking into an endpoint refiner that predicts each point's destination and static-versus-dynamic classification, followed by a lightweight trajectory refiner that decodes dense trajectories exclusively for dynamic points. Third, we propose 3D WAFT, replacing memory-prohibitive 4D correlation volumes with efficient feature sampling in the scene cloud. To the best of our knowledge, TrackEverything is the first 3D tracker capable of tracking all visible points across videos exceeding 1000 frames within 40 GB of GPU memory. On TAPVid-3D, TrackEverything outperforms all open-source all-frame dense 3D trackers by more than 20% APD on short clips, while remaining competitive with state-of-the-art sparse trackers on long sequences, despite tracking far more points.