Dense 3D tracking from monocular video requires establishing point correspondences over time, estimating their depth, and inferring their trajectories during occlusion. Existing methods typically learn these tasks jointly, a demanding objective given the scarcity of 4D supervision. We show that strong pretrained 2D trackers and 3D geometry models provide an effective foundation: lifted trajectories, with occluded segments simply interpolated, already rival dedicated 3D trackers using the same geometry. However, lifted trajectories inherit correspondence and depth errors, and under occlusion, their depth comes from the occluder rather than the tracked point. We introduce ReLift, an efficient feed-forward framework that recasts dense 3D tracking as geometric trajectory repair and completion. At its core, a trajectory corrector alternates attention along and across trajectories to preserve reliable observations, repair corrupted positions, and complete missing motion. Using only geometric inputs allows us to train directly on 3D trajectories with simulated lifting errors, decoupling training from images and from any specific tracker or geometry model. Concentrating 4D supervision on trajectory refinement yields a reusable model that consistently improves lifted trajectories across trackers and geometry providers without retraining, so it benefits directly from future advances in either component. On WorldTrack, ReLift achieves state-of-the-art mean accuracy, improving on the strongest baselines by 2.1 APD on all points and 3.0 on dynamic points, with faster configurations remaining competitive with leading 3D trackers while running faster than 4RC. Visit our supplementry page for more examples and code.
Figures & tables
Figure 1: ReLift combines pretrained 2D tracking and 3D geometry with learned trajectory repair and completion, achieving state-of-the-art dense 3D tracking accuracy on WorldTrack and a strong speed–accuracy trade-off. Left: Long-range 3D trajectories recovered from a real-world video. Right: WorldTrack accuracy versus full-pipeline FPS (see Appendix D.4 for details).
Figure 2: Inference pipeline of ReLift. Frozen pretrained models lift video observations (a–b) into corrupted and incomplete 3D trajectories (c). The corrector adds residuals to initialized XYZ coordinates to produce the final trajectories (d–e). Below: ray–point tokens ϕti are grouped into C -frame tracklets and embedded as tokens hni , then processed by alternating temporal and across-track attention. The output head predicts frame-level XYZ residuals, which are added to the initialized trajectories. We train the corrector on simulated lifts from clean trajectory annotations (Sec. 3.4 ).
ADT
PS
PO
DR
Mean
Method
All
Dyn
All
Dyn
All
Dyn
All
Dyn
All
Dyn
Time (s)
4RC
87.07
78.57
86.25
86.10
85.92
81.17
88.51
80.56
86.94
81.60
31.8
Any4D
76.17
78.32
77.57
77.61
73.96
73.56
83.54
72.83
77.81
75.58
3.2
MotionCrafter
67.04 ‡
83.34 ‡
62.60
62.63
46.29
49.09
70.83
62.39
61.69 ‡
64.36 ‡
20.4
OpenD4RT
71.71
79.68
84.36
85.27
68.39
74.21
77.55
79.69
75.50
79.71
284 †
St4RTrack
81.39
79.39
72.84
72.24
71.36
71.44
78.29
67.23
75.97
72.58
2.9
Table 1: Dense 3D tracking on WorldTrack. APD3D (%) on all (All) and dynamic (Dyn) points after RANSAC Sim(3) alignment; bold best, underline second best. Time includes the complete pipeline. † Timing at 2562 instead of 5122 . ‡ Restricted ADT coverage for MotionCrafter, also affecting its means (Appendix B.2 ).
Figure 3: Gains from trajectory repair and completion. APD3D gains (%) over the initialized lift (Sec. 3.2 ), per dataset and averaged, for all six tracker–geometry combinations; hatching denotes CoWTracker. Occluded selects ground-truth-occluded point–frames from All , retaining the All-point alignment. This subset may differ from the corrector’s masked inputs.
Figure 4: Qualitative comparison. Top to bottom: DyCheck spin ( Gao et al., 2022 ) and DAVIS car-shadow and libby ( Perazzi et al., 2016 ) . Trajectories of the same reference pixels are shown from matched novel viewpoints. Ours uses AllTracker and VGGT- Ω ; TrackCraft3R shares this geometry, while 4RC and V-DPM use their own reconstructions. Yellow boxes mark magnified regions.
Figure 5: Simulated versus provider-generated lifts. APD3D gains on all points, in percentage points over the initialized lift, with AllTracker and DA3 (left) or VGGT- Ω (right). “Real” denotes non-simulated, provider-generated training lifts. All runs use the same architecture and 150,000 training steps. Appendix D.1 reports a control with matched training windows.
AllTracker + DA3
AllTracker + VGGT- Ω
CoWTracker + DA3
CoWTracker + VGGT- Ω
Training corruptions
All
Dyn
All
Dyn
All
Dyn
All
Dyn
Lift (the corrector’s input)
82.1
79.3
84.6
81.6
82.2
79.5
84.9
82.3
Full simulation (ours)
83.7 (+1.6)
81.7 (+2.4)
86.6 (+2.0)
84.9 (+3.4)
84.5 (+2.4)
82.2 (+2.8)
87.5 (+2.6)
85.8 (+3.6)
without correlated noise
83.5 (+1.5)
81.3 (+2.0)
86.4 (+1.8)
84.5 (+2.9)
84.2 (+2.0)
82.1 (+2.7)
87.3 (+2.5)
85.7 (+3.5)
without wrong surface
82.3 (+0.2)
79.6 (+0.3)
84.9 (+0.3)
82.1 (+0.5)
82.8 (+0.6)
80.2 (+0.7)
85.5 (+0.6)
83.1 (+0.9)
without synthetic gaps
83.7 (+1.6)
81.5 (+2.2)
86.6 (+2.0)
84.9 (+3.4)
84.5 (+2.4)
82.3 (+2.8)
87.5 (+2.6)
85.8 (+3.6)
Table 2: Contribution of training corruptions. Removing wrong-surface corruptions causes the largest drop. APD3D (%), averaged over four datasets; gains over the initialized lift are shown in parentheses, in percentage points. Each variant is retrained with the same data, network, loss and budget (Appendix D.1 ).
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Frames per tracklet C
4
Feature width D
384
Factorized attention blocks
8
Attention heads
6, with width 64 each
Feed-forward hidden width / activation
1536 / GELU
Output normalization
Final layer norm
Appendix
Table 3: Corrector architecture settings.
Term
Distribution
Flicker ft
std U(0,0.03) , autoregressive with κ∼U(0.3,0.95)
Tilt bt
std U(0,0.03) , autoregressive with the flicker’s κ
Bias βti
std U(0,0.02) , autoregressive with κ=0.9
White noise ηti
std U(0.002,0.02)
Outliers jti
rate U(0,0.05) per cell, next-frame persistence probability 0.5 , jump ±U(0.1,0.5)
Angular ray noise
std U(0.0005,0.005) rad
Appendix
Table 4: Training corruption distributions. U denotes uniform sampling. Noise levels and outlier rates are drawn once per training example.
Method
PS / DR / PO
ADT
Inference settings
4RC ( Luo et al., 2026 )
280×504
5042
Any4D ( Karhade et al., 2025 ) V-DPM ( Sucar et al., 2026 )
294×518
5182
St4RTrack ( Feng et al., 2025 ) TraceAnything ( Liu et al., 2026 )
288×512
5122
MotionCrafter ( Zhu et al., 2026 )
320×640
320×640
Centre crop.
OpenD4RT ( RHOS Team, 2026 )
2562
2562
Resize, no crop; stride-2, 1282 grid.
TrackCraft3R ( Nam et al., 2026 )
480×832
480×832
Interleaved 12-frame clips, stride 8.
Appendix
Table 5: Baseline configurations. Input grids are H×W ; settings apply to the accuracy comparisons. Runtime measurements use the settings in Appendix D.4 .
RANSAC Sim(3)
Global scale
Method
All
Dyn
EPE
All
Dyn
EPE
4RC
85.33
78.81
0.299
82.40
73.68
0.342
Any4D
65.31
55.51
0.854
42.69
39.77
1.107
MotionCrafter
52.98
47.89
1.765
–
–
–
OpenD4RT
38.40
37.92
1.432
24.32
26.92
1.801
St4RTrack
59.45
54.24
3.640
56.56
51.77
3.585
Appendix
Table 6: Dense tracking on Kubric. TrackCraft3R’s ( Nam et al., 2026 ) 50 24-frame Kubric ( Greff et al., 2022 ) clips, with ground-truth trajectories for every reference pixel. APD3D (%) on all and dynamic tracks and EPE (scene units) under RANSAC Sim(3) and global-scale alignment; bold best, underline second best.
ADT
PS
PO
DR
Mean
Method
AJ
OA
AJ
OA
AJ
OA
AJ
OA
AJ
OA
OpenD4RT
58.82
91.04
69.01
87.34
45.52
78.63
63.04
90.03
59.10
86.76
Track4World
72.88
98.06
64.76
86.51
47.53
76.25
67.46
92.79
63.16
88.40
DELTAv2 + DA3
84.18
96.94
55.18
71.57
44.90
74.35
62.74
87.36
61.75
82.56
DELTAv2 + VGGT- Ω
83.83
95.09
57.91
71.32
45.61
74.21
68.54
87.94
63.97
82.14
TrackCraft3R + DA3
82.95
94.26
70.18
88.13
50.29
75.59
66.79
92.69
67.55
87.67
Appendix
Table 7: AJ and OA on WorldTrack (%). Predicted visibility is thresholded at 0.5 . AJ uses all-point RANSAC Sim(3) alignment; OA requires no geometric alignment. Higher is better.
Method
ADT
PS
PO
DR
Mean
4RC
0.153
0.138
0.249
0.149
0.172
Any4D
0.286
0.216
0.335
0.191
0.257
MotionCrafter
0.474 ‡
0.358
0.820
0.278
0.482 ‡
OpenD4RT
0.317
0.161
0.390
0.252
0.280
St4RTrack
0.242
0.258
0.311
0.236
0.262
Track4World
0.269
0.184
0.366
0.219
0.260
Appendix
Table 8: End-point error on WorldTrack. Mean 3D position error over evaluated point–frames, in metres after all-point RANSAC Sim(3) alignment. Lower is better. ‡ Restricted ADT coverage for MotionCrafter, also affecting its mean (Appendix B.2 ).
ADT
PS
PO
DR
Mean
Method
All
Dyn
All
Dyn
All
Dyn
All
Dyn
All
Dyn
4RC
83.09
68.24
69.15
69.15
80.23
76.44
82.95
73.13
78.86
71.74
Any4D
57.60
65.80
63.39
63.39
66.16
66.89
78.15
68.16
66.32
66.06
MotionCrafter
2.12 ‡
6.91 ‡
0.00
0.00
0.11
0.02
1.59
1.39
0.96 ‡
2.08 ‡
OpenD4RT
69.28
67.96
76.87
76.87
65.19
70.39
74.80
76.25
71.53
72.87
St4RTrack
74.33
71.13
69.12
69.12
67.86
68.34
74.10
67.31
71.35
68.97
Appendix
Table 9: WorldTrack under global-scale alignment. APD3D (%) on all (All) and dynamic (Dyn) points, fitting only scale (Appendix B.1 ). MotionCrafter’s coordinate-frame mismatch makes its scores non-comparable under this alignment. ‡ Restricted ADT coverage for MotionCrafter, also affecting its means (Appendix B.2 ).
Methods
Projection cameras
4RC, Any4D, Track4World, V-DPM
Predicted by the method
TrackCraft3R, DELTAv2
Input geometry cameras
Our configurations
Predicted by the geometry provider
MotionCrafter
Robust joint pose/focal fit to predicted point maps
St4RTrack, OpenD4RT, TraceAnything
Recovered from each method’s own outputs
Appendix
Table 10: Camera sources for APD2D. Cameras are predicted, supplied by a geometry provider, or recovered from model outputs; no ground-truth camera parameters are used.
ADT
PS
PO
DR
Mean
Method
All
Dyn
All
Dyn
All
Dyn
All
Dyn
All
Dyn
4RC
62.06
27.36
51.79
51.79
71.50
60.91
79.15
65.26
66.13
51.33
Any4D
40.90
23.60
31.46
31.46
56.99
44.39
76.93
50.10
51.57
37.39
MotionCrafter
17.47 ‡
10.34 ‡
15.11
15.11
21.88
13.70
40.69
30.58
23.79 ‡
17.43 ‡
OpenD4RT
45.94
21.77
46.90
46.90
56.13
49.43
72.58
57.41
55.39
43.88
St4RTrack
59.53
28.47
39.12
39.12
63.95
47.47
79.77
50.73
60.59
41.45
Appendix
Table 11: Image-plane accuracy on WorldTrack. APD2D (%) on all (All) and dynamic (Dyn) points. 3D trajectories are projected through the cameras in Table 10 ; native 2D references are scored without projection. Bold and underline rank 3D methods only. Scoring details are in Appendix C.4 . ‡ Restricted ADT coverage for MotionCrafter, also affecting its means (Appendix B.2 ).
Figure 6: Simulated versus AllTracker+DA3 training lifts with matched data. Both models train on the same 3,890 filtered windows for 150,000 steps. APD3D gains on all points are reported in percentage points over the initialized lift, with AllTracker and DA3 (left) or VGGT- Ω (right). “Mean” averages the four datasets.
AllTracker
CoWTracker
Training corruptions
DA3
VGGT- Ω
DA3
VGGT- Ω
Lift (the corrector’s input)
71.9
73.7
70.0
71.6
Full simulation (ours)
75.7 (+3.8)
79.0 (+5.2)
77.1 (+7.1)
80.8 (+9.2)
without correlated noise
75.3 (+3.4)
78.7 (+5.0)
76.7 (+6.7)
80.2 (+8.6)
without wrong surface
72.7 (+0.8)
75.3 (+1.5)
71.9 (+1.9)
74.0 (+2.4)
without synthetic gaps
75.6 (+3.7)
78.9 (+5.1)
77.0 (+7.0)
80.6 (+8.9)
Appendix
Table 12: Contribution of training corruptions on occluded points. APD3D (%) on ground-truth-occluded point–frames, averaged over the four datasets. Scoring retains the all-point RANSAC Sim(3) alignment. Parentheses report percentage-point gains over the initialized lift.
ADT
PS
PO
DR
Geometry
Subset
With masking
Without masking
With masking
Without masking
With masking
Without masking
With masking
Without masking
DA3
All
93.69
90.93
85.46
85.44
76.04
76.03
79.50
79.52
DA3
Occ
90.27
82.22
76.08
76.30
73.19
73.03
63.17
64.05
VGGT- Ω
All
94.04
91.23
90.40
90.36
76.10
76.10
85.74
85.75
VGGT- Ω
Occ
90.12
82.03
82.04
82.54
72.49
72.23
71.26
71.85
Appendix
Table 13: Ray masking. APD3D (%) for separately trained correctors that mask or retain rays where the input mask is zero, using AllTracker and C=4 . All and Occ share the all-point RANSAC Sim(3) alignment; Occ scores ground-truth-occluded point–frames.
DA3 input
VGGT- Ω input
Corrector time (s)
C
All
Dyn
All
Dyn
DA3
VGGT- Ω
1
82.93
81.08
85.83
83.79
10.0
9.7
4
83.68
81.74
86.57
84.92
3.1
3.0
8
83.59
81.48
86.48
84.75
2.0
1.9
Appendix
Table 14: Temporal chunking and pipeline runtime. AllTracker with DA3 or VGGT- Ω geometry. Left: correctors trained separately for each chunk size C (frames per token), reporting mean WorldTrack APD3D (%) under RANSAC Sim(3) alignment and corrector-only runtime at 5122 . Right: pipeline runtime by stage across resolutions, using our default C=4 . All timings use 84 frames on one RTX PRO 6000; profiling details appear in Appendix D.4 .
Figure 7: Group size at inference. Mean WorldTrack APD3D (%) on all and dynamic points under RANSAC Sim(3) alignment, varying tracks per group P with AllTracker and fixed corrector weights ( C=4 ). The dotted line marks our setting, P=1024 .
Run time (s)
Peak memory (GiB)
Method
2562
5122
10242
2562
5122
10242
Ours: the corrector on the lift of
AllTracker + DA3
2.1
11.4
82.6
10.1
21.3
63.7
AllTracker + VGGT- Ω
1.5
7.3
51.0
7.9
12.5
43.9
CoWTracker + DA3
2.8
15.3
OOM
10.6
36.5
OOM
CoWTracker + VGGT- Ω
2.2
11.7
OOM
10.6
36.5
OOM
Appendix
Table 15: Run time and peak allocated GPU memory. Complete pipelines at three nominal resolutions. OOM: out of memory; FAIL: St4RTrack’s upsampling exceeds an implementation tensor-size limit; –: not run. † Hard-coded input resolution overridden for this experiment. OpenD4RT runs only at 2562 ; ours uses C=4 .
Figure 8: Additional qualitative comparisons. Top: DAVIS soccerball ; bottom: lady-running . Ours uses CoWTracker with VGGT- Ω (top) and AllTracker with DA3 (bottom), with the same corrector weights ( C=4 ). TrackCraft3R uses VGGT- Ω in both rows (unlike ours in lady-running ); 4RC and V-DPM use their own reconstructions. Methods track the same reference pixels, shown on their respective reconstructions from matched novel viewpoints.
Figure 9: Effect of trajectory correction. Left to right: input frame, initialized lift, and corrected trajectories ( C=4 ). Top: DAVIS camel ; bottom: skate-park . CoWTracker’s 2D tracks and DA3 geometry are held fixed; both trajectory panels use the same point cloud and novel viewpoint.
Dense 3D tracking from monocular video is fundamental to dynamic scene understanding. While recent 3D foundation models provide reliable per-frame geometry, recovering object motion in this geometry remains challenging and benefits from strong motion priors learned from real-world videos. Existing 3D trackers either follow iterative paradigms trained from scratch on synthetic data or fine-tune 3D reconstruction models learned from static multi-view images, both lacking real-world motion priors. Pre-trained video diffusion transformers (video DiTs) offer rich spatio-temporal priors from internet-scale videos, making them a promising foundation for 3D tracking. However, their frame-anchored formulation, which generates each frame's content, is fundamentally mismatched with reference-anchored dense 3D tracking, which must follow the same physical points from a reference frame across time. We present TrackCraft3R, the first method to repurpose a video DiT as a feed-forward dense 3D tracker. Given a monocular video and its frame-anchored reconstruction pointmap, TrackCraft3R predicts a reference-anchored tracking pointmap that follows every pixel of the first frame across time in a single forward pass, along with its visibility. We achieve this through two designs: (i) a dual-latent representation that uses per-frame geometry latents and reference-anchored track latents as dense queries, and (ii) temporal RoPE alignment, which specifies the target timestamp of each track latent. Together, these designs convert the per-frame generative paradigm of video DiTs into a reference-anchored tracking formulation with LoRA fine-tuning. TrackCraft3R achieves state-of-the-art performance on standard sparse and dense 3D tracking benchmarks, while running 1.3x faster and using 4.6x less peak memory than the strongest prior method. We further demonstrate robustness to large motions and long videos.
We study how to repurpose video diffusion models (VDMs), pretrained on millions of videos, for dense 3D point tracking, where training data is scarce. Doing so is non-trivial, as it requires representing 3D tracks as video-like signals that may lie outside the natural-video distribution on which the VDM was trained. We thus systematically assess the effectiveness of two such video representations. The first, Residual 3D Tracking Video (RTV), encodes per-pixel 3D offsets relative to a query frame and allows recovery of 3D trajectories directly from the generated video. The second, Normalized Coordinate Map Video (NCMV), labels each pixel with a canonical coordinate invariant through time. This is closer to a natural video and easier for the VDM to learn, but requires additional post-processing with an off-the-shelf 2D tracker to recover 3D tracks. RTV tracks more accurately, while NCMV shows that a VDM can generate dense correspondence as a video. We instantiate both representations with LoRA adaptation of a frozen Wan backbone, conditioned on point maps from an off-the-shelf 3D reconstructor as a geometric scaffold. Trained with a single dynamic 3D tracking dataset, Track4D achieves competitive zero-shot tracking on four real and synthetic benchmarks. A regression readout that bypasses the video VAE and uses additional training data brings further improvement, demonstrating the potential of this direction for dense 3D tracking. We hope our analysis will inspire future work that leverages VDM priors for dense 3D tracking and 4D reconstruction. Project page: https://nirvanalan.github.io/projects/track4d
Yushi Lan, Zeren Jiang, Kelvin Li +3
Visual Geometry Group, University of Oxford · Nanyang Technological University
Dense 3D point tracking has been a prominent paradigm for modeling motion in dynamic scenes, but a point track is just a 3-DoF translation curve per pixel: it captures where pixels go, not the rotation of the underlying part, nor which pixels move together as one body. We propose MoSE3, the first feed-forward model that predicts dense SE(3) motion from monocular RGB video, producing full 6-DoF rigid transforms at every pixel in world space. Per-pixel SE(3) motion offers a richer view of how a scene moves: rotation, translation, and grouping all at once. Directly predicting SE(3) is challenging: rotations lie on a curved manifold that is ill-suited to Euclidean regression, and annotations for SE(3) are particularly difficult to acquire. To address these challenges, MoSE3 predicts per-pixel SE(3) through two jointly learned intermediates, 3D point tracks and rigidity embeddings, and recovers SE(3) by differentiably fitting transforms within each soft rigid cluster, enabling end-to-end prediction and supervision. To close the data gap, we introduce Art-Kubric, a large-scale synthetic dataset with dense SE(3) and rigidity labels for articulated objects with rich physical interactions. MoSE3 achieves state-of-the-art SE(3) estimation at pixel, part, and object levels on both rigid and articulated benchmarks, and state-of-the-art average 3D point tracking accuracy across three datasets, while showing strong generalization to real-world videos despite being trained solely on synthetic motion data.
Jiahuan Cheng, Zhiyi Li, Tian Xia +3
Harvard University · Johns Hopkins University · MIT +1