Dense 3D point tracking has been a prominent paradigm for modeling motion in dynamic scenes, but a point track is just a 3-DoF translation curve per pixel: it captures where pixels go, not the rotation of the underlying part, nor which pixels move together as one body. We propose MoSE3, the first feed-forward model that predicts dense SE(3) motion from monocular RGB video, producing full 6-DoF rigid transforms at every pixel in world space. Per-pixel SE(3) motion offers a richer view of how a scene moves: rotation, translation, and grouping all at once. Directly predicting SE(3) is challenging: rotations lie on a curved manifold that is ill-suited to Euclidean regression, and annotations for SE(3) are particularly difficult to acquire. To address these challenges, MoSE3 predicts per-pixel SE(3) through two jointly learned intermediates, 3D point tracks and rigidity embeddings, and recovers SE(3) by differentiably fitting transforms within each soft rigid cluster, enabling end-to-end prediction and supervision. To close the data gap, we introduce Art-Kubric, a large-scale synthetic dataset with dense SE(3) and rigidity labels for articulated objects with rich physical interactions. MoSE3 achieves state-of-the-art SE(3) estimation at pixel, part, and object levels on both rigid and articulated benchmarks, and state-of-the-art average 3D point tracking accuracy across three datasets, while showing strong generalization to real-world videos despite being trained solely on synthetic motion data.
Figures & tables
Figure 1: MoSE3 estimates dense SE(3) motion from monocular video in a single forward pass. It predicts a 3D rotation and translation at every pixel in a shared world coordinate frame, capturing rigid and articulated motion (left) as well as deformable motion (right). Colored coordinate axes visualize the predicted local transformations at sampled points; insets show the input frames and learned rigidity embeddings.
Figure 2: Method overview. MoSE3 extends the frozen π3 backbone with a trainable tracking branch that predicts dense 3D point tracks Xic and rigidity embeddings Ei . Per-pixel SE(3) motion is then recovered in closed form via a weighted Horn fit on the world-frame tracks, with weights given by rigidity-embedding similarity. Because the Horn fit is differentiable, the dense SE(3) prediction is supervised end-to-end together with the tracks and rigidity embeddings.
Figure 3: Art-Kubric overview. Two example scenes (one per row). Columns 1–3: three RGB frames sampled from the clip. Columns 4–7 (annotations rendered at the third frame): sampled long-range 3D tracks colored by their parent rigid body, per-pixel SE(3) motion as rotated coordinate frames at sparse anchors, rigid-part segmentation (per-pixel partition by shared rigid transform), and depth. Articulated objects from PartNet-Mobility [ 98 , 66 ] , Articraft [ 117 ] , and Infinigen-Articulated [ 37 ] interact with rigid Google Scanned Objects [ 16 ] under physics simulation while a moving camera observes the scene under varying HDRI lighting.
Per-pixel SE(3)
Object/Part-level SE(3)
HO3D
iTACO
HO3D
iTACO
Method
RRE ↓
ADD ↓
AUC r↑
AUC ADD↑
RRE ↓
ADD ↓
AUC r↑
AUC ADD↑
RRE ↓
ADD ↓
AUC ADD↑
IoU ↑
RRE ↓
ADD ↓
AUC ADD↑
IoU ↑
SpatialTrackerV2 [ 101 ]
36.45
3.73
0.376
0.633
36.71
28.45
0.408
0.309
32.56
3.46
0.670
0.385
27.25
23.01
0.421
0.239
POMATO [ 113 ]
60.53
6.59
0.216
0.405
46.42
46.66
0.319
0.121
59.99
6.18
0.446
0.197
41.53
44.48
0.152
0.190
ZeroMSF [ 54 ]
61.53
6.70
0.224
0.409
46.52
43.68
0.298
0.128
62.29
6.43
0.442
0.230
43.47
41.83
0.158
0.187
Any4D [ 41 ]
57.02
5.24
0.263
0.504
43.98
31.50
0.364
0.292
55.44
5.16
0.523
0.298
37.23
28.56
0.399
0.185
Table 1: SE(3) estimation on HO3D and iTACO. Per-pixel SE(3) evaluates a rigid transform at every pixel; Object/Part-level SE(3) clusters pixels into rigid bodies and recovers one transform per cluster, with IoU measured against ground-truth rigid masks. ADD is reported in centimeters; AUC r uses a 30∘ threshold, while AUC ADD uses a 10cm threshold on HO3D and 20cm on iTACO, reflecting the different translation scales of the two benchmarks. Baselines are evaluated with k -NN , which clusters predicted point tracks via k -NN; for our method we additionally report rigid clust. , which clusters our learned rigidity embeddings. Best in bold , second underlined .
Figure 4: Qualitative results on in-the-wild videos. MoSE3 generalizes to out-of-distribution real-world scenes spanning both rigid and non-rigid cases. For each clip, the three columns are three time steps: the top row shows the predicted per-pixel SE(3) motion and the bottom row the predicted rigidity embedding.
PointOdyssey [ 115 ]
ADT [ 69 ]
PStudio [ 36 ]
Avg.
Method
L-16
L-50
L-16
L-50
L-16
L-50
L-16
L-50
SpatialTrackerV2 [ 101 ]
0.1925
0.1763
0.2456
0.2163
0.3790
0.3689
0.2724
0.2538
POMATO [ 113 ]
0.4425
0.3905
0.3611
0.3548
0.5166
0.4713
0.4401
0.4055
ZeroMSF [ 54 ]
0.4053
0.3505
0.4530
0.3563
0.4828
0.4386
0.4471
0.3818
Any4D [ 41 ]
0.4769
0.4174
0.4460
0.3717
0.5707
0.5066
0.4979
0.4319
V-DPM [ 78 ]
0.4848
0.4233
0.4783
0.3759
0.6084
0.5795
0.5238
0.4596
Table 2: World-coordinate 3D point tracking on PointOdyssey, ADT, and PStudio. We follow the evaluation protocol of Track4World [ 61 ] . We report tracking accuracy at horizons L-16 ( ↑ ) and L-50 ( ↑ ). Best in bold , second underlined . Avg. is the unweighted mean across the three datasets.
Variant
RRE ↓
AUC r↑
ADD ↓
AUC ADD↑
(a) Architecture
Direct SE(3) regr.
24.57
0.525
19.05
0.393
(b) Training data
w/o Art-Kubric
22.89
0.566
17.01
0.462
(c) Training strategy
Track-only + k -NN
31.11
0.453
20.96
0.382
Table 3: Ablation study on iTACO (per-pixel SE(3) ). Trained with a reduced schedule, detailed in Appendix C.1 . ADD in cm; AUC r at 30∘ and AUC ADD at 20cm . Full results in Tab. 4 .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Per-pixel SE(3)
Object/Part-level SE(3)
HO3D
iTACO
HO3D
iTACO
Variant
RRE ↓
AUC r↑
ADD ↓
AUC ADD↑
RRE ↓
AUC r↑
ADD ↓
AUC ADD↑
RRE ↓
ADD ↓
AUC ADD↑
IoU ↑
RRE ↓
ADD ↓
AUC ADD↑
IoU ↑
(a) Architecture
Direct SE(3) regr.
29.79
0.449
3.65
0.655
24.57
0.525
19.05
0.393
36.18
3.21
0.701
0.600
22.49
15.01
0.511
0.620
(b) Training data
w/o Art-Kubric
25.07
0.480
2.94
0.716
22.89
0.566
17.01
0.462
23.24
2.86
0.734
0.643
19.06
15.32
0.502
0.670
Appendix
Table 4: Full ablation study. We ablate four design choices, each compared against our full model (bottom row). (a) Architecture: directly regressing SE(3) from an SE(3) head, instead of predicting rigidity embeddings and point tracks. (b) Training data: training without Art-Kubric dataset. (c) Training strategy: predicting tracks alone and recovering SE(3) posthoc, via either k -NN clustering of tracks or clustering of our rigidity embeddings, whose cluster IoU is inherited from our full model and therefore omitted (–). (d) Loss design: removing individual loss terms. ADD is reported in centimeters; AUC r uses a 30∘ threshold, while AUC ADD uses a 10cm threshold on HO3D and 20cm on iTACO. Best in each column in bold , second underlined .
PointOdyssey [ 115 ]
ADT [ 69 ]
PStudio [ 36 ]
Avg.
Method
L-16
L-50
L-16
L-50
L-16
L-50
L-16
L-50
Camera coordinate 3D tracking
SpatialTracker [ 100 ]
0.3116
0.2977
0.4962
0.4692
0.5390
0.4991
0.4489
0.4220
DELTA [ 67 ]
0.3529
0.3412
0.5116
0.4952
0.5922
0.5533
0.4856
0.4632
SpatialTrackerV2 [ 101 ]
0.1864
0.1785
0.2400
0.2330
0.3784
0.3690
0.2683
0.2602
MASt3R [ 50 ]
0.3546
0.3253
0.3368
0.3029
0.3293
0.2956
0.3402
0.3079
Appendix
Table 5: 3D point tracking results on PointOdyssey, ADT, and PStudio in both camera and world coordinates. We follow the evaluation protocol of Track4World [ 61 ] . We report tracking accuracy at horizons L-16 ( ↑ ) and L-50 ( ↑ ). Best in bold , second underlined . Avg. is the unweighted mean across the three datasets.
Per-pixel SE(3)
Object/Part-level SE(3)
Method
RRE ↓
ADD ↓
AUC r↑
AUC ADD↑
RRE ↓
ADD ↓
AUC ADD↑
IoU ↑
SpatialTrackerV2 [ 101 ]
21.93
4.93
0.461
0.576
22.93
6.55
0.508
0.251
POMATO [ 113 ]
46.47
10.78
0.223
0.336
50.11
10.89
0.309
0.054
ZeroMSF [ 54 ]
44.79
11.57
0.243
0.353
46.71
10.74
0.387
0.073
Any4D [ 41 ]
42.30
10.28
0.251
0.432
44.67
10.45
0.455
0.267
V-DPM [ 78 ]
30.42
7.58
0.329
0.504
31.60
8.18
0.501
0.234
Appendix
Table 6: SE(3) estimation on YCBInEOAT [ 95 ] . Nine clips of robot manipulation with rigid objects, evaluated with the protocol of Tab. 1 . ADD is reported in centimeters; AUC r uses a 30∘ threshold and AUC ADD a 10cm threshold. Best in bold , second underlined .
Figure 5: Qualitative comparison on in-the-wild videos. We compare MoSE3 against ProxyPose [ 112 ] on out-of-distribution real-world scenes, including deformable cases. For each method, we visualize the estimated per-pixel Tq→i(u) by drawing its SE(3) coordinate axes at sampled pixels.
Figure 6: Additional qualitative results on in-the-wild videos. In addition to Fig. 4 , we show further out-of-distribution real-world scenes spanning both rigid and non-rigid cases, here also visualizing the predicted 3D point tracks alongside the per-pixel rigidity embeddings and the per-pixel SE(3) motion overlaid on them.
Dataset
Source
Phys. multi-body
Articul. (multi-link)
Long-range tracks
3D track GT
Per-link SE(3) GT
Rigid- grouping
Cam. variety
FlyingThings3D [ 64 ]
Synth
X
X
X
✓ ♭
X
X
✓
Kubric MOVi-F [ 20 ]
Synth
✓
X
✓
✓ ♯
∘
X
✓
TAP-Vid Kubric [ 12 ]
Synth
✓
X
✓
X
X
X
✓
PointOdyssey [ 115 ]
Synth
∘ ♢
X †
✓
✓
X
X
✓
Dynamic Replica [ 38 ]
Synth
X †
X †
✓
✓
X
X
✓
SynthVerse [ 114 ]
Synth
X ‡
✓ ‡
✓
✓
X
X
X ‡
Appendix
Table 7: Comparison of supervision signals across point-tracking and articulation datasets. The point of this table is which annotation channels and scene regimes each dataset exposes, not raw scale. “Phys. multi-body” = multiple independently moving bodies that interact under physics simulation (collisions, contact). “Articul.” = bodies whose motion is a kinematic tree of multiple rigidly-moving links. “3D tracks” = dense 3D point trajectories over >2 frames. “Per-link SE(3) ” = world-frame 4×4 pose for each sub-object rigid link, every frame; ∘ = only an object-level (single-rigid-body) pose. “Rigid-grouping” = per-pixel partition of pixels by shared transform, beyond instance segmentation. “Cam. variety” = multiple distinct camera trajectory modes (mixing moving and stationary), in the spirit of Kubric’s MOVi setups. Among the datasets compared here, Art-Kubric is the only one that provides all seven properties.
Estimating the 3D trajectory of every pixel from a monocular video is crucial and promising for a comprehensive understanding of the 3D dynamics of videos. Recent monocular 3D tracking works demonstrate impressive performance, but are limited to either tracking sparse points on the first frame or a slow optimization-based framework for dense tracking. In this paper, we propose a feedforward model, called Track4World, enabling an efficient holistic 3D tracking of every pixel in the world-centric coordinate system. Built on the global 3D scene representation encoded by a VGGT-style ViT, Track4World applies a novel 3D correlation scheme to simultaneously estimate the pixel-wise 2D and 3D dense flow between arbitrary frame pairs. The estimated scene flow, along with the reconstructed 3D geometry, enables subsequent efficient 3D tracking of every pixel of this video. Extensive experiments on multiple benchmarks demonstrate that our approach consistently outperforms existing methods in 2D/3D flow estimation and 3D tracking, highlighting its robustness and scalability for real-world 4D reconstruction tasks.
Jiahao Lu, Jiayi Xu, Wenbo Hu +5
The Hong Kong University of Science and Technology · ARC Lab, Tencent PCG
Dense 3D tracking from monocular video is fundamental to dynamic scene understanding. While recent 3D foundation models provide reliable per-frame geometry, recovering object motion in this geometry remains challenging and benefits from strong motion priors learned from real-world videos. Existing 3D trackers either follow iterative paradigms trained from scratch on synthetic data or fine-tune 3D reconstruction models learned from static multi-view images, both lacking real-world motion priors. Pre-trained video diffusion transformers (video DiTs) offer rich spatio-temporal priors from internet-scale videos, making them a promising foundation for 3D tracking. However, their frame-anchored formulation, which generates each frame's content, is fundamentally mismatched with reference-anchored dense 3D tracking, which must follow the same physical points from a reference frame across time. We present TrackCraft3R, the first method to repurpose a video DiT as a feed-forward dense 3D tracker. Given a monocular video and its frame-anchored reconstruction pointmap, TrackCraft3R predicts a reference-anchored tracking pointmap that follows every pixel of the first frame across time in a single forward pass, along with its visibility. We achieve this through two designs: (i) a dual-latent representation that uses per-frame geometry latents and reference-anchored track latents as dense queries, and (ii) temporal RoPE alignment, which specifies the target timestamp of each track latent. Together, these designs convert the per-frame generative paradigm of video DiTs into a reference-anchored tracking formulation with LoRA fine-tuning. TrackCraft3R achieves state-of-the-art performance on standard sparse and dense 3D tracking benchmarks, while running 1.3x faster and using 4.6x less peak memory than the strongest prior method. We further demonstrate robustness to large motions and long videos.
Existing point tracking models face a fundamental tradeoff: they can either track a sparse set of query points over long horizons, or track all points across only short clips. We introduce TrackEverything, a 3D point tracker that breaks this trade-off by representing videos as persistent 3D scene tracks in world coordinates. Grounded in the insight that videos are 2D projections of an underlying 3D world, TrackEverything decouples model complexity from video duration, allowing it to scale with unique physical scene geometry instead. Our approach introduces three key innovations. First, we employ a voxelization-based de-duplication mechanism at sliding-window boundaries to merge co-located tracks, preventing repeated observations of the same surface from redundantly accumulating. Second, we decompose tracking into an endpoint refiner that predicts each point's destination and static-versus-dynamic classification, followed by a lightweight trajectory refiner that decodes dense trajectories exclusively for dynamic points. Third, we propose 3D WAFT, replacing memory-prohibitive 4D correlation volumes with efficient feature sampling in the scene cloud. To the best of our knowledge, TrackEverything is the first 3D tracker capable of tracking all visible points across videos exceeding 1000 frames within 40 GB of GPU memory. On TAPVid-3D, TrackEverything outperforms all open-source all-frame dense 3D trackers by more than 20% APD on short clips, while remaining competitive with state-of-the-art sparse trackers on long sequences, despite tracking far more points.