MoSE3: Learning World-Space SE(3) at Every Pixel

Jiahuan Cheng; Zhiyi Li; Tian Xia; Ruojin Cai; Yilun Du; Qianqian Wang

cs.CVOct 2, 2026

MoSE3: Learning World-Space SE(3) at Every Pixel

Authors: Jiahuan Cheng, Zhiyi Li, Tian Xia, Ruojin Cai, Yilun Du, Qianqian Wang

Organizations: Harvard University · Johns Hopkins University · MIT · Kempner Institute

Abstract

Dense 3D point tracking has been a prominent paradigm for modeling motion in dynamic scenes, but a point track is just a 3-DoF translation curve per pixel: it captures where pixels go, not the rotation of the underlying part, nor which pixels move together as one body. We propose MoSE3, the first feed-forward model that predicts dense SE(3) motion from monocular RGB video, producing full 6-DoF rigid transforms at every pixel in world space. Per-pixel SE(3) motion offers a richer view of how a scene moves: rotation, translation, and grouping all at once. Directly predicting SE(3) is challenging: rotations lie on a curved manifold that is ill-suited to Euclidean regression, and annotations for SE(3) are particularly difficult to acquire. To address these challenges, MoSE3 predicts per-pixel SE(3) through two jointly learned intermediates, 3D point tracks and rigidity embeddings, and recovers SE(3) by differentiably fitting transforms within each soft rigid cluster, enabling end-to-end prediction and supervision. To close the data gap, we introduce Art-Kubric, a large-scale synthetic dataset with dense SE(3) and rigidity labels for articulated objects with rich physical interactions. MoSE3 achieves state-of-the-art SE(3) estimation at pixel, part, and object levels on both rigid and articulated benchmarks, and state-of-the-art average 3D point tracking accuracy across three datasets, while showing strong generalization to real-world videos despite being trained solely on synthetic motion data.

Figures & tables

Figure 1: MoSE3 estimates dense $\mathrm{SE}(3)$ motion from monocular video in a single forward pass. It predicts a 3D rotation and translation at every pixel in a shared world coordinate frame, capturing rigid and articulated motion (left) as well as deformable motion (right). Colored coordinate axes visualize the predicted local transformations at sampled points; insets show the input frames and learned rigidity embeddings.

Figure 2: Method overview. MoSE3 extends the frozen $\pi^{3}$ backbone with a trainable tracking branch that predicts dense 3D point tracks $\mathbf{X}^{c}_{i}$ and rigidity embeddings $\mathbf{E}_{i}$ . Per-pixel $\mathrm{SE}(3)$ motion is then recovered in closed form via a weighted Horn fit on the world-frame tracks, with weights given by rigidity-embedding similarity. Because the Horn fit is differentiable, the dense $\mathrm{SE}(3)$ prediction is supervised end-to-end together with the tracks and rigidity embeddings.

Figure 3: Art-Kubric overview. Two example scenes (one per row). Columns 1–3: three RGB frames sampled from the clip. Columns 4–7 (annotations rendered at the third frame): sampled long-range 3D tracks colored by their parent rigid body, per-pixel

\mathrm{SE}(3)

motion as rotated coordinate frames at sparse anchors, rigid-part segmentation (per-pixel partition by shared rigid transform), and depth. Articulated objects from PartNet-Mobility [ 98 , 66 ] , Articraft [ 117 ] , and Infinigen-Articulated [ 37 ] interact with rigid Google Scanned Objects [ 16 ] under physics simulation while a moving camera observes the scene under varying HDRI lighting.

	Per-pixel $\mathrm{SE}(3)$								Object/Part-level $\mathrm{SE}(3)$
	HO3D				iTACO				HO3D				iTACO
Method	RRE $\downarrow$	ADD $\downarrow$	AUC ${}_{r}\!\uparrow$	AUC ${}_{\text{ADD}}\!\uparrow$	RRE $\downarrow$	ADD $\downarrow$	AUC ${}_{r}\!\uparrow$	AUC ${}_{\text{ADD}}\!\uparrow$	RRE $\downarrow$	ADD $\downarrow$	AUC ${}_{\text{ADD}}\!\uparrow$	IoU $\uparrow$	RRE $\downarrow$	ADD $\downarrow$	AUC ${}_{\text{ADD}}\!\uparrow$	IoU $\uparrow$
SpatialTrackerV2 [ 101 ]	36.45	3.73	0.376	0.633	36.71	28.45	0.408	0.309	32.56	3.46	0.670	0.385	27.25	23.01	0.421	0.239
POMATO [ 113 ]	60.53	6.59	0.216	0.405	46.42	46.66	0.319	0.121	59.99	6.18	0.446	0.197	41.53	44.48	0.152	0.190
ZeroMSF [ 54 ]	61.53	6.70	0.224	0.409	46.52	43.68	0.298	0.128	62.29	6.43	0.442	0.230	43.47	41.83	0.158	0.187
Any4D [ 41 ]	57.02	5.24	0.263	0.504	43.98	31.50	0.364	0.292	55.44	5.16	0.523	0.298	37.23	28.56	0.399	0.185

Table 1:

\mathrm{SE}(3)

estimation on HO3D and iTACO. Per-pixel

\mathrm{SE}(3)

evaluates a rigid transform at every pixel; Object/Part-level

\mathrm{SE}(3)

clusters pixels into rigid bodies and recovers one transform per cluster, with IoU measured against ground-truth rigid masks. ADD is reported in centimeters; AUC r uses a

30^{\circ}

threshold, while AUC

{}_{\text{ADD}}

uses a

10\,\text{cm}

threshold on HO3D and

20\,\text{cm}

on iTACO, reflecting the different translation scales of the two benchmarks. Baselines are evaluated with

k

-NN , which clusters predicted point tracks via

k

-NN; for our method we additionally report rigid clust. , which clusters our learned rigidity embeddings. Best in bold , second underlined .

Figure 4: Qualitative results on in-the-wild videos. MoSE3 generalizes to out-of-distribution real-world scenes spanning both rigid and non-rigid cases. For each clip, the three columns are three time steps: the top row shows the predicted per-pixel $\mathrm{SE}(3)$ motion and the bottom row the predicted rigidity embedding.

	PointOdyssey [ 115 ]		ADT [ 69 ]		PStudio [ 36 ]		Avg.
Method	L-16	L-50	L-16	L-50	L-16	L-50	L-16	L-50
SpatialTrackerV2 [ 101 ]	0.1925	0.1763	0.2456	0.2163	0.3790	0.3689	0.2724	0.2538
POMATO [ 113 ]	0.4425	0.3905	0.3611	0.3548	0.5166	0.4713	0.4401	0.4055
ZeroMSF [ 54 ]	0.4053	0.3505	0.4530	0.3563	0.4828	0.4386	0.4471	0.3818
Any4D [ 41 ]	0.4769	0.4174	0.4460	0.3717	0.5707	0.5066	0.4979	0.4319
V-DPM [ 78 ]	0.4848	0.4233	0.4783	0.3759	0.6084	0.5795	0.5238	0.4596

Table 2: World-coordinate 3D point tracking on PointOdyssey, ADT, and PStudio. We follow the evaluation protocol of Track4World [ 61 ] . We report tracking accuracy at horizons L-16 (

\uparrow

) and L-50 (

\uparrow

). Best in bold , second underlined . Avg. is the unweighted mean across the three datasets.

(a) Architecture
Variant	RRE $\downarrow$	AUC ${}_{r}\uparrow$	ADD $\downarrow$	AUC ${}_{\text{ADD}}\uparrow$
Direct $\mathrm{SE}(3)$ regr.	24.57	0.525	19.05	0.393
(b) Training data
w/o Art-Kubric	22.89	0.566	17.01	0.462
(c) Training strategy
Track-only + $k$ -NN	31.11	0.453	20.96	0.382

Table 3: Ablation study on iTACO (per-pixel

\mathrm{SE}(3)

). Trained with a reduced schedule, detailed in Appendix C.1 . ADD in cm; AUC r at

30^{\circ}

and AUC

{}_{\text{ADD}}

20\,\text{cm}

. Full results in Tab. 4 .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

(a) Architecture
	Per-pixel $\mathrm{SE}(3)$								Object/Part-level $\mathrm{SE}(3)$
	HO3D				iTACO				HO3D				iTACO
Variant	RRE $\downarrow$	AUC ${}_{r}\uparrow$	ADD $\downarrow$	AUC ${}_{\text{ADD}}\uparrow$	RRE $\downarrow$	AUC ${}_{r}\uparrow$	ADD $\downarrow$	AUC ${}_{\text{ADD}}\uparrow$	RRE $\downarrow$	ADD $\downarrow$	AUC ${}_{\text{ADD}}\uparrow$	IoU $\uparrow$	RRE $\downarrow$	ADD $\downarrow$	AUC ${}_{\text{ADD}}\uparrow$	IoU $\uparrow$
Direct $\mathrm{SE}(3)$ regr.	29.79	0.449	3.65	0.655	24.57	0.525	19.05	0.393	36.18	3.21	0.701	0.600	22.49	15.01	0.511	0.620
(b) Training data
w/o Art-Kubric	25.07	0.480	2.94	0.716	22.89	0.566	17.01	0.462	23.24	2.86	0.734	0.643	19.06	15.32	0.502	0.670

Appendix

Table 4: Full ablation study. We ablate four design choices, each compared against our full model (bottom row). (a) Architecture: directly regressing

\mathrm{SE}(3)

from an

\mathrm{SE}(3)

head, instead of predicting rigidity embeddings and point tracks. (b) Training data: training without Art-Kubric dataset. (c) Training strategy: predicting tracks alone and recovering

\mathrm{SE}(3)

posthoc, via either

k

-NN clustering of tracks or clustering of our rigidity embeddings, whose cluster IoU is inherited from our full model and therefore omitted (–). (d) Loss design: removing individual loss terms. ADD is reported in centimeters; AUC r uses a

30^{\circ}

threshold, while AUC

{}_{\text{ADD}}

uses a

10\,\text{cm}

threshold on HO3D and

20\,\text{cm}

on iTACO. Best in each column in bold , second underlined .

Camera coordinate 3D tracking
	PointOdyssey [ 115 ]		ADT [ 69 ]		PStudio [ 36 ]		Avg.
Method	L-16	L-50	L-16	L-50	L-16	L-50	L-16	L-50
SpatialTracker [ 100 ]	0.3116	0.2977	0.4962	0.4692	0.5390	0.4991	0.4489	0.4220
DELTA [ 67 ]	0.3529	0.3412	0.5116	0.4952	0.5922	0.5533	0.4856	0.4632
SpatialTrackerV2 [ 101 ]	0.1864	0.1785	0.2400	0.2330	0.3784	0.3690	0.2683	0.2602
MASt3R [ 50 ]	0.3546	0.3253	0.3368	0.3029	0.3293	0.2956	0.3402	0.3079

Appendix

Table 5: 3D point tracking results on PointOdyssey, ADT, and PStudio in both camera and world coordinates. We follow the evaluation protocol of Track4World [ 61 ] . We report tracking accuracy at horizons L-16 (

\uparrow

) and L-50 (

\uparrow

). Best in bold , second underlined . Avg. is the unweighted mean across the three datasets.

	Per-pixel $\mathrm{SE}(3)$				Object/Part-level $\mathrm{SE}(3)$
Method	RRE $\downarrow$	ADD $\downarrow$	AUC ${}_{r}\uparrow$	AUC ${}_{\text{ADD}}\uparrow$	RRE $\downarrow$	ADD $\downarrow$	AUC ${}_{\text{ADD}}\uparrow$	IoU $\uparrow$
SpatialTrackerV2 [ 101 ]	21.93	4.93	0.461	0.576	22.93	6.55	0.508	0.251
POMATO [ 113 ]	46.47	10.78	0.223	0.336	50.11	10.89	0.309	0.054
ZeroMSF [ 54 ]	44.79	11.57	0.243	0.353	46.71	10.74	0.387	0.073
Any4D [ 41 ]	42.30	10.28	0.251	0.432	44.67	10.45	0.455	0.267
V-DPM [ 78 ]	30.42	7.58	0.329	0.504	31.60	8.18	0.501	0.234

Appendix

Table 6:

\mathrm{SE}(3)

estimation on YCBInEOAT [ 95 ] . Nine clips of robot manipulation with rigid objects, evaluated with the protocol of Tab. 1 . ADD is reported in centimeters; AUC r uses a

30^{\circ}

threshold and AUC

{}_{\text{ADD}}

10\,\text{cm}

threshold. Best in bold , second underlined .

Figure 5: Qualitative comparison on in-the-wild videos. We compare MoSE3 against ProxyPose [ 112 ] on out-of-distribution real-world scenes, including deformable cases. For each method, we visualize the estimated per-pixel $\mathbf{T}_{q\to i}(\mathbf{u})$ by drawing its $\mathrm{SE}(3)$ coordinate axes at sampled pixels.

Figure 6: Additional qualitative results on in-the-wild videos. In addition to Fig. 4 , we show further out-of-distribution real-world scenes spanning both rigid and non-rigid cases, here also visualizing the predicted 3D point tracks alongside the per-pixel rigidity embeddings and the per-pixel

\mathrm{SE}(3)

motion overlaid on them.

Dataset	Source	Phys. multi-body	Articul. (multi-link)	Long-range tracks	3D track GT	Per-link $\mathrm{SE}(3)$ GT	Rigid- grouping	Cam. variety
FlyingThings3D [ 64 ]	Synth	X	X	X	$\checkmark$ ♭	X	X	$\checkmark$
Kubric MOVi-F [ 20 ]	Synth	$\checkmark$	X	$\checkmark$	$\checkmark$ ♯	$\circ$	X	$\checkmark$
TAP-Vid Kubric [ 12 ]	Synth	$\checkmark$	X	$\checkmark$	X	X	X	$\checkmark$
PointOdyssey [ 115 ]	Synth	$\circ$ ♢	X †	$\checkmark$	$\checkmark$	X	X	$\checkmark$
Dynamic Replica [ 38 ]	Synth	X †	X †	$\checkmark$	$\checkmark$	X	X	$\checkmark$
SynthVerse [ 114 ]	Synth	X ‡	$\checkmark$ ‡	$\checkmark$	$\checkmark$	X	X	X ‡

Appendix

Table 7: Comparison of supervision signals across point-tracking and articulation datasets. The point of this table is which annotation channels and scene regimes each dataset exposes, not raw scale. “Phys. multi-body” = multiple independently moving bodies that interact under physics simulation (collisions, contact). “Articul.” = bodies whose motion is a kinematic tree of multiple rigidly-moving links. “3D tracks” = dense 3D point trajectories over

>2

frames. “Per-link

\mathrm{SE}(3)

” = world-frame

4{\times}4

pose for each sub-object rigid link, every frame;

\circ

= only an object-level (single-rigid-body) pose. “Rigid-grouping” = per-pixel partition of pixels by shared transform, beyond instance segmentation. “Cam. variety” = multiple distinct camera trajectory modes (mixing moving and stationary), in the spirit of Kubric’s MOVi setups. Among the datasets compared here, Art-Kubric is the only one that provides all seven properties.

3D Point Tracking

3D point tracking estimates how scene points move through space and time, even under camera motion, occlusion, and deformation.

See all 30 papers

Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels
Mar 3, 2026Jiahao Lu, Jiayi Xu, Wenbo Hu +5Scene Flow Estimation 3D Point Tracking
The Hong Kong University of Science and Technology · ARC Lab, Tencent PCG
TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking
May 12, 2026Jisu Nam, Jahyeok Koo, Soowon Son +4Point Tracking 3D Point Tracking
KAIST AI · Google DeepMind
TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations
Sep 24, 2026Ayush Jain, Sreeharsha Paruchuri, Ishita Gupta +5Point Tracking 3D Point Tracking
Carnegie Mellon University · Meta

Mar 3, 2026J/K move · Enter open · S save

Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels

Jiahao Lu, Jiayi Xu, Wenbo Hu, Ruijie Zhu+4

The Hong Kong University of Science and Technology · ARC Lab, Tencent PCG

Scene Flow Estimation 3D Point Tracking

Open paper AskPDF arXiv

Estimating the 3D trajectory of every pixel from a monocular video is crucial and promising for a comprehensive understanding of the 3D dynamics of videos. Recent monocular 3D tracking works demonstrate impressive performance, but are limited to either tracking sparse points on the first frame or a slow optimization-based framework for dense tracking. In this paper, we propose a feedforward model, called Track4World, enabling an efficient holistic 3D tracking of every pixel in the world-centric coordinate system. Built on the global 3D scene representation encoded by a VGGT-style ViT, Track4World applies a novel 3D correlation scheme to simultaneously estimate the pixel-wise 2D and 3D dense flow between arbitrary frame pairs. The estimated scene flow, along with the reconstructed 3D geometry, enables subsequent efficient 3D tracking of every pixel of this video. Extensive experiments on multiple benchmarks demonstrate that our approach consistently outperforms existing methods in 2D/3D flow estimation and 3D tracking, highlighting its robustness and scalability for real-world 4D reconstruction tasks.

Abstract

3D Point Tracking

Explore similar work

Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels

TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking

TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations