Accurate 4D reconstruction remains challenging: jointly estimating camera motion, scene geometry, and non-rigid deformation entails highly nonconvex optimization, making global optimality difficult to guarantee. We present XM4D, a framework for accurate and efficient 4D reconstruction built upon certifiably globally optimal geometric optimization. By separating the estimation of camera motion from non-rigid deformation, we formulate both stages as structured quadratically constrained quadratic programs (QCQP) that admit semidefinite programming (SDP) relaxations and verifiable global optimality certificates. For camera motion estimation, we lift static point tracks into 3D using foundational depth prediction models and formulate a scaled bundle adjustment problem that jointly estimates camera poses, depth scales, and static landmarks. By analytically eliminating translations and landmarks, we reduce the problem to a QCQP over scaled rotations. For non-rigid reconstruction, we represent dynamic motion using an embedded deformation graph (EDG) and show that its tracking, rigidity, and temporal regularization objectives similarly reduce to a rotation-only QCQP. Both problems admit SDP relaxations that we solve efficiently with a GPU-accelerated Burer-Monteiro solver based on Riemannian optimization. The recovered geometry further initializes dynamic Gaussian splatting for photometric refinement. Extensive experiments across seven benchmarks demonstrate accurate camera motion estimation on both static and dynamic scenes, efficient optimization that scales to thousands of frames, and state-of-the-art novel-view rendering quality.
Figures & tables
Figure 2: The XM 4D view graph. (a) Schematic view graph. Camera poses (Ct,ct)∈SO(3)×R3 (red) observe static landmarks xis∈R3 (blue triangles) and dynamic points (pink circles) through the depth-lifted measurements u~its and u~itd (blue squares). Each dynamic keypoint is attached by skinning edges (dashed) to its neighboring nodes (Rj,t,pj,t) of the embedded deformation graph (EDG) (purple capsules), whose nodes are connected by EDG edges (thick solid lines). (b) The view graph instantiated on a dynamic scene: a train moves along a circular track (black arrow), colored curves show the dynamic point tracks, and each part of the train is an EDG node with a free SE(3) pose.
Figure 3: Certified versus local optimization. Results on a real DAVIS video [ 54 ] (top) and a synthetic deformation graph (bottom). Each panel shows the EDG nodes, their estimated rotations (red, green, and blue axes), and the graph edges at different frames. The local solver converges to a local minimum with wrong node rotations (orange boxes), whereas the certified XM solution recovers consistent rotations.
Dataset
Metric
Configuration
1
2
3
4
5
6
7
8
BAL
ATE-T
2.243
2.194
2.163
2.185
2.218
2.155
2.181
2.283
RPE-T
4.003
3.942
3.934
3.936
3.935
3.957
3.949
3.979
RPE-R
1.674
1.661
1.650
1.667
1.655
1.660
1.712
1.690
Sintel
ATE-T
53
53
54
51
52
56
49
74
RPE-T
72
71
75
69
71
74
69
105
Table 1: Mean camera accuracy ( ×10−3 ) of eight solver configurations of Section 5.2 , each determined by the sphere fraction ρ and the weight normalization: (1)–(5) raw weights at ρ=0.1 , 0.3 , 0.5 , 0.9 , and 1 ; (6) ρ=0.9 with the per-frame normalization ωit/wt ; (7) ρ=0.9 with the softer normalization ωit/wt ; and (8) the original XM solver [ 20 ] with raw weights. Configuration 6, our default, achieves the best or second-best accuracy in eight of the twelve rows.
Dataset
Metric
Method
COLMAP
VGGT
MegaSAM
MoSca
Ours
IMC
ATE-T
0.003
0.075
0.871
–
0.047
RPE-T
0.005
0.121
1.221
–
0.074
RPE-R
0.002
0.033
0.703
–
0.044
Replica
ATE-T
0.009
0.014
0.067
0.060
0.002
RPE-T
0.012
0.019
0.087
0.072
0.002
Table 2: Camera accuracy averaged per dataset. MoSca [ 35 ] crashes on IMC [ 28 ] (–), whose unordered photo collections are not video data. VGGT [ 67 ] is run on subsampled frames on DyCheck [ 18 ] (Section 6.3 ).
Dataset
Stage
Method
COLMAP
VGGT
MegaSAM
MoSca
Ours
IMC
Total
227.7
38.7
136.8
–
23.2
Replica
Total
125.4
54.4
83.5
1054.4
115.1
Preprocess
–
–
49.6
888.5
100.5
Solver
–
–
33.9
165.9
14.6
TUM
Total
1743.7
52.8
79.5
403.8
83.7
Table 3: Runtime of camera estimation. Top: runtime per scene in seconds, averaged per dataset and split into preprocessing (depth, masks, tracking) and the solver. Bottom: wall-clock time in seconds on Replica [ 63 ] room0 as the number of input frames N grows (contiguous prefix); a run is invalid if it exceeds the GPU memory ( OOM ), runs longer than 1 h ( > 1h ), or crashes ( err ). Best in bold, second best underlined.
Table 4: Novel-view rendering averaged over the scenes of each dataset; (w)/(w/o): with/without ground-truth camera poses. XM 4D is the best on every metric although it estimates its camera poses.
Dataset
Method
Preprocess
Optimization
Total
DyCheck
Shape of Motion
230.0
282.4
512.4
MoSca
9.2
31.3
40.5
Ours
8.8
25.8
34.6
NVIDIA
Shape of Motion
0.6
48.3
48.8
MoSca
1.5
32.1
33.6
Ours
1.5
20.4
21.9
Table 5: Per-scene runtime of the full dynamic reconstruction in minutes (averages over the scenes of each dataset), split into preprocessing (depth, masks, static and dynamic tracking) and optimization (camera estimation, the motion model, and photometric training). Best in bold.
Figure 4: Qualitative results of XM 4D . Sequences from DyCheck [ 18 ] , NVIDIA [ 81 ] , Syn4D, DAVIS [ 54 ] , Sintel [ 12 ] , EgoDexter [ 46 ] , a Boston Dynamics video [ 6 ] , and a generated video from Google Flow. For each sequence, from left to right: an input frame, the reconstructed static and dynamic structure, the 3D trajectories of the dynamic points over the video, a rendered novel view, and the rendered depth.
Figure 5: Performance overview. Each axis is the square-root-compressed ratio to the best method, averaged over the datasets of Tables 2 – 5 . Open polygons mark axes a method does not report. Farther out is better. XM 4D performs best overall, leading on four of the five axes.
Figure 6: Estimated camera trajectories. Estimated camera poses (blue) after Sim(3) alignment to the ground truth (red), shown for one scene per benchmark. IMC [ 28 ] consists of unordered images, whereas all other benchmarks contain temporally ordered sequences.
Figure 8: Articulated object reconstruction with XM 4D . A person opens the four doors of a refrigerator in turn in a casual monocular video. (a) Input frames. (b) The four moving doors, color-coded. (c) Rendered views of the reconstruction at the same time instants. (d) Rotation angle and translation of each door over the video. (e) Reconstructed geometry with the four doors and their estimated hinge axes.
Dataset
Static
Dynamic (EDG)
scenes
median η
nok
scenes
median η
nok
Sintel
23
9.9×10−12
23/23
–
–
–
TUM
6
3.6×10−11
6/6
–
–
–
Replica
8
9.7×10−10
8/8
–
–
–
BAL
3
3.9×10−5
3/3
–
–
–
IMC
6
5.5×10−12
6/6
–
–
–
Table 6: Median relative suboptimality η of ( 15 ) over every scene of each dataset. nok counts solves with η<10−3 . When Z(yr) is PSD and the solution has rank three with all blocks of positive determinant, the solve is certified optimal for the original problem, and we report η=10−12 , the numerical resolution of the eigenvalue solve. Otherwise, the static η uses the trace-restricted bound of Appendix 10 . For the dynamic solver, each entry is one segment solve (its final certificate), and the median and nok are over all segment solves.
Metric
Method
bear1
bear2
bear3
bear4
casa1
casa2
casa3
casa4
Mean
Chamfer
Ours
6.1
5.4
6.4
4.6
11.2
9.2
7.6
8.6
7.4
MoSca
27.3
29.1
75.2
45.6
112.0
125.1
50.2
60.4
65.6
SoM
34.4
214.7
90.8
172.4
38.3
33.3
69.3
2429.9
385.4
Train depth
Ours
0.012
0.012
0.026
0.015
0.077
0.029
0.054
0.045
0.034
MoSca
0.031
0.029
0.035
0.029
0.123
0.049
0.081
0.069
0.056
SoM
0.102
0.096
0.104
0.096
0.193
0.082
0.158
0.139
0.121
Table 7: Geometry accuracy on Syn4D per scene: Chamfer distance ( ×10−3 ) between the reconstructed and the ground-truth dynamic object, and relative L1 error of the rendered depth in the training and novel views; lower is better. All methods are scored with the same protocol: the Chamfer distance follows the camera-based similarity alignment with one global scale per scene chosen to minimize it, and the depth uses one median scale per scene, fitted on the training views and reused for the novel views. SoM is Shape of Motion [ 68 ] , and DA renders the Depth Anything 3 [ 38 ] depth lifted with its cameras.
We introduce Complet4R, a novel end-to-end framework for Geometric Complete 4D Reconstruction, which aims to recover temporally coherent and geometrically complete reconstruction for dynamic scenes. Our method formalizes the task of Geometric Complete 4D Reconstruction as a unified framework of reconstruction and completion, by directly accumulating full contexts onto each frame. Unlike previous approaches that rely on pairwise reconstruction or local motion estimation, Complet4R utilizes a decoder-only transformer to operate all context globally directly from sequential video input, reconstructing a complete geometry for every single timestamp, including occluded regions visible in other frames. Our method demonstrates the state-of-the-art performance on our proposed benchmark for Geometric Complete 4D Reconstruction and the 3D Point Tracking task. Code will be released to support future research.
Weibang Wang, Kenan Li, Zhuoguang Chen +2
IIIS, Tsinghua University · Shanghai Artificial Intelligence Laboratory · Shanghai Qi Zhi Institute
Geometry Foundation Models (GFMs) have substantially advanced monocular 3D reconstruction, yet extending this capability to 4D dynamic understanding remains a fundamental challenge. Most existing motion perception methods (e.g., sparse tracking, dense point-wise flow) treat motion as independent point-wise displacements, ignoring the structured nature of physical motion. However, real-world objects usually obey rigid-body kinematics, and points thus usually move collectively, not in isolation. Motion itself possesses geometric structure: physical objects undergo a set of rigid-body transformations governed by SE(3), rather than unstructured point-wise displacements. Building on this insight, we propose SM4RT, a Structured Motion 4D Reconstruction Transformer for end-to-end 3D reconstruction and structured motion perception. SM4RT introduces Structure-of-Motion to represent scene dynamics, where scene motion is decomposed into a compact set of motion bases, each represented as a temporal sequence of 6D twists in SE(3). Dense scene motion is then recovered by sparse, time-shared per-pixel assignment weights over these bases, ensuring points on the same object share a common rigid-body motion trajectory. SM4RT introduces a parallel motion geometry encoder and decoder that jointly infer 3D geometry, world-coordinate motion, and scene kinematic structure in a single forward pass from monocular RGB video. SM4RT achieves strong motion reconstruction performance while preserving the geometric structure of scene motion.
We introduce SM4RT, a multi-task cascade feed-forward model for dynamic scene reconstruction from RGB videos. While existing feed-forward approaches typically address geometry reconstruction and motion estimation separately--and often predict motions in a single coordinate frame that must account for camera movement--SM4RT solves these tasks sequentially by first reconstructing 3D geometry and then bootstrapping motion estimation from the predicted structure. By expressing each pixel's 3D motion in its corresponding camera frame, SM4RT reduces ambiguity arising from camera motion. To address the sparsity of real-world motion labels, SM4RT further incorporates a weakly supervised loss that regularizes unlabeled point motions using labeled point positions at the other time step. We evaluate motion estimation on three dynamic-scene datasets--Kubric, Waymo, and Point Odyssey--and SM4RT achieves the best performance on most metrics compared with prior baselines. We additionally benchmark geometry reconstruction on five dynamic scene datasets and show that SM4RT achieves competitive performance with state-of-the-art methods. Overall, our unified framework demonstrates that combining geometry reconstruction and motion estimation in a cascade not only avoids performance trade-offs but also enhances motion estimation through improved geometric reasoning.
Pang-Chi Lo, Jhao-Syun Lai, Tsung-Wei Ke
Carnegie Mellon University · National Taiwan University