Accurate 4D reconstruction remains challenging: jointly estimating camera motion, scene geometry, and non-rigid deformation entails highly nonconvex optimization, making global optimality difficult to guarantee. We present XM4D, a framework for accurate and efficient 4D reconstruction built upon certifiably globally optimal geometric optimization. By separating the estimation of camera motion from non-rigid deformation, we formulate both stages as structured quadratically constrained quadratic programs (QCQP) that admit semidefinite programming (SDP) relaxations and verifiable global optimality certificates. For camera motion estimation, we lift static point tracks into 3D using foundational depth prediction models and formulate a scaled bundle adjustment problem that jointly estimates camera poses, depth scales, and static landmarks. By analytically eliminating translations and landmarks, we reduce the problem to a QCQP over scaled rotations. For non-rigid reconstruction, we represent dynamic motion using an embedded deformation graph (EDG) and show that its tracking, rigidity, and temporal regularization objectives similarly reduce to a rotation-only QCQP. Both problems admit SDP relaxations that we solve efficiently with a GPU-accelerated Burer-Monteiro solver based on Riemannian optimization. The recovered geometry further initializes dynamic Gaussian splatting for photometric refinement. Extensive experiments across seven benchmarks demonstrate accurate camera motion estimation on both static and dynamic scenes, efficient optimization that scales to thousands of frames, and state-of-the-art novel-view rendering quality.
Figures & tables
Figure 2: The XM 4D view graph. (a) Schematic view graph. Camera poses (Ct,ct)∈SO(3)×R3 (red) observe static landmarks xis∈R3 (blue triangles) and dynamic points (pink circles) through the depth-lifted measurements u~its and u~itd (blue squares). Each dynamic keypoint is attached by skinning edges (dashed) to its neighboring nodes (Rj,t,pj,t) of the embedded deformation graph (EDG) (purple capsules), whose nodes are connected by EDG edges (thick solid lines). (b) The view graph instantiated on a dynamic scene: a train moves along a circular track (black arrow), colored curves show the dynamic point tracks, and each part of the train is an EDG node with a free SE(3) pose.
Figure 3: Certified versus local optimization. Results on a real DAVIS video [ 54 ] (top) and a synthetic deformation graph (bottom). Each panel shows the EDG nodes, their estimated rotations (red, green, and blue axes), and the graph edges at different frames. The local solver converges to a local minimum with wrong node rotations (orange boxes), whereas the certified XM solution recovers consistent rotations.
Dataset
Metric
Configuration
1
2
3
4
5
6
7
8
BAL
ATE-T
2.243
2.194
2.163
2.185
2.218
2.155
2.181
2.283
RPE-T
4.003
3.942
3.934
3.936
3.935
3.957
3.949
3.979
RPE-R
1.674
1.661
1.650
1.667
1.655
1.660
1.712
1.690
Sintel
ATE-T
53
53
54
51
52
56
49
74
RPE-T
72
71
75
69
71
74
69
105
Table 1: Mean camera accuracy ( ×10−3 ) of eight solver configurations of Section 5.2 , each determined by the sphere fraction ρ and the weight normalization: (1)–(5) raw weights at ρ=0.1 , 0.3 , 0.5 , 0.9 , and 1 ; (6) ρ=0.9 with the per-frame normalization ωit/wt ; (7) ρ=0.9 with the softer normalization ωit/wt ; and (8) the original XM solver [ 20 ] with raw weights. Configuration 6, our default, achieves the best or second-best accuracy in eight of the twelve rows.
Dataset
Metric
Method
COLMAP
VGGT
MegaSAM
MoSca
Ours
IMC
ATE-T
0.003
0.075
0.871
–
0.047
RPE-T
0.005
0.121
1.221
–
0.074
RPE-R
0.002
0.033
0.703
–
0.044
Replica
ATE-T
0.009
0.014
0.067
0.060
0.002
RPE-T
0.012
0.019
0.087
0.072
0.002
Table 2: Camera accuracy averaged per dataset. MoSca [ 35 ] crashes on IMC [ 28 ] (–), whose unordered photo collections are not video data. VGGT [ 67 ] is run on subsampled frames on DyCheck [ 18 ] (Section 6.3 ).
Dataset
Stage
Method
COLMAP
VGGT
MegaSAM
MoSca
Ours
IMC
Total
227.7
38.7
136.8
–
23.2
Replica
Total
125.4
54.4
83.5
1054.4
115.1
Preprocess
–
–
49.6
888.5
100.5
Solver
–
–
33.9
165.9
14.6
TUM
Total
1743.7
52.8
79.5
403.8
83.7
Table 3: Runtime of camera estimation. Top: runtime per scene in seconds, averaged per dataset and split into preprocessing (depth, masks, tracking) and the solver. Bottom: wall-clock time in seconds on Replica [ 63 ] room0 as the number of input frames N grows (contiguous prefix); a run is invalid if it exceeds the GPU memory ( OOM ), runs longer than 1 h ( > 1h ), or crashes ( err ). Best in bold, second best underlined.
Table 4: Novel-view rendering averaged over the scenes of each dataset; (w)/(w/o): with/without ground-truth camera poses. XM 4D is the best on every metric although it estimates its camera poses.
Dataset
Method
Preprocess
Optimization
Total
DyCheck
Shape of Motion
230.0
282.4
512.4
MoSca
9.2
31.3
40.5
Ours
8.8
25.8
34.6
NVIDIA
Shape of Motion
0.6
48.3
48.8
MoSca
1.5
32.1
33.6
Ours
1.5
20.4
21.9
Table 5: Per-scene runtime of the full dynamic reconstruction in minutes (averages over the scenes of each dataset), split into preprocessing (depth, masks, static and dynamic tracking) and optimization (camera estimation, the motion model, and photometric training). Best in bold.
Figure 4: Qualitative results of XM 4D . Sequences from DyCheck [ 18 ] , NVIDIA [ 81 ] , Syn4D, DAVIS [ 54 ] , Sintel [ 12 ] , EgoDexter [ 46 ] , a Boston Dynamics video [ 6 ] , and a generated video from Google Flow. For each sequence, from left to right: an input frame, the reconstructed static and dynamic structure, the 3D trajectories of the dynamic points over the video, a rendered novel view, and the rendered depth.
Figure 5: Performance overview. Each axis is the square-root-compressed ratio to the best method, averaged over the datasets of Tables 2 – 5 . Open polygons mark axes a method does not report. Farther out is better. XM 4D performs best overall, leading on four of the five axes.
Figure 6: Estimated camera trajectories. Estimated camera poses (blue) after Sim(3) alignment to the ground truth (red), shown for one scene per benchmark. IMC [ 28 ] consists of unordered images, whereas all other benchmarks contain temporally ordered sequences.
Figure 8: Articulated object reconstruction with XM 4D . A person opens the four doors of a refrigerator in turn in a casual monocular video. (a) Input frames. (b) The four moving doors, color-coded. (c) Rendered views of the reconstruction at the same time instants. (d) Rotation angle and translation of each door over the video. (e) Reconstructed geometry with the four doors and their estimated hinge axes.
Dataset
Static
Dynamic (EDG)
scenes
median η
nok
scenes
median η
nok
Sintel
23
9.9×10−12
23/23
–
–
–
TUM
6
3.6×10−11
6/6
–
–
–
Replica
8
9.7×10−10
8/8
–
–
–
BAL
3
3.9×10−5
3/3
–
–
–
IMC
6
5.5×10−12
6/6
–
–
–
Table 6: Median relative suboptimality η of ( 15 ) over every scene of each dataset. nok counts solves with η<10−3 . When Z(yr) is PSD and the solution has rank three with all blocks of positive determinant, the solve is certified optimal for the original problem, and we report η=10−12 , the numerical resolution of the eigenvalue solve. Otherwise, the static η uses the trace-restricted bound of Appendix 10 . For the dynamic solver, each entry is one segment solve (its final certificate), and the median and nok are over all segment solves.
Metric
Method
bear1
bear2
bear3
bear4
casa1
casa2
casa3
casa4
Mean
Chamfer
Ours
6.1
5.4
6.4
4.6
11.2
9.2
7.6
8.6
7.4
MoSca
27.3
29.1
75.2
45.6
112.0
125.1
50.2
60.4
65.6
SoM
34.4
214.7
90.8
172.4
38.3
33.3
69.3
2429.9
385.4
Train depth
Ours
0.012
0.012
0.026
0.015
0.077
0.029
0.054
0.045
0.034
MoSca
0.031
0.029
0.035
0.029
0.123
0.049
0.081
0.069
0.056
SoM
0.102
0.096
0.104
0.096
0.193
0.082
0.158
0.139
0.121
Table 7: Geometry accuracy on Syn4D per scene: Chamfer distance ( ×10−3 ) between the reconstructed and the ground-truth dynamic object, and relative L1 error of the rendered depth in the training and novel views; lower is better. All methods are scored with the same protocol: the Chamfer distance follows the camera-based similarity alignment with one global scale per scene chosen to minimize it, and the depth uses one median scale per scene, fitted on the training views and reused for the novel views. SoM is Shape of Motion [ 68 ] , and DA renders the Depth Anything 3 [ 38 ] depth lifted with its cameras.