Structure-from-Motion (SfM) is a cornerstone of 3D perception, yet current methods often fail when applied to complex videos involving challenging camera motions or dynamic scenes. Compounding the problem, the field lacks reliable ground-truth benchmarks for such difficult scenarios, making it hard to gauge real-world progress or to pinpoint where improvements are most needed. To address this gap, we introduce a new benchmark for evaluating camera pose estimation. Our key insight is to leverage online panoramic 360° video as a source of data from which to construct challenging clips, while still enabling robust ground-truth trajectory recovery. The panoramic nature of these videos provides richer visual context for tracking camera motion, even when parts of the view are affected by blur, motion, or dynamic objects. After tracking camera motion across full 360° videos, we crop and reproject selected portions to generate perspective-view clips that serve as our benchmark, called ORBIT. Experiments show that COLMAP, as well as recent optimization-based and feed-forward SfM methods struggle to accurately estimate camera poses on our benchmark. Hence, ORBIT provides a valuable testbed where researchers can meaningfully measure progress on truly challenging, real-world SfM problems.
Figures & tables
Figure 4: Example benchmark videos. Our benchmark video clips show a diversity of scenes with varying degrees of dynamic object presence, reprojected to different rotation patterns, fields of view, and aspect ratios.
Method
ATE Median ↓
ATE ↓
RPE-R ↓
RPE-T ↓
Failure Rate %↓
COLMAP
0.40
1.34 ± 2.16
1.19 ± 2.53
0.13 ± 0.28
56.49
MegaSaM
0.13
0.67 ± 1.24
0.44 ± 0.89
0.04 ± 0.10
39.09
RoMo+MegaSaM
0.11
0.59 ± 1.18
0.43 ± 0.93
0.05 ± 0.12
38.56
ORB-SLAM2
1.78
2.55 ± 2.71
1.20 ± 1.70
0.13 ± 0.25
95.45
VGGT-Long
0.84
1.30 ± 1.35
1.55 ± 1.58
0.15 ± 0.18
94.72
MonST3R
1.34
1.96 ± 1.79
1.71 ± 1.90
0.22 ± 0.44
98.44
Table 1: Evaluating camera pose estimates with ORBIT . We report the ATE median, as well as the mean and standard deviation of ATE, RPE-T, and RPE-R. Finally, we report failure rate as defined in Sec. 4.2 .
Figure 5: Visual verification. Given our estimated ground truth camera poses, we stabilize the 360° videos. Red lines depict the horizon in the input equirect frame (first row), as well as the equirect following stabilization (second row). We also plot reprojected tracked 3D points via our estimated ground truth trajectories. These augmented videos assist us in validating our pseudo-ground truth camera trajectories.
Figure 6: ATE Distribution. Here we sweep a threshold on a failure measure (defined in terms of the ATE being above a given threshold), and plot failure rate as a function of threshold. We observe different distributions between fully feedforward methods (VGGT-Long and MonST3R) and optimization-based methods (e.g., COLMAP, MegaSaM w/w.o Romo).
Challenge
COLMAP
MegaSaM
RoMo
ORB-SLAM2
VGGT-Long
MonST3R
ATE
RPE-R
ATE
RPE-R
ATE
RPE-R
ATE
RPE-R
ATE
RPE-R
ATE
RPE-R
Speed
0.90
1.21
0.49
0.79
0.43
0.78
3.59
1.41
1.61
2.11
2.33
1.62
Crowd
0.43
1.07
0.08
0.23
0.08
0.20
1.79
1.12
0.81
1.17
1.51
1.87
Ego
0.39
1.04
0.22
0.51
0.20
0.52
1.96
1.23
0.93
1.65
1.25
1.68
Fluid
0.20
1.73
0.05
0.45
0.06
0.40
0.85
1.05
0.59
1.75
0.89
1.63
Light
0.50
2.38
0.08
0.31
0.08
0.23
1.76
1.23
0.69
0.91
1.20
1.56
Table 2: Metrics by challenge category . Median ATE and average RPE-R highlighting easiest (blue) and hardest (red) challenge categories for each method and according to each metric.
Figure 7: Variance in method performance. We sort each ORBIT clip by average ATE across methods and plot this average and the standard deviation on each clip. A high standard deviation indicates a clip where different methods exhibit highly varying performance. The high variance of method performances observed in this plot agrees with our finding of distinct failure modes per method.
Figure 8: RPE-Rot based on Rotation Pattern in ORBIT. VGGT-Long shows a strong bias toward performance on original rotation patterns.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Mode
Axes
Description
Cinematic
Yaw, Pitch
Interpolation from start to end. Yaw: start ∈U(−30∘,30∘) , end ∈U(−60∘,60∘) . Pitch: 70%: start/end ∈U(−30∘,30∘) , 30%: ∈U(−60∘,60∘) .
Scan
Yaw or Pitch
Sinusoidal horizontal or vertical sweep. Amplitude ∈U(10∘,40∘) , frequency ∈U(0.05,0.3) Hz.
Waypoints
Yaw, Pitch
Interpolation through 3+ randomly placed waypoints. Per waypoint: yaw ∈U(−180∘,180∘) ; pitch 70%: U(−30∘,30∘) , 30%: U(−60∘,60∘) .
Roll
Roll
Barrel-roll ( 60 – 360∘ ).
Orbit
Yaw
Spin along the vertical direction.
Appendix
Table 3: Rotation modes for benchmark synthesis. For each clip, a mode is randomly selected and its parameters are sampled from the listed ranges. U(a,b) denotes uniform sampling.
Figure 9: ORBIT Rotation Pattern Distribution – The number of clips in the benchmark for each pattern.
Figure 10: Trajectory Comparisons. Sample frames from ORBIT clips alongside trajectory comparisons between ground truth and each method’s estimated trajectory. Ground truth is shown in blue.
Accurately recovering the camera's calibration and metric poses for any unconstrained video would unlock large-scale training data for navigation and scene understanding. The dominant approaches to this problem are severely limited: Simultaneous Localization and Mapping (SLAM) is sensitive to initialization and transient failures due to its causal, incremental nature; it is often over-optimized for real-time operation and generally requires known camera calibration; while Structure-from-Motion (SfM) typically forgoes any image ordering, enabling optimal initialization and global optimization, but lacks robustness to visual symmetries and extreme motions. To bridge this gap, we introduce a system that combines the strong sequential constraints of SLAM with the flexibility and global optimization of offline SfM, enabling the metric reconstruction of arbitrary, long, uncalibrated videos. This system leverages recent advances in wide-baseline dense image matching, treats temporal ordering as a first-class citizen for reliable loop closure, and augments global optimization with metric monocular depth priors. As a result, thorough evaluations on diverse, challenging datasets that exhibit extreme motion and visual symmetries reveal that our approach is significantly more robust and accurate than both state-of-the-art SLAM and SfM, classical or learned, with given or unknown camera calibration. The code is publicly available at https://github.com/cvg/vidmap.
Novel view synthesis (NVS) approaches such as NeRFs or 3DGS can produce photo-realistic 3D scene representation from a set of images with known extrinsic and intrinsic parameters. The necessary camera poses and calibrations are typically obtained from the images via Structure-from-Motion (SfM). Classical SfM approaches rely on local feature matches between the images to estimate both the poses and a sparse 3D model of the scene, using bundle adjustment to refine initial pose, intrinsics, and geometry estimates. In order to increase run-time efficiency, recent SfM systems forgo optimization via bundle adjustment. Instead, they train feed-forward (transformer-based) neural networks to directly regress camera parameters and the 3D structure. While orders of magnitude more efficient, such recent works produce significantly less accurate estimates. To stimulate research on developing SfM approaches that are both efficient and effective, this paper develops a benchmark focused on SfM for novel view synthesis. Using existing datasets and two simple strategies for making the reconstruction process more efficient, we show that: (1) simply using fewer features already significantly accelerates classical SfM methods while maintaining high pose accuracy. (2) using feed-forward networks to obtain initial estimates and refining them using classical SfM techniques leads to the best efficiency-effectiveness trade-off. We will make our benchmark and code publicly available.
Jhacson Meza, Martin R. Oswald, Torsten Sattler
Faculty of Electrical Engineering, Czech Technical University in Prague · Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague · University of Amsterdam
Structure from Motion (SfM) systems traditionally struggle with planar scenes, where standard epipolar geometry-based methods become degenerate. Rather than viewing planar surfaces as a limitation, we propose a unified framework that leverages them as a source of geometric constraints. Our key insight is that each planar surface visible across multiple views provides an independent estimate of relative camera poses through homography decomposition. By aggregating estimates from multiple planes or even from a single dominant plane we achieve robust pose recovery in scenarios where traditional methods fail. We introduce a novel graph-based approach that constructs a pose-graph from homography estimates and employs spectral embedding to identify and filter unreliable edges. Our method maps homography-based pose estimates onto the real line based on their geometric and visual consistency, enabling efficient extraction of a maximally consistent spanning tree for pose recovery. This approach naturally handles both highly planar scenes, such as indoor sports arenas, and general 3D environments. We demonstrate superior performance on basketball court imagery where existing methods struggle, while matching or exceeding state-of-the-art results on unconstrained outdoor scenes from the IMC Phototourism benchmark.