ORBIT++: Benchmarking SfM in the Wild with 360° Video

Sara Sabour; Linyi Jin; Richard Tucker; Amir Hertz; Marcus Brubaker; Saurabh Saxena; Junhwa Hur; Andrea Tagliasacchi

cs.CVAug 22, 2026

ORBIT++: Benchmarking SfM in the Wild with 360° Video

Authors: Sara Sabour, Linyi Jin, Richard Tucker, Amir Hertz, Marcus Brubaker, Saurabh Saxena, Junhwa Hur, Andrea Tagliasacchi, +4 more

Organizations: Google DeepMind · Simon Fraser University · University of Toronto

Abstract

Structure-from-Motion (SfM) is a cornerstone of 3D perception, yet current methods often fail when applied to complex videos involving challenging camera motions or dynamic scenes. Compounding the problem, the field lacks reliable ground-truth benchmarks for such difficult scenarios, making it hard to gauge real-world progress or to pinpoint where improvements are most needed. To address this gap, we introduce a new benchmark for evaluating camera pose estimation. Our key insight is to leverage online panoramic 360° video as a source of data from which to construct challenging clips, while still enabling robust ground-truth trajectory recovery. The panoramic nature of these videos provides richer visual context for tracking camera motion, even when parts of the view are affected by blur, motion, or dynamic objects. After tracking camera motion across full 360° videos, we crop and reproject selected portions to generate perspective-view clips that serve as our benchmark, called ORBIT. Experiments show that COLMAP, as well as recent optimization-based and feed-forward SfM methods struggle to accurately estimate camera poses on our benchmark. Hence, ORBIT provides a valuable testbed where researchers can meaningfully measure progress on truly challenging, real-world SfM problems.

Figures & tables

Figure 4: Example benchmark videos. Our benchmark video clips show a diversity of scenes with varying degrees of dynamic object presence, reprojected to different rotation patterns, fields of view, and aspect ratios.

Method	ATE Median $\downarrow$	ATE $\downarrow$	RPE-R $\downarrow$	RPE-T $\downarrow$	Failure Rate $\%\downarrow$
COLMAP	0.40	1.34 $\pm$ 2.16	1.19 $\pm$ 2.53	0.13 $\pm$ 0.28	56.49
MegaSaM	0.13	0.67 $\pm$ 1.24	0.44 $\pm$ 0.89	0.04 $\pm$ 0.10	39.09
RoMo+MegaSaM	0.11	0.59 $\pm$ 1.18	0.43 $\pm$ 0.93	0.05 $\pm$ 0.12	38.56
ORB-SLAM2	1.78	2.55 $\pm$ 2.71	1.20 $\pm$ 1.70	0.13 $\pm$ 0.25	95.45
VGGT-Long	0.84	1.30 $\pm$ 1.35	1.55 $\pm$ 1.58	0.15 $\pm$ 0.18	94.72
MonST3R	1.34	1.96 $\pm$ 1.79	1.71 $\pm$ 1.90	0.22 $\pm$ 0.44	98.44

Table 1: Evaluating camera pose estimates with ORBIT . We report the ATE median, as well as the mean and standard deviation of ATE, RPE-T, and RPE-R. Finally, we report failure rate as defined in Sec. 4.2 .

Panel of Figure 5 — Figure 5: Visual verification. Given our estimated ground truth camera poses, we stabilize the 360° videos. Red lines depict the horizon in the input equirect frame (first row), as well as the equirect following stabilization (second row). We also plot reprojected tracked 3D points via our estimated ground truth trajectories. These augmented videos assist us in validating our pseudo-ground truth camera trajectories.

Figure 6: ATE Distribution. Here we sweep a threshold on a failure measure (defined in terms of the ATE being above a given threshold), and plot failure rate as a function of threshold. We observe different distributions between fully feedforward methods (VGGT-Long and MonST3R) and optimization-based methods (e.g., COLMAP, MegaSaM w/w.o Romo).

Challenge	COLMAP		MegaSaM		RoMo		ORB-SLAM2		VGGT-Long		MonST3R
	ATE	RPE-R	ATE	RPE-R	ATE	RPE-R	ATE	RPE-R	ATE	RPE-R	ATE	RPE-R
Speed	0.90	1.21	0.49	0.79	0.43	0.78	3.59	1.41	1.61	2.11	2.33	1.62
Crowd	0.43	1.07	0.08	0.23	0.08	0.20	1.79	1.12	0.81	1.17	1.51	1.87
Ego	0.39	1.04	0.22	0.51	0.20	0.52	1.96	1.23	0.93	1.65	1.25	1.68
Fluid	0.20	1.73	0.05	0.45	0.06	0.40	0.85	1.05	0.59	1.75	0.89	1.63
Light	0.50	2.38	0.08	0.31	0.08	0.23	1.76	1.23	0.69	0.91	1.20	1.56

Table 2: Metrics by challenge category . Median ATE and average RPE-R highlighting easiest (blue) and hardest (red) challenge categories for each method and according to each metric.

Figure 7: Variance in method performance. We sort each ORBIT clip by average ATE across methods and plot this average and the standard deviation on each clip. A high standard deviation indicates a clip where different methods exhibit highly varying performance. The high variance of method performances observed in this plot agrees with our finding of distinct failure modes per method.

Figure 8: RPE-Rot based on Rotation Pattern in ORBIT. VGGT-Long shows a strong bias toward performance on original rotation patterns.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Mode	Axes	Description
Cinematic	Yaw, Pitch	Interpolation from start to end. Yaw: start $\in U(-30^{\circ},30^{\circ})$ , end $\in U(-60^{\circ},60^{\circ})$ . Pitch: 70%: start/end $\in U(-30^{\circ},30^{\circ})$ , 30%: $\in U(-60^{\circ},60^{\circ})$ .
Scan	Yaw or Pitch	Sinusoidal horizontal or vertical sweep. Amplitude $\in U(10^{\circ},40^{\circ})$ , frequency $\in U(0.05,0.3)$ Hz.
Waypoints	Yaw, Pitch	Interpolation through 3+ randomly placed waypoints. Per waypoint: yaw $\in U(-180^{\circ},180^{\circ})$ ; pitch 70%: $U(-30^{\circ},30^{\circ})$ , 30%: $U(-60^{\circ},60^{\circ})$ .
Roll	Roll	Barrel-roll ( $60$ – $360^{\circ}$ ).
Orbit	Yaw	Spin along the vertical direction.

Appendix

Table 3: Rotation modes for benchmark synthesis. For each clip, a mode is randomly selected and its parameters are sampled from the listed ranges.

U(a,b)

denotes uniform sampling.

Figure 9: ORBIT Rotation Pattern Distribution – The number of clips in the benchmark for each pattern.

Figure 10: Trajectory Comparisons. Sample frames from ORBIT clips alongside trajectory comparisons between ground truth and each method’s estimated trajectory. Ground truth is shown in blue.

Explore similar work

VidMap: Exploiting Temporal Structure for Video-Based Structure-from-Motion
Jul 29, 2026Zador Pataki, Paul-Edouard Sarlin, Marc PollefeysMulti-View 3D Reconstruction Camera Pose Estimation
ETH Zurich · Google · Microsoft Spatial AI Lab
Benchmarking Efficient & Effective Camera Pose Estimation Strategies for Novel View Synthesis
Mar 20, 2026Jhacson Meza, Martin R. Oswald, Torsten SattlerCamera Pose Estimation Structure from Motion
Faculty of Electrical Engineering, Czech Technical University in Prague · Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague · University of Amsterdam
Planar-SfM: Camera Pose Estimation via Homography Graph Embeddings
Jun 30, 2026Gabi Pragier, Matan Karklinsky, David Ungarish +1Multi-View Geometry Camera Pose Estimation
Amazon Prime Video Live Events

Jul 29, 2026J/K move · Enter open · S save

VidMap: Exploiting Temporal Structure for Video-Based Structure-from-Motion

Zador Pataki, Paul-Edouard Sarlin, Marc Pollefeys

ETH Zurich · Google · Microsoft Spatial AI Lab

Multi-View 3D Reconstruction Camera Pose Estimation Visual SLAM Structure from Motion

Open paper AskPDF arXiv

Accurately recovering the camera's calibration and metric poses for any unconstrained video would unlock large-scale training data for navigation and scene understanding. The dominant approaches to this problem are severely limited: Simultaneous Localization and Mapping (SLAM) is sensitive to initialization and transient failures due to its causal, incremental nature; it is often over-optimized for real-time operation and generally requires known camera calibration; while Structure-from-Motion (SfM) typically forgoes any image ordering, enabling optimal initialization and global optimization, but lacks robustness to visual symmetries and extreme motions. To bridge this gap, we introduce a system that combines the strong sequential constraints of SLAM with the flexibility and global optimization of offline SfM, enabling the metric reconstruction of arbitrary, long, uncalibrated videos. This system leverages recent advances in wide-baseline dense image matching, treats temporal ordering as a first-class citizen for reliable loop closure, and augments global optimization with metric monocular depth priors. As a result, thorough evaluations on diverse, challenging datasets that exhibit extreme motion and visual symmetries reveal that our approach is significantly more robust and accurate than both state-of-the-art SLAM and SfM, classical or learned, with given or unknown camera calibration. The code is publicly available at https://github.com/cvg/vidmap.

Abstract

Explore similar work

VidMap: Exploiting Temporal Structure for Video-Based Structure-from-Motion

Benchmarking Efficient & Effective Camera Pose Estimation Strategies for Novel View Synthesis

Planar-SfM: Camera Pose Estimation via Homography Graph Embeddings