Initialization-Free Bundle Adjustment Revisited: A Controlled Experimental Study
Authors: Simon Weber, Mateo de Mayo, Je Hyeong Hong, Carl Olsson, Daniel Cremers, Ronald Clark
Organizations: department of Computer Science at the University of Oxford, United Kingdom · Technical University of Munich and the Munich Center for Machine Learning, Germany · department of Electronic Engineering at Hanyang University, South Korea · Centre for Mathematical Sciences at Lund University, Sweden
Initialization-free bundle adjustment (InitFree BA) aims to recover camera poses and scene structure directly from image observations, avoiding the geometric initialization stages of conventional structure-from-motion pipelines. Recent methods based on Object-Space Error (OSE) formulations and Variable Projection (VarPro) show encouraging optimization behavior from random camera configurations. However, existing evaluations primarily measure optimization success, leaving unclear whether a low OSE objective yields a valid metric 3D reconstruction. We revisit InitFree BA experimentally through a unified evaluation framework combining a C++ implementation of existing OSE formulations with a Blender-based dataset generator providing exact ground truth and controlled camera configurations and observation densities. Our experiments reveal a previously overlooked optimization--reconstruction gap: projective solutions with similarly low OSE values can lead to substantially different Euclidean reconstructions after metric upgrade. We identify initialization priors, landmark observation density, and metric-upgrade stability as key factors governing reconstruction success. Overall, our results suggest that the main challenge of InitFree BA is not merely minimizing OSE objectives, but obtaining projective reconstructions that admit reliable metric upgrade. We believe that the proposed benchmark, implementation, and analysis establish stronger experimental foundations for future research on initialization-free bundle adjustment, a problem largely unexplored within the computer vision community. Project page is available at https://github.com/simonwebertum/InitFreeBA.git.
Figures & tables
Fig. 1 : Traditional BA and initialization-free BA solve different input problems. (a) Traditional BA minimizes reprojection error from image observations and an initial estimate of cameras and landmarks. It is a well-studied problem [ 1 ] . (b) In contrast, initialization-free BA starts only from image observations and must recover cameras and structure without a geometric initialization. This problem is recent [ 2 , 3 ] and still largely uncharted.
Fig. 2 : Initialization-free BA can recover structure where randomly initialized traditional BA fails. (a) We apply a recent traditional BA solver [ 4 ] randomly initialized on our simple Set3 problem. It collapses to an invalid solution. InitFree BA, here with pOSE [ 3 ] , instead estimates (b) a projective reconstruction, and can then be upgraded (c) to a metric reconstruction.
Fig. 3 : Initialization-free BA is highly sensitive to formulation and initialization, even on a controlled scene. Panels show (a), (b) Set2 and (c), (d) Set5 after optimization and linear metric upgrade, with the corresponding normalized OSE objective shown below each reconstruction. Although the OSE objective decreases rapidly and reaches close final errors for each corresponding pair, the resulting Euclidean reconstructions can differ dramatically. This illustrates that a decreasing OSE objective is not by itself a certificate of successful 3D reconstruction.
Fig. 4 : Metric upgrade failure rate as a function of the final OSE residual. A run is counted as failed when Rot. AUC@20 <20 , Trans. AUC@20 <20 , or landmark RMSE >30 . In contrast to an underlying assumption of previous works, a low OSE error does not necessarily ensure a valid reconstruction.
Fig. 5 : Overview of the dataset generator. Each row shows a rendered image, 2D observations (blue) and projected 3D landmarks (red), depth map, covisibility matrix, and reconstructed points and cameras. From top to bottom: Set 6; an artist-made realistic scene 4 4 4 Model courtesy of Aurélien Martel, licensed CC BY-NC 4.0, Sketchfab. ; and a photogrammetry scan of Skokloster Castle 5 5 5 Scan courtesy of Skoklosters slott, licensed CC BY 4.0, Sketchfab. . The examples illustrate that the generator supports both controlled and realistic scene geometries. Our generator is not restricted to InitFree BA problem, and is a reusable tool for controlled evaluation in SfM, SLAM, and related 3D reconstruction problems
Fig. 6 : The eight sequences used in our experiments. Sets 1–2 and 3–4 test the effect of observation sparsity on convergence, Set 5 is a real-world COLMAP reconstruction used to validate our controlled findings in a realistic setting, and Sets 6–8 test the effect of camera distribution and lookat targets on convergence. Bottom row: covisibility matrix for each sequence, giving an indication of the density of the Schur complement and of how well connected the cameras are (cyan: weakly covisible, magenta: highly covisible; normalized per matrix).
Formulation
Rot. A5
Rot. A20
Trans. A5
Trans. A20
pOSE
25.43
34.25
26.43
34.10
rOSE
24.33
31.99
28.82
37.56
RpOSE
9.28
15.24
12.29
19.85
expOSE
25.74
35.89
26.39
35.94
pOSE+rot
42.14
53.87
45.80
57.08
TABLE I : Average pose accuracy by formulation. Results are averaged over all datasets, initialization distributions, and non-robust/Cauchy variants after the linear metric upgrade. We report AUC 5° and AUC 20°, for rotation and translation. Higher is better.
Formulation
Lmk. Med. ↓
Lmk. RMSE ↓
Lmk. P90 ↓
pOSE
22.68
24.19
33.28
rOSE
20.52
21.82
29.92
RpOSE
27.93
29.58
40.66
expOSE
19.71
21.40
29.81
pOSE+rot
15.27
16.63
22.46
TABLE II : Average structure accuracy by formulation. Results are averaged over all datasets, initialization distributions, and non-robust/Cauchy variants after the linear metric upgrade. Lower is better.
Fig. 7 : Effect of initialization. On Set8 with pOSE+rot, normal randomization collapses, whereas unit-circle initialization recovers a coherent reconstruction.
Initialization
Rot. A5
Rot. A20
Trans. A5
Trans. A20
Lmk. RMSE ↓
Normal
24.23
29.60
26.39
32.21
24.63
Uniform
23.47
30.54
25.71
33.04
24.27
Unit circle
28.45
42.60
31.74
45.47
19.27
TABLE III : Average effect of initialization. Results are averaged over all datasets, formulations, and non-robust/Cauchy variants after the linear metric upgrade. Higher is better for AUC; lower is better for landmark RMSE.
Initialization
Rot. A5
Rot. A20
Trans. A5
Trans. A20
Normal
25.01 / 23.46
30.18 / 29.01
26.29 / 26.49
31.64 / 32.79
Uniform
25.72 / 21.22
32.59 / 28.49
27.35 / 24.08
34.27 / 31.80
Unit circle
27.95 / 28.94
42.24 / 42.96
31.58 / 31.89
45.80 / 45.13
TABLE IV : Effect of initialization and Cauchy robustification on pose recovery. Results are averaged over all datasets and formulations after the linear metric upgrade. Each entry reports without Cauchy/with Cauchy, and the best value for each initialization is bolded.
Initialization
Lmk. Med. ↓
Lmk. RMSE ↓
Lmk. P90 ↓
Normal
22.66 / 23.19
24.50 / 24.76
33.96 / 34.17
Uniform
22.50 / 22.86
23.98 / 24.56
33.18 / 33.98
Unit circle
17.81 / 18.31
18.99 / 19.55
25.60 / 26.48
TABLE V : Effect of initialization and Cauchy robustification on structure accuracy. Results are averaged over all datasets and formulations after the linear metric upgrade. Each entry reports without Cauchy/with Cauchy, and the best value for each initialization is bolded. Lower is better.
Fig. 8 : Effect of robustification. On Set2 with pOSE and normal initialization, the non-robust solver collapses, while the Cauchy loss recovers a valid metric reconstruction.
Dataset
Form.
Init.
Loss
Rot. A20
Trans. A20
Lmk. RMSE ↓
Set2
pOSE
Normal
None
0.4
3.9
40.66
Cauchy
97.8
99.0
0.10
Set2
pOSE+rot
Uniform
None
2.5
9.9
40.66
Cauchy
99.3
99.7
0.03
Set2
rOSE
Normal
None
3.7
7.0
40.66
Cauchy
96.4
98.3
0.17
TABLE VI : Robustification can prevent catastrophic failures with linear metric upgrade. Each row compares the non-robust formulation with the Cauchy variant for the same dataset, formulation, and initialization.
Fig. 9 : Effect of observation density. On Set2 with rOSE, filtering weakly observed landmarks stabilizes the metric upgrade and produces a more coherent reconstruction.
Loss
Δ Rot. A5
Δ Rot. A20
Δ Trans. A5
Δ Trans. A20
Δ Lmk. RMSE ↓
None
+6.31
+6.21
+6.46
+5.10
-2.63
Cauchy
+8.55
+10.08
+9.38
+9.54
-4.67
TABLE VII : Effect of densification. We report the change obtained by increasing the landmark visibility threshold from three to six observations, computed on the six complete common datasets: Set1–Set5 and Set8. Positive values are better for AUC metrics; negative values are better for landmark RMSE. Results use linear metric upgrade.
Form.
Δ Rot. A20
Δ Trans. A20
Δ Lmk. RMSE ↓
pOSE
+1.04
+2.35
-2.53
rOSE
+12.60
+12.01
-5.68
RpOSE
+18.34
+14.17
-5.15
expOSE
+1.99
+3.61
-1.37
pOSE+rot
+6.75
+4.46
-3.53
TABLE VIII : Model-wise effect of densification. We report the change induced by increasing the minimum landmark visibility from three to six observations, using the six complete common datasets. Results are averaged over initialization distributions and non-robust/Cauchy variants after linear metric upgrade.
Dataset
Form.
Init.
Loss
Rot. A20
Trans. A20
Lmk. RMSE ↓
Set4
expOSE
Normal
None
0.6
0.3
41.52
Cauchy
100.0
100.0
0.00
Set2
RpOSE
Normal
None
1.3
2.2
41.00
Cauchy
99.9
99.9
0.01
Set3
pOSE
Normal
None
0.9
3.8
41.19
Cauchy
100.0
100.0
0.00
TABLE IX : Robustification can prevent catastrophic failures for minimum 6 obs. per landmark, with linear metric upgrade. Each row compares the non-robust formulation with the best robust variant for the same dataset, formulation, and initialization. Metrics are reported after the linear metric upgrade.
Stage
Rot. A20
Trans. A20
Lmk. RMSE ↓
Before metric upgrade
16.34
6.23
25.86
Linear metric upgrade
36.30
39.06
22.14
Mean change
+19.96
+32.83
-3.73
TABLE X : Effect of linear metric upgrade. We compare the projective solution before metric upgrade with the reconstruction obtained after the linear metric upgrade. Results are averaged over all datasets, formulations, initializations, and losses.
Fig. 10 : Metric upgrade failure rate as a function of the final refinement residual. We use the final mean residual after the refinement optimization and evaluate failure after linear metric upgrade with the same criterion as in Fig. 4 .
Fig. 11 : Failure and recovery modes on BAL after linear metric upgrade. Each row varies one factor while keeping the scene and formulation fixed, for Trafalgar-150 (first row), and Venice-52 (other rows). The OSE residual decreases in both successful and failed cases, whereas the refinement residual more clearly separates successful metric reconstructions from collapsed ones.
TABLE XI : Median runtime decomposition. Values are seconds and are computed over all datasets, initialization distributions, and the non-robust/Cauchy variants. The pre-stage-2 column includes formulation-specific preprocessing and intermediate steps before reprojection refinement, including the RpOSE local radial refinement and the pOSE+rot pair/matrix construction.
Variant
Rot. A5
Rot. A20
Trans. A5
Trans. A20
Lmk. RMSE
Wall time
pOSE+rot
42.14
53.87
45.81
57.08
16.63
38.39
pOSE+rot est.
41.80
52.90
45.30
54.83
17.04
69.03
Appendix
TABLE XII : pOSE+rot with ground-truth and estimated relative rotations. Results use the linear metric upgrade and are averaged over all datasets, initialization distributions, and the non-robust/Cauchy variants. Higher is better for AUC; lower is better for landmark RMSE.
Fig. 12 : Metric upgrade failure rate as a function of the final OSE residual. We consider non-robust and Cauchy losses, followed by selected metric upgrade. A run is counted as failed when Rot. AUC@20 <20 , Trans. AUC@20 <20 , or landmark RMSE >30 .
Fig. 13 : Metric upgrade failure rate as a function of the final refinement residual. We use the final mean residual after the refinement optimization and evaluate failure after selected metric upgrade with the same criterion as in Fig. 12 .
Formulation
Rot. A5
Rot. A20
Trans. A5
Trans. A20
pOSE
24.89
34.15
26.73
35.64
rOSE
24.95
33.48
31.24
41.45
RpOSE
9.18
15.01
12.49
20.36
expOSE
25.08
35.04
26.68
37.28
pOSE+rot
41.99
54.16
46.08
57.63
Appendix
TABLE XIII : Average pose accuracy by formulation. Results are averaged over all datasets, initialization distributions, and non-robust/Cauchy variants after the selected linear/refined metric upgrade. Higher is better.
Formulation
Lmk. Med. ↓
Lmk. RMSE ↓
Lmk. P90 ↓
pOSE
21.06
22.80
31.03
rOSE
18.67
20.06
27.30
RpOSE
28.01
29.62
40.75
expOSE
18.39
20.37
28.14
pOSE+rot
15.34
16.67
22.44
Appendix
TABLE XIV : Average structure accuracy by formulation. Results are averaged over all datasets, initialization distributions, and non-robust/Cauchy variants after the selected linear/refined metric upgrade. Lower is better.
Initialization
Rot. A5
Rot. A20
Trans. A5
Trans. A20
Lmk. RMSE ↓
Normal
24.30
30.29
27.48
34.62
22.81
Uniform
23.30
30.38
26.57
34.68
23.81
Unit circle
28.06
42.43
31.89
46.10
19.09
Appendix
TABLE XV : Average effect of initialization. Results are averaged over all datasets, formulations, and non-robust/Cauchy variants after the selected linear/refined metric upgrade. Higher is better for AUC; lower is better for landmark RMSE.
Loss
Rot. A20
Trans. A20
Lmk. Med. ↓
Lmk. RMSE ↓
Lmk. P90 ↓
None
34.87
37.69
20.53
22.06
30.22
Cauchy
33.87
39.24
20.07
21.74
29.65
Appendix
TABLE XVI : Average effect of robustification. Results are averaged over all datasets, formulations, and initialization distributions after the selected linear/refined metric upgrade. Higher is better for AUC; lower is better for landmark errors.
pOSE
rOSE
RpOSE
expOSE
pOSE+rot
Dataset
Initialization
A5
A10
A20
A5
A10
A20
A5
A10
A20
A5
A10
A20
A5
A10
A20
Set1
Normal
0.2/86.6/86.6%
1.7/93.3/93.3%
5.5/96.6/96.6%
0.0/76.8/76.8%
1.4/88.4/88.4%
4.5/94.2/94.2%
1.3/0.1/0.0%
4.3/1.9/1.2%
10.5/5.4/3.6%
0.2/71.2/71.2%
1.8/85.6/85.6%
5.2/92.8/92.8%
0.4/56.3/56.3%
2.5/78.1/78.1%
6.7/89.1/89.1%
Uniform
0.9/83.1/83.1%
3.4/91.6/91.6%
8.7/95.8/95.8%
2.6/70.8/70.8%
6.7/85.4/85.4%
15.0/92.7/92.7%
0.8/1.7/1.7%
3.6/5.2/5.2%
10.3/12.6/12.6%
1.8/88.5/88.5%
5.1/94.2/94.2%
12.1/97.1/97.1%
2.0/90.8/90.8%
5.8/95.4/95.4%
14.9/97.7/97.7%
Unit circle
81.8/6.2/6.2%
90.9/14.9/14.9%
95.5/34.4/34.4%
17.6/5.6/5.6%
50.1/13.5/13.5%
75.0/31.6/31.6%
86.3/0.0/0.0%
93.1/1.3/1.4%
96.6/3.9/4.1%
83.5/8.0/8.0%
91.8/19.4/19.4%
95.9/41.4/41.4%
32.2/18.5/18.5%
63.0/37.8/37.8%
81.5/64.6/64.6%
Set2
Normal
0.0/0.0/0.0%
0.1/0.0/0.0%
0.2/0.4/0.4%
0.4/0.0/0.0%
2.7/1.2/1.2%
9.0/3.7/3.7%
0.0/0.0/0.0%
0.4/0.4/0.4%
1.9/1.6/1.6%
0.1/0.0/0.0%
0.8/0.4/0.4%
2.7/1.4/1.4%
3.0/89.5/89.5%
7.2/94.7/94.7%
15.4/97.4/97.4%
Uniform
0.0/0.0/0.0%
0.5/1.1/1.0%
2.5/3.3/3.2%
0.0/0.0/0.0%
0.4/0.7/0.7%
2.2/2.6/2.6%
0.0/0.0/0.0%
0.3/0.5/0.5%
1.8/1.5/1.5%
0.0/0.0/0.0%
0.2/1.2/1.2%
1.8/3.6/3.6%
0.1/0.0/0.0%
0.6/0.7/0.7%
2.3/2.5/2.3%
Appendix
TABLE XVII : Rotation AUC results without robust loss. Each entry reports before/linear/selected metric upgrade.
pOSE
rOSE
RpOSE
expOSE
pOSE+rot
Dataset
Initialization
A5
A10
A20
A5
A10
A20
A5
A10
A20
A5
A10
A20
A5
A10
A20
Set1
Normal
0.7/0.6/0.0%
3.0/2.8/1.5%
10.1/7.0/4.5%
0.5/42.7/42.7%
2.5/69.0/69.0%
7.4/84.5/84.5%
0.8/0.0/0.0%
3.4/1.4/1.4%
11.2/4.2/4.2%
0.2/26.3/26.3%
1.7/48.9/48.9%
5.6/73.8/73.8%
0.1/79.3/79.3%
1.5/89.7/89.7%
4.6/94.8/94.8%
Uniform
0.6/19.6/19.6%
3.2/39.3/39.3%
10.5/66.0/66.0%
3.0/64.4/44.7%
7.6/82.2/71.5%
16.4/91.1/85.7%
0.1/0.1/0.1%
0.8/0.9/0.9%
3.3/3.5/3.5%
0.6/35.7/2.8%
3.1/61.2/7.2%
10.1/80.6/16.7%
0.1/19.8/19.8%
1.9/37.8/37.8%
7.5/63.6/63.6%
Unit circle
82.2/5.7/5.7%
91.1/13.7/13.7%
95.6/32.0/32.0%
17.6/5.9/5.9%
49.9/14.1/14.1%
75.0/32.8/32.8%
86.3/0.0/0.0%
93.1/1.3/1.3%
96.6/3.9/4.0%
83.5/7.2/7.2%
91.8/17.4/17.4%
95.9/38.5/38.5%
31.4/18.3/18.3%
62.3/37.5/37.5%
81.2/64.3/64.3%
Set2
Normal
0.1/91.4/91.4%
1.1/95.7/95.7%
3.9/97.8/97.8%
3.1/85.7/85.7%
9.7/92.9/92.9%
35.5/96.4/96.4%
0.0/0.0/0.0%
0.5/0.8/0.8%
2.3/3.0/3.0%
0.5/0.0/0.0%
2.7/1.2/1.2%
7.2/3.6/3.6%
0.1/0.0/0.0%
0.6/0.7/0.4%
2.4/2.4/1.9%
Uniform
0.0/0.0/0.0%
0.5/1.1/1.2%
2.7/3.5/3.7%
0.0/0.0/0.0%
0.3/1.1/1.0%
2.3/3.3/3.2%
0.1/0.0/0.0%
0.8/0.8/0.8%
3.3/3.0/3.0%
0.0/0.1/0.0%
0.6/0.4/0.5%
2.6/1.9/2.2%
0.8/97.2/97.2%
3.3/98.6/98.6%
8.5/99.3/99.3%
Appendix
TABLE XVIII : Rotation AUC results with Cauchy loss. Each entry reports before/linear/selected metric upgrade.
pOSE
rOSE
RpOSE
expOSE
pOSE+rot
Dataset
Initialization
T5
T10
T20
T5
T10
T20
T5
T10
T20
T5
T10
T20
T5
T10
T20
Set1
Normal
0.2/93.8/93.8%
0.5/96.9/96.9%
1.9/98.5/98.5%
0.2/89.3/89.3%
1.0/94.6/94.6%
4.1/97.3/97.3%
0.2/1.8/1.9%
1.1/3.3/3.6%
3.8/6.4/7.2%
1.5/86.6/86.6%
6.7/93.3/93.3%
19.5/96.7/96.7%
0.9/79.9/79.9%
4.0/89.9/89.9%
11.2/95.0/95.0%
Uniform
0.4/92.2/92.2%
1.2/96.1/96.1%
3.3/98.1/98.1%
0.2/86.5/86.5%
0.5/93.3/93.3%
2.1/96.6/96.6%
0.6/12.1/12.1%
1.6/21.4/21.4%
3.6/36.6/36.6%
0.2/94.7/94.7%
0.8/97.3/97.3%
3.1/98.7/98.7%
0.3/95.7/95.7%
1.3/97.9/97.9%
4.1/98.9/98.9%
Unit circle
1.0/26.9/26.9%
4.2/43.5/43.5%
10.7/66.8/66.8%
0.0/25.6/25.6%
0.2/41.5/41.5%
1.5/64.3/64.3%
2.0/2.0/1.8%
4.2/3.8/3.7%
8.0/7.5/7.3%
0.3/30.2/30.2%
1.5/48.6/48.6%
4.5/72.1/72.1%
1.4/43.8/43.8%
3.1/68.1/68.1%
6.6/84.0/84.0%
Set2
Normal
0.1/0.7/0.7%
0.2/1.7/1.7%
1.1/3.9/3.9%
0.1/1.6/1.6%
0.3/3.5/3.5%
0.7/7.0/7.0%
0.2/0.6/0.6%
0.4/2.2/2.2%
1.0/5.5/5.5%
0.2/0.2/0.2%
0.3/0.8/0.8%
1.1/2.8/2.8%
0.2/95.1/95.1%
0.8/97.6/97.6%
2.9/98.8/98.8%
Uniform
0.2/2.2/2.0%
0.7/4.3/3.5%
1.7/8.3/6.6%
0.1/0.5/0.5%
0.1/2.6/2.6%
0.5/7.4/7.4%
0.1/0.6/0.6%
0.3/1.8/1.8%
1.0/4.9/4.9%
0.1/1.6/1.6%
0.3/3.5/3.5%
2.4/7.2/7.2%
0.6/2.1/1.5%
1.8/4.5/3.9%
5.9/9.9/8.9%
Appendix
TABLE XIX : Translation-direction AUC results without robust loss. Each entry reports before/linear/selected metric upgrade.
pOSE
rOSE
RpOSE
expOSE
pOSE+rot
Dataset
Initialization
T5
T10
T20
T5
T10
T20
T5
T10
T20
T5
T10
T20
T5
T10
T20
Set1
Normal
0.1/2.7/2.2%
0.2/4.8/4.2%
1.3/9.3/8.4%
0.3/71.5/71.5%
1.0/85.8/85.8%
4.0/92.9/92.9%
0.1/1.9/1.9%
0.9/3.8/3.8%
3.4/7.6/7.6%
0.2/53.4/53.4%
0.7/76.2/76.2%
3.6/88.1/88.1%
0.1/90.5/90.5%
0.2/95.2/95.2%
1.0/97.6/97.6%
Uniform
0.1/44.8/44.8%
0.5/69.2/69.2%
4.0/84.6/84.6%
3.9/83.5/76.3%
8.1/91.7/88.2%
15.3/95.9/94.1%
0.2/0.1/0.1%
0.3/0.5/0.5%
1.5/3.8/3.8%
0.2/63.9/33.2%
0.7/81.9/54.7%
3.0/91.0/77.1%
0.9/41.9/41.9%
3.5/66.1/66.1%
8.7/83.0/83.0%
Unit circle
1.0/25.8/25.8%
4.1/41.8/41.8%
10.5/64.6/64.6%
0.0/26.2/26.2%
0.2/42.4/42.4%
1.8/65.4/65.4%
2.0/1.9/1.8%
4.2/3.7/3.7%
8.0/7.4/7.3%
0.3/28.8/28.8%
1.6/46.4/46.4%
4.5/70.0/70.0%
1.4/43.5/43.5%
3.0/67.8/67.8%
6.4/83.9/83.9%
Set2
Normal
0.3/96.0/96.0%
1.0/98.0/98.0%
3.4/99.0/99.0%
0.1/93.4/93.4%
0.3/96.7/96.7%
1.7/98.3/98.3%
1.0/1.7/1.7%
2.1/4.0/4.0%
4.7/7.7/7.7%
0.1/1.8/1.8%
0.4/3.6/3.6%
1.4/7.2/7.2%
0.1/1.5/0.5%
0.1/3.1/2.1%
1.1/5.8/6.6%
Uniform
0.1/1.9/1.7%
0.3/3.7/3.4%
0.5/7.4/6.9%
0.1/1.4/1.7%
0.2/3.6/3.6%
0.6/7.5/7.4%
0.1/0.3/0.3%
0.5/1.2/1.2%
2.3/4.0/4.0%
0.4/1.2/0.5%
1.3/3.1/2.2%
3.5/7.0/6.0%
0.6/98.7/98.7%
1.9/99.3/99.3%
5.6/99.7/99.7%
Appendix
TABLE XX : Translation-direction AUC results with Cauchy loss. Each entry reports before/linear/selected metric upgrade.
pOSE
rOSE
RpOSE
expOSE
pOSE+rot
Dataset
Initialization
Med
RMSE
P90
Med
RMSE
P90
Med
RMSE
P90
Med
RMSE
P90
Med
RMSE
P90
Set1
Normal
27.89/0.13/0.13
32.26/0.17/0.17
45.94/0.24/0.24
21.73/0.23/0.23
26.31/0.29/0.29
38.77/0.41/0.41
28.22/39.68/39.68
32.39/41.16/41.16
44.84/58.11/58.11
32.51/0.29/0.29
35.84/0.36/0.36
51.25/0.52/0.52
24.64/0.44/0.44
26.34/0.54/0.54
34.17/0.78/0.78
Uniform
30.02/0.17/0.17
33.92/0.21/0.21
45.62/0.30/0.30
22.07/0.29/0.29
27.38/0.36/0.36
38.55/0.52/0.52
27.49/7.19/7.19
31.32/8.85/8.85
43.70/13.31/13.31
27.39/0.12/0.12
31.82/0.14/0.14
42.62/0.20/0.20
22.26/0.09/0.09
26.60/0.11/0.11
35.64/0.16/0.16
Unit circle
0.94/3.87/3.87
1.13/4.32/4.32
1.56/6.24/6.24
2.99/4.20/4.20
3.98/4.70/4.70
5.89/6.78/6.78
3.01/39.68/39.68
5.26/41.16/41.16
8.38/58.11/58.11
2.10/3.17/3.17
2.43/3.54/3.54
3.55/5.11/5.11
2.49/1.60/1.60
2.72/1.82/1.82
3.96/2.65/2.65
Set2
Normal
39.10/39.12/39.12
40.65/40.66/40.66
58.19/58.19/58.19
19.03/39.12/39.12
21.59/40.66/40.66
31.34/58.19/58.19
38.23/39.12/39.12
39.25/40.66/40.66
54.92/58.19/58.19
39.12/39.12/39.12
40.66/40.66/40.66
58.20/58.19/58.19
14.85/0.10/0.10
17.19/0.13/0.13
23.09/0.18/0.18
Uniform
38.76/39.12/39.12
40.03/40.66/40.66
58.12/58.19/58.20
38.42/39.12/39.12
39.77/40.66/40.66
57.30/58.19/58.19
39.03/39.12/39.12
40.50/40.66/40.66
57.94/58.19/58.19
39.12/39.12/39.12
40.66/40.66/40.66
58.19/58.19/58.19
38.84/39.12/39.12
40.28/40.66/40.66
57.99/58.19/58.19
Appendix
TABLE XXI : Landmark accuracy without robust loss. Each entry reports before/linear/selected metric upgrade; lower is better.
pOSE
rOSE
RpOSE
expOSE
pOSE+rot
Dataset
Initialization
Med
RMSE
P90
Med
RMSE
P90
Med
RMSE
P90
Med
RMSE
P90
Med
RMSE
P90
Set1
Normal
19.52/39.68/39.67
19.52/41.15/41.16
26.30/58.11/58.11
25.98/0.64/0.64
29.38/0.77/0.77
39.68/1.12/1.12
20.65/39.68/39.68
25.58/41.16/41.16
37.85/58.11/58.11
29.72/1.14/1.14
32.48/1.32/1.32
44.53/1.93/1.93
32.04/0.21/0.21
33.38/0.26/0.26
43.63/0.37/0.37
Uniform
20.88/1.51/1.51
20.57/1.73/1.73
28.12/2.52/2.52
14.07/0.36/0.52
16.27/0.44/0.65
22.23/0.64/0.92
33.31/39.68/39.68
36.94/41.16/41.16
53.32/58.11/58.11
17.66/0.81/2.24
19.39/0.98/2.73
25.88/1.42/3.94
18.99/1.62/1.62
21.18/1.90/1.90
27.47/2.78/2.78
Unit circle
0.94/4.14/4.14
1.13/4.63/4.63
1.55/6.69/6.69
3.00/4.05/4.05
3.99/4.53/4.53
5.90/6.54/6.54
3.01/39.68/39.68
5.26/41.16/41.16
8.38/58.11/58.11
2.12/3.43/3.43
2.44/3.84/3.84
3.57/5.53/5.53
2.53/1.61/1.61
2.77/1.83/1.83
4.04/2.68/2.68
Set2
Normal
27.78/0.09/0.09
31.38/0.10/0.10
44.39/0.15/0.15
6.38/0.14/0.14
8.07/0.17/0.17
12.05/0.25/0.25
36.04/39.12/39.12
37.71/40.66/40.66
52.40/58.18/58.18
26.61/39.12/39.12
27.55/40.66/40.66
34.47/58.19/58.19
38.66/39.12/39.12
39.99/40.66/40.66
57.89/58.19/58.19
Uniform
38.61/39.12/39.12
39.59/40.66/40.66
54.64/58.19/58.20
38.62/39.12/39.12
39.90/40.66/40.66
57.95/58.19/58.19
36.73/39.12/39.12
38.51/40.66/40.66
54.20/58.19/58.19
38.94/39.12/39.12
40.46/40.66/40.66
57.60/58.19/58.19
30.97/0.03/0.03
33.76/0.03/0.03
44.19/0.05/0.05
Appendix
TABLE XXII : Landmark accuracy with Cauchy loss. Each entry reports before/linear/selected metric upgrade; lower is better.
pOSE
rOSE
RpOSE
expOSE
pOSE+rot
Dataset
Initialization
A5
A10
A20
A5
A10
A20
A5
A10
A20
A5
A10
A20
A5
A10
A20
Set1
Normal
0.0/0.4%
1.5/2.4%
5.6/6.4%
0.3/43.4%
2.2/69.4%
6.9/84.7%
2.5/95.0%
7.0/97.5%
21.0/98.7%
0.7/33.0%
3.3/57.5%
10.3/78.7%
1.5/63.8%
5.7/81.9%
13.7/90.9%
Uniform
2.1/14.9%
8.7/30.2%
25.4/53.4%
3.0/74.2%
7.9/87.1%
18.2/93.6%
0.4/92.1%
2.5/96.1%
7.2/98.0%
1.1/58.8%
3.8/79.4%
9.2/89.7%
0.9/22.2%
3.6/41.6%
9.1/67.6%
Unit circle
82.2/8.4%
91.1/20.4%
95.6/42.9%
17.4/7.7%
49.7/18.6%
74.8/40.4%
86.2/0.0%
93.1/1.5%
96.5/4.3%
83.1/8.7%
91.5/21.2%
95.8/44.0%
30.9/21.3%
61.9/42.1%
80.9/68.6%
Set2
Normal
0.0/0.0%
0.7/1.2%
3.0/3.5%
5.7/98.2%
14.8/99.1%
39.6/99.6%
0.8/99.4%
3.5/99.7%
9.0/99.9%
1.5/0.0%
4.7/1.5%
11.4/4.4%
1.7/84.9%
5.1/92.5%
12.3/96.2%
Uniform
1.4/85.7%
4.7/92.9%
14.0/96.4%
0.7/76.5%
3.0/88.2%
8.1/94.1%
0.0/0.0%
0.4/0.2%
2.6/1.3%
1.6/78.9%
5.0/89.5%
11.7/94.7%
1.4/85.5%
4.5/92.8%
11.1/96.4%
Appendix
TABLE XXIII : Rotation AUC with minimum 6 observations per landmark and Cauchy loss. Each entry reports before/linear metric upgrade.
pOSE
rOSE
RpOSE
expOSE
pOSE+rot
Dataset
Initialization
A5
A10
A20
A5
A10
A20
A5
A10
A20
A5
A10
A20
A5
A10
A20
Set1
Normal
0.6/67.4%
3.0/83.7%
7.8/91.8%
1.0/78.7%
3.6/89.3%
9.0/94.7%
1.4/95.2%
4.4/97.6%
10.6/98.8%
1.1/81.8%
3.7/90.9%
9.3/95.5%
1.4/57.9%
4.5/78.9%
14.1/89.5%
Uniform
1.3/74.9%
4.1/87.5%
10.0/93.7%
2.9/65.9%
7.6/83.0%
17.8/91.5%
1.3/41.0%
4.4/67.2%
10.7/83.6%
2.6/70.0%
7.1/85.0%
16.7/92.5%
1.1/6.9%
3.9/16.9%
9.7/33.8%
Unit circle
81.8/8.9%
90.9/21.7%
95.5/44.7%
17.4/8.0%
49.8/19.5%
74.9/41.7%
86.2/0.0%
93.1/1.3%
96.6/3.9%
83.1/8.7%
91.5/21.3%
95.8/44.1%
31.7/17.7%
62.6/36.6%
81.3/63.4%
Set2
Normal
2.1/92.7%
6.0/96.4%
14.1/98.2%
4.5/98.5%
11.8/99.3%
31.7/99.6%
0.1/0.0%
0.7/0.2%
2.8/1.3%
4.0/96.1%
9.9/98.1%
21.8/99.0%
2.2/54.3%
6.2/77.0%
14.7/88.5%
Uniform
0.0/0.0%
1.1/1.1%
3.7/3.4%
1.5/95.1%
4.9/97.6%
11.5/98.8%
0.0/0.0%
0.9/0.7%
3.0/2.6%
2.7/88.1%
6.9/94.0%
16.0/97.0%
1.7/94.1%
5.1/97.1%
12.2/98.5%
Appendix
TABLE XXIV : Rotation AUC with minimum 6 observations per landmark without robust loss. Each entry reports before/linear metric upgrade. Note that Set6 and Set7 are too sparse for supporting this densification.
pOSE
rOSE
RpOSE
expOSE
pOSE+rot
Dataset
Initialization
A5
A10
A20
A5
A10
A20
A5
A10
A20
A5
A10
A20
A5
A10
A20
Set1
Normal
0.1/6.9%
0.1/11.4%
0.6/18.3%
0.3/71.5%
1.6/85.8%
6.8/92.9%
0.1/97.7%
0.5/98.8%
3.7/99.4%
0.1/60.4%
0.5/80.2%
1.7/90.1%
0.6/83.2%
2.4/91.6%
6.9/95.8%
Uniform
0.1/34.9%
0.3/56.4%
0.9/78.0%
0.1/88.0%
0.4/94.0%
1.6/97.0%
0.1/96.4%
0.4/98.2%
1.8/99.1%
0.2/80.9%
1.0/90.4%
5.4/95.2%
5.5/45.4%
11.0/69.8%
19.7/84.9%
Unit circle
1.0/31.1%
4.1/49.8%
10.7/73.2%
0.0/29.8%
0.2/47.9%
1.6/71.4%
2.0/1.9%
4.2/3.8%
8.1/7.5%
0.3/31.6%
1.6/50.6%
4.6/73.8%
1.4/47.4%
2.9/71.6%
6.4/85.8%
Set2
Normal
0.1/2.0%
0.3/3.8%
1.2/7.5%
0.0/99.2%
0.0/99.6%
0.5/99.8%
0.2/99.7%
0.5/99.9%
1.5/99.9%
0.2/1.5%
0.3/3.4%
0.5/7.3%
0.6/93.0%
1.6/96.5%
3.7/98.3%
Uniform
0.1/93.4%
0.4/96.7%
1.3/98.4%
0.3/89.1%
0.9/94.6%
3.3/97.3%
0.2/0.3%
0.5/1.0%
1.8/3.0%
1.1/90.2%
3.9/95.1%
11.2/97.6%
0.3/93.3%
1.1/96.6%
3.4/98.3%
Appendix
TABLE XXV : Translation-direction AUC with minimum 6 observations per landmark with Cauchy loss. Each entry reports before/linear metric upgrade.
pOSE
rOSE
RpOSE
expOSE
pOSE+rot
Dataset
Initialization
A5
A10
A20
A5
A10
A20
A5
A10
A20
A5
A10
A20
A5
A10
A20
Set1
Normal
0.3/84.9%
1.3/92.4%
4.1/96.2%
2.2/90.1%
5.3/95.1%
13.2/97.5%
0.1/97.8%
1.6/98.9%
5.1/99.4%
0.3/91.6%
1.3/95.8%
3.7/97.9%
0.1/80.5%
0.3/90.2%
1.2/95.1%
Uniform
0.1/88.4%
0.3/94.2%
0.9/97.1%
0.2/84.2%
0.6/92.1%
2.1/96.0%
0.4/69.5%
1.5/84.8%
4.3/92.4%
0.3/86.1%
0.6/93.1%
1.8/96.5%
4.3/24.4%
11.2/39.7%
22.6/61.7%
Unit circle
1.0/32.0%
4.3/51.2%
10.8/74.3%
0.0/30.4%
0.1/48.9%
1.3/72.3%
2.0/2.0%
4.3/3.8%
8.1/7.4%
0.4/31.6%
1.6/50.7%
4.6/73.9%
1.4/42.7%
3.0/66.9%
6.6/83.5%
Set2
Normal
0.8/96.7%
2.3/98.3%
5.8/99.2%
0.4/99.3%
1.4/99.7%
6.6/99.8%
0.2/0.2%
0.3/0.7%
0.9/2.2%
0.3/98.2%
1.1/99.1%
4.7/99.6%
0.7/78.7%
1.9/89.3%
4.5/94.7%
Uniform
0.6/1.6%
1.7/3.4%
4.0/6.7%
0.1/97.7%
0.5/98.9%
3.0/99.4%
0.2/1.6%
0.8/4.2%
2.0/9.6%
0.1/94.5%
0.5/97.3%
1.7/98.6%
0.7/97.3%
1.8/98.6%
4.0/99.3%
Appendix
TABLE XXVI : Translation-direction AUC with minimum 6 observations per landmark without robust loss. Each entry reports before/linear metric upgrade.
Bundle Adjustment (BA) is a cornerstone of 3D computer vision and has benefited from decades of advances in sparse optimization and numerical methods. It was originally developed for jointly optimizing camera intrinsics, poses and sparse 3D points. While extensions incorporate lines and other primitives, integrating richer geometric structures such as parallelism, coplanarity, or wireframes often introduces significantly increased computational cost and reduced numerical stability. In this paper, we propose a unified framework that extends bundle adjustment to jointly optimize geometric features and higher-order relations. We first introduce a taxonomy that distinguishes scalable geometric features with direct 2D measurements (e.g., points and lines), from groups encoding higher-order relations (e.g., coplanarity, parallelism, etc.), where we show that groups can be modeled as camera-like entities within the bundle adjustment framework. Building on this formulation, we propose that both group constraints and cross-feature relations (i.e., point-line associations) can be expressed through 2D reprojection measurements. By formulating group-induced and cross-feature reprojection errors, we preserve the sparsity structure of classical point-based BA under Schur elimination, while avoiding direct 3D regularization that degrades the conditioning and stability. Experiments on both real-world and synthetic datasets demonstrate runtime performance comparable to classical point-only bundle adjustment, while producing significantly richer 3D structures and improved geometric accuracy.
Shaohui Liu, Rémi Pautrat, Daniel Barath +3
ETH Zurich · Microsoft Spatial AI Lab · Australian National University +1
Recent hybrid Structure-from-Motion (SfM) systems combine the robustness of feed-forward 3D reconstruction with the accuracy of traditional bundle adjustment (BA) with pixel matching. They are usually the best performing methods however their scalability and usability remains limited since estimating dense correspondences between views is prohibitively costly, especially considering time constraints inherent to online applications like Visual SLAM (VSLAM). In this paper, we introduce a regularized BA framework that leverages a fast multi-view matcher and monocular priors for initialization and regularization. In contrast to existing systems, our unified approach seamlessly supports both online VSLAM and offline reconstruction from unordered image collections within the same optimization framework and sharing common hyperparameters for all tasks. Extensive experiments across both domains demonstrate improved performance and speed tradeoffs over traditional, feed-forward, and hybrid baselines. Notably for VSLAM, our uncalibrated method outperforms all previous calibrated approaches.
Vincent Leroy, Philippe Weinzaepfel, Lojze Zust +2
Bundle adjustment (BA) remains a critical refinement module for image-based 3D reconstruction and continues to improve geometric accuracy even in learning-based pipelines. However, in low-parallax and near-rotational regimes, classical Schur-based Levenberg--Marquardt (LM) often becomes ill-conditioned and yields unreliable pose and calibration estimates. We propose Gate-Guided CSS-BA, a solver-side modification of Schur-LM that preserves the classical BA objective and trust-region framework while constraining each update to a geometrically informed low-dimensional subspace. By integrating Column Space Search (CSS) with geometry-aware gating, the method stabilizes the Schur-LM update without altering the estimation problem. In contrast to keyframe or state-selection approaches, all camera and point parameters remain in the optimization problem; only the update direction is restricted. The method serves as a drop-in replacement for existing BA pipelines. Experiments on both generic and challenging weak-geometry scenarios show more stable optimization, improved relative pose accuracy, and competitive calibration behavior while maintaining reprojection quality.