Authors: Wanhui Luo, Hongbo Zhao, Yi Zhou, Qijun Chen, Yiming Nie, Rui Fan
Organizations: Department of Control Science & Engineering, the College of Electronic & Information Engineering, Tongji University, Shanghai 201804, China · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, Shanghai 201210, China · School of Artificial Intelligence and Robotics, Hunan University, 410082, Changsha, China · Unmanned Systems Technology Research Center, Defense Innovation Institute, Academy of Military Science, Beijing 100071, China · Department of Control Science & Engineering, the College of Electronic & Information Engineering, Shanghai Research Institute for Intelligent Autonomous Systems, the State Key Laboratory of Intelligent Autonomous Systems, and Frontiers Science Center for Intelligent Autonomous Systems, Tongji University, Shanghai 201804, China
The wide field of view of fisheye cameras and the simplicity of a monocular setup make monocular fisheye visual odometry (VO) attractive for robotic localization and mapping. Nevertheless, scale drift remains a major challenge over long trajectories. Unsupervised depth learning can help to mitigate this issue, but the pronounced distortion of fisheye images poses challenges for accurate depth estimation. This article presents S2D-FVO, a monocular fisheye VO framework integrating spherical bundle adjustment (SBA)-guided unsupervised depth learning to improve scale consistency and depth estimation accuracy. Given that pixel reprojection residuals do not uniformly reflect angular deviations between viewing rays and that the first-order linearization of a scalar angular reprojection residual is insensitive to perturbations along one tangent direction, SBA employs tangent-space angular reprojection residuals derived from the spherical logarithmic map. These residuals retain sensitivity along both tangent directions, with their norms equal to the corresponding scalar angular residuals. A median depth-based scale regularization term is further integrated into SBA to constrain the reconstruction scale. Building on these formulations, SBA replaces the pose estimation network during unsupervised depth learning, providing more reliable relative poses for view synthesis. In turn, depth estimates from the well-trained depth estimation network provide scale references for SBA, improving scale consistency while allowing individual 3-D landmark positions to be refined. Extensive experiments conducted on multiple public datasets, as well as two newly collected large-scale datasets, demonstrate that the proposed framework outperforms representative existing approaches in both depth and trajectory estimation accuracy and is applicable to different camera models.
Figures & tables
Fig. 1: Illustration of the proposed S 2 D-FVO framework for monocular fisheye visual odometry and depth estimation. An NIO ES8 vehicle is used for data collection. The estimated trajectory and depth map are compared with the ground truth, and a reconstructed local 3-D scene is visualized.
Fig. 2: An overview of the proposed S 2 D-FVO framework. Fisheye images are first remapped to ERP images for both camera tracking and unsupervised depth learning. Meanwhile, visual correspondences are unprojected onto the unit sphere for spherical bundle adjustment. Subsequently, the refined camera poses are incorporated into the unsupervised depth learning process, and the well-trained DepthNet is used to provide a scale reference for monocular fisheye visual odometry.
Fig. 3: An illustration of the tangent-space angular reprojection residual. The angle between the observed viewing ray bij and the predicted viewing ray b^ij is denoted by αij . The predicted viewing ray is then mapped to the tangent space TbijS2 via the logarithmic map Logbij(⋅) , and the coordinates of the resulting tangent vector with respect to the orthonormal basis {qij1,qij2} form the residual used in the spherical bundle adjustment.
Fig. 4: Training process within the n -th window of the proposed unsupervised fisheye depth learning framework. Spherical bundle adjustment refines the camera poses and 3-D landmarks, and the refined camera poses are subsequently used for view synthesis to train the DepthNet.
Fig. 5: Architecture of the proposed monocular fisheye visual odometry framework, which consists of camera tracking, local mapping, and loop closing. Feature track management is shown in green, while the spherical geometry pipeline is shown in blue.
Dataset
Camera Model
Method
Abs Rel ↓
Sq Rel ↓
RMSE ↓
RMSE log ↓
δ<1.25↑
δ<1.252↑
δ<1.253↑
KITTI-360
Perspective
Lite-Mono [ 89 ]
0.123
0.586
3.526
0.194
0.860
0.958
0.983
Monodepth2 [ 25 ]
0.212
1.186
4.616
0.307
0.656
0.896
0.957
SC-DepthV3 [ 90 ]
0.147
0.849
5.217
0.240
0.790
0.929
0.970
RA-Depth [ 91 ]
0.132
0.791
4.159
0.201
0.847
0.955
0.982
Manydepth2 [ 92 ]
0.145
0.656
3.821
0.216
0.804
0.946
0.980
Fisheye
FisheyeDepth [ 23 ]
0.128
0.647
3.587
0.202
0.850
0.954
0.981
TABLE I: Quantitative comparison with SoTA networks on the KITTI-360, NIO-Fisheye, and ScanNet++ datasets. The best results are shown in bold font. The symbols ↑ and ↓ indicate that higher and lower values correspond to better performance, respectively.
Fig. 6: Qualitative comparison of monocular fisheye depth estimation on the KITTI-360 dataset: (i) estimated depth maps; (ii) corresponding error maps. White and yellow dashed contours highlight regions with substantial improvements.
Fig. 7: Qualitative comparison of monocular fisheye depth estimation on the NIO-Fisheye dataset: (i) estimated depth maps; (ii) corresponding error maps. White and yellow dashed contours highlight regions with substantial improvements.
Fig. 8: Qualitative comparison of monocular fisheye depth estimation on the ScanNet++ dataset: (i) estimated depth maps; (ii) corresponding error maps. White and yellow dashed contours highlight regions with substantial improvements.
Dataset
Subset
Method
Abs Rel ↓
Sq Rel ↓
RMSE ↓
RMSE log ↓
δ<1.25↑
δ<1.252↑
δ<1.253↑
KITTI-360
–
Metric3D v2 [ 93 ]
0.172
0.837
5.012
0.254
0.735
0.931
0.972
UniK3D † [ 20 ]
0.093
0.426
2.930
0.159
0.924
0.976
0.988
DAC † [ 19 ]
0.193
1.042
5.097
0.283
0.689
0.903
0.963
UniDAC † [ 21 ]
0.186
0.990
5.128
0.275
0.698
0.908
0.966
Pretrained DepthNet
0.179
0.808
4.588
0.251
0.703
0.922
0.977
Ours
0.122
0.582
3.613
0.183
0.858
0.959
0.984
TABLE III: Quantitative comparison with SoTA foundation models tailored for metric depth estimation. Depth estimates are scaled by aligning their median values before evaluation. The best results are shown in bold font, and the second-best results are underlined. The symbol † denotes methods supporting different camera models.
Fig. 9: Qualitative comparison of monocular fisheye depth estimation on the TartanAir V2 dataset: (i) estimated depth maps; (ii) corresponding error maps. White and yellow dashed contours highlight regions with substantial improvements.
Fig. 10: Qualitative comparison of monocular fisheye depth estimation on the MIAS-FisheyeVO dataset: (i) estimated depth maps; (ii) corresponding error maps. White and yellow dashed contours highlight regions with substantial improvements.
Fig. 11: Comparison of estimated camera trajectories on the KITTI-360, NIO-Fisheye, TUM-VI, TartanAir V2, and MIAS-FisheyeVO datasets.
Fig. 12: Trade-off between depth estimation accuracy and model complexity among representative monocular fisheye depth estimation methods.
KITTI-360
Type
Method
et (%)
er ( ∘ /100 m)
Geometry-based
OV 2 SLAM [ 57 ]
57.691
6.865
Ours
4.451
0.288
Learning-based
Lite-Mono [ 89 ]
13.908
3.169
Monodepth2 [ 25 ]
14.015
2.448
SC-DepthV3 [ 90 ]
18.366
5.554
TABLE IV: Quantitative comparison of monocular visual odometry in terms of translational and rotational drift on the KITTI-360 and NIO-Fisheye datasets.
Type
Method
Seq. 00
Seq. 02
Seq. 03
Seq. 04
Seq. 05
Seq. 06
Seq. 07
Seq. 09
Seq. 10
Average
ATE (m) ↓
Geometry-based
ORB-SLAM3 [ 56 ]
×
×
×
×
×
×
×
×
×
×
OpenVSLAM [ 54 ]
×
×
×
×
×
×
×
×
×
×
CubemapSLAM [ 5 ]
308.178
354.433
×
869.561
287.203
×
×
×
×
454.844
OV 2 SLAM [ 57 ]
242.963
289.912
140.861
661.126
209.020
244.015
645.683
213.020
372.146
335.416
SVO [ 3 ]
336.580
×
215.743
×
×
300.569
885.366
240.658
736.450
452.561
TABLE V: Quantitative comparison of monocular visual odometry on the KITTI-360 dataset. The best results among fully tracked runs are shown in bold font. The symbol × indicates tracking failures, while † indicates partially tracked results with lower errors than the best fully tracked results. Errors for partially tracked runs are computed only over the successfully tracked trajectory segments.
Type
Method
Seq. 00
Seq. 01
Seq. 02
Seq. 03
Seq. 04
Average
ATE (m) ↓
Geometry-based
ORB-SLAM3 [ 56 ]
×
936.253
×
×
×
936.253
OpenVSLAM [ 54 ]
×
×
20.570
×
×
20.570
CubemapSLAM [ 5 ]
×
1352.358
222.228
369.410
×
647.999
OV 2 SLAM [ 57 ]
39.003
232.083
224.715
207.867
2182.091
577.152
SVO [ 3 ]
225.474
1275.617
215.834
306.793
2072.314
819.206
TABLE VI: Quantitative comparison of monocular visual odometry on the NIO-Fisheye dataset. The best results among fully tracked runs are shown in bold font. The symbol × indicates tracking failures, while † indicates partially tracked results with lower errors than the best fully tracked results. Errors for partially tracked runs are computed only over the successfully tracked trajectory segments.
Method
Room 1
Room 2
Room 3
Room 4
Room 5
Room 6
Average
ATE (m) ↓
ORB-SLAM3 [ 56 ]
0.013
0.014
0.157
0.010
0.104
0.010
0.051
OpenVSLAM [ 54 ]
0.013
0.014
0.014
0.014
0.014
0.017
0.014
CubemapSLAM [ 5 ]
0.042
0.007
0.009
0.006
0.174
0.004
0.040
OV 2 SLAM [ 57 ]
1.003
0.875
1.307
0.988
0.868
0.216
0.876
SVO [ 3 ]
0.912
1.235
1.279
0.987
0.869
0.892
1.029
TABLE VII: Quantitative comparison of monocular visual odometry on the TUM-VI dataset. The best results among fully tracked runs are shown in bold font. The symbol × indicates tracking failures, while † indicates partially tracked results with lower errors than the best fully tracked results. Errors for partially tracked runs are computed only over the successfully tracked trajectory segments.
Type
Method
House Subset
RetroOffice Subset
H0
H1
H2
H3
H4
H5
H6
H7
Average
R0
R1
R2
R3
R4
R5
Average
ATE (m) ↓
Geometry-based
ORB-SLAM3 [ 56 ]
×
×
1.653
×
×
×
0.012
0.185 †
0.617
0.026
0.006 †
×
×
0.011
×
0.014
OpenVSLAM [ 54 ]
×
×
×
×
×
×
0.055
×
0.055
0.035
0.016
0.012
0.013
0.016
0.062
0.026
CubemapSLAM [ 5 ]
×
0.017 †
0.058
×
0.027 †
0.026
0.012
×
0.028
0.006
0.003 †
0.005 †
0.004
0.004
0.005
0.005
OV 2 SLAM [ 57 ]
2.565
2.757
2.884
1.810
3.895
2.139
1.677
2.846
2.572
0.616
0.048
0.023
0.020
0.012
0.115
0.139
TABLE VIII: Quantitative comparison of monocular visual odometry on the House and RetroOffice scenarios of the TartanAir V2 dataset. Sequences P000–P007 and P000–P005 are denoted as H0–H7 and R0–R5 in the House and RetroOffice scenarios, respectively. The best results among fully tracked runs are shown in bold font. The symbol × indicates tracking failures, while † indicates partially tracked results with lower errors than the best fully tracked results. Errors for partially tracked runs are computed only over the successfully tracked trajectory segments.
Type
Method
S0
S1
S2
S3
S4
S5
S6
S7
Average
ATE (m) ↓
Geometry-based
ORB-SLAM3 [ 56 ]
×
0.033 †
×
×
×
×
×
×
0.033
OpenVSLAM [ 54 ]
×
0.248
0.601
0.409
1.114
×
0.916
0.706
0.666
CubemapSLAM [ 5 ]
0.017
0.036
0.013
0.031
0.022 †
0.031 †
0.039 †
0.023
0.026
OV 2 SLAM [ 57 ]
0.172
0.209
0.807
0.112
0.226
2.228
4.968
2.005
1.341
SVO [ 3 ]
1.992
0.154
8.231
9.688
9.337
6.277
7.192
7.047
6.240
TABLE IX: Quantitative comparison of monocular visual odometry on the SeasideTown scenario of the TartanAir V2 dataset. Sequences P000–P007 are denoted as S0–S7, respectively. The best results among fully tracked runs are shown in bold font. The symbol × indicates tracking failures, while † indicates partially tracked results with lower errors than the best fully tracked results. Errors for partially tracked runs are computed only over the successfully tracked trajectory segments.
Type
Method
Indoor Scenario
Outdoor Scenario
Seq. 00
Seq. 01
Seq. 02
Seq. 03
Seq. 04
Seq. 05
Average
Seq. 00
Seq. 01
Seq. 02
Seq. 03
Seq. 04
Average
ATE (m) ↓
Geometry-based
ORB-SLAM3 [ 56 ]
0.037
0.084
0.271
0.135
0.029
1.186
0.290
45.997
×
16.386
4.141
0.136
16.665
OpenVSLAM [ 54 ]
0.076
×
×
0.294
0.142
0.029
0.135
×
0.054 †
×
0.370
0.125
0.183
CubemapSLAM [ 5 ]
0.223
×
×
×
0.006
0.002 †
0.077
×
0.729
×
0.032 †
0.013 †
0.258
OV 2 SLAM [ 57 ]
0.077
0.888
1.021
0.093
0.944
0.046
0.511
43.257
1.693
14.996
0.299
0.142
12.077
TABLE X: Quantitative comparison of monocular visual odometry on the MIAS-FisheyeVO dataset. The best results among fully tracked runs are shown in bold font. The symbol × indicates tracking failures, while † indicates partially tracked results with lower errors than the best fully tracked results. Errors for partially tracked runs are computed only over the successfully tracked trajectory segments.
Fig. 13: Comparison of image representations adopted by different monocular fisheye visual odometry systems: (a) original fisheye image; (b) undistorted perspective image; (c) unfolded cubemap image; (d) ERP image.
Module
Main Component
Time (ms)
Data Preprocessing
ERP Remapping
1.5±0.4
Depth Estimation
DepthNet
8.5±6.9
Camera Tracking
SuperPoint
17.8±1.6
SphereGlue
38.9±5.4
Pose Estimation
47.9±5.3
Local Map Tracking
0.01±0.05
TABLE XI: Runtime of the main components of S 2 D-FVO.
Configuration
Abs Rel ↓
Sq Rel ↓
RMSE ↓
RMSE log ↓
δ<1.25↑
δ<1.252↑
δ<1.253↑
(a) Baseline
0.212
1.186
4.616
0.307
0.656
0.896
0.957
(b) Pretrained DepthNet
0.179
0.808
4.588
0.251
0.703
0.922
0.977
(c) Pretrained DepthNet + PoseNet
0.190
0.904
4.110
0.272
0.696
0.917
0.968
(d) Pretrained DepthNet + BS
0.135
0.610
3.708
0.204
0.831
0.953
0.982
(e) Pretrained DepthNet + BS + SBA
0.122
0.582
3.613
0.183
0.858
0.959
0.984
TABLE XII: Ablation study of different pose estimation strategies used for view synthesis during unsupervised monocular fisheye depth learning on the KITTI-360 dataset. “BS” denotes the birotation solution, while “SBA” denotes the spherical bundle adjustment.
Fig. 14: Qualitative results of the monocular fisheye depth learning ablation study on the KITTI-360 dataset: (i) estimated depth maps; (ii) corresponding error maps. (a)–(e) correspond to the ablation study configurations. White and yellow dashed contours highlight regions with substantial improvements.
Configuration
Seq. 03
Seq. 05
ATE (m) ↓
RPE Trans (m) ↓
et (%) ↓
ATE (m) ↓
RPE Trans (m) ↓
et (%) ↓
(a) No Scale Reference
6.108
1.233
2.420
55.019
0.221
5.866
(b) Scale Reference for Pose Initialization
4.978
0.087
1.907
44.176
0.194
7.174
(c) Scale Regularization Using Pointwise Depth
10.066
0.145
4.024
28.579
0.074
3.355
(d) Scale Regularization Using Median Depth (Ours)
3.052
0.031
1.183
23.577
0.094
4.593
TABLE XIII: Ablation study on different scale regularization strategies on the KITTI-360 dataset.
Fig. 15: Ablation study on different scale regularization strategies for monocular fisheye visual odometry on the KITTI-360 dataset.
Fig. 16: Qualitative comparison of monocular depth estimation on perspective video sequences from the TartanAir V2 dataset: (i) estimated depth maps; (ii) corresponding error maps. White and yellow dashed contours highlight regions with substantial improvements in depth estimation accuracy.
Fig. 17: Qualitative comparison of monocular depth estimation on equirectangular video sequences from the TartanAir V2 dataset: (i) estimated depth maps; (ii) corresponding error maps. White and yellow dashed contours highlight regions with substantial improvements in depth estimation accuracy.
Fig. 18: Qualitative comparison of monocular visual odometry on perspective and equirectangular video sequences from the TartanAir V2 dataset.
Camera Model
Method
House
RetroOffice
SeasideTown
H0
H5
H6
R0
R2
R4
S0
S1
S3
Perspective
OpenVSLAM [ 54 ]
×
×
0.015
0.121
0.007
0.013
3.474
0.059
0.104
Ours
2.978
1.660
1.249
0.364
0.077
0.009
2.856
5.447
2.691
Equirectangular
OpenVSLAM [ 54 ]
0.010
0.008
0.005
0.005
0.003
0.002
0.048
0.035
0.023
Ours
0.047
0.015
0.016
0.003
0.003
0.002
0.017
0.023
0.017
TABLE XIV: Quantitative comparison of monocular visual odometry on perspective and equirectangular video sequences from the TartanAir V2 dataset. Results are reported in terms of ATE (m), with lower values indicating better performance. The best results are shown in bold font.
Camera Model
Method
Abs Rel ↓
RMSE (m) ↓
δ1↑
Indoor
Perspective
Monodepth2 [ 25 ]
0.291
0.673
0.536
UniDAC [ 21 ]
0.155
0.412
0.791
Pretrained DepthNet
0.290
0.664
0.526
Ours
0.230
0.688
0.536
Equirectangular
Monodepth2 [ 25 ]
0.222
0.660
0.653
TABLE XV: Quantitative comparison of monocular depth estimation on perspective and equirectangular video sequences from the indoor and outdoor scenarios of the TartanAir V2 dataset. The best results are shown in bold font.
We propose a method to extend foundational monocular depth estimators (FMDEs), trained on perspective images, to fisheye images. Despite being trained on tens of millions of images, FMDEs are susceptible to the covariate shift introduced by changes in camera calibration (intrinsic, distortion) parameters, leading to erroneous depth estimates. Our method aligns the distribution of latent embeddings encoding fisheye images to those of perspective images, enabling the reuse of FMDEs for fisheye cameras without retraining or finetuning. To this end, we introduce a set of Calibration Tokens as a light-weight adaptation mechanism that modulates the latent embeddings for alignment. By exploiting the already expressive latent space of FMDEs, we posit that modulating their embeddings avoids the negative impact of artifacts and loss introduced in conventional recalibration or map projection to a canonical reference frame in the image space. Our method is self-supervised and does not require fisheye images but leverages publicly available large-scale perspective image datasets. This is done by recalibrating perspective images to fisheye images, and enforcing consistency between their estimates during training. We evaluate our approach with several FMDEs, on both indoors and outdoors, where we consistently improve over state-of-the-art methods using a single set of tokens for both. Code available at: https://github.com/JungHeeKim29/calibration-token; https://github.com/Suchisrit/CalibrationTokens.
Deep learning-based visual odometry (VO) has achieved significant progress, yet most existing methods focus on a monocular approach, which suffers from scale ambiguity. Stereo VO provides real metric by its nature, but remains less studied in deep learning VO due to its high computational cost and modeling complexity. Recent advances in stereo matching and optical flow estimation have made dense visual correspondence increasingly accurate and reliable, but their complementary geometric information has not been fully exploited for VO. In this paper, we present SFVO, a correspondence-driven stereo VO framework that directly builds upon pretrained stereo matching and optical flow models. SFVO exploits pretrained stereo matching and optical flow models to estimate stereo and temporal correspondences. Instead of learning pose directly from images, SFVO maps learned correspondences into geometric constraints and predicts which points are trustworthy. To improve the reliability of visual correspondence-based geometric constraints, we introduce decoupled confidence maps for rotation and translation. This design better aligns the characteristics of visual correspondence and 6-DoF transformations. Extensive experiments on outdoor and indoor datasets demonstrate that SFVO achieves robust and accurate pose estimation with strong generalization capability. The code will be released.
Kai Zhang, Guoyang Zhao, Jun Ma
Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · Cheng Kar-Shun Robotics Institute, The Hong Kong University of Science and Technology, Hong Kong SAR, China
Fisheye cameras are increasingly adopted in robotics for near-field manipulation, navigation, and immersive perception, yet indoor depth benchmarks with accurate ground truth are still missing. To address this, we introduce WideDepth - the first indoor dataset for fisheye depth estimation, featuring 101 scenes containing 5K high-resolution stereo pairs labeled with millimeter-level ground truth depth and disparity. Our dataset also includes paired pinhole and fisheye samples across varying fields of view and baselines in both horizontal and vertical stereo setups. We further propose a method to adapt pinhole-trained stereo models to fisheye images and introduce a novel stereo fisheye image generation pipeline based on high-resolution LiDAR scans. Leveraging these methods, we thoroughly evaluate state-of-the-art monocular depth, stereo matching, and depth completion models on our benchmark. Additionally, we provide 18K LiDAR-derived sparse depth training samples, achieving up to a 62% performance boost on fisheye data when fine-tuning pinhole-based stereo models. In summary, the high precision and versatility of our benchmark set a strong foundation for advancing research in fisheye depth estimation and robotics perception. Project page: https://ilyaind.github.io/WideDepth
Ilia Indyk, Ignat Penshin, Ivan Sosin +3
Robotics Center, Moscow, Russia · AXXX, Moscow, Russia · Trusted AI Research Center, RAS, Moscow, Russia