Reliable omnidirectional depth estimation from multi-fisheye stereo matching is pivotal to many applications, such as embodied robotics. Existing approaches either rely on spherical sweeping with heuristic fusion strategies to build the cost columns or perform reference-centric stereo matching based on rectified views. However, these methods fail to explicitly exploit geometric relationships between multiple views, rendering them less capable of capturing the global dependencies, visibility, or scale changes. In this paper, we shift to a new perspective and propose a novel reference-free framework, dubbed FreeOmniMVS, via multi-view consistency maximization. The highlight of FreeOmniMVS is that it can aggregate pair-wise correlations into a robust, visibility-aware, and global consensus. As such, it is tolerant to occlusions, partial overlaps, and varying baselines. Specifically, to achieve global coherence, we introduce a novel View-pair Correlation Transformer (VCT) that explicitly models pairwise correlation volumes across all camera view pairs, allowing us to drop unreliable pairs caused by occlusion or out-of-focus observations. To realize scalable and visibility-aware consensus, we propose a lightweight attention mechanism that adaptively fuses the correlation vectors, eliminating the need for a designated reference view and allowing all cameras to contribute equally to the stereo matching process. Extensive experiments on diverse benchmark datasets demonstrate the superiority of our method for globally consistent, visibility-aware, and scale-aware omnidirectional depth estimation.
Figures & tables
Fig. 1: (a,c) Fisheye inputs with randomly injected local blur and noise, simulating out-of-focus or corrupted observations. (b,d) Depth predictions from our backbone (top) versus FreeOmniMVS (bottom). Our method suppresses artifacts around degraded regions and preserves thin structures and cross-camera consistency across the full 360∘ field of view.
Fig. 2: Overall architecture of FreeOmniMVS. Given four fisheye images, Stage 1 extracts multi-scale unary features and performs OmniMVS-style spherical sweeping to build per-view feature volumes on the ERP sphere. Stage 2 applies the proposed VCT to construct pairwise correlation volumes over all camera combinations and aggregates them, via sparse Top- k attention, into a visibility-aware consistency volume Cfused . Stage 3 uses a lightweight context fuser to build a global context volume and feeds both context and consistency features into a RAFT-Stereo-style recurrent updater, which starts from a zero inverse-depth map and iteratively refines it, followed by convex upsampling to obtain the final high-resolution omnidirectional depth.
Fig. 3: Overview of the proposed VCT. For each voxel (d,x) , VCT takes the vector of pairwise correlations across all camera pairs, builds a self-similarity matrix over view pairs, and applies attention with Top- k sparsification, with Gumbel perturbation during training, to aggregate them into a fused scalar consistency score.
Dataset
OmniThings
OmniHouse
Sunny
Metric
>1
>3
>5
MAE
RMSE
>1
>3
>5
MAE
RMSE
>1
>3
>5
MAE
RMSE
Trained on OmniThings only
OmniMVS [ 9 ]
47.72
15.12
8.91
2.40
5.27
30.53
10.29
6.27
1.72
4.05
27.16
6.13
3.98
1.24
3.09
S-OmniMVS [ 16 ]
28.03
10.40
6.33
1.48
3.68
18.86
8.05
4.90
1.06
2.41
17.19
6.03
3.89
1.11
3.60
OmniMVS 32+ [ 14 ]
20.70
8.18
5.49
1.37
4.11
19.89
5.89
3.99
1.30
2.64
13.57
4.81
3.10
0.88
2.56
RomniStereo 32 [ 17 ]
20.42
8.49
5.81
1.39
4.22
12.13
4.73
3.02
0.80
1.85
12.28
5.59
3.79
0.80
2.68
TABLE I: Quantitative comparison with existing omnidirectional depth estimation methods on OmniThings, OmniHouse, and Sunny. We report the percentage of pixels whose inverse-depth index error exceeds 1, 3, and 5 bins ( >1 , >3 , >5 ), as well as MAE and RMSE. All models are either trained on OmniThings only (middle block) or additionally fine-tuned on OmniHouse and Sunny (bottom block). The best results are shown in bold , and the second best are underlined .
Fig. 4: Qualitative comparison on clean test images from OmniThings, OmniHouse, and Sunny.
Dataset (Occ)
OmniHouse
Sunny
RomniStereo 32 -ft [ 17 ]
2.90
2.01
FreeOmniMVS-ft
1.71
1.61
TABLE II: RMSE on occlusion-augmented test sets (lower is better). FreeOmniMVS-ft significantly reduces the error compared to RomniStereo 32 -ft, especially on OmniHouse-Occ, where RMSE drops from 2.90 to 1.71.
Fig. 5: Corner cases on occlusion-augmented OmniHouse and Sunny. (a,c) Fisheye inputs with strong local blur or noise. (b,d) Depth predictions from RomniStereo 32 -ft (top) and FreeOmniMVS-ft (bottom). Our method better preserves thin structures and reduces large local errors around degraded regions.
Dataset
OH
Sn
OH (Occ)
Sn (Occ)
Metric
>1
MAE
>1
MAE
RMSE
RMSE
top-k ( k=1 )
13.14
1.04
16.19
1.11
2.13
3.04
top-k ( k=2 )
13.47
0.97
14.21
1.02
2.01
2.80
w/o top-k
12.74
0.94
13.52
0.89
1.95
2.74
w/o VCT
12.17
0.79
12.33
0.80
2.85
3.98
w/o context fuser
12.39
0.93
12.98
0.84
2.04
2.83
TABLE III: Ablation study on OmniHouse, Sunny, and their occlusion-augmented versions.
Department of Computer Science and Artificial Intelligence, Dongguk University, Seoul 04620, Republic of Korea · 42dot, Seongnam-si, Gyeonggi-do 13449, Republic of Korea