Reference-Free Omnidirectional Stereo Matching via Multi-View Consistency Maximization

Lehuai Xu; Weiming Zhang; Yang Li; Sidan Du; Lin Wang

cs.CVMar 16, 2026

Reference-Free Omnidirectional Stereo Matching via Multi-View Consistency Maximization

Authors: Lehuai Xu, Weiming Zhang, Yang Li, Sidan Du, Lin Wang

Organizations: Nanjing University, Nanjing, China · Nanyang Technological University, Singapore · HKUST(GZ), Guangzhou, China

Abstract

Reliable omnidirectional depth estimation from multi-fisheye stereo matching is pivotal to many applications, such as embodied robotics. Existing approaches either rely on spherical sweeping with heuristic fusion strategies to build the cost columns or perform reference-centric stereo matching based on rectified views. However, these methods fail to explicitly exploit geometric relationships between multiple views, rendering them less capable of capturing the global dependencies, visibility, or scale changes. In this paper, we shift to a new perspective and propose a novel reference-free framework, dubbed FreeOmniMVS, via multi-view consistency maximization. The highlight of FreeOmniMVS is that it can aggregate pair-wise correlations into a robust, visibility-aware, and global consensus. As such, it is tolerant to occlusions, partial overlaps, and varying baselines. Specifically, to achieve global coherence, we introduce a novel View-pair Correlation Transformer (VCT) that explicitly models pairwise correlation volumes across all camera view pairs, allowing us to drop unreliable pairs caused by occlusion or out-of-focus observations. To realize scalable and visibility-aware consensus, we propose a lightweight attention mechanism that adaptively fuses the correlation vectors, eliminating the need for a designated reference view and allowing all cameras to contribute equally to the stereo matching process. Extensive experiments on diverse benchmark datasets demonstrate the superiority of our method for globally consistent, visibility-aware, and scale-aware omnidirectional depth estimation.

Figures & tables

Panel of Figure 1 — Fig. 1: (a,c) Fisheye inputs with randomly injected local blur and noise, simulating out-of-focus or corrupted observations. (b,d) Depth predictions from our backbone (top) versus FreeOmniMVS (bottom). Our method suppresses artifacts around degraded regions and preserves thin structures and cross-camera consistency across the full $360^{\circ}$ field of view.

Fig. 2: Overall architecture of FreeOmniMVS. Given four fisheye images, Stage 1 extracts multi-scale unary features and performs OmniMVS-style spherical sweeping to build per-view feature volumes on the ERP sphere. Stage 2 applies the proposed VCT to construct pairwise correlation volumes over all camera combinations and aggregates them, via sparse Top- $k$ attention, into a visibility-aware consistency volume $\mathcal{C}_{\text{fused}}$ . Stage 3 uses a lightweight context fuser to build a global context volume and feeds both context and consistency features into a RAFT-Stereo-style recurrent updater, which starts from a zero inverse-depth map and iteratively refines it, followed by convex upsampling to obtain the final high-resolution omnidirectional depth.

Fig. 3: Overview of the proposed VCT. For each voxel $(d,\mathbf{x})$ , VCT takes the vector of pairwise correlations across all camera pairs, builds a self-similarity matrix over view pairs, and applies attention with Top- $k$ sparsification, with Gumbel perturbation during training, to aggregate them into a fused scalar consistency score.

Trained on OmniThings only
Dataset	OmniThings					OmniHouse					Sunny
Metric	$>1$	$>3$	$>5$	MAE	RMSE	$>1$	$>3$	$>5$	MAE	RMSE	$>1$	$>3$	$>5$	MAE	RMSE
OmniMVS [ 9 ]	47.72	15.12	8.91	2.40	5.27	30.53	10.29	6.27	1.72	4.05	27.16	6.13	3.98	1.24	3.09
S-OmniMVS [ 16 ]	28.03	10.40	6.33	1.48	3.68	18.86	8.05	4.90	1.06	2.41	17.19	6.03	3.89	1.11	3.60
OmniMVS ${}^{+}_{32}$ [ 14 ]	20.70	8.18	5.49	1.37	4.11	19.89	5.89	3.99	1.30	2.64	13.57	4.81	3.10	0.88	2.56
RomniStereo 32 [ 17 ]	20.42	8.49	5.81	1.39	4.22	12.13	4.73	3.02	0.80	1.85	12.28	5.59	3.79	0.80	2.68

TABLE I: Quantitative comparison with existing omnidirectional depth estimation methods on OmniThings, OmniHouse, and Sunny. We report the percentage of pixels whose inverse-depth index error exceeds 1, 3, and 5 bins (

>1

>3

>5

), as well as MAE and RMSE. All models are either trained on OmniThings only (middle block) or additionally fine-tuned on OmniHouse and Sunny (bottom block). The best results are shown in bold , and the second best are underlined .

Panel of Figure 5 — Fig. 4: Qualitative comparison on clean test images from OmniThings, OmniHouse, and Sunny.

Dataset (Occ)	OmniHouse	Sunny
RomniStereo 32 -ft [ 17 ]	2.90	2.01
FreeOmniMVS-ft	1.71	1.61

TABLE II: RMSE on occlusion-augmented test sets (lower is better). FreeOmniMVS-ft significantly reduces the error compared to RomniStereo 32 -ft, especially on OmniHouse-Occ, where RMSE drops from 2.90 to 1.71.

Panel of Figure 7 — Fig. 5: Corner cases on occlusion-augmented OmniHouse and Sunny. (a,c) Fisheye inputs with strong local blur or noise. (b,d) Depth predictions from RomniStereo 32 -ft (top) and FreeOmniMVS-ft (bottom). Our method better preserves thin structures and reduces large local errors around degraded regions.

Dataset	OH		Sn		OH (Occ)	Sn (Occ)
Metric	$>1$	MAE	$>1$	MAE	RMSE	RMSE
top-k ( $k=1$ )	13.14	1.04	16.19	1.11	2.13	3.04
top-k ( $k=2$ )	13.47	0.97	14.21	1.02	2.01	2.80
w/o top-k	12.74	0.94	13.52	0.89	1.95	2.74
w/o VCT	12.17	0.79	12.33	0.80	2.85	3.98
w/o context fuser	12.39	0.93	12.98	0.84	2.04	2.83

TABLE III: Ablation study on OmniHouse, Sunny, and their occlusion-augmented versions.

Panoramic Depth Estimation

See all 12 papers

Jul 3, 2026

OmniDS: Dual-Stream Context Fusion for Omnidirectional Depth from Fisheye Cameras

Omnidirectional depth estimation from multi-fisheye camera rigs is complicated by visibility conflicts: wide baselines cause different cameras to observe different portions, or even different faces, of the same object, so aggregating their features into a unified equirectangular (ERP) representation under fixed projection produces ambiguous matching evidence near occlusion boundaries and thin structures. Although existing methods mitigate this by down-weighting unreliable views, they do not resolve the underlying discrepancy because context formation and cross-view fusion remain tied to rigid fisheye-to-ERP sampling. We present OmniDS, an iterative depth refinement framework that replaces rigid aggregation by combining dynamic context fusion with consensus-aware multi-view similarity. A dual-stream encoder pairs a lightweight CNN for geometric detail with a frozen DINOv3 for semantic priors; their features are reprojected into ERP space at each refinement step via learned view weighting and deformable cross-attention with geometric distortion bias. In parallel, a multi-view consensus volume captures global cross-camera agreement through group-wise correlation and feature variance, regularized by a 3D U-Net. For efficient deployment, we distill the dual-stream representation into a single MobileNet-based encoder. OmniDS achieves state-of-the-art performance on the OmniThings, OmniHouse, and Sunny benchmarks while maintaining competitive inference speed. Project page and codes are available at https://parkchaesong.github.io/omnids.

Chaesong Park, Jihyeon Hwang, Muyeol Sung +1

Seoul National University, Seoul, Republic of Korea

Multi-View Depth Estimation Multi-View Feature Fusion Depth Estimation Feature Fusion Panoramic Depth Estimation

Ask PDF

Aug 21, 2026

M2Depth: Unifying Monocular Depth Foundation Priors with Multi-View Stereo

Deep learning-based Multi-View Stereo (MVS) has advanced significantly but often generalizes poorly to unseen scenes, particularly in occluded areas or regions with limited view overlap. To mitigate this, recent approaches integrate Depth Foundation Models (DFMs) into MVS pipelines to provide monocular depth priors. However, existing methods typically rely on a static, one-way fusion scheme, which fails to fully exploit the complementary strengths of both modalities. We propose a novel framework that overcomes this limitation by tightly coupling a DFM with a cascade MVS pipeline through a bidirectional mutual refinement strategy. Our method leverages MVS depth to resolve the scale ambiguity in monocular predictions, while the monocular depth, in turn, enhances the structural completeness and fine-grained detail of the MVS estimate. Furthermore, we introduce a prior-guided cost volume refinement mechanism that effectively integrates multi-view and monocular information via attention-based fusion and discretized depth bins, thereby promoting local geometric consistency. Extensive experiments demonstrate that our method outperforms state-of-the-art MVS approaches on standard benchmarks, producing more complete and generalizable depth maps with sharp boundaries. Furthermore, although not explicitly designed for sparse-view settings, our framework generalizes remarkably well, competing favorably with even dedicated sparse-view methods while maintaining a superior accuracy-efficiency trade-off.

Byeonggwon Lee, Sanggi Lee, Siwoo Lee +2

Department of Computer Science and Artificial Intelligence, Dongguk University, Seoul 04620, Republic of Korea · 42dot, Seongnam-si, Gyeonggi-do 13449, Republic of Korea

Multi-View Depth Estimation Multi-View 3D Reconstruction Multi-View Consistency

Ask PDF

May 14, 2026

H-OmniStereo: Zero-Shot Omnidirectional Stereo Matching with Heading-Aligned Normal Priors

Stereo matching on top-bottom equirectangular images provides an effective framework for full-surround perception, as vertically aligned epipolar lines enable the use of advanced perspective stereo architectures that are largely driven by large-scale datasets and monocular priors. However, the performance of such adaptations is severely limited by the scarcity of omnidirectional stereo datasets and the degradation of perspective monocular priors under spherical distortions. To address these challenges, we propose H-OmniStereo, a zero-shot omnidirectional stereo matching framework. First, we construct high-quality synthetic dataset comprising over 2.8 million top-bottom equirectangular stereo pairs to scale up training. Second, we introduce an equirectangular monocular normal estimator, specifically operating in a heading-aligned coordinate system. Beyond providing distortion-robust and cross-view-consistent geometric priors for establishing reliable correspondences in stereo matching, this design boosts training efficiency and accommodates train-test FoV mismatches. Extensive experiments show that our approach achieves higher accuracy than existing methods on out-of-domain datasets and successfully generalizes to real-world consumer camera setups using a single model. The model and dataset will be released at https://github.com/JIANG-CX/H-OmniStereo.

Chenxing Jiang, Zhe Tong, Pusen Gao +5

Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China

Stereo Matching

Ask PDF

Abstract

Panoramic Depth Estimation

Explore similar work

OmniDS: Dual-Stream Context Fusion for Omnidirectional Depth from Fisheye Cameras

M2Depth: Unifying Monocular Depth Foundation Priors with Multi-View Stereo

H-OmniStereo: Zero-Shot Omnidirectional Stereo Matching with Heading-Aligned Normal Priors