From Perspective to Fisheye Depth Estimation and Open-Vocabulary Segmentation

Rit Gangopadhyay; Alex Wong

cs.CVAug 28, 2026

From Perspective to Fisheye Depth Estimation and Open-Vocabulary Segmentation

Authors: Rit Gangopadhyay, Alex Wong

Organizations: Yale Vision Laboratory, Yale University, New Haven, CT 06520, USA

Abstract

Vision foundation models are capable of generalizing across 3-dimensional (3D) scenes with high-fidelity estimates; their empirical success can be attributed to training on large-scale datasets of perspective images. However, when transferred to wide field-of-view (FoV) images, such as those captured by fisheye cameras, they return erroneous outputs due to a covariate shift stemming from the radial distortion on the image pixels. We propose a method to generalize vision foundation models to fisheye cameras. The crux of our method lies in a set of learnable parameters, termed Distortion Extenders (DEX), that model the fisheye distortion coefficients and the distributional shift between fisheye and perspective images encoded in the latent space. By minimizing a self-supervised alignment loss, DEX transforms the latent embeddings of fisheye images to resemble those of perspective images to recover high-fidelity estimates. DEX is architecture- and task-agnostic: We demonstrate DEX on monocular depth estimation and open-vocabulary segmentation for convolution- and Transformer-based architectures, where we consistently improve over baselines across indoor and outdoor fisheye datasets. As a byproduct, the activations of DEX can also be decoded to distortion coefficients to support camera calibration. Code available at: https://github.com/Suchisrit/DEX.

Figures & tables

Figure 1: 3D reconstructions and open-vocabulary segmentation of fisheye imagery produced by foundational models, UniDepthV2 [ 42 ] and LSeg [ 27 ] . The distortion in the input leads to erroneous and distorted reconstructions and segmentations.

Figure 2: Distortion Extenders. The figure illustrates the DEX training pipeline and architecture. During training, we pass perspective inputs and fisheye inputs augmented by DEX into a frozen backbone model. The fisheye features select convex combinations of Extenders for translation in the latent space. We optimize the Distortion Extenders to minimize the difference between the fisheye and perspective output.

Figure 3: MDE Qualitative Results. We show 3D reconstructions of indoor and outdoor samples from the evaluation in Tab. 1 . Previous methods struggle with the outermost regions in the fisheye images, which have the most severe distortion. DEX is more robust to these distortions. In the outdoor scenes with the highest distortions, we are still able to accurately reconstruct many of the points at the edges of the images.

Figure 4: Segmentation Qualitative Results. We evaluate LSeg and SED with and without DEX on the WoodScape outdoor dataset. We see improvements in the open-vocabulary segmentation, highlighting DEX’s generalizability to this recognition-based task. We evaluate on the valid regions of the images specified by the ground truth.

Testset	Experiment	Model	Train Dataset	RMSE $\downarrow$	$\delta_{1}$ $\uparrow$
ScanNet++ [ 78 ]	Base Model	MiDaS [ 46 ]	Mix 1.4M	0.672	0.479
	w/ LoRA [ 21 ]	MiDaS [ 46 ]	Mix 200K	0.561	0.599
	w/ Calibration Tokens [ 11 ]	MiDaS [ 46 ]	Mix 200K	0.413	0.688
	w/ Distortion Extenders	MiDaS [ 46 ]	Mix 200K	0.397	0.702
	Base Model	DepthAnything [ 75 ]	Mix 63.5M	0.705	0.454
	w/ LoRA [ 21 ]	DepthAnything [ 75 ]	Mix 200K	0.706	0.445

Table 1: MDE Performance on Indoor and Outdoor. After training a set of Distortion Extenders, we test their zero-shot performance on ScanNet++ and KITTI-360, which contain real fisheye images with varying scenes and distortion parameters.

Testset	Experiment	Model	Train Dataset	mIoU $\uparrow$	weighted IoU $\uparrow$
WoodScape [ 80 ]	Baseline	LSeg [ 27 ]	Mix 500K	0.305	0.819
	w/ Calibration Tokens	LSeg [ 27 ]	Mix 50K	0.321	0.823
	w/ Distortion Extenders	LSeg [ 27 ]	Mix 50K	0.362	0.838
	Baseline	SED [ 71 ]	Mix 120K	0.439	0.829
	w/ Distortion Extenders	SED [ 71 ]	Mix 50K	0.449	0.832

Table 2: Open Vocabulary Segmentation. We evaluate the zero-shot performance of our method on the WoodScape dataset, which contains real fisheye images with varying scenes and distortion parameters.

Method	Depth		Segmentation
Method	RMSE $\downarrow$	$\delta_{1}\uparrow$	mIoU $\uparrow$	weighted IoU $\uparrow$
PanopticDepth [ 12 ]	7.273	0.267	0.160	0.300
w/ Distortion Extenders	3.583	0.354	0.282	0.833

Table 3: DEX with PanopticDepth. We apply DEX to PanopticDepth and evaluate both depth estimation and segmentation on WoodScape.

Panel of Figure 5 — Figure 5: t-SNE Plot of Fisheye and Perspective Image Features. When using DEX, fisheye image features align much closer with corresponding perspective features.

ScanNet++	$k_{1}$	$k_{2}$	$k_{3}$	$k_{4}$	Average
MSE $\downarrow$	$2.04{\times}10^{-6}$	$1.53{\times}10^{-6}$	$3.53{\times}10^{-7}$	$8.87{\times}10^{-9}$	$9.81{\times}10^{-7}$
MAE $\downarrow$	$1.17{\times}10^{-3}$	$9.88{\times}10^{-4}$	$5.27{\times}10^{-4}$	$7.05{\times}10^{-5}$	$6.88{\times}10^{-4}$
MAPE (%) $\downarrow$	3.91	14.30	41.86	78.61	34.67

Table 6: KB Distortion Parameter Decoding. Distortion Extenders are able to recover KB distortion parameters for an image input.

Figure 6: Error in KB Distortion Prediction Visualized. We provide an example for the visual difference in distortion when we generate a fisheye image with a set of distortion parameters representative of ScanNet++, and when we change all of the distortion coefficients by our average error in prediction. Absolute difference between the images generated by the two different sets of distortion coefficients has little visual difference. Absolute difference is best viewed in 5 $\times$ .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Affinity Maps of DEX. Each Extender exhibits different affinity toward different regions of the fisheye image features, enabling a selective, yet cooperative modulation of the fisheye features to align them back to those of perspective images.

	$M$	$d$	ScanNet++		KITTI-360
	$M$	$d$	RMSE $\downarrow$	$\delta_{1}\uparrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$
Sensitivity	12	64	0.209	0.845	1.704	0.826
	12	20	0.200	0.860	1.663	0.842
	12	10	0.207	0.858	1.695	0.844
	12	8	0.207	0.852	1.709	0.847
	12	4	0.212	0.847	1.746	0.840

Appendix

Table 7: Sensitivity study on hyperparameters and layer placement. We analyze the effect of the number of extenders

M

, decomposition dimension

d

, and different layer placement strategies on monocular depth estimation performance for ScanNet++ and KITTI-360. Within the "Layers" section of the table, "Every Other" refers to inserting DEX at every other encoder block; "First" refers to only using DEX at the first encoder block; "Shared" refers to training one set of Extenders and reusing it for every encoder block.

Label Set	Method	mIoU $\uparrow$	weighted IoU $\uparrow$
[ miscellaneous, street, lanemarks, curbstone, person, rider, vehicles, bike, motorbike, traffic sign ]	LSeg	0.289	0.812
	w/ DEX	0.305	0.822
[ background, roadway, road markings, sidewalk edge, pedestrian, riding person, automobiles, cycle, moto, road sign ]	LSeg	0.227	0.792
	w/ DEX	0.293	0.816
[ other, road, lane markings, curb, human, cyclist, cars, bicycle, motorcycle, street sign ]	LSeg	0.305	0.819
	w/ DEX	0.362	0.838

Appendix

Table 8: Sensitivity to label wording on WoodScape open-vocabulary segmentation. We evaluate how different textual label sets affect the performance of LSeg with DEX on WoodScape. We note that the sensitivity of outcomes for different label sets is more of a property of the base model (LSeg) than it is of DEX.

Figure 8: Additional Qualitative Examples. We show additional 3D reconstructions of diverse indoor and outdoor samples using DEX.

Metric	Definition
RMSE $\downarrow$	$\displaystyle\sqrt{\frac{1}{\|\Omega\|}\sum_{p\in\Omega}\bigl(\hat{d}(p)-d(p)\bigr)^{2}}$
$\delta_{1}$ $\uparrow$	$\displaystyle\frac{1}{\|\Omega\|}\sum_{p\in\Omega}\mathbf{1}\!\Bigl(\max\!\Bigl(\frac{\hat{d}(p)}{d(p)},\,\frac{d(p)}{\hat{d}(p)}\Bigr)<1.25\Bigr)$
$\text{mIoU}\uparrow$	$\displaystyle\frac{1}{K}\sum_{k=1}^{K}\frac{TP_{k}}{TP_{k}+FP_{k}+FN_{k}}$
$\text{weighted IoU}\uparrow$	$\displaystyle\sum_{k=1}^{K}w_{k}\,\frac{TP_{k}}{TP_{k}+FP_{k}+FN_{k}},$ $w_{k}=\frac{n_{k}}{\sum_{j=1}^{K}n_{j}}$

Appendix

Table 9: Error metrics for depth estimation and open-vocabulary segmentation. These evaluation metrics compute the error between predicted output values and the ground truth. For depth estimation metrics (e.g. RMSE and

\delta_{1}

\hat{d}

denotes the prediction and

d

the ground truth. For open-vocabulary segmentation metrics (e.g., mIoU, weighted IoU), TP denotes true positive, FP denotes false positive, FN denotes false negative, and

n_{k}

denotes the sum of pixels in the ground truth for the class

k

Base Model	Method	Dataset	Chamfer $\downarrow$	F1 $\uparrow$
UniDepthV2	Undistortion	ScanNet++	0.360	0.317
UniDepthV2	DEX	ScanNet++	0.117	0.841
UniDepthV2	Undistortion	KITTI-360	2.428	0.196
UniDepthV2	DEX	KITTI-360	0.343	0.887

Appendix

Table 10: Comparison to standard undistortion preprocessing pipelines. We compare undistortion preprocessing followed by inference with UniDepthV2 against UniDepthV2 with DEX using Chamfer distance and F1 score.

VNL	ScanNet++		KITTI-360
VNL	RMSE $\downarrow$	$\delta_{1}\uparrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$
NYUv2	0.372	0.704	6.283	0.301
KITTI	1.153	0.377	2.222	0.605

Appendix

Table 11: Limitations. We evaluate the performance of the indoor and outdoor VNL models with DEX on both the indoor and outdoor test sets.

Vision Foundation Model Adaptation

Vision foundation model adaptation tailors pretrained image and video models to new tasks and domains while balancing accuracy, efficiency, and generalization.

See all 326 papers

Aug 6, 2025

Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokens

We propose a method to extend foundational monocular depth estimators (FMDEs), trained on perspective images, to fisheye images. Despite being trained on tens of millions of images, FMDEs are susceptible to the covariate shift introduced by changes in camera calibration (intrinsic, distortion) parameters, leading to erroneous depth estimates. Our method aligns the distribution of latent embeddings encoding fisheye images to those of perspective images, enabling the reuse of FMDEs for fisheye cameras without retraining or finetuning. To this end, we introduce a set of Calibration Tokens as a light-weight adaptation mechanism that modulates the latent embeddings for alignment. By exploiting the already expressive latent space of FMDEs, we posit that modulating their embeddings avoids the negative impact of artifacts and loss introduced in conventional recalibration or map projection to a canonical reference frame in the image space. Our method is self-supervised and does not require fisheye images but leverages publicly available large-scale perspective image datasets. This is done by recalibrating perspective images to fisheye images, and enforcing consistency between their estimates during training. We evaluate our approach with several FMDEs, on both indoors and outdoors, where we consistently improve over state-of-the-art methods using a single set of tokens for both. Code available at: https://github.com/JungHeeKim29/calibration-token; https://github.com/Suchisrit/CalibrationTokens.

Rit Gangopadhyay, Jung-Hee Kim, Xien Chen +3

Yale University1 · Michigan State University2

Domain Adaptation Depth Estimation Monocular Depth Estimation Vision Foundation Model Adaptation Visual Prompt Learning

Ask PDF

Mar 30, 2026

Fisheye3R: Adapting Unified 3D Feed-Forward Foundation Models to Fisheye Lenses

Feed-forward foundation models for multi-view 3-dimensional (3D) reconstruction have been trained on large-scale datasets of perspective images; when tested on wide field-of-view images, e.g., from a fisheye camera, their performance degrades. This degradation arises from changes in spatial arrangements of pixels induced by the non-linear projection model that maps 3D points onto the 2D image plane. While one may surmise that training on fisheye images would resolve this problem, there are far fewer fisheye images with ground truth than perspective images, which limits generalization. To enable inference on imagery exhibiting high radial distortion, we propose Fisheye3R, a novel adaptation framework that extends these multi-view 3D reconstruction foundation models to natively accommodate fisheye inputs without performance regression on perspective images. To address the scarcity of fisheye images and ground truth, we introduce flexible learning schemes that support self-supervised adaptation using only unlabeled perspective images and supervised adaptation without any fisheye training data. Extensive experiments across three foundation models, including VGGT,

π^3

, and MapAnything, demonstrate that our approach consistently improves camera pose, depth, point map, and field-of-view estimation on fisheye images. Code is available at https://github.com/android-xr/fisheye3r.

Ruxiao Duan, Erin Hong, Dongxu Zhao +3

Yale University · Google

3D Foundation Models Domain Adaptation 3D Reconstruction Multi-View 3D Reconstruction Camera Pose Estimation Vision Foundation Model Adaptation

Ask PDF

Apr 12, 2026

FishRoPE: Projective Rotary Position Embeddings for Omnidirectional Visual Perception

Vision foundation models (VFMs) and Bird's Eye View (BEV) representation have advanced visual perception substantially, yet their internal spatial representations assume the rectilinear geometry of pinhole cameras. Fisheye cameras, widely deployed on production autonomous vehicles for their surround-view coverage, exhibit severe radial distortion that renders these representations geometrically inconsistent. At the same time, the scarcity of large-scale fisheye annotations makes retraining foundation models from scratch impractical. We present \ours, a lightweight framework that adapts frozen VFMs to fisheye geometry through two components: a frozen DINOv2 backbone with Low-Rank Adaptation (LoRA) that transfers rich self-supervised features to fisheye without task-specific pretraining, and Fisheye Rotary Position Embedding (FishRoPE), which reparameterizes the attention mechanism in the spherical coordinates of the fisheye projection so that both self-attention and cross-attention operate on angular separation rather than pixel distance. FishRoPE is architecture-agnostic, introduces negligible computational overhead, and naturally reduces to the standard formulation under pinhole geometry. We evaluate \ours on WoodScape 2D detection (54.3 mAP) and SynWoodScapes BEV segmentation (65.1 mIoU), where it achieves state-of-the-art results on both benchmarks.

Rahul Ahuja, Mudit Jain, Bala Murali Manoghar Sai Sudhakar +4

Automated Driving, Qualcomm Technologies, Inc · Automated Driving, Qualcomm India Private Limited

Rotary Positional Embeddings Vision Foundation Model Adaptation

Ask PDF

Abstract

Vision Foundation Model Adaptation

Explore similar work

Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokens

Fisheye3R: Adapting Unified 3D Feed-Forward Foundation Models to Fisheye Lenses

FishRoPE: Projective Rotary Position Embeddings for Omnidirectional Visual Perception