From Perspective to Fisheye Depth Estimation and Open-Vocabulary Segmentation

Rit Gangopadhyay; Alex Wong

cs.CVAug 28, 2026

From Perspective to Fisheye Depth Estimation and Open-Vocabulary Segmentation

Authors: Rit Gangopadhyay, Alex Wong

Organizations: Yale Vision Laboratory, Yale University, New Haven, CT 06520, USA

Abstract

Vision foundation models are capable of generalizing across 3-dimensional (3D) scenes with high-fidelity estimates; their empirical success can be attributed to training on large-scale datasets of perspective images. However, when transferred to wide field-of-view (FoV) images, such as those captured by fisheye cameras, they return erroneous outputs due to a covariate shift stemming from the radial distortion on the image pixels. We propose a method to generalize vision foundation models to fisheye cameras. The crux of our method lies in a set of learnable parameters, termed Distortion Extenders (DEX), that model the fisheye distortion coefficients and the distributional shift between fisheye and perspective images encoded in the latent space. By minimizing a self-supervised alignment loss, DEX transforms the latent embeddings of fisheye images to resemble those of perspective images to recover high-fidelity estimates. DEX is architecture- and task-agnostic: We demonstrate DEX on monocular depth estimation and open-vocabulary segmentation for convolution- and Transformer-based architectures, where we consistently improve over baselines across indoor and outdoor fisheye datasets. As a byproduct, the activations of DEX can also be decoded to distortion coefficients to support camera calibration. Code available at: https://github.com/Suchisrit/DEX.

Figures & tables

Figure 1: 3D reconstructions and open-vocabulary segmentation of fisheye imagery produced by foundational models, UniDepthV2 [ 42 ] and LSeg [ 27 ] . The distortion in the input leads to erroneous and distorted reconstructions and segmentations.

Figure 2: Distortion Extenders. The figure illustrates the DEX training pipeline and architecture. During training, we pass perspective inputs and fisheye inputs augmented by DEX into a frozen backbone model. The fisheye features select convex combinations of Extenders for translation in the latent space. We optimize the Distortion Extenders to minimize the difference between the fisheye and perspective output.

Figure 3: MDE Qualitative Results. We show 3D reconstructions of indoor and outdoor samples from the evaluation in Tab. 1 . Previous methods struggle with the outermost regions in the fisheye images, which have the most severe distortion. DEX is more robust to these distortions. In the outdoor scenes with the highest distortions, we are still able to accurately reconstruct many of the points at the edges of the images.

Figure 4: Segmentation Qualitative Results. We evaluate LSeg and SED with and without DEX on the WoodScape outdoor dataset. We see improvements in the open-vocabulary segmentation, highlighting DEX’s generalizability to this recognition-based task. We evaluate on the valid regions of the images specified by the ground truth.

Testset	Experiment	Model	Train Dataset	RMSE $\downarrow$	$\delta_{1}$ $\uparrow$
ScanNet++ [ 78 ]	Base Model	MiDaS [ 46 ]	Mix 1.4M	0.672	0.479
	w/ LoRA [ 21 ]	MiDaS [ 46 ]	Mix 200K	0.561	0.599
	w/ Calibration Tokens [ 11 ]	MiDaS [ 46 ]	Mix 200K	0.413	0.688
	w/ Distortion Extenders	MiDaS [ 46 ]	Mix 200K	0.397	0.702
	Base Model	DepthAnything [ 75 ]	Mix 63.5M	0.705	0.454
	w/ LoRA [ 21 ]	DepthAnything [ 75 ]	Mix 200K	0.706	0.445

Table 1: MDE Performance on Indoor and Outdoor. After training a set of Distortion Extenders, we test their zero-shot performance on ScanNet++ and KITTI-360, which contain real fisheye images with varying scenes and distortion parameters.

Testset	Experiment	Model	Train Dataset	mIoU $\uparrow$	weighted IoU $\uparrow$
WoodScape [ 80 ]	Baseline	LSeg [ 27 ]	Mix 500K	0.305	0.819
	w/ Calibration Tokens	LSeg [ 27 ]	Mix 50K	0.321	0.823
	w/ Distortion Extenders	LSeg [ 27 ]	Mix 50K	0.362	0.838
	Baseline	SED [ 71 ]	Mix 120K	0.439	0.829
	w/ Distortion Extenders	SED [ 71 ]	Mix 50K	0.449	0.832

Table 2: Open Vocabulary Segmentation. We evaluate the zero-shot performance of our method on the WoodScape dataset, which contains real fisheye images with varying scenes and distortion parameters.

Method	Depth		Segmentation
Method	RMSE $\downarrow$	$\delta_{1}\uparrow$	mIoU $\uparrow$	weighted IoU $\uparrow$
PanopticDepth [ 12 ]	7.273	0.267	0.160	0.300
w/ Distortion Extenders	3.583	0.354	0.282	0.833

Table 3: DEX with PanopticDepth. We apply DEX to PanopticDepth and evaluate both depth estimation and segmentation on WoodScape.

Panel of Figure 5 — Figure 5: t-SNE Plot of Fisheye and Perspective Image Features. When using DEX, fisheye image features align much closer with corresponding perspective features.

ScanNet++	$k_{1}$	$k_{2}$	$k_{3}$	$k_{4}$	Average
MSE $\downarrow$	$2.04{\times}10^{-6}$	$1.53{\times}10^{-6}$	$3.53{\times}10^{-7}$	$8.87{\times}10^{-9}$	$9.81{\times}10^{-7}$
MAE $\downarrow$	$1.17{\times}10^{-3}$	$9.88{\times}10^{-4}$	$5.27{\times}10^{-4}$	$7.05{\times}10^{-5}$	$6.88{\times}10^{-4}$
MAPE (%) $\downarrow$	3.91	14.30	41.86	78.61	34.67

Table 6: KB Distortion Parameter Decoding. Distortion Extenders are able to recover KB distortion parameters for an image input.

Figure 6: Error in KB Distortion Prediction Visualized. We provide an example for the visual difference in distortion when we generate a fisheye image with a set of distortion parameters representative of ScanNet++, and when we change all of the distortion coefficients by our average error in prediction. Absolute difference between the images generated by the two different sets of distortion coefficients has little visual difference. Absolute difference is best viewed in 5 $\times$ .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Affinity Maps of DEX. Each Extender exhibits different affinity toward different regions of the fisheye image features, enabling a selective, yet cooperative modulation of the fisheye features to align them back to those of perspective images.

	$M$	$d$	ScanNet++		KITTI-360
	$M$	$d$	RMSE $\downarrow$	$\delta_{1}\uparrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$
Sensitivity	12	64	0.209	0.845	1.704	0.826
	12	20	0.200	0.860	1.663	0.842
	12	10	0.207	0.858	1.695	0.844
	12	8	0.207	0.852	1.709	0.847
	12	4	0.212	0.847	1.746	0.840

Appendix

Table 7: Sensitivity study on hyperparameters and layer placement. We analyze the effect of the number of extenders

M

, decomposition dimension

d

, and different layer placement strategies on monocular depth estimation performance for ScanNet++ and KITTI-360. Within the "Layers" section of the table, "Every Other" refers to inserting DEX at every other encoder block; "First" refers to only using DEX at the first encoder block; "Shared" refers to training one set of Extenders and reusing it for every encoder block.

Label Set	Method	mIoU $\uparrow$	weighted IoU $\uparrow$
[ miscellaneous, street, lanemarks, curbstone, person, rider, vehicles, bike, motorbike, traffic sign ]	LSeg	0.289	0.812
	w/ DEX	0.305	0.822
[ background, roadway, road markings, sidewalk edge, pedestrian, riding person, automobiles, cycle, moto, road sign ]	LSeg	0.227	0.792
	w/ DEX	0.293	0.816
[ other, road, lane markings, curb, human, cyclist, cars, bicycle, motorcycle, street sign ]	LSeg	0.305	0.819
	w/ DEX	0.362	0.838

Appendix

Table 8: Sensitivity to label wording on WoodScape open-vocabulary segmentation. We evaluate how different textual label sets affect the performance of LSeg with DEX on WoodScape. We note that the sensitivity of outcomes for different label sets is more of a property of the base model (LSeg) than it is of DEX.

Figure 8: Additional Qualitative Examples. We show additional 3D reconstructions of diverse indoor and outdoor samples using DEX.

Metric	Definition
RMSE $\downarrow$	$\displaystyle\sqrt{\frac{1}{\|\Omega\|}\sum_{p\in\Omega}\bigl(\hat{d}(p)-d(p)\bigr)^{2}}$
$\delta_{1}$ $\uparrow$	$\displaystyle\frac{1}{\|\Omega\|}\sum_{p\in\Omega}\mathbf{1}\!\Bigl(\max\!\Bigl(\frac{\hat{d}(p)}{d(p)},\,\frac{d(p)}{\hat{d}(p)}\Bigr)<1.25\Bigr)$
$\text{mIoU}\uparrow$	$\displaystyle\frac{1}{K}\sum_{k=1}^{K}\frac{TP_{k}}{TP_{k}+FP_{k}+FN_{k}}$
$\text{weighted IoU}\uparrow$	$\displaystyle\sum_{k=1}^{K}w_{k}\,\frac{TP_{k}}{TP_{k}+FP_{k}+FN_{k}},$ $w_{k}=\frac{n_{k}}{\sum_{j=1}^{K}n_{j}}$

Appendix

Table 9: Error metrics for depth estimation and open-vocabulary segmentation. These evaluation metrics compute the error between predicted output values and the ground truth. For depth estimation metrics (e.g. RMSE and

\delta_{1}

\hat{d}

denotes the prediction and

d

the ground truth. For open-vocabulary segmentation metrics (e.g., mIoU, weighted IoU), TP denotes true positive, FP denotes false positive, FN denotes false negative, and

n_{k}

denotes the sum of pixels in the ground truth for the class

k

Base Model	Method	Dataset	Chamfer $\downarrow$	F1 $\uparrow$
UniDepthV2	Undistortion	ScanNet++	0.360	0.317
UniDepthV2	DEX	ScanNet++	0.117	0.841
UniDepthV2	Undistortion	KITTI-360	2.428	0.196
UniDepthV2	DEX	KITTI-360	0.343	0.887

Appendix

Table 10: Comparison to standard undistortion preprocessing pipelines. We compare undistortion preprocessing followed by inference with UniDepthV2 against UniDepthV2 with DEX using Chamfer distance and F1 score.

VNL	ScanNet++		KITTI-360
VNL	RMSE $\downarrow$	$\delta_{1}\uparrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$
NYUv2	0.372	0.704	6.283	0.301
KITTI	1.153	0.377	2.222	0.605

Appendix

Table 11: Limitations. We evaluate the performance of the indoor and outdoor VNL models with DEX on both the indoor and outdoor test sets.

Vision Foundation Model Adaptation

Vision foundation model adaptation tailors pretrained image and video models to new tasks and domains while balancing accuracy, efficiency, and generalization.

See all 326 papers

Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokens
Aug 6, 2025Rit Gangopadhyay, Jung-Hee Kim, Xien Chen +3Domain Adaptation Depth Estimation
Yale University1 · Michigan State University2
Fisheye3R: Adapting Unified 3D Feed-Forward Foundation Models to Fisheye Lenses
Mar 30, 2026Ruxiao Duan, Erin Hong, Dongxu Zhao +33D Foundation Models Domain Adaptation
Yale University · Google
FishRoPE: Projective Rotary Position Embeddings for Omnidirectional Visual Perception
Apr 12, 2026Rahul Ahuja, Mudit Jain, Bala Murali Manoghar Sai Sudhakar +4Rotary Positional Embeddings Vision Foundation Model Adaptation
Automated Driving, Qualcomm Technologies, Inc · Automated Driving, Qualcomm India Private Limited

Aug 6, 2025J/K move · Enter open · S save

Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokens

Rit Gangopadhyay, Jung-Hee Kim, Xien Chen, Patrick Rim+2

Yale University1 · Michigan State University2

Domain Adaptation Depth Estimation Monocular Depth Estimation Vision Foundation Model Adaptation Visual Prompt Learning

Open paper AskPDF arXiv

We propose a method to extend foundational monocular depth estimators (FMDEs), trained on perspective images, to fisheye images. Despite being trained on tens of millions of images, FMDEs are susceptible to the covariate shift introduced by changes in camera calibration (intrinsic, distortion) parameters, leading to erroneous depth estimates. Our method aligns the distribution of latent embeddings encoding fisheye images to those of perspective images, enabling the reuse of FMDEs for fisheye cameras without retraining or finetuning. To this end, we introduce a set of Calibration Tokens as a light-weight adaptation mechanism that modulates the latent embeddings for alignment. By exploiting the already expressive latent space of FMDEs, we posit that modulating their embeddings avoids the negative impact of artifacts and loss introduced in conventional recalibration or map projection to a canonical reference frame in the image space. Our method is self-supervised and does not require fisheye images but leverages publicly available large-scale perspective image datasets. This is done by recalibrating perspective images to fisheye images, and enforcing consistency between their estimates during training. We evaluate our approach with several FMDEs, on both indoors and outdoors, where we consistently improve over state-of-the-art methods using a single set of tokens for both. Code available at: https://github.com/JungHeeKim29/calibration-token; https://github.com/Suchisrit/CalibrationTokens.

Abstract

Vision Foundation Model Adaptation

Explore similar work

Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokens

Fisheye3R: Adapting Unified 3D Feed-Forward Foundation Models to Fisheye Lenses

FishRoPE: Projective Rotary Position Embeddings for Omnidirectional Visual Perception