Vision foundation models (VFMs) and Bird's Eye View (BEV) representation have advanced visual perception substantially, yet their internal spatial representations assume the rectilinear geometry of pinhole cameras. Fisheye cameras, widely deployed on production autonomous vehicles for their surround-view coverage, exhibit severe radial distortion that renders these representations geometrically inconsistent. At the same time, the scarcity of large-scale fisheye annotations makes retraining foundation models from scratch impractical. We present \ours, a lightweight framework that adapts frozen VFMs to fisheye geometry through two components: a frozen DINOv2 backbone with Low-Rank Adaptation (LoRA) that transfers rich self-supervised features to fisheye without task-specific pretraining, and Fisheye Rotary Position Embedding (FishRoPE), which reparameterizes the attention mechanism in the spherical coordinates of the fisheye projection so that both self-attention and cross-attention operate on angular separation rather than pixel distance. FishRoPE is architecture-agnostic, introduces negligible computational overhead, and naturally reduces to the standard formulation under pinhole geometry. We evaluate \ours on WoodScape 2D detection (54.3 mAP) and SynWoodScapes BEV segmentation (65.1 mIoU), where it achieves state-of-the-art results on both benchmarks.
Figures & tables
Figure 1: Architecture overview. FishRoPE comprises (1) a frozen DINOv2 backbone with LoRA adaptation for multi-scale feature extraction from fisheye images, (2) a FishRoPE-enhanced feature encoder that embeds fisheye-aware angular geometry into self-attention, and (3) task-specific heads for 2D detection and BEV segmentation via Kannala–Brandt view transformation.
Method
Backbone
Reported score ↑
FisheyeYOLO (24-sided polygon) [ 18 ]
R-18
44.65 †
FisheyeDetNet (polygon) [ 23 ]
R-18
49.5 ‡
FishRoPE (R-18 + FishRoPE)
R-18
51.4
FishRoPE (DINOv2-B + 2D RoPE)
DINOv2-B
52.5
FishRoPE (DINOv2-B + FishRoPE)
DINOv2-B
54.2
Table 1: 2D fisheye detection results. We report our WoodScape [email protected] and prior published numbers under their original protocols. Best in bold .
Method
Backbone
mIoU ↑
F2BEV [ 20 ]
R-18
53.39 †
FishBEV [ 1 ]
DINOv2-L
64.22 §
FishRoPE (R-18 + FishRoPE)
R-18
56.8
FishRoPE (DINOv2-B + 2D RoPE)
DINOv2-B
61.4
FishRoPE (DINOv2-B + FishRoPE)
DINOv2-B
65.1
Table 2: BEV semantic segmentation on SynWoodScapes. mIoU across foreground classes. Best in bold .
Position Encoding
mAP ↑
mIoU ↑
Learned absolute PE
51.8
59.2
Sinusoidal 2D PE
52.1
59.6
2D Axial RoPE [ 5 ]
52.5
61.4
FishRoPE ( θ only)
53.6
64.7
FishRoPE ( θ,ϕ )
54.2
65.1
Table 3: Position encoding ablation. Detection mAP (WoodScape) and BEV segmentation mIoU (SynWoodScapes). All use DINOv2-B backbone.
Backbone
Params
mAP ↑
mIoU ↑
Swin-T [ 15 ]
29M
52.9
59.8
DINOv2-S ∗ [ 16 ]
8.1M
53.4
62.7
DINOv2-B ∗ [ 16 ]
12.4M
54.2
65.1
Table 4: Backbone comparison. All use FishRoPE. mAP on WoodScape, mIoU on SynWoodScapes.
Figure 2: Qualitative results. FishRoPE (right) correctly handles peripheral image regions where baselines (left) fail due to fisheye distortion.
Rotary positional embeddings (RoPE) encode token positions by rotating attention features, supporting relative-position reasoning in language and multimodal models.
Vision foundation models are capable of generalizing across 3-dimensional (3D) scenes with high-fidelity estimates; their empirical success can be attributed to training on large-scale datasets of perspective images. However, when transferred to wide field-of-view (FoV) images, such as those captured by fisheye cameras, they return erroneous outputs due to a covariate shift stemming from the radial distortion on the image pixels. We propose a method to generalize vision foundation models to fisheye cameras. The crux of our method lies in a set of learnable parameters, termed Distortion Extenders (DEX), that model the fisheye distortion coefficients and the distributional shift between fisheye and perspective images encoded in the latent space. By minimizing a self-supervised alignment loss, DEX transforms the latent embeddings of fisheye images to resemble those of perspective images to recover high-fidelity estimates. DEX is architecture- and task-agnostic: We demonstrate DEX on monocular depth estimation and open-vocabulary segmentation for convolution- and Transformer-based architectures, where we consistently improve over baselines across indoor and outdoor fisheye datasets. As a byproduct, the activations of DEX can also be decoded to distortion coefficients to support camera calibration. Code available at: https://github.com/Suchisrit/DEX.
Rit Gangopadhyay, Alex Wong
Yale Vision Laboratory, Yale University, New Haven, CT 06520, USA
Feed-forward foundation models for multi-view 3-dimensional (3D) reconstruction have been trained on large-scale datasets of perspective images; when tested on wide field-of-view images, e.g., from a fisheye camera, their performance degrades. This degradation arises from changes in spatial arrangements of pixels induced by the non-linear projection model that maps 3D points onto the 2D image plane. While one may surmise that training on fisheye images would resolve this problem, there are far fewer fisheye images with ground truth than perspective images, which limits generalization. To enable inference on imagery exhibiting high radial distortion, we propose Fisheye3R, a novel adaptation framework that extends these multi-view 3D reconstruction foundation models to natively accommodate fisheye inputs without performance regression on perspective images. To address the scarcity of fisheye images and ground truth, we introduce flexible learning schemes that support self-supervised adaptation using only unlabeled perspective images and supervised adaptation without any fisheye training data. Extensive experiments across three foundation models, including VGGT, π3, and MapAnything, demonstrate that our approach consistently improves camera pose, depth, point map, and field-of-view estimation on fisheye images. Code is available at https://github.com/android-xr/fisheye3r.
Fisheye cameras are widely deployed in autonomous driving perception suites for their low cost and full-coverage field of view (FOV), yet their potential remains underleveraged in 3D object detection. Severe radial distortion challenges most BEV detectors by violating the fundamental assumption of uniform sampling. To bridge this gap, we propose Distortion-Aware PETR (DAPETR), a projection-free detector tailored for mixed pinhole-fisheye camera setups. DAPETR incorporates two key learned-adaptive modules: a unified distortion-aware positional embedding that harmonizes positional encodings for image representations with fisheye geometry, and a bidirectional feature-geometry co-modulation module that mutually adapts image features and 3D positional embeddings. In our experiments on a converted KITTI-360 benchmark, we systematically compare our learned adaptive approach against PETR in polar coordinates (PolarPETR). We find that while both methods improve over the baseline, our learned modules achieve superior performance. Crucially, we uncover a negative interaction when combining both strategies, revealing that learned adaptation and explicit geometric reparameterization can conflict. Our final DAPETR model significantly advances the research and benchmark for fisheye BEV detection, providing critical insights into effective distortion-aware 3D perception design other than image rectification.