Achieving 360° coverage is critical for the visual perception systems of autonomous vehicles. Fisheye cameras offer a cost-effective solution by enabling full surround coverage with as few as two sensors. However, existing multi-view fisheye datasets are limited, and synthesizing rare corner cases typically requires computationally expensive 3D simulations, hindering the training. While generative models have achieved significant success in standard perspective imagery, their application to wide-angle distortion remains unexplored. In this work, we formally introduce the novel problem of multi-view fisheye image generation conditioned on volumetric semantic representations and present two distinct methods. We first propose SyntheOcc-FE, which adapts the SyntheOcc architecture to fisheye data. While effective, this method is constrained by the scarcity of fisheye datasets, which limits its generalization. To overcome these limitations, we propose our second method, MIVIFI (multi-view fisheye), which leverages cross-domain learning with Equirectangular Projections. By bridging the gap between dataset domains using KITTI-360 fisheye images alongside nuScenes multi-view standard images, our approach enables high-fidelity manipulation of scene content. This framework enables the structural modification of semantic occupancy inputs to introduce or eliminate specific actors and facilitates the rendering of diverse meteorological conditions and illumination scenarios absent in the limited fisheye datasets. Quantitative and qualitative experiments demonstrate that our methods achieve robust photorealistic multi-view fisheye image generation and highlight the specific advantages of our cross-domain strategy for handling data scarcity.
Figures & tables
Figure 1: Controllable multi-view fisheye image generation. We introduce the task of generating controllable multi-view fisheye images and present the first methods for it, (1) SyntheOcc-FE , an adaptation of SyntheOcc [ 12 ] trained on fisheye datasets. To mitigate the limited availability of multi-view fisheye data and improve generalization, we further propose (2) MIVIFI , which builds on SyntheOcc-FE and uses equirectangular projections to enable training on both fisheye and standard-lens datasets (a). (b) Both methods are conditioned on semantic occupancy maps and text prompts and (c) train a diffusion model to synthesize multi-view fisheye images (d). The occupancy conditioning provides the image semantic layout, supporting various downstream applications (f).
Figure 2: SyntheOcc-FE and MIVIFI overview. (a) Geometric conditioning: a 3D semantic occupancy map is projected into a fisheye-aware multi-plane image (MPI) by discretizing depth and stacking the per-plane semantic layers (for each target view), yielding the structural control signal. (b) Semantic enrichment: a vision-language model provides training captions, while arbitrary prompts are used at inference to steer appearance. The fisheye MPI and text prompt jointly condition a diffusion model to generate multi-view fisheye images (d). (c) Post-processing: an optional binary vignette mask mimics mechanical lens vignetting and suppresses boundary artifacts. MIVIFI uses the same MPI+text conditioning but maps both fisheye and perspective data to a shared equirectangular projection (ERP) canvas, enabling joint training with large-scale perspective datasets via stitched ERP views and masked supervision in unobserved regions.
Figure 5: Qualitative Results of MIVIFI. Comparison of generated outputs for KITTI-360 (a)–(d) and nuScenes (e)–(i). Both datasets display (a, e) the ERP MPI, (b, f) the generated ERP, and (c, g) the ERP converted to fisheye. For comparison, KITTI-360 includes (d) the real fisheye image, while nuScenes includes (h) the converted perspective and (i) the real reference image. These results demonstrate visual consistency, high quality, and faithful input adherence, as well as highlighting the ability to convert perspective images into multi-view fisheye images covering a 360°×180° FoV.
Figure 6: Comparison of Object Insertion with SyntheOcc-FE and MIVIFI. Left: Results using SyntheOcc-FE. (a, d) Modified MPIs for the left and right views, showing an inserted car (blue) and person (red). (b, e) The respective generated images, demonstrating successful car insertion (b) but failed person insertion (e). (c, f) The original multi-view reference images before modification. Right: Results using MIVIFI. (g) Original MPI and (h) the corresponding real image. (i) Modified MPI with an inserted person (red) and (j) the generated output. The expanded training dataset allows MIVIFI to successfully synthesize pedestrians.
mIoU Classes ↑
Method
FID ↓
SSIM ↑
mIoU ↑
Road ↑
Car ↑
Background ↑
LPIPS ↑
SyntheOcc-FE (ours)
10.77
0.49
0.34
0.65
0.49
0.74
0.30
MIVIFI (ours)
17.05
0.41
0.27
0.56
0.35
0.64
0.36
Table I: Image quality results for SyntheOcc-FE and MIVIFI. SyntheOcc-FE demonstrates superior generation quality, while MIVIFI achieves greater diversity. Best results are in bold.
Figure 7: Nighttime Synthesis Comparison. Transformation of daytime multi-view fisheye imagery via text-prompt adaptation, comparing SyntheOcc-FE (a)–(d) and MIVIFI (e)–(h). (a, b) and (e, f) show the generated multi-view fisheye images conditioned by including “nighttime” in the prompt, while (c, d) and (g, h) are the real daytime images for comparison.
Figure 8: Rain Synthesis Comparison. Transformation of sunny multi-view fisheye imagery via text-prompt adaptation to rain, comparing SyntheOcc-FE (a)–(d) and MIVIFI (e)–(h). (a, b) and (e, f) show the generated multi-view fisheye images conditioned by including “rain” in the prompt, while (c, d) and (g, h) are the real sunny images for comparison. SyntheOcc-FE is unable to generate rain, while MIVIFI successfully renders the weather modification realistic.
Table II: Ablation Study on SyntheOcc-FE. Removing VLM prompts and lens masks degrades overall performance. Conversely, introducing depth conditioning and multi-view cosine similarity yields no significant quantitative gains. SSIM and mIoU are overall more stable than FID. Best in bold.
Training data
Vehicle ↑
Person ↑
Bicycle ↑
Avg ↑
KITTI-360 dataset
27.7
10.7
0.1
12.9
SyntheOcc-FE
25.6
0.6
5.5
10.6
MIVIFI
20.7
4.0
1.1
8.6
Table III : Downstream 2D Object Detection. mAP on the KITTI-360 test set for models trained on real KITTI-360 versus generated training data. Best in bold.
Figure A.1 : Single-View ERP visualization. Top: perspective image and corresponding MPI. Bottom: single-view mapped to ERP with masked regions and full-screen MPI.
Feed-forward foundation models for multi-view 3-dimensional (3D) reconstruction have been trained on large-scale datasets of perspective images; when tested on wide field-of-view images, e.g., from a fisheye camera, their performance degrades. This degradation arises from changes in spatial arrangements of pixels induced by the non-linear projection model that maps 3D points onto the 2D image plane. While one may surmise that training on fisheye images would resolve this problem, there are far fewer fisheye images with ground truth than perspective images, which limits generalization. To enable inference on imagery exhibiting high radial distortion, we propose Fisheye3R, a novel adaptation framework that extends these multi-view 3D reconstruction foundation models to natively accommodate fisheye inputs without performance regression on perspective images. To address the scarcity of fisheye images and ground truth, we introduce flexible learning schemes that support self-supervised adaptation using only unlabeled perspective images and supervised adaptation without any fisheye training data. Extensive experiments across three foundation models, including VGGT, π3, and MapAnything, demonstrate that our approach consistently improves camera pose, depth, point map, and field-of-view estimation on fisheye images. Code is available at https://github.com/android-xr/fisheye3r.
Vision foundation models are capable of generalizing across 3-dimensional (3D) scenes with high-fidelity estimates; their empirical success can be attributed to training on large-scale datasets of perspective images. However, when transferred to wide field-of-view (FoV) images, such as those captured by fisheye cameras, they return erroneous outputs due to a covariate shift stemming from the radial distortion on the image pixels. We propose a method to generalize vision foundation models to fisheye cameras. The crux of our method lies in a set of learnable parameters, termed Distortion Extenders (DEX), that model the fisheye distortion coefficients and the distributional shift between fisheye and perspective images encoded in the latent space. By minimizing a self-supervised alignment loss, DEX transforms the latent embeddings of fisheye images to resemble those of perspective images to recover high-fidelity estimates. DEX is architecture- and task-agnostic: We demonstrate DEX on monocular depth estimation and open-vocabulary segmentation for convolution- and Transformer-based architectures, where we consistently improve over baselines across indoor and outdoor fisheye datasets. As a byproduct, the activations of DEX can also be decoded to distortion coefficients to support camera calibration. Code available at: https://github.com/Suchisrit/DEX.
Rit Gangopadhyay, Alex Wong
Yale Vision Laboratory, Yale University, New Haven, CT 06520, USA
Vision foundation models (VFMs) and Bird's Eye View (BEV) representation have advanced visual perception substantially, yet their internal spatial representations assume the rectilinear geometry of pinhole cameras. Fisheye cameras, widely deployed on production autonomous vehicles for their surround-view coverage, exhibit severe radial distortion that renders these representations geometrically inconsistent. At the same time, the scarcity of large-scale fisheye annotations makes retraining foundation models from scratch impractical. We present \ours, a lightweight framework that adapts frozen VFMs to fisheye geometry through two components: a frozen DINOv2 backbone with Low-Rank Adaptation (LoRA) that transfers rich self-supervised features to fisheye without task-specific pretraining, and Fisheye Rotary Position Embedding (FishRoPE), which reparameterizes the attention mechanism in the spherical coordinates of the fisheye projection so that both self-attention and cross-attention operate on angular separation rather than pixel distance. FishRoPE is architecture-agnostic, introduces negligible computational overhead, and naturally reduces to the standard formulation under pinhole geometry. We evaluate \ours on WoodScape 2D detection (54.3 mAP) and SynWoodScapes BEV segmentation (65.1 mIoU), where it achieves state-of-the-art results on both benchmarks.
Rahul Ahuja, Mudit Jain, Bala Murali Manoghar Sai Sudhakar +4