Benchmarking Multi-View BEV Object Detection with Mixed Pinhole and Fisheye Cameras

Xiangzhong Liu; Hao Shen

cs.CVMar 29, 2026

Benchmarking Multi-View BEV Object Detection with Mixed Pinhole and Fisheye Cameras

Authors: Xiangzhong Liu, Hao Shen

Organizations: Machine Learning Group, fortiss GmbH, Guerickestraße 25, 80805 Munich, Germany

Abstract

Modern autonomous driving systems increasingly rely on mixed camera configurations with pinhole and fisheye cameras for full view perception. However, Bird's-Eye View (BEV) 3D object detection models are predominantly designed for pinhole cameras, leading to performance degradation under fisheye distortion. To bridge this gap, we introduce a multi-view BEV detection benchmark with mixed cameras by converting KITTI-360 into nuScenes format. Our study encompasses three adaptations: rectification for zero-shot evaluation and fine-tuning of nuScenes-trained models, distortion-aware view transformation modules (VTMs) via the MEI camera model, and polar coordinate representations to better align with radial distortion. We systematically evaluate three representative BEV architectures, BEVFormer, BEVDet and PETR, across these strategies. We demonstrate that projection-free architectures are inherently more robust and effective against fisheye distortion than other VTMs. This work establishes the first real-data 3D detection benchmark with fisheye and pinhole images and provides systematic adaptation and practical guidelines for designing robust and cost-effective 3D perception systems. The code is available at https://github.com/CesarLiu/FishBEVOD.git.

Figures & tables

Fig. 1 : Qualitative detection results on KITTI-360 mixed camera configuration. 3D bounding box and point cloud rendering overlaid on pinhole and fisheye images. The visualization is created with a customized nuScenes devkit. Our method successfully handles the severe radial distortion in fisheye cameras while maintaining consistent detection accuracy across different camera types.

Fig. 2 : Camera configuration and full field-of-view coverage comparison. Left: Original KITTI-360 with stereo cameras and fisheye cameras. Middle: Rectified KITTI-360 with 6 pinhole cameras. Right: NuScenes configuration with 6 pinhole cameras.

Fig. 3 : Overview of distortion-aware BEV 3D object detection framework. Multi-view images (pinhole+fisheye) are processed by a shared backbone encoder, then fed into three distortion-aware view transformation modules via MEI camera model. The resulting BEV features can be represented in either Cartesian or polar coordinates to better align with fisheye geometry. A detection head(Transformer or CNN) processes BEV features to produce final 3D object detection outputs.

Fig. 4 : Class distribution in KITTI-360 dataset (log scale). The annotation distribution exhibits significant class imbalance, heavily skewed toward static infrastructure objects, while dynamic objects relevant to autonomous driving other than car represent a smaller portion. Blue bars indicate training samples, orange bars for validation, with consistent ratios maintained across splits.

Model	Backbone	Method	mAP $\uparrow$	NDS $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	mAVE $\downarrow$	AP ${}_{\text{car}}$ $\uparrow$	AP ${}_{\text{ped}}$ $\uparrow$	AP ${}_{\text{bus}}$ $\uparrow$
		Zero-Shot	0.008	0.030	1.036	0.830	1.139	0.957	0.055	0.000	0.000
		Baseline	0.121	0.159	0.736	0.481	1.031	1.017	0.462	0.096	0.000
BEVDet	ResNet-50	DA-VTM	0.150	0.212	0.686	0.400	1.089	0.816	0.513	0.138	0.015
		+ Polar	0.153	0.219	0.655	0.384	0.979	0.846	0.517	0.153	0.002
		Rectification	0.169	0.232	0.673	0.388	1.004	0.760	0.538	0.189	0.024
		Zero-Shot	0.034	0.083	0.998	0.469	1.031	1.770	0.195	0.009	0.000

TABLE I : Comprehensive evaluation on KITTI-360 fisheye benchmark. Results show zero-shot performance of nuScenes-trained models, baseline training ignoring distortion, our distortion-aware adaptations with MEI camera model, polar coordinate enhancements, and upper-bound performance on rectified data.

Fig. 5 : Model robustness under camera failure scenarios. FL, FB and SL denote the front-left, front-both and side-left.

Method	0–10 m	10–20 m	20–30 m	30–40 m	40–50 m
BEVFormer(rect)	49.54	26.08	7.87	2.17	0.63
BEVFormer (ours)	47.38	20.17	5.43	1.83	0.40
PETR(rect)	55.87	37.70	14.95	4.83	1.45
PETR (ours)	56.57	35.52	12.49	4.50	1.44
BEVDet(rect)	35.17	19.90	6.41	2.35	0.57
BEVDet (ours)	34.22	18.90	5.64	1.65	0.36

TABLE II : mAP (%) across distance ranges on KITTI-360. (rect) for models on rectified images and (ours) represent distortion-aware models on mixed inputs.

Method	Front $120^{\circ}$	Back $120^{\circ}$	Sides $120^{\circ}$
BEVFormer baseline	19.42	12.56	18.08
BEVFormer (ours)	19.36	14.06	21.44
PETR baseline	29.84	21.87	28.77
PETR (ours)	31.35	22.68	29.84
BEVDet baseline	14.43	7.81	13.67
BEVDet (ours)	17.58	12.05	17.30

TABLE III : Angular-stratified mAP (%) on KITTI-360. Baseline for standard training on mixed images and (ours) = distortion-aware models.

Multi-View 3D Object Detection

Multi-View 3D Object Detection combines images from multiple cameras to locate and classify objects in three-dimensional space.

See all 25 papers

Distortion-Aware PETR for BEV Object Detection with Mixed Pinhole-Fisheye Cameras
Jun 7, 2026Xiangzhong Liu3D ViTs Autonomous Driving Perception
Machine Learning Group, fortiss GmbH, Guerickestraße 25, 80805 Munich, Germany
FishRoPE: Projective Rotary Position Embeddings for Omnidirectional Visual Perception
Apr 12, 2026Rahul Ahuja, Mudit Jain, Bala Murali Manoghar Sai Sudhakar +4Rotary Positional Embeddings Vision Foundation Model Adaptation
Automated Driving, Qualcomm Technologies, Inc · Automated Driving, Qualcomm India Private Limited
Geometry-Aware Fisheye-LiDAR Fusion for Robust 3D Object Detection in Low-Overlap Setups
Jun 7, 2026Xiangzhong Liu, Xihao Wang, Hao ShenCamera-LiDAR Fusion Bird's-Eye-View Representation
Technical University of Munich, 80333 Munich, Germany

Jun 7, 2026J/K move · Enter open · S save

Distortion-Aware PETR for BEV Object Detection with Mixed Pinhole-Fisheye Cameras

Xiangzhong Liu

Machine Learning Group, fortiss GmbH, Guerickestraße 25, 80805 Munich, Germany

3D ViTs Autonomous Driving Perception 3D Object Detection Object Detection

Open paper AskPDF arXiv

Fisheye cameras are widely deployed in autonomous driving perception suites for their low cost and full-coverage field of view (FOV), yet their potential remains underleveraged in 3D object detection. Severe radial distortion challenges most BEV detectors by violating the fundamental assumption of uniform sampling. To bridge this gap, we propose Distortion-Aware PETR (DAPETR), a projection-free detector tailored for mixed pinhole-fisheye camera setups. DAPETR incorporates two key learned-adaptive modules: a unified distortion-aware positional embedding that harmonizes positional encodings for image representations with fisheye geometry, and a bidirectional feature-geometry co-modulation module that mutually adapts image features and 3D positional embeddings. In our experiments on a converted KITTI-360 benchmark, we systematically compare our learned adaptive approach against PETR in polar coordinates (PolarPETR). We find that while both methods improve over the baseline, our learned modules achieve superior performance. Crucially, we uncover a negative interaction when combining both strategies, revealing that learned adaptation and explicit geometric reparameterization can conflict. Our final DAPETR model significantly advances the research and benchmark for fisheye BEV detection, providing critical insights into effective distortion-aware 3D perception design other than image rectification.

Abstract

Multi-View 3D Object Detection

Explore similar work

Distortion-Aware PETR for BEV Object Detection with Mixed Pinhole-Fisheye Cameras

FishRoPE: Projective Rotary Position Embeddings for Omnidirectional Visual Perception

Geometry-Aware Fisheye-LiDAR Fusion for Robust 3D Object Detection in Low-Overlap Setups