FUSEye: Training-Light Fisheye Detection with Overlapping Views and Zero-Initialized Adapters
Authors: Wenya Su, Kai Luo, Di Wen, Ruiping Liu, Yufan Chen, Junwei Zheng, Kunyu Peng, Kailun Yang
Organizations: School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China · Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Germany
Fisheye cameras give mobile robots a single-sensor, low-cost view of their surroundings, yet the COCO-pretrained detectors that practitioners routinely reuse fail on them: strong radial distortion warps local image structure, while boundary compression shrinks objects to near-invisible sizes. Full fine-tuning closes much of the gap but requires abundant fisheye labels and compute. We present FUSEye, a training-light framework that turns a frozen-backbone COCO-pretrained extra-large YOLO26 detector (YOLO26-x) into a fisheye detector. FUSEye adds roughly 227k new parameters while updating the inserted modules and the pretrained detection head. It addresses the transfer gap at three causally linked levels. At the input level, overlapping grid view generation and box remapping (GridViews) enlarge compressed boundary regions. At the feature level, zero-initialized residual adapters (Z-Adapters) correct distortion-induced feature misalignment. At the decision level, learned cross-projection agreement fusion (AgreeFusion) promotes low-confidence detections only when they are supported by consistent evidence across multiple views. On the WoodScape surround-view fisheye benchmark, FUSEye raises YOLO26-x from 0.148 to 0.266 mAP50 and retains 84.3% fully fine-tuned accuracy. Moreover, randomly using only 25% of the labeled training images, FUSEye achieves 0.2597 mAP50, retaining 97.6% of its full-label performance. FUSEye also consistently improves YOLOv8-11 detectors, showing that the recipe is architecture-agnostic. Source code will be available at https://github.com/Su-wenya/FUSEye.
Figures & tables
Fig. 1: Overall architecture of FUSEye. FUSEye adapts a COCO-pretrained detector to fisheye imagery while keeping its backbone weights frozen. It combines overlapping-view inference, residual feature adaptation, and detection-level agreement fusion. All views share the same adapted detector, and their predictions are remapped to a common coordinate system before fusion. The adapters and detection head are updated during detector adaptation, while a lightweight scorer learns to re-score the resulting multi-view clusters.
Fig. 2: Detailed designs of the three FUSEye modules. (a) GridViews generates the full view and four 857×647 corner crops from a 1280×966 image. Adjacent crops overlap by 423 pixels horizontally and 319 pixels vertically. All views are resized to 640×640 , and their detections are remapped to the original image coordinates. (b) Zero-initialized residual Z-Adapters are inserted after P4, P5, and C2PSA to model local distortion while preserving the pretrained features. (c) AgreeFusion clusters aligned boxes by class and IoU and uses learned agreement features to re-score and fuse multi-view detections, followed by class-wise non-maximum suppression.
Model
Variant
Backbone blocks
Overall
Per-class AP50:95
AP50↑
AP50:95↑
Car ↑
Person ↑
Bus ↑
YOLOv8 [ 37 ]
Nano
C2f
5.44
3.94
6.66
5.15
0.00
Small
9.12
6.26
10.37
8.40
0.00
Medium
12.71
8.95
13.47
12.39
0.99
Large
13.07
9.66
14.25
13.73
0.99
Extra-large
12.67
9.34
14.14
13.76
0.12
TABLE 1: Direct-transfer results of COCO-pretrained detectors on WoodScape MVR [ 1 ] . Results are reported as percentages. Backbone blocks list the main feature-extraction components; slashes separate component types. Bold and underline denote the best and second-best distinct results, respectively; ties receive the same formatting.
Method
Backbone frozen
Overall
Per-class AP50:95
AP50↑
AP50:95↑
Car ↑
Person ↑
Bus ↑
Direct transfer
Yes
14.80
10.96
15.74
15.67
1.49
FUSEye
Yes
26.61
17.26
26.83
24.06
0.87
Full fine-tuning
No
31.56
22.82
42.48
22.44
3.54
TABLE 2: Comparison of adaptation strategies for YOLO26-x on WoodScape MVR. Bold and underline denote the best and second-best results, respectively.
GV
ZA
AF
Overall
Per-class AP50:95
AP50↑
AP50:95↑
Car ↑
Person ↑
Bus ↑
–
–
–
14.62
11.05
15.63
16.27
1.25
✓
–
–
15.49
11.22
13.60
18.51
1.55
–
✓
–
17.75
13.01
20.37
17.76
0.89
✓
✓
–
20.39
14.03
19.74
21.45
0.89
✓
–
✓
18.91
12.95
17.36
20.43
1.06
TABLE 3: Component ablation on YOLO26-x. GV, ZA, and AF denote GridViews, Z-Adapters, and AgreeFusion, respectively; the first row disables all three components. Bold and underline denote the best and second-best distinct results, respectively.
Metric
25% training set
100%
Subset 1
Subset 2
Subset 3
Mean ± SD
Full
AP50↑
26.21
26.12
25.59
25.97±0.34
26.61
AP50:95↑
17.04
16.91
16.56
16.84±0.25
17.26
AP50:95car↑
26.21
26.19
26.11
26.17±0.05
26.83
AP50:95person↑
23.10
23.04
22.24
22.79±0.48
24.06
AP50:95bus↑
1.82
1.48
1.33
1.55±0.25
0.87
TABLE 4: Results with 25% and 100% of the training set. Each 25% subset contains 1,520 images; mean ± sample standard deviation is computed over three subsets.
Model
Method
Overall
Per-class AP50:95
AP50↑
AP50:95↑
Car ↑
Person ↑
Bus ↑
YOLOv8-x [ 37 ]
Base
11.89
9.05
12.79
13.36
0.99
FUSEye
17.69
11.50
17.70
16.35
0.45
YOLOv9-e [ 18 ]
Base
13.84
10.23
14.11
15.43
1.16
FUSEye
22.13
13.89
19.95
20.90
0.83
YOLOv10-x [ 19 ]
Base
12.25
9.16
13.40
13.09
0.99
TABLE 5: Cross-model evaluation on WoodScape MVR using the protocol in Sec. IV-A . The evaluated variants are Extra-large for YOLOv8, YOLOv10, and YOLO11, and Extended for YOLOv9. Base denotes COCO-pretrained direct transfer. Results are percentages; bold marks the better result in each pair.
Fig. 3: C2PSA activation maps from the YOLO26-x backbone for direct transfer and FUSEye. Warmer colors indicate stronger responses.
Fig. 4: Columns show ground truth, zero-shot COCO baseline, and complete FUSEye; insets enlarge representative detections on WoodScape MVR.
Vision foundation models (VFMs) and Bird's Eye View (BEV) representation have advanced visual perception substantially, yet their internal spatial representations assume the rectilinear geometry of pinhole cameras. Fisheye cameras, widely deployed on production autonomous vehicles for their surround-view coverage, exhibit severe radial distortion that renders these representations geometrically inconsistent. At the same time, the scarcity of large-scale fisheye annotations makes retraining foundation models from scratch impractical. We present \ours, a lightweight framework that adapts frozen VFMs to fisheye geometry through two components: a frozen DINOv2 backbone with Low-Rank Adaptation (LoRA) that transfers rich self-supervised features to fisheye without task-specific pretraining, and Fisheye Rotary Position Embedding (FishRoPE), which reparameterizes the attention mechanism in the spherical coordinates of the fisheye projection so that both self-attention and cross-attention operate on angular separation rather than pixel distance. FishRoPE is architecture-agnostic, introduces negligible computational overhead, and naturally reduces to the standard formulation under pinhole geometry. We evaluate \ours on WoodScape 2D detection (54.3 mAP) and SynWoodScapes BEV segmentation (65.1 mIoU), where it achieves state-of-the-art results on both benchmarks.
Rahul Ahuja, Mudit Jain, Bala Murali Manoghar Sai Sudhakar +4
Vision foundation models are capable of generalizing across 3-dimensional (3D) scenes with high-fidelity estimates; their empirical success can be attributed to training on large-scale datasets of perspective images. However, when transferred to wide field-of-view (FoV) images, such as those captured by fisheye cameras, they return erroneous outputs due to a covariate shift stemming from the radial distortion on the image pixels. We propose a method to generalize vision foundation models to fisheye cameras. The crux of our method lies in a set of learnable parameters, termed Distortion Extenders (DEX), that model the fisheye distortion coefficients and the distributional shift between fisheye and perspective images encoded in the latent space. By minimizing a self-supervised alignment loss, DEX transforms the latent embeddings of fisheye images to resemble those of perspective images to recover high-fidelity estimates. DEX is architecture- and task-agnostic: We demonstrate DEX on monocular depth estimation and open-vocabulary segmentation for convolution- and Transformer-based architectures, where we consistently improve over baselines across indoor and outdoor fisheye datasets. As a byproduct, the activations of DEX can also be decoded to distortion coefficients to support camera calibration. Code available at: https://github.com/Suchisrit/DEX.
Rit Gangopadhyay, Alex Wong
Yale Vision Laboratory, Yale University, New Haven, CT 06520, USA
Real-time object detectors achieve remarkable accuracy under controlled conditions, yet degrade sharply on non-ideal inputs: fisheye distortion, game-renderedcharacters, aerial viewpoints, and 360°panoramas. We present YOLOv14, a detection framework with four adaptive mechanisms designed for specific types of inputvariation:(1) Deformable Area-Attention with windowed computation and shiftedwindows for geometric distortion;(2) Multi-level Game2Real Alignment with progressive adversarial training for domain shift;(3) View-Aware Contrastive Learning with adaptive temperature for viewpoint invariance; and (4) Scene-Adaptive Augmentation with dynamic loss balancing for scene diversity. Together, YOLOv14 achieves 49.1 mAP on COCO val2017 at 2.91 ms (T4 GPU), and delivers substantial gains on fisheye (+4.1 mAP), panorama (+6.6 mAP), drone (+6.4 mAP), andour synthesized game-character benchmark (+26.1 mAP). We release code and models to facilitate reproducible research.
Jinling Jia, Jian Lu, Jone Yawl +1
Nanjing University of Posts and Telecommunications