Rethinking Camera Choice: An Empirical Study on Fisheye Camera Properties in Robotic Manipulation

Han Xue; Nan Min; Xiaotong Liu; Wendi Chen; Yuan Fang; Jun Lv; Cewu Lu; Chuan Wen

cs.ROMar 2, 2026

Rethinking Camera Choice: An Empirical Study on Fisheye Camera Properties in Robotic Manipulation

Authors: Han Xue, Nan Min, Xiaotong Liu, Wendi Chen, Yuan Fang, Jun Lv, Cewu Lu, Chuan Wen

Organizations: Shanghai Jiao Tong University · Southeast University · USTC · Shanghai Innovation Institute · Noematrix Ltd.

Abstract

The adoption of fisheye cameras in robotic manipulation, driven by their exceptionally wide Field of View (FoV), is rapidly outpacing a systematic understanding of their downstream effects on policy learning. This paper presents the first comprehensive empirical study to bridge this gap, rigorously analyzing the properties of wrist-mounted fisheye cameras for imitation learning. Through extensive experiments in both simulation and the real world, we investigate three critical research questions: spatial localization, scene generalization, and hardware generalization. Our investigation reveals that: (1) The wide FoV significantly enhances spatial localization, but this benefit is critically contingent on the visual complexity of the environment. (2) Fisheye-trained policies, while prone to overfitting in simple scenes, unlock superior scene generalization when trained with sufficient environmental diversity. (3) While naive cross-camera transfer leads to failures, we identify the root cause as scale overfitting and demonstrate that hardware generalization performance can be improved with a simple Random Scale Augmentation (RSA) strategy. Collectively, our findings provide concrete, actionable guidance for the large-scale collection and effective use of fisheye datasets in robotic learning. More results and videos are available on https://robo-fisheye.github.io/

Figures & tables

Figure 1: Overview of the four factors analyzed to address our Research Questions (RQs). We study: (a) Camera Model (fisheye vs. pinhole) as our primary comparison; (b) Scene Complexity (poor vs. rich) for spatial localization (RQ1); (c) Scene Diversity (1 vs. N scenes) for scene generalization (RQ2); and (d) Camera Parameters (varied intrinsics) for hardware generalization (RQ3).

Figure 2: The implementation pipeline of fisheye camera simulation in MuJoCo [ 48 ] .

Figure 3: Random Crop Augmentation (fixed scale) vs. Random Scale Augmentation (RSA) for cross-camera generalization.

Figure 4: The six tasks in simulation experiments.

Figure 5: (a) The real-world experiment setup, which includes changeable backgrounds for scene complexity (RQ1) and scene generalization (RQ2) experiments. (b) The three tasks in real-world experiments: Pick Cup , Fold Towel and Hang Chinese Knot .

Figure 6: Real-world performance of fisheye / pinhole cameras with different scene complexity ( feature-poor v.s. feature-rich ) in three tasks (RQ1).

Experimental Factors		Simulation Task Success Rate						Average
Camera	Feature	Square	Tool_Hang	Coffee	Threading	Assemly	Mug_clean	Average
Pinhole(Single)	Poor	0.40	0.52	0.36	0.04	0.14	0.40	0.31
Pinhole(Single)	Rich	0.48 (+0.08)	0.56 (+0.04)	0.34 (-0.02)	0.18 (+0.14)	0.12 (-0.02)	0.38 (-0.02)	0.34 (+0.03)
Fisheye(Single)	Poor	0.68	0.80	0.80	0.30	0.24	0.58	0.57
Fisheye(Single)	Rich	0.74 (+0.06)	0.84 (+0.04)	0.76 (-0.04)	0.56 (+0.26)	0.48 (+0.24)	0.60 (+0.02)	0.66 (+0.09)
Pinhole(Double)	Poor	0.50	0.44	0.26	0.22	0.44	0.40	0.38

Table 1: Simulation performance of fisheye / pinhole cameras with different scene complexity ( feature-poor v.s. feature-rich ) (RQ1). Performance in feature-rich backgrounds is shown with the absolute difference (in parentheses) compared to the feature-poor background baseline for the same camera.

Task	Camera	Feature	Trans. Err(cm) $\downarrow$	Rot. Err( ∘ ) $\downarrow$
Pick Cup	Pinhole	Poor	12.309	15.345
	Pinhole	Rich	5.367	7.612
	Fisheye	Poor	3.369	3.677
	Fisheye	Rich	2.362	3.394
Fold Towel	Pinhole	Poor	4.204	6.829
	Pinhole	Rich	5.329	6.434

Table 2: Quantitative probing of visual encoder spatial awareness using proprioception prediction as a proxy task (RQ1). We evaluate the quality of learned spatial representations by fine-tuning a lightweight MLP head on the pre-trained visual encoder to predict the robot’s proprioceptive state in three real-world tasks.

Figure 7: The unseen scenes for evaluation in (a) simulation and (b) real-world experiments (RQ2).

Figure 8: The policy performance improves with the number of training scenes in (a) simulation experiments on Coffee task and (b) real-world experiments on Pick Cup task (RQ2).

Figure 9: The failure cases of cross-hardware generalization (RQ3). The policy tends to overfit the absolute scale.

Figure 10: The policy performance under different unseen camera parameters , averaged across six tasks in simulation (RQ3).

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Trajectory Counts	Data Source	Data Type
Square	200	RoboMimic	PH
Tool Hang	200	RoboMimic	PH
Coffee	500	MimicGen	D1
Threading	500	MimicGen	D0
Assembly	500	MimicGen	D0
Mug Cleanup	500	MimicGen	D0

Appendix

Table S1 : Simulation Tasks Overview. PH: Proficient-Human datasets; D0: Default reset distribution; D1: Broadened reset distribution.

Figure S1 : Examples of simulation tasks. Each row illustrates sequential snapshots of a distinct task: (a) Square, grasp the yellow square block and insert it into the yellow target slot. (b) Tool Hang, insert the needle-shaped hook and hang the tool. (c) Coffee, place the coffee pod into the machine and close the lid. (d) Threading, grasp the needle and insert it into the pinhole. (e) Assembly, insert two irregular blocks in sequence. (f) Mug Cleanup, open the drawer, place the mug inside, and close it.

Figure S2 : Visualization of different cameras . The orange area represents the field of view of the pinhole wrist camera, and the blue area represents the field of view of the newly added symmetrically arranged camera. The upper part shows the images from the pinhole wrist camera, and the lower part shows the images from the double fisheye cameras, with blue and orange borders distinguishing the imaging effects corresponding to their respective fields of view.

Model Architecture
Config	Simulation	Real-World
Visual Backbone	ResNet-18 (No Pretrain)	CLIP ViT-B/16
Pooling Method	Spatial Softmax	Spatial Softmax
Denoising Network	Conditional U-Net1D	Conditional U-Net1D
Action Space	Relative Action	Relative Action
Action Horizon ( $T_{act}$ )	8	8

Appendix

Table S2 : Detailed hyperparameters. We report the specific settings used for Simulation (Sim) and Real-World (Real) experiments. The hyperparameter values are aligned with [ 4 ] and [ 18 ] .

Figure S3 : Scene comparison in RQ1. Visualization of sideview, pinhole, fisheye, dual-pinhole, and dual-fisheye cameras in poor (single dark) vs. rich (diverse elements) scenes.

Experimental Factors		Simulation Task Success Rate						Average
Camera	State	Square	Tool_Hang	Coffee	Threading	Assembly	Mug_Clean	Average
Pinhole(Single)	w/ State	0.82	0.68	0.38	0.72	0.46	0.66	0.62
Pinhole(Single)	w/o State	0.48 (-41%)	0.56 (-18%)	0.34 (-11%)	0.18 (-75%)	0.12 (-74%)	0.38 (-42%)	0.34 (-45%)
Fisheye(Single)	w/ State	0.86	0.88	0.88	0.72	0.58	0.60	0.75
Fisheye(Single)	w/o State	0.74 (-14%)	0.84 (-5%)	0.76 (-14%)	0.56 (-22%)	0.48 (-17%)	0.60 (0%)	0.66 (-12%)
Pinhole(Double)	w/ State	0.92	0.54	0.38	0.76	0.58	0.66	0.64

Appendix

Table S3 : Ablation study on proprioception input in simulation. We compare the success rates between ( w/ State ) and without ( w/o State ) proprioceptive input. Values in

(\cdot)

indicate the performance drop (or gain) relative to the baseline w/ State performance. Pinhole cameras show a sharp relative decline (e.g., -41% performance drop on average), whereas Fisheye cameras maintain robust performance (e.g., only -6% drop on average).

Experimental Factors		Real-World Normalized Score			Average
Camera	State	Pick Cup	Fold Towel	Hang Knot	Average
Pinhole	w/ State	0.75	0.37	0.45	0.52
Pinhole	w/o State	0.65 (-13%)	0.32 (-14%)	0.15 (-67%)	0.37 (-29%)
Fisheye	w/ State	0.98	0.92	0.70	0.87
Fisheye	w/o State	0.80 (-18%)	0.70 (-24%)	0.50 (-29%)	0.67 (-23%)

Appendix

Table S4 : Real-World ablation on proprioception. We report the Normalized Score in the feature-rich setting. Values in

(\cdot)

indicate the performance drop relative to the baseline w/ State performance. Notably, the Fisheye policy without proprioception (0.67) outperforms the Pinhole policy even with proprioception (0.52).

Config (Double Cam + 3rd)	Sqr	Tool	Cof	Thrd	Asm	Mug	Mean
Pinhole baseline	0.94	0.78	0.78	0.80	0.56	0.66	0.75
Fisheye ( Ours )	0.96	0.84	0.82	0.82	0.66	0.72	0.80
Improvement	+0.02	+0.06	+0.04	+0.02	+0.10	+0.06	+0.05

Appendix

Table S5 : Third-view Camera Ablation in Simulation.

Panel of Figure 21 — Figure S4 : Comprehensive visualization of scene diversity. We explicitly bridge the visual gap between simulation and real-world by ensuring high scene complexity in both domains. (a) In simulation, we randomize 32 distinct textures (wood, marble, fabric, tiles) onto background surfaces. (b) In the real world, we utilize a changeable background with visually distinct textured cloths to rigorously test spatial localization and generalization.

Figure S5 : Visualization of scene distribution in the generalization experiment. We visualize the scene distributions by clustering the CLIP embeddings of scene images with K-Means. The Red Stars highlight the Held-out Test Scenes .

Figure S6 : Generalization Performance on each scene. We visualize the performance scaling curves for each individual held-out test scene to investigate the consistency of generalization. The Dashed Lines represent the aggregated mean performance, while solid lines represent specific test environments.

Typical Test Scene	Feature Density	Score		Fisheye
(Real-world Variations)	(ORB [ 38 ] pts/frame)	Pinhole	Fisheye	Improvement
Textureless (Poor)	1299.35 $\pm$ 176.81 (Low)	0.1250	0.5250	+0.4000
Wooden Desk (Typical)	2111.43 $\pm$ 341.87 (Median)	0.1250	0.5500	+0.4250
Highly Textured (Rich)	$3574.99\pm 102.54$ (High)	0.1813	0.9875	+0.8062

Appendix

Table S6 : Performance in Different Real-world Environments.

Domain	Simulation (Success Rate)		Real-World (Mean Score)
Feat. Density (ORB [ 38 ] pts/frame)	$268.77\pm 30.27$		$3574.99\pm 102.54$
Encoder	ResNet-18	CLIP	ResNet-18	CLIP
Pinhole	0.4467	0.5333	0.4250	0.7000
Fisheye	0.7533	0.77	0.7000	0.8875

Appendix

Table S7 : Ablation Study on Visual Encoders.

Config Name	Method	Focal Length	Distortion	Scale
Seen Param	EUCM	$45$	$a\_:0.4$ $b\_:2.0$	$0.9$
Param 1	EUCM	$60$	$a\_:0.5$ $b\_:2.0$	$1.0$
Param 2	DS	$50$	$a\_:0.5$ $xi\_:0.1$	$1.0$
Param 3	EUCM	$45$	$a\_:0.4$ $b\_:2.0$	$1.0$
Param 4	EUCM	$45$	$a\_:0.4$ $b\_:2.5$	$1.0$
Param 5	EUCM	$35$	$a\_:0.4$ $b\_:1.2$	$1.0$

Appendix

Table S8 : Simulation camera parameters for RQ3. We define one seen configuration for training and distinct configurations for zero-shot evaluation. Variations in focal length and distortion simulate significant geometric shifts.

Figure S7 : Simulated fisheye effects using projection models with different distortion parameters. Each subfigure demonstrates the visual output when applying a distinct set of parameters: “Seen param” represents a baseline parameter set, while “Param1” to “Param5” illustrate variations in distortion intensity, field of view, and other projection characteristics, showcasing how different parameter configurations alter the fisheye rendering.

Figure S8 : Cross-Camera generalization performance in simulation for each task. We evaluate the zero-shot camera-transfer performance of policies trained on a single “Seen Param” configuration across several unseen camera settings. The Baseline (Blue) exhibits a sharp performance drop as camera parameters deviate from the training domain, indicating severe overfitting to absolute object scales. In contrast, our RSA method (Orange) demonstrates a broad generalization plateau, maintaining robust performance across a wide range of unseen camera parameters.

Figure S9 : Real-world Scale Sensitivity Analysis. To investigate the root cause of cross-camera transfer failure, we simulate geometric domain shifts by applying center crops with varying scale factors (

S

) to the fisheye input. The results show a characteristic “inverted-V” performance drop : the policy performs robustly near the training scale (

S=1.0

) but suffers catastrophic degradation as the scale deviates significantly (e.g.,

0\%

success rate at

S=0.7

). This confirms that the standard policy is highly sensitive to absolute object scale, validating the necessity of our Random Scale Augmentation (RSA) strategy.

Figure S10 : Qualitative Visualization of Scale-Induced Failures. We visualize the specific failure modes of the baseline policy under geometric scale shifts (simulated via center cropping). (Left) When zoomed in (Crop= $0.7$ ), the object appears larger, misleading the policy to perceive it as closer; consequently, the robot attempts to grasp at a shallower depth (undershooting the target). (Right) When zoomed out (Crop= $1.3$ ), the object appears smaller, causing the policy to perceive it as farther away; this leads to grasping at a deeper depth (overshooting or colliding with the target). (Middle) Accurate manipulation is only achieved at the training scale (Crop= $1.0$ ). These behaviors explicitly confirm that the cross-camera failure stems from the policy’s overfitting to the absolute pixel scale of objects. Please see the video in the supplementary files for more details.

Policy Model	Aug. Strategy	$S=0.70$	Param 1	$S=1.0$	$S=1.15$	$S=1.30$
		(Zoom-in)		(Seen)		(Zoom-out)
Diffusion Policy [ 4 ]	Standard Aug.	0.000	0.350	0.750	0.750	0.500
	RSA (Ours)	0.725	0.950	1.000	0.750	0.650
$\pi_{0.5}$ [ 2 ]	Standard Aug.	0.375	0.875	1.000	0.600	0.150
	RSA (Ours)	0.900	1.000	1.000	0.975	1.000

Appendix

Table S9 : Normalized scores for the “Pick and Place” task under simulated scale shifts. We compare Standard Aug. (Standard Augmentation) and RSA (Random Scale Augmentation) across different parameters (

S

Test Camera	FOV Angle	Induced Scale Shift	Baseline(Standard Aug.)	RSA (Ours)
Seen Camera	180 ∘	1.0 $\times$ (Seen)	1.0000	1.0000
Narrow Lens	150 ∘	$\sim$ 1.2 $\times$ (Zoom In)	0.5000	0.9500
Wide Lens	220 ∘	$\sim$ 0.8 $\times$ (Zoom Out)	0.0025	0.6000

Appendix

Table S10 : Real-world Cross-camera Generalization with RSA.

Explore similar work

Fisheye-VLA: Decoupling Coverage and Acuity for Manipulation with a Single Fisheye Camera
Sep 22, 2026Ziang Ren, Zike Yan, Raymond Zhang +2Robotic Manipulation Vision-Based Robot Control
Hong Kong Embodied AI Lab · The Chinese University of Hong Kong, Hong Kong SAR, China · DeepCybo +1
An Empirical Study on What Matters for Viewpoint-Generalizable Policies in Visual Imitation Learning
Sep 26, 2026Mino Nakura, Sriram Krishna, Yufei Wang +3Robot Imitation Learning Generalization in Robotic Manipulation
Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA.
One Demo is Worth a Thousand Trajectories: Action-View Augmentation for Visuomotor Policies
Jun 17, 2026Chuer Pan, Litian Liang, Dominik Bauer +4Data Augmentation Generalization in Robotic Manipulation
Stanford University · Columbia University · Toyota Research Institute

Sep 22, 2026J/K move · Enter open · S save

Fisheye-VLA: Decoupling Coverage and Acuity for Manipulation with a Single Fisheye Camera

Ziang Ren, Zike Yan, Raymond Zhang, Xuguo He+1

Hong Kong Embodied AI Lab · The Chinese University of Hong Kong, Hong Kong SAR, China · DeepCybo+1

Robotic Manipulation Vision-Based Robot Control Active Visual Perception

Open paper AskPDF arXiv

Manipulation requires both broad scene awareness and detailed local feedback, yet conventional camera rigs provide them through separate front and wrist cameras. We present Fisheye-VLA, a visual interface that brings these capabilities together using a single passive fisheye. A global view preserves the workspace, while local perspective crops direct detail toward the interaction. The key design question is where this local visual budget should go. We answer it through a controlled re-rendering study, comparing alternative crop directions on the same recorded observations. The study finds that end-effector-centered views capture most of the estimated benefit of a much larger candidate pool, motivating a compact allocation around both hands. Our interface uses calibrated end-effector projection and motion lead to track the crops, while a shared ray encoding preserves their spatial meaning as they move. Integrated with a pretrained VLA, it achieves 84% and 82% success in the two expanded tabletop regions, where some target placements extend beyond the front-camera coverage, and supports shelf and conveyor manipulation. Ablations show that local crops and their viewing directions become more important in the larger workspace regions. The results demonstrate that a single fisheye can support these manipulation tasks without physical wrist cameras.

Abstract

Explore similar work

Fisheye-VLA: Decoupling Coverage and Acuity for Manipulation with a Single Fisheye Camera

An Empirical Study on What Matters for Viewpoint-Generalizable Policies in Visual Imitation Learning

One Demo is Worth a Thousand Trajectories: Action-View Augmentation for Visuomotor Policies