cs.ROOct 2, 2026
SaveEyeRobot 2.0: Active Gaze for Precise Manipulation without Wrist Cameras
Organizations: UC Berkeley · Amazon FAR · Stanford University
Abstract
Inspired by human vision, we introduce a framework using active gaze to enable fine-grained bimanual manipulation with only a single stereo camera. EyeRobot 2.0 physically attends to a 3D fixation point in the scene by swiveling two eye viewpoints to center their gaze on it. The resulting images are processed foveally by allocating more visual tokens to the image centers, focusing computation on task-relevant features. Such Active Visual Fixation (AVF) requires carefully coordinated gaze during task execution, which we accomplish hierarchically by first training a low-level gaze servoing policy conditioned on a goal object, then training a target selector which emits fixation goals based on task progress. Both modules are trained with RL on real-world data: the first is trained with a dense geometric reward and the second co-trains with the BC gripper policy which allows it to discover fixation sequences that can resemble a human's fixation sequence while performing the task. EyeRobot 2.0 further takes advantage of fixation by canonicalizing gripper information into a fixation-relative SE(3) frame, which compacts the size of the action distribution to learn. We collect teleoperation data for 7 real-world and 6 simulated tasks, and conduct over 1000 physical and 1800 simulated robot trials comparing EyeRobot 2.0 against passive stereo and ego + wrist camera policies trained on the same data. Removing wrist cameras is costly for standard policies: with only passive stereo, real-world success drops from 52% to 27%. EyeRobot 2.0 closes this gap with only stereo, outperforming passive stereo by 40% in real and 20% in sim. It matches ego + wrist policies when their wrist views are clear (69% vs. 64%), and more than doubles their success when grasped objects occlude the wrist cameras (48% vs. 22%)
Figures & tables
Figure 1: EyeRobot 2.0 enables fine-grained bimanual manipulation like inserting a straw into a boba cup using only a fixed stereo camera. It coordinates a sequence of gaze targets, for example here first at the boba cup, then the straw, then the insertion point. Gaze targets are output from a semantic object selector, which conditions a low-level gaze policy to servo precisely. Its design draws inspiration from human foveated vision by leveraging multi-resolution image crops to provide high detail where the policy gazes.
Figure 2: Gaze target outputs over an autonomous physical trial which picks up a marker cap and body, caps it, repositions a bag, inserts the marker, and zips the bag. The target selector outputs goals that are relevant to the current stage of the task: for example the policy looks at the marker when capping or the bag when repositioning. Wrist cameras are present but unused.
Figure 3: Learning gaze control . EyeRobot 2.0 trains a goal-conditioned gaze policy to continuously servo two eyes’ gaze in 3D. We train this behavior via RL, with a spatial reward derived from detecting the object and estimating its 3D centroid with stereo depth.
Figure 4: Sequencing fixation for manipulation. ( Left ) EyeRobot 2.0 trains a lightweight per-task target selector to switch between objects in the scene. This target selector is trained with RL to maximize the prediction accuracy of a gripper BC policy, which is co-trained from scratch at the same time on viewpoints generated from the fixation rollout. Action chunks are supervised with teleop demos, and negative prediction error is supplied as the reward for the selector to optimize. ( Right ) Action chunks are represented as SE(3) chunks in the eye frame, aligned with the current fixation point.
Figure 5: We collect teleop demonstrations for 7 real and 6 simulated tasks. Selected images show the raw perspective of the left stereo camera. Real-world tasks contain precise manipulation requirements, multi-stage behavior, articulated object interaction, and manipulation across the entire workspace. See the Appendix for a full breakdown of task stages and success criteria.
Figure 6: End-to-end success on real and sim tasks. Each real task is evaluated on 25 trials for each policy, all sim tasks over 100. Full per-stage conditional success is reported in the Appendix.
Figure 7: Wrist occlusion. When grasped objects block the wrist cameras, ego+wrist falls to near the stereo baseline while EyeRobot 2.0 maintains performance.
| wrench | tea | tape | ||||
| Policy | 1st grasp | success | 1st grasp | success | 1st grasp | success |
| Ego-only | 3/25 | 0/25 | 3/25 | 0/25 | 6/25 | 0/25 |
| Ego + Wrist | 10/25 | 4/25 | 11/25 | 4/25 | 14/25 | 0/25 |
| EyeRobot 2.0 | 20/25 | 5/25 | 13/25 | 4/25 | 23/25 | 8/25 |
Table 1: Real-world performance with distractors. EyeRobot 2.0 is better able to ignore distractor objects because of its physical attention directed towards relevant objects.
| Policy | boba | marker | pot | tea | toaster | tape | wrench | Mean |
| EyeRobot 2.0 foveation | 12/25 | 5/25 | 10/25 | 12/25 | 9/25 | 47/64 | 15/25 | 46.5% |
| EyeRobot 2.0 stereo | 10/25 | 8/25 | 10/25 | 15/25 | 6/25 | 51/64 | 16/25 | 48.5% |
| EyeRobot 2.0 | 11/25 | 17/25 | 13/25 | 20/25 | 13/25 | 60/64 | 19/25 | 66.5% |
Table 2: Real-world ablations show that foveation and stereo inputs are both important.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Per-stage gaze selection in simulated evaluations . For each policy trained on sim data, we compute the mean gaze pointing error to each object in the scene, and consider the object with the lowest mean error as the “chosen” object during that stage. EyeRobot 2.0 reliably learns to pre-attend grasping, as evidenced by the fact it looks at the first object 100% of the time. In sim, it sometimes does not look at the second stage of the task, i.e., the place action, for example in the pot and spoon hanging tasks. The tiger task is omitted as it only has one stage.
| Periphery (cm) | Foveal (cm) | |
| Sim Tasks | 4.73 | 3.51 |
| Real Tasks | 4.44 | 3.55 |
Table 3: 3D fixation Euclidean error during fixation pretraining, averaged over the final 100 training iterations of the best seed per task.
Figure 9: Per-stage conditional success for EyeRobot 2.0 across 25 physical trials per task.
Figure 10: Per-stage conditional success for Ego-only gaze-free baseline across 25 physical trials per task.
Figure 11: Per-stage conditional success for Ego Wrist baseline across 25 physical trials per task.
Figure 12: Baseline action space grid-search over all options in 5 simulated tasks. Note that faded bars are evaluated on the exact same positional distributions as the training set to understand train-test overfitting in the simulated environments. Main-paper results are reported with the “Test Set” distribution, which is a different random seed from teleop demonstrations. Across the board, EE-relative (rightmost in each section) outperforms other action spaces, so we use it for all real-world baselines. All baseline models exhibit signs of overfitting to the train positions on some tasks (particularly tape and plate ), even for tasks where teleop positions were randomly initialized.
Figure 13: Real tasks and their stages . The tasks that we study in EyeRobot 2.0 represent a suite of complex manipulation behaviors, including bimanual coordination (tape handover, grasping and opening a lid), fine-grained tolerance requirements (capping a marker, grasping a zipper, inserting a straw into a hole, inserting a marker into a pencil pouch), challenging visual scenarios (picking up a white teabag on a white table), and multi-stage tasks (6 for both marker and toaster oven tasks).
Figure 14: Simulated tasks and their stages . Simulated tasks represent a range of simple pick-place behaviors including single-arm (pot, spoon), single-arm-with-handover (tape), and bimanual coordination (tray, spoon).
Figure 15: Training distribution of objects in the training set. Each dot represents an actual computed ground truth 3D location of an object which has been projected onto the table surface for visualization. Shadows of the YAM arms have been provided for scale. The distribution of objects is approximately uniformly random, except in tape where the distribution is a grid.
Figure 16: Test distributions , visualized by average-compositing all 25 test overlay images on top of each other. During each evaluation we standardize reset positions across policies using these.
Figure 17: Subsample of distractor positions during the wrench task (25 total).
| Simulation | Real | |||||||||||||
| plate | tape | tiger | hang_spoon | medical_tray | pot_lid | tape | boba | wrench | marker_bag | pot_lid | tea | toaster | Total | |
| # Demos | 100 | 64 | 50 | 64 | 63 | 64 | 64 | 108 | 114 | 126 | 95 | 213 | 89 | 1278 |
| Minutes | 21.6 | 10.8 | 4.0 | 8.1 | 18.4 | 7.5 | 9.5 | 31.1 | 26.3 | 53.4 | 11.3 | 26.5 | 52.0 | 287.7 |
Table 4: Per-task demonstration counts and total duration of demonstrations.
| EyeRobot 2.0 (full) | EyeRobot 2.0 ( fix. relative) | Object gaze (worst obj.) | Object gaze (best obj.) | EyeRobot 2.0 stereo | EyeRobot 2.0 foveation |
| 0.74 | 0.53 | 0.50 | 0.69 | 0.74 | 0.72 |
Table 5: Simulated Ablations . Getting rid of fixation-relative actions (“ fix. relative”) severely harms performance even on relatively simple pick and place tasks, dropping by 21% overall. Restraining gaze to one constant object during training produces a 24% drop for the worst option, and even for the best constant-gaze choice results in a 5% reduction. Interestingly, both stereo and foveation ablations have little effect in simulated tasks, which we hypothesize is because they do not possess the visual complexity or fine-grained tolerance to require precise visual features or spatial understanding.
| Policy | Subtask 1 | Subtask 2 | Subtask 3 | Success |
| wrench | grab tab | open box | grab wrench | complete |
| Ego-only | 3/25 | 0/25 | 0/25 | 0/25 |
| Ego + Wrist | 10/25 | 6/25 | 4/25 | 4/25 |
| EyeRobot 2.0 | 20/25 | 7/25 | 5/25 | 5/25 |
| tea | grab tea | tea in cup | – | complete |
| Ego-only | 3/25 | 0/25 | – | 0/25 |
Table 6: Task success with distractors. Each cell reports the number of trials out of 25 reaching the corresponding subtask. Success denotes end-to-end task completion. Bold indicates the best policy for each task-subtask.
| Policy | Plate | Tape | Tiger | ||||||
| med | mn sd | max | med | mn sd | max | med | mn sd | max | |
| BC ego-mono | 54 | 52 5 | 55 | 38 | 38 5 | 41 | 76 | 78 5 | 83 |
| BC ego-stereo | 41 | 41 1 | 41 | 52 | 52 4 | 55 | 86 | 86 2 | 87 |
| BC ego+wrist | 71 | 71 3 | 74 | 79 | 79 3 | 81 | 96 | 95 2 | 96 |
| EyeRobot 2.0 | 67 | 67 3 | 70 | 75 | 76 2 | 79 | 97 | 97 1 | 98 |
Table 7: Full sim success statistics (%). Median, mean, and std are reported over 3 seeds for each policy/task combination. Each evaluation of a seed is done over 100 trials.