Inspired by human vision, we introduce a framework using active gaze to enable fine-grained bimanual manipulation with only a single stereo camera. EyeRobot 2.0 physically attends to a 3D fixation point in the scene by swiveling two eye viewpoints to center their gaze on it. The resulting images are processed foveally by allocating more visual tokens to the image centers, focusing computation on task-relevant features. Such Active Visual Fixation (AVF) requires carefully coordinated gaze during task execution, which we accomplish hierarchically by first training a low-level gaze servoing policy conditioned on a goal object, then training a target selector which emits fixation goals based on task progress. Both modules are trained with RL on real-world data: the first is trained with a dense geometric reward and the second co-trains with the BC gripper policy which allows it to discover fixation sequences that can resemble a human's fixation sequence while performing the task. EyeRobot 2.0 further takes advantage of fixation by canonicalizing gripper information into a fixation-relative SE(3) frame, which compacts the size of the action distribution to learn. We collect teleoperation data for 7 real-world and 6 simulated tasks, and conduct over 1000 physical and 1800 simulated robot trials comparing EyeRobot 2.0 against passive stereo and ego + wrist camera policies trained on the same data. Removing wrist cameras is costly for standard policies: with only passive stereo, real-world success drops from 52% to 27%. EyeRobot 2.0 closes this gap with only stereo, outperforming passive stereo by 40% in real and 20% in sim. It matches ego + wrist policies when their wrist views are clear (69% vs. 64%), and more than doubles their success when grasped objects occlude the wrist cameras (48% vs. 22%)
Figures & tables
Figure 1: EyeRobot 2.0 enables fine-grained bimanual manipulation like inserting a straw into a boba cup using only a fixed stereo camera. It coordinates a sequence of gaze targets, for example here first at the boba cup, then the straw, then the insertion point. Gaze targets are output from a semantic object selector, which conditions a low-level gaze policy to servo precisely. Its design draws inspiration from human foveated vision by leveraging multi-resolution image crops to provide high detail where the policy gazes.
Figure 2: Gaze target outputs over an autonomous physical trial which picks up a marker cap and body, caps it, repositions a bag, inserts the marker, and zips the bag. The target selector outputs goals that are relevant to the current stage of the task: for example the policy looks at the marker when capping or the bag when repositioning. Wrist cameras are present but unused.
Figure 3: Learning gaze control . EyeRobot 2.0 trains a goal-conditioned gaze policy to continuously servo two eyes’ gaze in 3D. We train this behavior via RL, with a spatial reward derived from detecting the object and estimating its 3D centroid with stereo depth.
Figure 4: Sequencing fixation for manipulation. ( Left ) EyeRobot 2.0 trains a lightweight per-task target selector to switch between objects in the scene. This target selector is trained with RL to maximize the prediction accuracy of a gripper BC policy, which is co-trained from scratch at the same time on viewpoints generated from the fixation rollout. Action chunks are supervised with teleop demos, and negative prediction error is supplied as the reward for the selector to optimize. ( Right ) Action chunks are represented as SE(3) chunks in the eye frame, aligned with the current fixation point.
Figure 5: We collect teleop demonstrations for 7 real and 6 simulated tasks. Selected images show the raw perspective of the left stereo camera. Real-world tasks contain precise manipulation requirements, multi-stage behavior, articulated object interaction, and manipulation across the entire workspace. See the Appendix for a full breakdown of task stages and success criteria.
Figure 6: End-to-end success on real and sim tasks. Each real task is evaluated on 25 trials for each policy, all sim tasks over 100. Full per-stage conditional success is reported in the Appendix.
Figure 7: Wrist occlusion. When grasped objects block the wrist cameras, ego+wrist falls to near the stereo baseline while EyeRobot 2.0 maintains performance.
wrench
tea
tape
Policy
1st grasp
success
1st grasp
success
1st grasp
success
Ego-only
3/25
0/25
3/25
0/25
6/25
0/25
Ego + Wrist
10/25
4/25
11/25
4/25
14/25
0/25
EyeRobot 2.0
20/25
5/25
13/25
4/25
23/25
8/25
Table 1: Real-world performance with distractors. EyeRobot 2.0 is better able to ignore distractor objects because of its physical attention directed towards relevant objects.
Policy
boba
marker
pot
tea
toaster
tape
wrench
Mean
EyeRobot 2.0 − foveation
12/25
5/25
10/25
12/25
9/25
47/64
15/25
46.5%
EyeRobot 2.0 − stereo
10/25
8/25
10/25
15/25
6/25
51/64
16/25
48.5%
EyeRobot 2.0
11/25
17/25
13/25
20/25
13/25
60/64
19/25
66.5%
Table 2: Real-world ablations show that foveation and stereo inputs are both important.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Per-stage gaze selection in simulated evaluations . For each policy trained on sim data, we compute the mean gaze pointing error to each object in the scene, and consider the object with the lowest mean error as the “chosen” object during that stage. EyeRobot 2.0 reliably learns to pre-attend grasping, as evidenced by the fact it looks at the first object 100% of the time. In sim, it sometimes does not look at the second stage of the task, i.e., the place action, for example in the pot and spoon hanging tasks. The tiger task is omitted as it only has one stage.
Periphery (cm) ↓
Foveal (cm) ↓
Sim Tasks
4.73
3.51
Real Tasks
4.44
3.55
Appendix
Table 3: 3D fixation Euclidean error during fixation pretraining, averaged over the final 100 training iterations of the best seed per task.
Figure 9: Per-stage conditional success for EyeRobot 2.0 across 25 physical trials per task.
Figure 10: Per-stage conditional success for Ego-only gaze-free baseline across 25 physical trials per task.
Figure 11: Per-stage conditional success for Ego + Wrist baseline across 25 physical trials per task.
Figure 12: Baseline action space grid-search over all options in 5 simulated tasks. Note that faded bars are evaluated on the exact same positional distributions as the training set to understand train-test overfitting in the simulated environments. Main-paper results are reported with the “Test Set” distribution, which is a different random seed from teleop demonstrations. Across the board, EE-relative (rightmost in each section) outperforms other action spaces, so we use it for all real-world baselines. All baseline models exhibit signs of overfitting to the train positions on some tasks (particularly tape and plate ), even for tasks where teleop positions were randomly initialized.
Figure 13: Real tasks and their stages . The tasks that we study in EyeRobot 2.0 represent a suite of complex manipulation behaviors, including bimanual coordination (tape handover, grasping and opening a lid), fine-grained tolerance requirements (capping a marker, grasping a zipper, inserting a straw into a hole, inserting a marker into a pencil pouch), challenging visual scenarios (picking up a white teabag on a white table), and multi-stage tasks (6 for both marker and toaster oven tasks).
Figure 14: Simulated tasks and their stages . Simulated tasks represent a range of simple pick-place behaviors including single-arm (pot, spoon), single-arm-with-handover (tape), and bimanual coordination (tray, spoon).
Figure 15: Training distribution of objects in the training set. Each dot represents an actual computed ground truth 3D location of an object which has been projected onto the table surface for visualization. Shadows of the YAM arms have been provided for scale. The distribution of objects is approximately uniformly random, except in tape where the distribution is a grid.
Figure 16: Test distributions , visualized by average-compositing all 25 test overlay images on top of each other. During each evaluation we standardize reset positions across policies using these.
Figure 17: Subsample of distractor positions during the wrench task (25 total).
Simulation
Real
plate
tape
tiger
hang_spoon
medical_tray
pot_lid
tape
boba
wrench
marker_bag
pot_lid
tea
toaster
Total
# Demos
100
64
50
64
63
64
64
108
114
126
95
213
89
1278
Minutes
21.6
10.8
4.0
8.1
18.4
7.5
9.5
31.1
26.3
53.4
11.3
26.5
52.0
287.7
Appendix
Table 4: Per-task demonstration counts and total duration of demonstrations.
EyeRobot 2.0 (full)
EyeRobot 2.0 ( − fix. relative)
Object gaze (worst obj.)
Object gaze (best obj.)
EyeRobot 2.0 − stereo
EyeRobot 2.0 − foveation
0.74
0.53
0.50
0.69
0.74
0.72
Appendix
Table 5: Simulated Ablations . Getting rid of fixation-relative actions (“ − fix. relative”) severely harms performance even on relatively simple pick and place tasks, dropping by 21% overall. Restraining gaze to one constant object during training produces a 24% drop for the worst option, and even for the best constant-gaze choice results in a 5% reduction. Interestingly, both stereo and foveation ablations have little effect in simulated tasks, which we hypothesize is because they do not possess the visual complexity or fine-grained tolerance to require precise visual features or spatial understanding.
Policy
Subtask 1
Subtask 2
Subtask 3
Success
wrench
grab tab
open box
grab wrench
complete
Ego-only
3/25
0/25
0/25
0/25
Ego + Wrist
10/25
6/25
4/25
4/25
EyeRobot 2.0
20/25
7/25
5/25
5/25
tea
grab tea
tea in cup
–
complete
Ego-only
3/25
0/25
–
0/25
Appendix
Table 6: Task success with distractors. Each cell reports the number of trials out of 25 reaching the corresponding subtask. Success denotes end-to-end task completion. Bold indicates the best policy for each task-subtask.
Policy
Plate
Tape
Tiger
med
mn ± sd
max
med
mn ± sd
max
med
mn ± sd
max
BC ego-mono
54
52 ± 5
55
38
38 ± 5
41
76
78 ± 5
83
BC ego-stereo
41
41 ± 1
41
52
52 ± 4
55
86
86 ± 2
87
BC ego+wrist
71
71 ± 3
74
79
79 ± 3
81
96
95 ± 2
96
EyeRobot 2.0
67
67 ± 3
70
75
76 ± 2
79
97
97 ± 1
98
Appendix
Table 7: Full sim success statistics (%). Median, mean, and std are reported over 3 seeds for each policy/task combination. Each evaluation of a seed is done over 100 trials.
Vision-Language-Action (VLA) fine-tuning pairs images with actions at every step, yet typically provides only a task-level language instruction, leaving moment-to-moment visual relevance implicit. We introduce eye-tracker-supervised gaze prompting, which uses gaze recorded during VR teleoperation to provide frame-level visual guidance for VLA fine-tuning. During training, recorded gaze locations are rendered as crosshairs on the robot's head-camera images. At deployment, a lightweight predictor estimates gaze locations from recent images and the instruction, supplying the same type of visual prompt without an eye tracker or changes to the policy architecture. Instantiated with π0, gaze prompting increases mean success from 26.3% to 56.0% across six real-world bimanual manipulation tasks, with gains also observed when a single policy is trained on all six tasks. We release GazeMani, a dataset of 1,200 teleoperated trajectories with synchronized gaze.
Yihan Zhou, Rui Yan, Mingcong Li +4
Department of Automation, Tsinghua University · Beijing Key Laboratory of Embodied Intelligence Systems · Institute for Embodied Intelligence and Robotics, Tsinghua University +1
Large-scale real-world robot data collection is a prerequisite for bringing robots into everyday deployment. However, existing pipelines often rely on specialized handheld devices to bridge the embodiment gap, which not only increases operator burden and limits scalability, but also makes it difficult to capture the naturally coordinated perception-manipulation behaviors of human daily interaction. This challenge calls for a more natural system that can faithfully capture human manipulation and perception behaviors while enabling zero-shot transfer to robotic platforms. We introduce ActiveGlasses, a system for learning robot manipulation from ego-centric human demonstrations with active vision. A stereo camera mounted on smart glasses serves as the sole perception device for both data collection and policy inference: the operator wears it during bare-hand demonstrations, and the same camera is mounted on a 6-DoF perception arm during deployment to reproduce human active vision. To enable zero-transfer, we extract object trajectories from demonstrations and use an object-centric point-cloud policy to jointly predict manipulation and head movement. Across several challenging tasks involving occlusion and precise interaction, ActiveGlasses achieves zero-shot transfer with active vision, consistently outperforms strong baselines under the same hardware setup, and generalizes across two robot platforms.
Yanwen Zou, Chenyang Shi, Wenye Yu +5
Shanghai Jiao Tong University · Shanghai Innovation Institute · Noematrix Ltd.
Current Vision-Language-Action (VLA) models often struggle with high-precision robotic manipulation. We attribute this limitation primarily to their visual attention being dispersed across task-irrelevant regions. To address this issue, we propose ActGaze, a training approach that guides VLA policies to gaze on task-relevant regions, much like humans gaze on critical visual cues while executing precise movements. Unlike prior methods that rely on external labels for gaze supervision, ActGaze derives spatial supervision directly from the VLA's own action objective by using counterfactual visual interventions to identify regions that are critical for action prediction. Extensive real-robot experiments on four high-precision robotic manipulation tasks demonstrate that ActGaze induces more focused visual attention on task-relevant regions and consistently outperforms the base VLA policy and other visual-grounding approaches.
Jinxuan Zhu, Jiaheng Wang, Chao Tang +7
Samsung Robotics eXperience · National University of Singapore