Vision-language-action (VLA) policies must ground language instructions in the right objects, contact points, and target locations for reliable manipulation. Human gaze provides a natural signal for this grounding, but existing robot datasets rarely pair real human gaze with the multi-camera observations used by modern policies. We introduce GazeAct, a gaze-equipped real-robot manipulation dataset and training framework for multi-view VLA policies. Using screen-based teleoperation, we record gaze directly on the robot camera feed viewed by the operator, yielding 12000 demonstrations across 40 tasks and five domains with 300 Hz gaze synchronized with images, proprioception, and actions. We transfer gaze from the operator-selected view to the remaining synchronized cameras, producing spatially consistent targets across the policy's visual inputs. We then use these gaze-derived targets to regularize language-conditioned visual attention during training while keeping inference gaze-free. Across 18 real-world tasks, our method improves average success from 48.1% to 58.6%, with gains up to 25 percentage points. Ablations show that gaze works best as soft supervision and that real human gaze provides stronger attention targets than generic attention-shaping priors. The gains transfer to LIBERO-Spatial (57.9% to 69.6%) and different VLA backbones.These results show that gaze collected in the robot observation space can provide effective training-time targets for improving visual grounding in VLA manipulation.
Figures & tables
Fig. 1: Overview of GazeAct. Left: We collect teleoperation demonstrations with synchronized robot observations, proprioception, actions, and eye gaze recorded on the active robot camera view. Middle: Recorded gaze is first aligned to the remaining camera views and converted into patch-level gaze distributions. During training, these distributions regularize the policy’s language-conditioned visual attention toward task-relevant regions. Right: At inference, gaze is not required as a signal and the policy predicts actions from standard VLA inputs: images, language, and proprioception.
Fig. 2: Data collection and synchronization system. During ALOHA teleoperation, robot observations, proprioception, and actions are logged on the robot side, while a screen-based eye tracker records the operator’s gaze on the displayed camera feed. Hardware triggers define shared trial boundaries, and synchronized timestamps support post hoc alignment of gaze with images, proprioception, and actions.
Fig. 3: Cross-View Gaze Alignment for Multi-View Robot Observations. We visualize a teleoperated trajectory for the task “open the rice cooker and pour the contents of the bowl inside.” At each timestep, gaze is recorded only on the active camera view (blue edges); our cross-view alignment propagates it to the remaining synchronized views (orange edges). The overlays show that the propagated gaze remains spatially consistent across viewpoints and task-relevant regions.
Fig. 4: Attention Regularization. Top: the transformer-based policy processes language and image patch tokens, producing visual attention over robot observations while learning to predict actions. Bottom: during training, recorded teleoperation gaze is converted into patch-level heatmaps and used as an auxiliary signal to guide the policy’s visual attention toward human-fixated, task-relevant regions. At inference, gaze is not required and the policy predicts actions from standard VLA inputs.
Dataset
Recorded Human Gaze
Robot-View Gaze
Multi-View Gaze
Gaze Freq.
Gaze Acq.
Multi-View Obs.
Demos
Tasks
Avg. Dur. (s)
Participant
Set.
HARMONIC [ 7 ]
✓
✗
✗
120 Hz
HMD
✗
480
1
37.5
24
Real
RoboTurk-SIM [ 29 ]
✗
✗
✗
–
–
✓
3,224
3
153.5
n/a
Sim
RoboTurk-Real [ 30 ]
✗
✗
✗
–
–
✓
2,144
3
187.2
54
Real
BridgeData V2 [ 5 ]
✗
✗
✗
–
–
✓
60,096
13
7.6
n/a
Real
DROID [ 6 ]
✗
✗
✗
–
–
✓
76,000
86
16.6
50
Real
RT-1 [ 4 ]
✗
✗
✗
–
–
✓
130,000
744
n/a
n/a
Real
TABLE I: Comparison of GazeAct with representative robot manipulation and gaze-augmented datasets. Recorded Human Gaze denotes whether real gaze is collected; Robot-View Gaze whether gaze is recorded on robot camera observations; Multi-View Gaze whether gaze is available across multiple camera views; and Multi-View Obs. whether the dataset includes multi-view robot observations. Gaze Freq. , Gaze Acq. , and Set. denote eye-tracker frequency, acquisition setup, and data setting (Real or Simulated).
Standard
Robustness
Overall
Gain
Task
Base
Ours
Base
Ours
Base [95% CI]
Ours [95% CI]
pp
Pick cup and place on plate
25/50
37/50
14/50
27/50
39 [30.0,48.8]
64 [54.2,72.7]
+25
Pick cup and place inside container
36/50
40/50
30/50
33/50
66 [56.3,74.5]
73 [63.6,80.7]
+7
Pick up towel and place in basket
42/50
47/50
36/50
43/50
78 [68.9,85.0]
90 [82.6,94.5]
+12
Pick up cube and place on plate
40/50
44/50
33/50
38/50
73 [63.6,80.7]
82 [73.3,88.3]
+9
Open rice cooker
34/50
41/50
21/50
30/50
55 [45.3,64.4]
71 [61.5,79.0]
+16
TABLE II: Expanded real-world evaluation with 100 rollouts per task and method. Standard and robustness columns report successes out of 50 trials; overall columns report successes out of 100 trials with 95% Wilson confidence intervals. Improvement is the absolute percentage-point gain in overall success.
Task Instruction
Base
Ours
λ=0
0.001
0.01
10
Pick cup and place on plate
39
64
42
24
Pick cup and place inside container
66
73
63
30
Pick up towel and place in basket
78
90
72
40
Pick up cube and place on plate
73
82
77
56
Pick up injection place on tray
46
61
41
33
TABLE III: Top: varying the regularization strength λ shows that gaze-derived target is most effective as a soft prior. Bottom: comparing supervision sources shows that real human gaze provides the strongest attention targets.
Fig. 5: Attention Visualization. Qualitative attention maps for rice-cooker and injection-tray rollouts. Warmer regions indicate higher language-conditioned visual attention. Compared with the baseline, the attention-regularized policy places more attention on actionable regions, including the rice-cooker button and the injection tool, while using no gaze input at inference.
π0.5
OpenVLA
Task
Base
Ours
Base
Ours
Open rice cooker
55
71
47
60
Transfer tennis balls
15
28
10
17
Place towel in basket
78
90
63
77
Pick cube, place on plate
73
82
57
70
Pick cup, place on plate
39
64
51
60
TABLE IV: Cross-architecture evaluation on real-world tasks. Success rates (%) for π0.5 and OpenVLA with and without attention regularization. Best result per task and backbone is in bold .
Object Location
Base
Ours
Between plate and ramekin
83.0
90.3
Next to ramekin
73.0
84.0
Table center
83.0
90.7
On cookie box
64.0
74.0
In cabinet drawer
62.0
72.0
On ramekin
36.0
50.0
TABLE V: LIBERO-Spatial success rates (%) at 30 k steps. Real human gaze as supervision for attention regularization improves performance across all spatial configurations.
Current Vision-Language-Action (VLA) models often struggle with high-precision robotic manipulation. We attribute this limitation primarily to their visual attention being dispersed across task-irrelevant regions. To address this issue, we propose ActGaze, a training approach that guides VLA policies to gaze on task-relevant regions, much like humans gaze on critical visual cues while executing precise movements. Unlike prior methods that rely on external labels for gaze supervision, ActGaze derives spatial supervision directly from the VLA's own action objective by using counterfactual visual interventions to identify regions that are critical for action prediction. Extensive real-robot experiments on four high-precision robotic manipulation tasks demonstrate that ActGaze induces more focused visual attention on task-relevant regions and consistently outperforms the base VLA policy and other visual-grounding approaches.
Jinxuan Zhu, Jiaheng Wang, Chao Tang +7
Samsung Robotics eXperience · National University of Singapore
Vision-Language-Action (VLA) fine-tuning pairs images with actions at every step, yet typically provides only a task-level language instruction, leaving moment-to-moment visual relevance implicit. We introduce eye-tracker-supervised gaze prompting, which uses gaze recorded during VR teleoperation to provide frame-level visual guidance for VLA fine-tuning. During training, recorded gaze locations are rendered as crosshairs on the robot's head-camera images. At deployment, a lightweight predictor estimates gaze locations from recent images and the instruction, supplying the same type of visual prompt without an eye tracker or changes to the policy architecture. Instantiated with π0, gaze prompting increases mean success from 26.3% to 56.0% across six real-world bimanual manipulation tasks, with gains also observed when a single policy is trained on all six tasks. We release GazeMani, a dataset of 1,200 teleoperated trajectories with synchronized gaze.
Yihan Zhou, Rui Yan, Mingcong Li +4
Department of Automation, Tsinghua University · Beijing Key Laboratory of Embodied Intelligence Systems · Institute for Embodied Intelligence and Robotics, Tsinghua University +1
Vision-Language-Action (VLA) models have recently shown strong potential for robot learning by following language instructions. However, in practice, language alone is often insufficient to precisely convey human intent. It is difficult to describe which exact object to interact with among similar candidates, where to act on the object, or how the target may change during execution. To address this limitation, we propose Gaze2Act, a novel VLA framework that leverages human gaze as a dynamic and intuitive intent signal for complex interactive manipulation. Gaze2Act first bridges the ego-exo view gap by mapping first-person gaze into the robot's perspective through cross-view semantic matching, producing both an object mask and a gaze point for coarse-to-fine target specification. These cues are then integrated into the policy through perception-level prompting and action-level conditioning, allowing the robot to attend to relevant regions and execute precise interactions under dynamic intent. In a systematic evaluation across seven task categories and 16 real-robot tasks on a Unitree G1 humanoid, Gaze2Act achieves state-of-the-art performance in both intent accuracy and task success rate. It notably outperforms baselines in object disambiguation, fine-grained interaction, and dynamic intent steering. These results demonstrate that human gaze provides a natural, low-burden, and highly expressive modality for human-in-the-loop VLA control.