GazeAct: A Human-Gaze-Annotated Manipulation Dataset for Attention-Regularized Policies

Anupam Pani; Wang Yuan; Qifan Wang; Yanchao Yang

cs.ROOct 17, 2026

GazeAct: A Human-Gaze-Annotated Manipulation Dataset for Attention-Regularized Policies

Authors: Anupam Pani, Wang Yuan, Qifan Wang, Yanchao Yang

Organizations: HKU Musketeers Foundation Institute of Data Science · Department of Electrical and Electronic Engineering, The University of Hong Kong

Abstract

Vision-language-action (VLA) policies must ground language instructions in the right objects, contact points, and target locations for reliable manipulation. Human gaze provides a natural signal for this grounding, but existing robot datasets rarely pair real human gaze with the multi-camera observations used by modern policies. We introduce GazeAct, a gaze-equipped real-robot manipulation dataset and training framework for multi-view VLA policies. Using screen-based teleoperation, we record gaze directly on the robot camera feed viewed by the operator, yielding 12000 demonstrations across 40 tasks and five domains with 300 Hz gaze synchronized with images, proprioception, and actions. We transfer gaze from the operator-selected view to the remaining synchronized cameras, producing spatially consistent targets across the policy's visual inputs. We then use these gaze-derived targets to regularize language-conditioned visual attention during training while keeping inference gaze-free. Across 18 real-world tasks, our method improves average success from 48.1% to 58.6%, with gains up to 25 percentage points. Ablations show that gaze works best as soft supervision and that real human gaze provides stronger attention targets than generic attention-shaping priors. The gains transfer to LIBERO-Spatial (57.9% to 69.6%) and different VLA backbones.These results show that gaze collected in the robot observation space can provide effective training-time targets for improving visual grounding in VLA manipulation.

Figures & tables

Fig. 1: Overview of GazeAct. Left: We collect teleoperation demonstrations with synchronized robot observations, proprioception, actions, and eye gaze recorded on the active robot camera view. Middle: Recorded gaze is first aligned to the remaining camera views and converted into patch-level gaze distributions. During training, these distributions regularize the policy’s language-conditioned visual attention toward task-relevant regions. Right: At inference, gaze is not required as a signal and the policy predicts actions from standard VLA inputs: images, language, and proprioception.

Fig. 2: Data collection and synchronization system. During ALOHA teleoperation, robot observations, proprioception, and actions are logged on the robot side, while a screen-based eye tracker records the operator’s gaze on the displayed camera feed. Hardware triggers define shared trial boundaries, and synchronized timestamps support post hoc alignment of gaze with images, proprioception, and actions.

Fig. 3: Cross-View Gaze Alignment for Multi-View Robot Observations. We visualize a teleoperated trajectory for the task “open the rice cooker and pour the contents of the bowl inside.” At each timestep, gaze is recorded only on the active camera view (blue edges); our cross-view alignment propagates it to the remaining synchronized views (orange edges). The overlays show that the propagated gaze remains spatially consistent across viewpoints and task-relevant regions.

Fig. 4: Attention Regularization. Top: the transformer-based policy processes language and image patch tokens, producing visual attention over robot observations while learning to predict actions. Bottom: during training, recorded teleoperation gaze is converted into patch-level heatmaps and used as an auxiliary signal to guide the policy’s visual attention toward human-fixated, task-relevant regions. At inference, gaze is not required and the policy predicts actions from standard VLA inputs.

Dataset	Recorded Human Gaze	Robot-View Gaze	Multi-View Gaze	Gaze Freq.	Gaze Acq.	Multi-View Obs.	Demos	Tasks	Avg. Dur. (s)	Participant	Set.
HARMONIC [ 7 ]	✓	✗	✗	$120$ Hz	HMD	✗	$480$	$1$	$37.5$	$24$	Real
RoboTurk-SIM [ 29 ]	✗	✗	✗	–	–	✓	$3{,}224$	$3$	$153.5$	n/a	Sim
RoboTurk-Real [ 30 ]	✗	✗	✗	–	–	✓	$2{,}144$	$3$	$187.2$	$54$	Real
BridgeData V2 [ 5 ]	✗	✗	✗	–	–	✓	$60{,}096$	$13$	$7.6$	n/a	Real
DROID [ 6 ]	✗	✗	✗	–	–	✓	$76{,}000$	$86$	$16.6$	$50$	Real
RT-1 [ 4 ]	✗	✗	✗	–	–	✓	$130{,}000$	$744$	n/a	n/a	Real

TABLE I: Comparison of GazeAct with representative robot manipulation and gaze-augmented datasets. Recorded Human Gaze denotes whether real gaze is collected; Robot-View Gaze whether gaze is recorded on robot camera observations; Multi-View Gaze whether gaze is available across multiple camera views; and Multi-View Obs. whether the dataset includes multi-view robot observations. Gaze Freq. , Gaze Acq. , and Set. denote eye-tracker frequency, acquisition setup, and data setting (Real or Simulated).

	Standard		Robustness		Overall		Gain
Task	Base	Ours	Base	Ours	Base [95% CI]	Ours [95% CI]	pp
Pick cup and place on plate	25/50	37/50	14/50	27/50	39 [30.0,48.8]	64 [54.2,72.7]	+25
Pick cup and place inside container	36/50	40/50	30/50	33/50	66 [56.3,74.5]	73 [63.6,80.7]	+7
Pick up towel and place in basket	42/50	47/50	36/50	43/50	78 [68.9,85.0]	90 [82.6,94.5]	+12
Pick up cube and place on plate	40/50	44/50	33/50	38/50	73 [63.6,80.7]	82 [73.3,88.3]	+9
Open rice cooker	34/50	41/50	21/50	30/50	55 [45.3,64.4]	71 [61.5,79.0]	+16

TABLE II: Expanded real-world evaluation with

100

rollouts per task and method. Standard and robustness columns report successes out of

50

trials; overall columns report successes out of

100

trials with

95\%

Wilson confidence intervals. Improvement is the absolute percentage-point gain in overall success.

Task Instruction	Base	Ours
	$\lambda{=}0$	0.001	0.01	10
Pick cup and place on plate	39	64	42	24
Pick cup and place inside container	66	73	63	30
Pick up towel and place in basket	78	90	72	40
Pick up cube and place on plate	73	82	77	56
Pick up injection place on tray	46	61	41	33

TABLE III: Top: varying the regularization strength

\lambda

shows that gaze-derived target is most effective as a soft prior. Bottom: comparing supervision sources shows that real human gaze provides the strongest attention targets.

Fig. 5: Attention Visualization. Qualitative attention maps for rice-cooker and injection-tray rollouts. Warmer regions indicate higher language-conditioned visual attention. Compared with the baseline, the attention-regularized policy places more attention on actionable regions, including the rice-cooker button and the injection tool, while using no gaze input at inference.

	$\pi_{0.5}$		OpenVLA
Task	Base	Ours	Base	Ours
Open rice cooker	55	71	47	60
Transfer tennis balls	15	28	10	17
Place towel in basket	78	90	63	77
Pick cube, place on plate	73	82	57	70
Pick cup, place on plate	39	64	51	60

TABLE IV: Cross-architecture evaluation on real-world tasks. Success rates (%) for

\pi_{0.5}

and OpenVLA with and without attention regularization. Best result per task and backbone is in bold .

Object Location	Base	Ours
Between plate and ramekin	83.0	90.3
Next to ramekin	73.0	84.0
Table center	83.0	90.7
On cookie box	64.0	74.0
In cabinet drawer	62.0	72.0
On ramekin	36.0	50.0

TABLE V: LIBERO-Spatial success rates (%) at

30

k steps. Real human gaze as supervision for attention regularization improves performance across all spatial configurations.

Spatial Grounding for VLA Models

See all 26 papers

ActGaze: Learning Action-Grounded Gaze through Counterfactual Visual Interventions for High-Precision Manipulation
Sep 24, 2026Jinxuan Zhu, Jiaheng Wang, Chao Tang +7Visuomotor Policy Learning Robotic Manipulation
Samsung Robotics eXperience · National University of Singapore
Gaze Prompts: Temporally Dense Human Attention for Vision-Language-Action Fine-Tuning
Sep 28, 2026Yihan Zhou, Rui Yan, Mingcong Li +4Visuomotor Policy Learning Robotic Manipulation
Department of Automation, Tsinghua University · Beijing Key Laboratory of Embodied Intelligence Systems · Institute for Embodied Intelligence and Robotics, Tsinghua University +1
Gaze2Act: Gaze-Conditioned Vision-Language-Action Policies for Interactive Robot Manipulation
May 28, 2026Kuangji Zuo, Gen Li, Bofan Lyu +9Language-Conditioned Robot Manipulation Robotic Manipulation
1MARS Lab, Nanyang Technological University

Sep 24, 2026J/K move · Enter open · S save

ActGaze: Learning Action-Grounded Gaze through Counterfactual Visual Interventions for High-Precision Manipulation

Jinxuan Zhu, Jiaheng Wang, Chao Tang, Mengfan Wang+6

Samsung Robotics eXperience · National University of Singapore

Visuomotor Policy Learning Robotic Manipulation Vision-Language-Action Models

Open paper AskPDF arXiv

Current Vision-Language-Action (VLA) models often struggle with high-precision robotic manipulation. We attribute this limitation primarily to their visual attention being dispersed across task-irrelevant regions. To address this issue, we propose ActGaze, a training approach that guides VLA policies to gaze on task-relevant regions, much like humans gaze on critical visual cues while executing precise movements. Unlike prior methods that rely on external labels for gaze supervision, ActGaze derives spatial supervision directly from the VLA's own action objective by using counterfactual visual interventions to identify regions that are critical for action prediction. Extensive real-robot experiments on four high-precision robotic manipulation tasks demonstrate that ActGaze induces more focused visual attention on task-relevant regions and consistently outperforms the base VLA policy and other visual-grounding approaches.

Abstract

Spatial Grounding for VLA Models

Explore similar work

ActGaze: Learning Action-Grounded Gaze through Counterfactual Visual Interventions for High-Precision Manipulation

Gaze Prompts: Temporally Dense Human Attention for Vision-Language-Action Fine-Tuning

Gaze2Act: Gaze-Conditioned Vision-Language-Action Policies for Interactive Robot Manipulation