Despite advances in Vision-Language-Action (VLA) models, robotic manipulation struggles with fine-grained tasks because current models lack mechanisms for active visual attention allocation. Human gaze naturally encodes intent, planning, and execution patterns -- offering a powerful supervisory signal for guiding robot perception. We introduce a gaze-regularized training framework that aligns VLA models' internal attention with human visual patterns without architectural modifications or inference-time overhead. Our method transforms temporally aggregated gaze heatmaps into patch-level distributions and regularizes the transformer's attention through KL divergence, creating an inductive bias toward task-relevant features while preserving deployment efficiency. When integrated into existing VLA architectures, our approach yields 4-12% improvements across manipulation benchmarks. The gaze-regularized models reach equivalent performance with fewer training steps and maintain robustness under lighting variations and sensor noise. Beyond performance metrics, the learned attention patterns produce interpretable visualizations that mirror human strategies, enhancing trust in robotic systems. Moreover, our framework requires no eye-tracking equipment and applies directly to existing datasets. These results demonstrate that human perceptual priors can significantly accelerate robot learning while improving both task performance and system interpretability.
Figures & tables
Figure 1: Effect of Gaze Regularization. The baseline (middle) exhibits scattered attention across the scene, while the gaze-regularized model (right) concentrates on task-relevant regions (the plate and its immediate surroundings). This focused attention pattern not only improves task performance but also provides interpretable visual grounding that enhances trust in the model.
Figure 2: Overview of the Proposed Gaze-Regularized VLA Framework. Left: During training, gaze priors are converted into patch-level gaze distributions that match the transformer’s attention resolution. The KL divergence between gaze and model attention is minimized, guiding the model to align its visual focus with human fixation patterns over time. Right: During inference, the policy operates without any gaze input. Visual, language, and proprioceptive tokens are processed by the vision–language backbone and action head, and fused through causal attention to produce action features, which are mapped by the action decoder to control outputs. This training-time regularization yields gaze-aligned internal representations while maintaining a lightweight, gaze-free inference pipeline.
Figure 3: Temporally Aggregated Gaze Prior Generation. A sequence of k video frames is tokenized and processed by the GLC [ 25 ] module, which predicts per-frame gaze heatmaps using both past and future context. These heatmaps are temporally aggregated to yield a gaze distribution that captures attention over time and serves as the supervision signal for training- time regularization.
Location of Object
w Gaze
w/o Gaze
10k
20k
30k
10k
20k
30k
Between plate and ramekin
73.3
80
100
70
76.7
83.3
Next to ramekin
60.3
71.3
100
50
63.3
85.7
Table center
76.7
80
100
70
80
100
On cookie box
63.3
70.3
91.3
76.7
80
100
In cabinet drawer
60
70
73.3
43.3
50
80
Table 1: Per-task success rates on LIBERO Spatial [ 29 ] The model significantly performs better with spatially-modulated attention.
w Gaze
w/o Gaze
Δ Improvement
Dataset
10k
20k
30k
10k
20k
30k
10k
20k
30k
Libero Suite
LIBERO-Spatial
59.3
70.2
95.5
53.2
62.6
85.9
↑ 6.1
↑ 7.6
↑ 9.6
LIBERO-Object
76.4
86.1
97.3
69.5
81.2
91.7
↑ 6.9
↑ 4.9
↑ 5.6
LIBERO-Goal
72.8
83.5
92.6
66.9
77.4
84.3
↑ 5.9
↑ 6.1
↑ 8.3
LIBERO-10
41.7
58.3
77.9
42.5
53.8
66.1
↓ 0.8
↑ 4.5
↑ 11.8
Table 2: Comparison on LIBERO [ 29 ] and ALOHA Suites [ 52 ] (Success Rate %). Each value reports the mean success rate over three seeds at different training steps. Columns on the right show the improvement of the Gaze-Regularized Model over the Base Model.
Dataset
w/o Gaze
w Gaze
Δ Improvement
LIBERO-Spatial
76.0
82.2
↑ 6.2
LIBERO-Object
79.5
86.1
↑ 6.6
LIBERO-Goal
72.5
76.8
↑ 4.3
LIBERO-10
45.9
51.5
↑ 5.6
Overall Avg.
68.5
74.2
↑ 5.7
Table 3: Comparison of Base and Gaze-Regularized models with OpenVLA [ 23 ] . Our proposed method achieves higher performance even under a different architectural setup.
Task
Steps
Base Model
Gaze Model
Place the cube
20,000
4%
6%
on the plate
40,000
32%
44%
Pick cup and place
20,000
24%
28%
it in container
40,000
64%
72%
Pick multiple cups
20,000
5%
5%
and place in container
40,000
30%
40%
Table 4: Comparison of Base and Gaze-Regularized models on real-world tasks using Pi-0.
Regularization Scale
Suite
Baseline
Low
Moderate
High
LIBERO-Spatial
85.9
95.5
88.4
44.2
LIBERO-Object
91.7
97.3
90.8
50.6
LIBERO-Goal
84.3
92.6
85.1
41.7
LIBERO-10
66.1
77.9
64.4
30.1
Overall Avg.
82
90.8
82.2
41.6
Table 5: Effect of gaze-regularization strength ( λ ) on task success rate (%). The model shows improved performance when gaze is used as a soft prior rather than a hard constraint.
Perturbation
LIBERO-Spatial
LIBERO-Object
LIBERO-Goal
w/o Gaze
w Gaze
w/o Gaze
w Gaze
w/o Gaze
w Gaze
No Perturbation
85.9
95.5
91.7
97.3
84.3
92.6
Lighting Variation
77.2
89.1
84.5
92.8
80.4
89.7
Camera Noise
82.1
91.3
85.8
93.5
79.6
88.9
Table 6: Performance comparison under different visual perturbations when noise or visual degradations are simulated. Results are reported across three LIBERO benchmarks [ 29 ] .
Symbol
Description
t
Timestep index of the current observation
i
Camera/view index
Nv
Number of visual tokens (e.g., 16×16=256 )
P
Patch grid dimension (e.g., P=16 )
Xl∈RNl×d
Language token sequence
Xvi∈RNv×d
Visual tokens from camera view i
Table 7: Summary of key notations used in gaze-to-attention regularization and VLA token interactions.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Location of Object
w Gaze
DeepGaze
w/o Gaze
Uniform
30k
30k
30k
30k
Between plate and ramekin
100
85.7
83.3
69.7
Next to ramekin
100
86.7
85.7
59.7
Table center
100
100
100
80.3
On cookie box
91.3
100
100
79.3
In cabinet drawer
73.3
82.0
80
39.3
Appendix
Table 8: Per-task success rates on LIBERO Spatial [ 29 ] at 30k training steps. We compare the baseline model, our gaze-regularized model, a DeepGaze-based gaze variant, and a uniform-distribution variant.
Figure 4: Closer look at Gaze Prior Generation A sequence of k video frames is tokenized and processed by the GLC [ 25 ] module, where it utilizes global tokens (derived from the sequence) and local tokens, and undergoes self attention as well as Global-Local Correlation to then predict per-frame gaze heatmaps. These heatmaps are temporally aggregated to yield a gaze distribution that captures attention over time and serves as the supervision signal for training- time regularization.
Figure 5: Additional Visualisations of Attention. Given the input observation, we show the spatial attention from the baseline model (second), the attention obtained when a perturbed gaze variant is used (third, corresponding to Table 8 ), and finally the sharper, task-relevant attention produced by our gaze-regularized model (fourth).
Location of Object
w Gaze (All Layers)
10k
20k
30k
Between plate and ramekin
65.0
75.0
90.3
Next to ramekin
55.0
70.0
89.7
Table center
70.0
85.0
100.0
On cookie box
58.3
65.0
70.3
In cabinet drawer
43.3
56.3
60.7
Appendix
Table 9: Per-task success rates on LIBERO Spatial with regularization applied to all layers. The model shows competitive performance with comprehensive regularization.
Figure 6: Reliability of Synthetic Gaze on Simulation Videos Given the input task, we show the the predicted gaze is accurate and even on similar visual settings, produces different gaze results depending on the language instruction. The model utilizes a temporal sequence of frames, rather than a single frame, and then computes the gaze prediction thus the prediction occurs due to the conditioning through the global context by operating on a sequence of frames
Location of Object
w Gaze (Distractors)
w/o Gaze (Distractors)
30k
30k
Between plate and ramekin
96.7
78.3
Next to ramekin
95.0
80.0
Table center
97.0
96.7
On cookie box
90.0
96.7
In cabinet drawer
70.0
70.0
Appendix
Table 10: Per-task success rates on LIBERO Spatial under prompt distractors (e.g., replacing “pick” with “grab”, “lift”, etc.). Both the baseline and gaze-regularized models exhibit performance degradation, but the gaze model remains more robust.
Location of Object
w/o Gaze
w Gaze
Foveated
30k
30k
30k
Between plate and ramekin
83.3
100
80.0
Next to ramekin
85.7
100
81.3
Table center
100
100
95.7
On cookie box
100
91.3
90.0
In cabinet drawer
80
73.3
65.3
Appendix
Table 11: Per-task success rates on LIBERO Spatial [ 29 ] at 30k training steps. We compare the baseline model, our gaze-regularized model, and a foveated-vision variant.
Figure 7: Additional Visualisations of Attention. Given the input observation (first), we show the spatial attention from the baseline model (second) and task-relevant attention produced by our gaze-regularized model (third).
Location of Object
w Gaze
w/o Gaze
30k
30k
Between plate and ramekin
90.3
81.3
Next to ramekin
80.7
71.0
Table center
90.7
81.7
On cookie box
70.7
62.0
In cabinet drawer
69.3
60.7
Appendix
Table 12: Per-task success rates on the Missing Views experiment at 30k training steps. The gaze-regularized model consistently outperforms the baseline across all spatial configurations.
Figure 8: Visualization of Real-world Task on Aloha Robot In the figure, we provide some frames from a real world task performed using our gaze-regularized policy to show that our method works outside of simulation as well. Here, the task is to pick up the cube and place it on the correct plate.
Location of Object
w/o Gaze
w Gaze
Human Gaze
30k
30k
30k
Between plate and ramekin
83.3
100
100
Next to ramekin
85.7
100
100
Table center
100
100
100
On cookie box
100
91.3
89.3
In cabinet drawer
80
73.3
78.3
Appendix
Table 13: Per-task success rates on LIBERO Spatial [ 29 ] at 30k training steps. We compare the baseline model, our gaze-regularized model, and the human-gaze-trained variant.
Figure 9: Visualization of Real-world Task on Aloha Robot In this figure, we present a short horizon task of picking up a cup and placing it in a container(top) and also another longer horizon task to pick up multiple cups one-by-one, and place them in the container. Both visualisations are obtained using our gaze-regularized policy, highlighting its working functionality even in real-world scenarios
Figure 10: Visualization Results In the figure, we provide some visualization results to show how the policy performs on the Libero-Spatial [ 29 ] task suites. We provide the task instructions, and some important frames to show the task success. The baseline model performs admirably, but our method enhances the results by using gaze-regularization.
Z←∑x=1Hg∑y=1WgH(x,y).
Appendix
Algorithm 1 Heatmap-to-Token Projection
{(I1:n,t,ℓt,qt,At∗)}b=1B.
Appendix
Algorithm 2 Training Loop with Gaze Regularization
Figure 11: Failure Case. We show a failure example from the Libero-Spatial [ 29 ] task suite. In this task, the baseline model outperforms the gaze-regularized model, suggesting that stronger or more accurate gaze priors could further improve reliability. The bottom sequence illustrates the failure case where the robot hand fails to grab the bowl in the top drawer and proceeds to carry out the intended action.
Figure 12: Attention Comparison. The baseline model displays diffuse attention spread across the scene, with a single sharp point that is largely task-irrelevant. In contrast, the gaze-regularized model produces noticeably sharper, more concentrated, and consistently task-relevant attention, leading to clearer visual grounding for the instructed action.
Figure 13: Visualisation during a failure case. In this figure, we provide a visualisation of attention during a specific case of failure, where it can be seen that even though the task is to pick up the bowl, attention is not properly distributed on the bowl but rather than on the cabinet handle. Such cases can be mitigated using a better predictor or using a model trained with human supervision on simulated videos
Current Vision-Language-Action (VLA) models often struggle with high-precision robotic manipulation. We attribute this limitation primarily to their visual attention being dispersed across task-irrelevant regions. To address this issue, we propose ActGaze, a training approach that guides VLA policies to gaze on task-relevant regions, much like humans gaze on critical visual cues while executing precise movements. Unlike prior methods that rely on external labels for gaze supervision, ActGaze derives spatial supervision directly from the VLA's own action objective by using counterfactual visual interventions to identify regions that are critical for action prediction. Extensive real-robot experiments on four high-precision robotic manipulation tasks demonstrate that ActGaze induces more focused visual attention on task-relevant regions and consistently outperforms the base VLA policy and other visual-grounding approaches.
Jinxuan Zhu, Jiaheng Wang, Chao Tang +7
Samsung Robotics eXperience · National University of Singapore
Vision-language-action (VLA) policies must ground language instructions in the right objects, contact points, and target locations for reliable manipulation. Human gaze provides a natural signal for this grounding, but existing robot datasets rarely pair real human gaze with the multi-camera observations used by modern policies. We introduce GazeAct, a gaze-equipped real-robot manipulation dataset and training framework for multi-view VLA policies. Using screen-based teleoperation, we record gaze directly on the robot camera feed viewed by the operator, yielding 12000 demonstrations across 40 tasks and five domains with 300 Hz gaze synchronized with images, proprioception, and actions. We transfer gaze from the operator-selected view to the remaining synchronized cameras, producing spatially consistent targets across the policy's visual inputs. We then use these gaze-derived targets to regularize language-conditioned visual attention during training while keeping inference gaze-free. Across 18 real-world tasks, our method improves average success from 48.1% to 58.6%, with gains up to 25 percentage points. Ablations show that gaze works best as soft supervision and that real human gaze provides stronger attention targets than generic attention-shaping priors. The gains transfer to LIBERO-Spatial (57.9% to 69.6%) and different VLA backbones.These results show that gaze collected in the robot observation space can provide effective training-time targets for improving visual grounding in VLA manipulation.
Anupam Pani, Wang Yuan, Qifan Wang +1
HKU Musketeers Foundation Institute of Data Science · Department of Electrical and Electronic Engineering, The University of Hong Kong
Vision-Language-Action (VLA) models have recently shown strong potential for robot learning by following language instructions. However, in practice, language alone is often insufficient to precisely convey human intent. It is difficult to describe which exact object to interact with among similar candidates, where to act on the object, or how the target may change during execution. To address this limitation, we propose Gaze2Act, a novel VLA framework that leverages human gaze as a dynamic and intuitive intent signal for complex interactive manipulation. Gaze2Act first bridges the ego-exo view gap by mapping first-person gaze into the robot's perspective through cross-view semantic matching, producing both an object mask and a gaze point for coarse-to-fine target specification. These cues are then integrated into the policy through perception-level prompting and action-level conditioning, allowing the robot to attend to relevant regions and execute precise interactions under dynamic intent. In a systematic evaluation across seven task categories and 16 real-robot tasks on a Unitree G1 humanoid, Gaze2Act achieves state-of-the-art performance in both intent accuracy and task success rate. It notably outperforms baselines in object disambiguation, fine-grained interaction, and dynamic intent steering. These results demonstrate that human gaze provides a natural, low-burden, and highly expressive modality for human-in-the-loop VLA control.