Gaze-Regularized Vision-Language-Action Models for Robotic Manipulation

Anupam Pani; Yanchao Yang

cs.CVMar 24, 2026

Gaze-Regularized Vision-Language-Action Models for Robotic Manipulation

Authors: Anupam Pani, Yanchao Yang

Organizations: Institute of Data Science, University Of Hong Kong

Abstract

Despite advances in Vision-Language-Action (VLA) models, robotic manipulation struggles with fine-grained tasks because current models lack mechanisms for active visual attention allocation. Human gaze naturally encodes intent, planning, and execution patterns -- offering a powerful supervisory signal for guiding robot perception. We introduce a gaze-regularized training framework that aligns VLA models' internal attention with human visual patterns without architectural modifications or inference-time overhead. Our method transforms temporally aggregated gaze heatmaps into patch-level distributions and regularizes the transformer's attention through KL divergence, creating an inductive bias toward task-relevant features while preserving deployment efficiency. When integrated into existing VLA architectures, our approach yields 4-12% improvements across manipulation benchmarks. The gaze-regularized models reach equivalent performance with fewer training steps and maintain robustness under lighting variations and sensor noise. Beyond performance metrics, the learned attention patterns produce interpretable visualizations that mirror human strategies, enhancing trust in robotic systems. Moreover, our framework requires no eye-tracking equipment and applies directly to existing datasets. These results demonstrate that human perceptual priors can significantly accelerate robot learning while improving both task performance and system interpretability.

Figures & tables

Figure 1: Effect of Gaze Regularization. The baseline (middle) exhibits scattered attention across the scene, while the gaze-regularized model (right) concentrates on task-relevant regions (the plate and its immediate surroundings). This focused attention pattern not only improves task performance but also provides interpretable visual grounding that enhances trust in the model.

Figure 2: Overview of the Proposed Gaze-Regularized VLA Framework. Left: During training, gaze priors are converted into patch-level gaze distributions that match the transformer’s attention resolution. The KL divergence between gaze and model attention is minimized, guiding the model to align its visual focus with human fixation patterns over time. Right: During inference, the policy operates without any gaze input. Visual, language, and proprioceptive tokens are processed by the vision–language backbone and action head, and fused through causal attention to produce action features, which are mapped by the action decoder to control outputs. This training-time regularization yields gaze-aligned internal representations while maintaining a lightweight, gaze-free inference pipeline.

Figure 3: Temporally Aggregated Gaze Prior Generation. A sequence of $k$ video frames is tokenized and processed by the GLC [ 25 ] module, which predicts per-frame gaze heatmaps using both past and future context. These heatmaps are temporally aggregated to yield a gaze distribution that captures attention over time and serves as the supervision signal for training- time regularization.

Location of Object	w Gaze			w/o Gaze
Location of Object	10k	20k	30k	10k	20k	30k
Between plate and ramekin	73.3	80	100	70	76.7	83.3
Next to ramekin	60.3	71.3	100	50	63.3	85.7
Table center	76.7	80	100	70	80	100
On cookie box	63.3	70.3	91.3	76.7	80	100
In cabinet drawer	60	70	73.3	43.3	50	80

Table 1: Per-task success rates on LIBERO Spatial [ 29 ] The model significantly performs better with spatially-modulated attention.

	w Gaze			w/o Gaze			$\Delta$ Improvement
Dataset	10k	20k	30k	10k	20k	30k	10k	20k	30k
Libero Suite
LIBERO-Spatial	59.3	70.2	95.5	53.2	62.6	85.9	$\uparrow$ 6.1	$\uparrow$ 7.6	$\uparrow$ 9.6
LIBERO-Object	76.4	86.1	97.3	69.5	81.2	91.7	$\uparrow$ 6.9	$\uparrow$ 4.9	$\uparrow$ 5.6
LIBERO-Goal	72.8	83.5	92.6	66.9	77.4	84.3	$\uparrow$ 5.9	$\uparrow$ 6.1	$\uparrow$ 8.3
LIBERO-10	41.7	58.3	77.9	42.5	53.8	66.1	$\downarrow$ 0.8	$\uparrow$ 4.5	$\uparrow$ 11.8

Table 2: Comparison on LIBERO [ 29 ] and ALOHA Suites [ 52 ] (Success Rate %). Each value reports the mean success rate over three seeds at different training steps. Columns on the right show the improvement of the Gaze-Regularized Model over the Base Model.

Dataset	w/o Gaze	w Gaze	$\Delta$ Improvement
LIBERO-Spatial	76.0	82.2	$\uparrow$ 6.2
LIBERO-Object	79.5	86.1	$\uparrow$ 6.6
LIBERO-Goal	72.5	76.8	$\uparrow$ 4.3
LIBERO-10	45.9	51.5	$\uparrow$ 5.6
Overall Avg.	68.5	74.2	$\uparrow$ 5.7

Table 3: Comparison of Base and Gaze-Regularized models with OpenVLA [ 23 ] . Our proposed method achieves higher performance even under a different architectural setup.

Task	Steps	Base Model	Gaze Model
Place the cube	20,000	4%	6%
on the plate	40,000	32%	44%
Pick cup and place	20,000	24%	28%
it in container	40,000	64%	72%
Pick multiple cups	20,000	5%	5%
and place in container	40,000	30%	40%

Table 4: Comparison of Base and Gaze-Regularized models on real-world tasks using Pi-0.

		Regularization Scale
Suite	Baseline	Low	Moderate	High
LIBERO-Spatial	85.9	95.5	88.4	44.2
LIBERO-Object	91.7	97.3	90.8	50.6
LIBERO-Goal	84.3	92.6	85.1	41.7
LIBERO-10	66.1	77.9	64.4	30.1
Overall Avg.	82	90.8	82.2	41.6

Table 5: Effect of gaze-regularization strength (

\lambda

) on task success rate (%). The model shows improved performance when gaze is used as a soft prior rather than a hard constraint.

Perturbation	LIBERO-Spatial		LIBERO-Object		LIBERO-Goal
Perturbation	w/o Gaze	w Gaze	w/o Gaze	w Gaze	w/o Gaze	w Gaze
No Perturbation	85.9	95.5	91.7	97.3	84.3	92.6
Lighting Variation	77.2	89.1	84.5	92.8	80.4	89.7
Camera Noise	82.1	91.3	85.8	93.5	79.6	88.9

Table 6: Performance comparison under different visual perturbations when noise or visual degradations are simulated. Results are reported across three LIBERO benchmarks [ 29 ] .

Symbol	Description
$t$	Timestep index of the current observation
$i$	Camera/view index
$N_{v}$	Number of visual tokens (e.g., $16\times 16=256$ )
$P$	Patch grid dimension (e.g., $P=16$ )
$X_{l}\in\mathbb{R}^{N_{l}\times d}$	Language token sequence
$X_{v}^{i}\in\mathbb{R}^{N_{v}\times d}$	Visual tokens from camera view $i$

Table 7: Summary of key notations used in gaze-to-attention regularization and VLA token interactions.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Location of Object	w Gaze	DeepGaze	w/o Gaze	Uniform
Location of Object	30k	30k	30k	30k
Between plate and ramekin	100	85.7	83.3	69.7
Next to ramekin	100	86.7	85.7	59.7
Table center	100	100	100	80.3
On cookie box	91.3	100	100	79.3
In cabinet drawer	73.3	82.0	80	39.3

Appendix

Table 8: Per-task success rates on LIBERO Spatial [ 29 ] at 30k training steps. We compare the baseline model, our gaze-regularized model, a DeepGaze-based gaze variant, and a uniform-distribution variant.

Figure 4: Closer look at Gaze Prior Generation A sequence of $k$ video frames is tokenized and processed by the GLC [ 25 ] module, where it utilizes global tokens (derived from the sequence) and local tokens, and undergoes self attention as well as Global-Local Correlation to then predict per-frame gaze heatmaps. These heatmaps are temporally aggregated to yield a gaze distribution that captures attention over time and serves as the supervision signal for training- time regularization.

Figure 5: Additional Visualisations of Attention. Given the input observation, we show the spatial attention from the baseline model (second), the attention obtained when a perturbed gaze variant is used (third, corresponding to Table 8 ), and finally the sharper, task-relevant attention produced by our gaze-regularized model (fourth).

Location of Object	w Gaze (All Layers)
Location of Object	10k	20k	30k
Between plate and ramekin	65.0	75.0	90.3
Next to ramekin	55.0	70.0	89.7
Table center	70.0	85.0	100.0
On cookie box	58.3	65.0	70.3
In cabinet drawer	43.3	56.3	60.7

Appendix

Table 9: Per-task success rates on LIBERO Spatial with regularization applied to all layers. The model shows competitive performance with comprehensive regularization.

Figure 6: Reliability of Synthetic Gaze on Simulation Videos Given the input task, we show the the predicted gaze is accurate and even on similar visual settings, produces different gaze results depending on the language instruction. The model utilizes a temporal sequence of frames, rather than a single frame, and then computes the gaze prediction thus the prediction occurs due to the conditioning through the global context by operating on a sequence of frames

Location of Object	w Gaze (Distractors)	w/o Gaze (Distractors)
Location of Object	30k	30k
Between plate and ramekin	96.7	78.3
Next to ramekin	95.0	80.0
Table center	97.0	96.7
On cookie box	90.0	96.7
In cabinet drawer	70.0	70.0

Appendix

Table 10: Per-task success rates on LIBERO Spatial under prompt distractors (e.g., replacing “pick” with “grab”, “lift”, etc.). Both the baseline and gaze-regularized models exhibit performance degradation, but the gaze model remains more robust.

Location of Object	w/o Gaze	w Gaze	Foveated
Location of Object	30k	30k	30k
Between plate and ramekin	83.3	100	80.0
Next to ramekin	85.7	100	81.3
Table center	100	100	95.7
On cookie box	100	91.3	90.0
In cabinet drawer	80	73.3	65.3

Appendix

Table 11: Per-task success rates on LIBERO Spatial [ 29 ] at 30k training steps. We compare the baseline model, our gaze-regularized model, and a foveated-vision variant.

Figure 7: Additional Visualisations of Attention. Given the input observation (first), we show the spatial attention from the baseline model (second) and task-relevant attention produced by our gaze-regularized model (third).

Location of Object	w Gaze	w/o Gaze
Location of Object	30k	30k
Between plate and ramekin	90.3	81.3
Next to ramekin	80.7	71.0
Table center	90.7	81.7
On cookie box	70.7	62.0
In cabinet drawer	69.3	60.7

Appendix

Table 12: Per-task success rates on the Missing Views experiment at 30k training steps. The gaze-regularized model consistently outperforms the baseline across all spatial configurations.

Figure 8: Visualization of Real-world Task on Aloha Robot In the figure, we provide some frames from a real world task performed using our gaze-regularized policy to show that our method works outside of simulation as well. Here, the task is to pick up the cube and place it on the correct plate.

Location of Object	w/o Gaze	w Gaze	Human Gaze
Location of Object	30k	30k	30k
Between plate and ramekin	83.3	100	100
Next to ramekin	85.7	100	100
Table center	100	100	100
On cookie box	100	91.3	89.3
In cabinet drawer	80	73.3	78.3

Appendix

Table 13: Per-task success rates on LIBERO Spatial [ 29 ] at 30k training steps. We compare the baseline model, our gaze-regularized model, and the human-gaze-trained variant.

Figure 9: Visualization of Real-world Task on Aloha Robot In this figure, we present a short horizon task of picking up a cup and placing it in a container(top) and also another longer horizon task to pick up multiple cups one-by-one, and place them in the container. Both visualisations are obtained using our gaze-regularized policy, highlighting its working functionality even in real-world scenarios

Figure 10: Visualization Results In the figure, we provide some visualization results to show how the policy performs on the Libero-Spatial [ 29 ] task suites. We provide the task instructions, and some important frames to show the task success. The baseline model performs admirably, but our method enhances the results by using gaze-regularization.

Z\leftarrow\sum_{x=1}^{H_{g}}\sum_{y=1}^{W_{g}}H(x,y).

Appendix

Algorithm 1 Heatmap-to-Token Projection

\{(I_{1:n,t},\ell_{t},q_{t},A_{t}^{\ast})\}_{b=1}^{B}.

Appendix

Algorithm 2 Training Loop with Gaze Regularization

Figure 11: Failure Case. We show a failure example from the Libero-Spatial [ 29 ] task suite. In this task, the baseline model outperforms the gaze-regularized model, suggesting that stronger or more accurate gaze priors could further improve reliability. The bottom sequence illustrates the failure case where the robot hand fails to grab the bowl in the top drawer and proceeds to carry out the intended action.

Figure 12: Attention Comparison. The baseline model displays diffuse attention spread across the scene, with a single sharp point that is largely task-irrelevant. In contrast, the gaze-regularized model produces noticeably sharper, more concentrated, and consistently task-relevant attention, leading to clearer visual grounding for the instructed action.

Figure 13: Visualisation during a failure case. In this figure, we provide a visualisation of attention during a specific case of failure, where it can be seen that even though the task is to pick up the bowl, attention is not properly distributed on the bowl but rather than on the cabinet handle. Such cases can be mitigated using a better predictor or using a model trained with human supervision on simulated videos

VLMs for Robotics

Vision-language models for robotics connect visual and linguistic understanding to grounded navigation, manipulation, and robot control.

See all 627 papers

ActGaze: Learning Action-Grounded Gaze through Counterfactual Visual Interventions for High-Precision Manipulation
Sep 24, 2026Jinxuan Zhu, Jiaheng Wang, Chao Tang +7Visuomotor Policy Learning Robotic Manipulation
Samsung Robotics eXperience · National University of Singapore
GazeAct: A Human-Gaze-Annotated Manipulation Dataset for Attention-Regularized Policies
Oct 17, 2026Anupam Pani, Wang Yuan, Qifan Wang +1Spatial Grounding for VLA Models Robot Manipulation Datasets
HKU Musketeers Foundation Institute of Data Science · Department of Electrical and Electronic Engineering, The University of Hong Kong
Gaze2Act: Gaze-Conditioned Vision-Language-Action Policies for Interactive Robot Manipulation
May 28, 2026Kuangji Zuo, Gen Li, Bofan Lyu +9Language-Conditioned Robot Manipulation Robotic Manipulation
1MARS Lab, Nanyang Technological University

Sep 24, 2026J/K move · Enter open · S save

ActGaze: Learning Action-Grounded Gaze through Counterfactual Visual Interventions for High-Precision Manipulation

Jinxuan Zhu, Jiaheng Wang, Chao Tang, Mengfan Wang+6

Samsung Robotics eXperience · National University of Singapore

Visuomotor Policy Learning Robotic Manipulation Vision-Language-Action Models

Open paper AskPDF arXiv

Current Vision-Language-Action (VLA) models often struggle with high-precision robotic manipulation. We attribute this limitation primarily to their visual attention being dispersed across task-irrelevant regions. To address this issue, we propose ActGaze, a training approach that guides VLA policies to gaze on task-relevant regions, much like humans gaze on critical visual cues while executing precise movements. Unlike prior methods that rely on external labels for gaze supervision, ActGaze derives spatial supervision directly from the VLA's own action objective by using counterfactual visual interventions to identify regions that are critical for action prediction. Extensive real-robot experiments on four high-precision robotic manipulation tasks demonstrate that ActGaze induces more focused visual attention on task-relevant regions and consistently outperforms the base VLA policy and other visual-grounding approaches.

Abstract

VLMs for Robotics

Explore similar work

ActGaze: Learning Action-Grounded Gaze through Counterfactual Visual Interventions for High-Precision Manipulation

GazeAct: A Human-Gaze-Annotated Manipulation Dataset for Attention-Regularized Policies

Gaze2Act: Gaze-Conditioned Vision-Language-Action Policies for Interactive Robot Manipulation