Gaze coordinates attention in human interaction through coupled eye, head, and eyelid motion. A humanoid robot must choose where to look while generating a suitable movement toward that target; fixed tracking rules alone do not model the temporal structure of human gaze. We separate these tasks with an asynchronous vision-language model that selects a scene referent, a reactive perception module that maintains fixation, and a Realistic Gaze Transformer (RGT) that generates target-conditioned motion. RGT is a Transformer-VAE trained on markerless human recordings retargeted to the robot's 11-DoF joint space: self-supervised pretraining encodes recent full-head history, and autoregressive fine-tuning exposes the model to its own predictions during rollout. At inference, a four-dimensional binocular-eye trajectory conditions coordinated neck and eye motion; an auxiliary blink predictor triggers eyelid templates during playback. On a two-second benchmark in which all controllers receive the same ground-truth target, RGT-Base obtains the lowest reported overall seven-joint MSE, and physical demonstrations show the integrated targeting and control pipeline. The evidence establishes robot-space kinematic fidelity and real-time RGT inference; perceived naturalness and open-vocabulary target-selection accuracy remain questions for controlled evaluation.
Figures & tables
Figure 1: Overview of the Realistic Gaze Transformer (RGT) pipeline. The left panel illustrates our hierarchical perception system operating on a real humanoid robot. Egocentric video from the camera-embedded eyes, together with high-level natural language instructions, is processed by a slow-system VLM that reasons over the environmental context and designates salient targets. A fast-system Visual Perception Module then tracks the selected faces in real time to produce a continuous target gaze direction. The right panel depicts the RGT architecture: a History Full-head Motion Tokenizer encodes recent facial dynamics while an Eyeball Trajectory Generator encodes the target gaze, and a Transformer-VAE fuses both streams into a latent representation from which it predicts future motion. The Action De-Tokenizer maps the near-future tokens to 11-DoF joint commands on the physical robot. The quantitative benchmark evaluates motion generation with ground-truth eye targets; target selection is illustrated by the physical demonstration.
Figure 2: Full pipeline of the Realistic Gaze Transformer (RGT). Top: Data preprocessing stage. Raw human motion sequences (head pose, eye gaze, and blinks) are segmented into clips. Blink events are detected and extracted as separate blink templates (small eyelid motion sequences), and those blink segments are removed from the sequences to produce blink-free training data. Bottom: Model architecture and inference. Given a past full-head motion sequence and a planned eyeball motion sequence, the Transformer-VAE model (encoder and decoder) predicts a future blink-free trajectory along with blink probabilities for each future frame. At inference time, when a blink is predicted (high probability), a corresponding blink template is injected into the generated trajectory, yielding the final full-head motion with realistic blinks (red frames show the injected blink).
Figure 3: Qualitative comparison of gaze behaviors across controllers. Each row shows a time sequence of robot face images during a representative leftward gaze shift. Top : Ground-truth human motion (GT) exhibits coordinated eye–head motion, with eyes leading the head. Second : RGT reproduces the illustrated sequence’s eye–head timing. Third : Eyes-Only rotates the eyes to mechanical extremes while keeping the head fixed, resulting in strained, unnatural motion. Bottom : Head-Only turns the head abruptly toward the target while leaving the eyes static, exaggerating head movement and yielding sluggish gaze redirection. Each column represents 1/3 second of motion.
Method
BID 1
BID 2
BID 3
BID 4
BID 5
BID 6
Overall
Head-Only
5.8e-2
5.9e-2
7.3e-2
7.2e-2
7.2e-2
6.8e-2
6.8e-2
Eye-Only
5.1e-2
5.4e-2
5.7e-2
5.6e-2
5.5e-2
5.2e-2
5.2e-2
Eye-Head Coord.
1.3e-2
1.2e-2
3.9e-2
1.5e-2
1.7e-2
1.4e-2
1.8e-2
Head-Only (w/o fixed)
1.6e-2
1.3e-2
5.8e-2
2.3e-2
3.2e-2
2.1e-2
3.1e-2
Eye-Only (w/o fixed)
1.7e-2
1.8e-2
1.5e-2
2.3e-2
1.1e-2
1.6e-2
1.7e-2
RNN
1.3e-4
2.0e-4
1.6e-4
2.2e-4
3.7e-4
9.8e-4
3.0e-4
Table 1: Closed-loop seven-joint MSE ( rad2 ) over 1,848 S1 clips. A single stochastic sample is evaluated per model call. Values are from one run and are rounded; no uncertainty intervals are available. “w/o fixed” rows score only active joints and are not directly comparable with the seven-joint rows.
Figure 5
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Method
BID 1
BID 2
BID 3
BID 4
BID 5
BID 6
Overall
RNN
8.6e-5
1.3e-4
1.2e-4
1.9e-4
3.2e-4
9.9e-4
2.6e-4
VAE-CNN
1.2e-5
1.5e-5
3.7e-5
2.0e-5
5.2e-5
4.8e-4
8.7e-5
Flow Matching (16-step)
4.2e-6
1.0e-5
1.4e-5
2.6e-5
1.9e-5
1.9e-5
1.2e-5
RGT-Base
1.6e-5
2.5e-5
3.8e-5
3.9e-5
9.5e-5
3.9e-4
8.5e-5
RGT-Large
1.4e-5
2.1e-5
4.1e-5
2.9e-5
8.9e-5
6.0e-4
1.1e-4
RGT-Small
1.4e-5
2.3e-5
4.3e-5
3.2e-5
8.5e-5
6.5e-4
1.2e-4
Appendix
Table 4: Single-step sliding-window evaluation. Per-BID and overall joint-angle MSE ( rad2 ). The model always receives ground-truth history; there is no error compounding. Best per column in bold.
Behavior (Subject)
Description
Rows
Behavior 1 (Subject 1)
This sequence mimics natural human eye tracking behavior. The head oscillates within ± 10 ∘ , while the eyes move in coordination, turning left and right with an anticipatory expression.
31487
Behavior 2 (Subject 1)
Similar to Behavior 1, this sequence replicates eye tracking behavior but over a shorter duration. The head and eye motions follow the same pattern.
26739
Behavior 3 (Subject 1)
This sequence captures a Head-Down Scan gesture. The motion begins by lowering the head approximately 30 ∘ , followed by a slow left-right head rotation of about 15 ∘ , expressing contemplation.
8358
Behavior 4 (Subject 1)
This sequence represents a Head-Up Survey gesture. The head raises approximately 45 ∘ and then moves left and right, coordinated with eye movements, expressing alertness.
6750
Behavior 5 (Subject 1)
This sequence features a Head-Mid Survey , using head oscillations within ± 10 ∘ along with coordinated eye movements.
10014
Behavior 6 (Subject 1)
This sequence follows an All Survey pattern, combining head oscillations within ± 10 ∘ and coordinated eye movements to express anticipation.
14310
Appendix
Table 5: Summary of S1 human motion data; counts are processed 60-Hz rows.
Figure 5: (Top) Pearson correlation matrix across all head, eye, and eyelid degrees of freedom (DOFs), revealing strong coupling between head yaw and eye yaw, as well as head pitch and eye pitch. (Bottom) Lagged Pearson correlation peaks at a positive lag of approximately 20 frames (333 ms), consistent with eye motion preceding head motion in these recordings. Per-frame significance labels shown within the supplied plots are exploratory because adjacent frames are temporally correlated.
Robot motion generation creates feasible, task-directed trajectories for robots to move, manipulate objects, and interact with people and environments.
Leveraging auditory and visual feedback for attention reorientation is essential for natural gaze shifts in social interaction. However, enabling humanoid robots to perform natural and context-appropriate gaze shifts in unconstrained human--robot interaction (HRI) remains challenging, as it requires the coupling of cognitive attention mechanisms and biomimetic motion generation. In this work, we propose the Robot Gaze-Shift (RGS) framework, which integrates these two components into a unified pipeline. First, RGS employs a vision--language model (VLM)-based gaze reasoning pipeline to infer context-appropriate gaze targets from multimodal interaction cues, ensuring consistency with human gaze-orienting regularities. Second, RGS introduces a conditional Vector Quantized-Variational Autoencoder (VQ-VAE) model for eye--head coordinated gaze-shift motion generation, producing diverse and human-like gaze-shift behaviors. Experiments validate that RGS effectively replicates human-like target selection and generates realistic, diverse gaze-shift motions.
Jingchao Wei, Jingkai Qin, Yuxiao Cao +4
School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China
Human vision is a highly active process driven by gaze, which directs attention to task-relevant regions through foveation, dramatically reducing visual processing. In contrast, robot learning systems typically rely on passive, uniform processing of raw camera images. In this work, we explore how incorporating human-like active gaze into robotic policies can enhance efficiency and robustness. We develop GIAVA (Gaze Integrated Active-Vision ALOHA), a robot vision system that emulates human head and neck movement, and gaze adjustment for foveated processing. Extending the AV-ALOHA robot platform, we introduce a framework for simultaneously collecting eye-tracking, perspective control, and robot manipulation demonstration data from a human operator. We also open-source a simulation benchmark and dataset for training robot policies that incorporate human gaze. Inspired by recent work in foveated image segmentation and given the widespread use of Vision Transformers (ViTs) in robot learning, we integrate gaze information into ViTs using a foveated patch tokenization scheme. Compared to uniform patch tokenization, this significantly reduces the number of tokens, and thus computation. Our results show that our method for foveated robot vision drastically reduces computational overhead, and enhances robustness to background distractors. Notably, on certain high-precision tasks, foveated vision also improves performance, as reflected in higher success rates. Together, these findings suggest that human-inspired foveated visual processing offers untapped potential and should be further considered as a useful inductive bias in robotic vision systems. https://soltanilara.github.io/giava/
Ian Chuang, Jinyu Zou, Andrew Lee +2
University of California, Berkeley · 2Tongji University · University of California, Davis
Despite advances in Vision-Language-Action (VLA) models, robotic manipulation struggles with fine-grained tasks because current models lack mechanisms for active visual attention allocation. Human gaze naturally encodes intent, planning, and execution patterns -- offering a powerful supervisory signal for guiding robot perception. We introduce a gaze-regularized training framework that aligns VLA models' internal attention with human visual patterns without architectural modifications or inference-time overhead. Our method transforms temporally aggregated gaze heatmaps into patch-level distributions and regularizes the transformer's attention through KL divergence, creating an inductive bias toward task-relevant features while preserving deployment efficiency. When integrated into existing VLA architectures, our approach yields 4-12% improvements across manipulation benchmarks. The gaze-regularized models reach equivalent performance with fewer training steps and maintain robustness under lighting variations and sensor noise. Beyond performance metrics, the learned attention patterns produce interpretable visualizations that mirror human strategies, enhancing trust in robotic systems. Moreover, our framework requires no eye-tracking equipment and applies directly to existing datasets. These results demonstrate that human perceptual priors can significantly accelerate robot learning while improving both task performance and system interpretability.
Anupam Pani, Yanchao Yang
Institute of Data Science, University Of Hong Kong