cs.ROOct 14, 2026

Eye, Robot: Learning Expressive Gaze Dynamics via a VLM-Guided Transformer-VAE

Authors: Yuhang Hu, Junlin Lyu, Yuan Fang, Xiaoying Shen, Lechen Zhang, Jiong Lin, Judah Goldfeder, Steven Tian, 

Organizations: Creative Machines Lab, Department of Mechanical Engineering, Columbia University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Gaze coordinates attention in human interaction through coupled eye, head, and eyelid motion. A humanoid robot must choose where to look while generating a suitable movement toward that target; fixed tracking rules alone do not model the temporal structure of human gaze. We separate these tasks with an asynchronous vision-language model that selects a scene referent, a reactive perception module that maintains fixation, and a Realistic Gaze Transformer (RGT) that generates target-conditioned motion. RGT is a Transformer-VAE trained on markerless human recordings retargeted to the robot's 11-DoF joint space: self-supervised pretraining encodes recent full-head history, and autoregressive fine-tuning exposes the model to its own predictions during rollout. At inference, a four-dimensional binocular-eye trajectory conditions coordinated neck and eye motion; an auxiliary blink predictor triggers eyelid templates during playback. On a two-second benchmark in which all controllers receive the same ground-truth target, RGT-Base obtains the lowest reported overall seven-joint MSE, and physical demonstrations show the integrated targeting and control pipeline. The evidence establishes robot-space kinematic fidelity and real-time RGT inference; perceived naturalness and open-vocabulary target-selection accuracy remain questions for controlled evaluation.

Figures & tables

Figure 1: Overview of the Realistic Gaze Transformer (RGT) pipeline. The left panel illustrates our hierarchical perception system operating on a real humanoid robot. Egocentric video from the camera-embedded eyes, together with high-level natural language instructions, is processed by a slow-system VLM that reasons over the environmental context and designates salient targets. A fast-system Visual Perception Module then tracks the selected faces in real time to produce a continuous target gaze direction. The right panel depicts the RGT architecture: a History Full-head Motion Tokenizer encodes recent facial dynamics while an Eyeball Trajectory Generator encodes the target gaze, and a Transformer-VAE fuses both streams into a latent representation from which it predicts future motion. The Action De-Tokenizer maps the near-future tokens to 11-DoF joint commands on the physical robot. The quantitative benchmark evaluates motion generation with ground-truth eye targets; target selection is illustrated by the physical demonstration.
Figure 2: Full pipeline of the Realistic Gaze Transformer (RGT). Top: Data preprocessing stage. Raw human motion sequences (head pose, eye gaze, and blinks) are segmented into clips. Blink events are detected and extracted as separate blink templates (small eyelid motion sequences), and those blink segments are removed from the sequences to produce blink-free training data. Bottom: Model architecture and inference. Given a past full-head motion sequence and a planned eyeball motion sequence, the Transformer-VAE model (encoder and decoder) predicts a future blink-free trajectory along with blink probabilities for each future frame. At inference time, when a blink is predicted (high probability), a corresponding blink template is injected into the generated trajectory, yielding the final full-head motion with realistic blinks (red frames show the injected blink).
Figure 3: Qualitative comparison of gaze behaviors across controllers. Each row shows a time sequence of robot face images during a representative leftward gaze shift. Top : Ground-truth human motion (GT) exhibits coordinated eye–head motion, with eyes leading the head. Second : RGT reproduces the illustrated sequence’s eye–head timing. Third : Eyes-Only rotates the eyes to mechanical extremes while keeping the head fixed, resulting in strained, unnatural motion. Bottom : Head-Only turns the head abruptly toward the target while leaving the eyes static, exaggerating head movement and yielding sluggish gaze redirection. Each column represents 1/3 second of motion.
MethodBID 1BID 2BID 3BID 4BID 5BID 6Overall
Head-Only5.8e-25.9e-27.3e-27.2e-27.2e-26.8e-26.8e-2
Eye-Only5.1e-25.4e-25.7e-25.6e-25.5e-25.2e-25.2e-2
Eye-Head Coord.1.3e-21.2e-23.9e-21.5e-21.7e-21.4e-21.8e-2
Head-Only (w/o fixed)1.6e-21.3e-25.8e-22.3e-23.2e-22.1e-23.1e-2
Eye-Only (w/o fixed)1.7e-21.8e-21.5e-22.3e-21.1e-21.6e-21.7e-2
RNN1.3e-42.0e-41.6e-42.2e-43.7e-49.8e-43.0e-4
Table 1: Closed-loop seven-joint MSE ( rad2\mathrm{rad}^{2} ) over 1,848 S1 clips. A single stochastic sample is evaluated per model call. Values are from one run and are rounded; no uncertainty intervals are available. “w/o fixed” rows score only active joints and are not directly comparable with the seven-joint rows.
Figure 5
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

MethodBID 1BID 2BID 3BID 4BID 5BID 6Overall
RNN8.6e-51.3e-41.2e-41.9e-43.2e-49.9e-42.6e-4
VAE-CNN1.2e-51.5e-53.7e-52.0e-55.2e-54.8e-48.7e-5
Flow Matching (16-step)4.2e-61.0e-51.4e-52.6e-51.9e-51.9e-51.2e-5
RGT-Base1.6e-52.5e-53.8e-53.9e-59.5e-53.9e-48.5e-5
RGT-Large1.4e-52.1e-54.1e-52.9e-58.9e-56.0e-41.1e-4
RGT-Small1.4e-52.3e-54.3e-53.2e-58.5e-56.5e-41.2e-4
Appendix
Table 4: Single-step sliding-window evaluation. Per-BID and overall joint-angle MSE ( rad2\mathrm{rad}^{2} ). The model always receives ground-truth history; there is no error compounding. Best per column in bold.
Behavior (Subject)DescriptionRows
Behavior 1 (Subject 1)This sequence mimics natural human eye tracking behavior. The head oscillates within ±\pm 10 ∘ , while the eyes move in coordination, turning left and right with an anticipatory expression.31487
Behavior 2 (Subject 1)Similar to Behavior 1, this sequence replicates eye tracking behavior but over a shorter duration. The head and eye motions follow the same pattern.26739
Behavior 3 (Subject 1)This sequence captures a Head-Down Scan gesture. The motion begins by lowering the head approximately 30 ∘ , followed by a slow left-right head rotation of about 15 ∘ , expressing contemplation.8358
Behavior 4 (Subject 1)This sequence represents a Head-Up Survey gesture. The head raises approximately 45 ∘ and then moves left and right, coordinated with eye movements, expressing alertness.6750
Behavior 5 (Subject 1)This sequence features a Head-Mid Survey , using head oscillations within ±\pm 10 ∘ along with coordinated eye movements.10014
Behavior 6 (Subject 1)This sequence follows an All Survey pattern, combining head oscillations within ±\pm 10 ∘ and coordinated eye movements to express anticipation.14310
Appendix
Table 5: Summary of S1 human motion data; counts are processed 60-Hz rows.
Figure 5: (Top) Pearson correlation matrix across all head, eye, and eyelid degrees of freedom (DOFs), revealing strong coupling between head yaw and eye yaw, as well as head pitch and eye pitch. (Bottom) Lagged Pearson correlation peaks at a positive lag of approximately 20 frames (333 ms), consistent with eye motion preceding head motion in these recordings. Per-frame significance labels shown within the supplied plots are exploratory because adjacent frames are temporally correlated.
More in this topic

Robot Motion Generation

Robot motion generation creates feasible, task-directed trajectories for robots to move, manipulate objects, and interact with people and environments.

See all 188 papers

Explore similar work

CardsList

  1. Humanizing Robot Gaze Shifts: A Framework for Natural Gaze Shifts in Humanoid Robots

    Feb 25, 2026Jingchao Wei, Jingkai Qin, Yuxiao Cao +4Human-Robot InteractionHumanoid Robotics

    School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China

  2. Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers

    Jul 21, 2025Ian Chuang, Jinyu Zou, Andrew Lee +2Visuomotor Policy LearningEfficient ViTs

    University of California, Berkeley · 2Tongji University · University of California, Davis

  3. Gaze-Regularized Vision-Language-Action Models for Robotic Manipulation

    Mar 24, 2026Anupam Pani, Yanchao YangVLMs for RoboticsRobotic Manipulation

    Institute of Data Science, University Of Hong Kong