Leveraging auditory and visual feedback for attention reorientation is essential for natural gaze shifts in social interaction. However, enabling humanoid robots to perform natural and context-appropriate gaze shifts in unconstrained human--robot interaction (HRI) remains challenging, as it requires the coupling of cognitive attention mechanisms and biomimetic motion generation. In this work, we propose the Robot Gaze-Shift (RGS) framework, which integrates these two components into a unified pipeline. First, RGS employs a vision--language model (VLM)-based gaze reasoning pipeline to infer context-appropriate gaze targets from multimodal interaction cues, ensuring consistency with human gaze-orienting regularities. Second, RGS introduces a conditional Vector Quantized-Variational Autoencoder (VQ-VAE) model for eye--head coordinated gaze-shift motion generation, producing diverse and human-like gaze-shift behaviors. Experiments validate that RGS effectively replicates human-like target selection and generates realistic, diverse gaze-shift motions.
Figures & tables
Fig. 1: Overview of our RGS framework.
Fig. 2: VLM-Based Gaze Reasoning Pipeline. Notation: superscripts f , m and ∗ denote the final frame of a cycle, instance masking, and mark indexing, respectively.
Fig. 3: Overview of the proposed gaze-shift motion generation model, trained in two stages. Top: a conditional VQ-VAE is trained to reconstruct eye–head rotation increments from ground-truth motions and conditioning inputs. Bottom: a conditional prior predicts a distribution over codebook entries from the conditioning inputs. The prior selects the maximum-probability code to compute the loss during training, while using stochastic sampling at inference time to enable behavioral diversity.
Fig. 4: Qualitative examples of the proposed VLM-based gaze reasoning pipeline across four gaze-orienting regularities (H1–H4). Each row corresponds to one regularity and shows three representative inference cycles (T1–T3) selected at different times. The three cycles are temporally ordered but not necessarily consecutive. The red semi-transparent overlay indicates the gaze target selected by the pipeline at each illustrated inference cycle.
Regularity
Clips
Correct
Success rate (%)
(H1) Deictic orienting of attention
20
17
85.0
(H2) Social orienting reflex
19
19
100.0
(H3) Turn-taking
20
17
85.0
(H4) Responsive joint attention
17
12
70.6
TABLE I: Human-likeness evaluation of gaze-target selection across four gaze-orienting regularities.
Fig. 5: Validation-set MGD curves for the two-stage training. (a) Training Stage-1: conditional VQ-VAE reconstruction errors. (b) Training Stage-2: conditional prior inference errors. Both stages exhibit consistent error reduction and stable convergence.
Fig. 6: Qualitative visualization of inference-time diversity in eye–head coordinated gaze-shift motion generation. Given identical initial conditions c={θe,θh,xg,3d} , the model generates multiple distinct yet plausible eye–head coordination modes via stochastic code sampling. Red numbers denote sampled discrete code indices, and P indicates the corresponding prior probability (we show samples with probability >5% ).
The BioRobotics Institute, Scuola Superiore Sant’Anna; Pisa, Italy · Department of Excellence in Robotics and AI, Scuola Superiore Sant’Anna; Pisa, Italy