Gaze-Aware Task Progression Detection Framework for Human-Robot Interaction Using RGB Cameras
Authors: Linlin Cheng, Koen Hindriks, Artem V. Belopolsky
Organizations: Department of Computer Science, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands. · Department of Movement Sciences, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands.
In human-robot interaction (HRI), detecting a human's gaze helps robots interpret user attention and intent. However, most gaze detection approaches rely on specialized eye-tracking hardware, limiting deployment in everyday settings. Appearance-based gaze estimation methods remove this dependency by using standard RGB cameras, but their practicality in HRI remains underexplored. We present a calibration-free framework for detecting task progression when information is conveyed via integrated display interfaces. The framework uses only the robot's built-in monocular RGB camera (640x480 resolution) and state-of-the-art gaze estimation to monitor attention patterns. It leverages natural behavior, where users shift focus from task interfaces to the robot's face to signal task completion, formalized through three Areas of Interest (AOI): tablet, robot face, and elsewhere. Systematic parameter optimization identifies configurations that balance detection accuracy and interaction latency. We validate our framework in a "First Day at Work" scenario, comparing it to button-based interaction. Results show a task completion detection accuracy of 77.6%. Compared to button-based interaction, the proposed system exhibits slightly higher response latency but preserves information retention and significantly improves comfort, social presence, and perceived naturalness. Notably, most participants reported that they did not consciously use eye movements to guide the interaction, underscoring the intuitive role of gaze as a communicative cue. This work demonstrates the feasibility of intuitive, low-cost, RGB-only gaze-based HRI for natural and engaging interactions.
Figures & tables
Fig. 1: The overview of the proposed framework. The system uses the humanoid robot’s built-in monocular camera to continuously track human facial features and estimate gaze direction using a pretrained deep learning model. The resulting 3D gaze vector undergoes smoothing filtration to reduce noise, then is mathematically projected onto the robot’s 2D interaction plane and mapped to predefined AOIs. In the state detection algorithm(left flowchart), the user’s level of engagement was determined by analyzing their gaze patterns over time within predefined AOIs using a simple threshold-based approach.
Fig. 2: The relationship between 3D gaze directions and their projection onto a 2D screen
Parameter
Symbol
Value
Smooth Window
N
3
Engagement Confirmation Window
We
1s
Disengagement Confirmation Window
Wd
1s
Engagement Threshold
τe
40%
Disengagement Threshold
τd
50%
TABLE I: Final Parameter Settings Used in the Study
Fig. 3: Flowchart of the experimental procedure for two interaction conditions, gaze-based and button-based. The gaze-based condition was presented to participants as a “no-button” interaction to conceal the underlying gaze detection mechanism.
Fig. 4: Example of a gaze-based interaction task. Top: Smoothed pitch (blue) and yaw (red) over time. The semi-transparent blue areas indicate periods when the robot displayed Page 1 and Page 2. Bottom: Frame sequences from real-time recordings corresponding to three moments ( t1 , t2 , t3 ) in the top plot. Panels (a) and (b) show views from a third-person perspective (a) and the robot’s perspective (b). In (b), the arrows represent the original 3D gaze (red) and the smoothed gaze (green). These three moments illustrate the transition from Page1 to Page2 using the proposed framework.
Fig. 5: Success Rate Distribution of the Proposed System Across Participants. The violin plot shows the probability density of success rates, with wider sections indicating where more participants cluster. Individual data points (blue dots) represent each participant’s success rate, calculated as the percentage of successful state changes detected across all trials. The red line indicates the mean success rate, while the orange line shows the median. The main reasons for the four participants with lower success rates are indicated in the figure with arrows and text.
Fig. 6: Comparison of gaze-based and button-based interaction conditions in memory task. (A) Page duration by condition and content. Left: Mean duration by input condition. Middle: Mean duration by content type. Right: Condition × content interaction. (B) Memory task accuracy by condition and question type. Left: Accuracy by input condition. Middle: Accuracy by question type. Right: Condition × question type interaction. (C) User experience. Top: Bar plots showing subjective ratings on a 7-point Likert scale. Bottom: Radar chart summarizing mean scores across dimensions. Higher values indicate more positive evaluations. (D) Visual attention patterns. Top: Gaze allocation to face, tablet, and elsewhere regions. Bottom: Spatial gaze density heatmaps by condition. *p < 0.05, **p < 0.01, ***p < 0.005; n.s., not significant (p > 0.05).
Leveraging auditory and visual feedback for attention reorientation is essential for natural gaze shifts in social interaction. However, enabling humanoid robots to perform natural and context-appropriate gaze shifts in unconstrained human--robot interaction (HRI) remains challenging, as it requires the coupling of cognitive attention mechanisms and biomimetic motion generation. In this work, we propose the Robot Gaze-Shift (RGS) framework, which integrates these two components into a unified pipeline. First, RGS employs a vision--language model (VLM)-based gaze reasoning pipeline to infer context-appropriate gaze targets from multimodal interaction cues, ensuring consistency with human gaze-orienting regularities. Second, RGS introduces a conditional Vector Quantized-Variational Autoencoder (VQ-VAE) model for eye--head coordinated gaze-shift motion generation, producing diverse and human-like gaze-shift behaviors. Experiments validate that RGS effectively replicates human-like target selection and generates realistic, diverse gaze-shift motions.
Jingchao Wei, Jingkai Qin, Yuxiao Cao +4
School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China
Gaze-following in child-robot interaction improves attention, recall, and learning, but requires expensive platforms ($30,000+), sensors, algorithms, and raises privacy concerns. We propose a framework that avoids sensors and computation entirely, instead relying on the human visual system's assumption of convexity to produce perceptual gaze-following between a robot and its viewer. Specifically, we motivate sub-dollar cardboard robot design that directly implements the brain's own gaze computation pipeline in reverse, making the viewer's perceptual system the robot's "actuator", with no sensors, no power, and no privacy concerns. We ground this framework in three converging lines of theoretical and empirical neuroscience evidence. Namely, the distributed face processing network that computes gaze direction via the superior temporal sulcus, the high-precision convexity prior that causes the brain to perceive concave faces as convex, and the predictive processing hierarchy in which top-down face knowledge overrides bottom-up depth signals. These mechanisms explain why a concave eye socket with a painted pupil produces the perception of mutual gaze from any viewing angle. We derive design constraints from perceptual science, present a sub-dollar open-template robot with parameterized interchangeable eye inserts, and identify boundary conditions (developmental, clinical, and geometric) that predict where the framework will succeed and where it will fail. If leveraged, two decades of HRI gaze findings become deliverable at population scale.
Mason Kadem
Computing and Software, Faculty of Engineering, McMaster University
Gaze and speech are promising interaction modalities for individuals with motor impairments, yet robust intent recognition in multi-object environments remains challenging due to micro-saccades, semantic ambiguity, and viewpoint changes. This paper presents a multimodal interaction framework for assistive robotic manipulation. We propose a sticky-glance algorithm that stabilizes gaze-based intent by jointly accumulating geometric distance and directional evidence, enabling robust real-time target selection and switching. We further introduce Glance-Say, a gaze-speech interaction paradigm in which gaze specifies objects and speech specifies actions, together with a continuous shared-control scheme that provides high-readiness robot motion and human-in-the-loop feedback. Experiments demonstrate a tracking rate of 0.92 for moving targets, selection accuracy of 0.97 for static targets, and reduced task duration. These results indicate improved robustness, efficiency, and usability over representative interaction paradigms.
Yuzhi Lai, Shenghai Yuan, Peizheng Li +2
University of Tuebingen, Geschwister-Scholl-Platz, 72074 Germany · Nanyang Technological University, 50 Nanyang Avenue, Singapore 639798