GUIDER: Evaluating Goal-Free Human Intent Inference for Teleoperated Manipulation on Real-Robot Data

Nicholas Kenny; Cesar Alan Contreras; Basile Ouedraogo; Rustam Stolkin; Manolis Chiou; Maria Kyrarini

cs.ROAug 15, 2026

GUIDER: Evaluating Goal-Free Human Intent Inference for Teleoperated Manipulation on Real-Robot Data

Authors: Nicholas Kenny, Cesar Alan Contreras, Basile Ouedraogo, Rustam Stolkin, Manolis Chiou, Maria Kyrarini

Organizations: Human–Machine Interaction and Innovation Lab (HMI2), Dept. of Electrical & Computer Engineering, Santa Clara University, Santa Clara, CA, USA. · Extreme Robotics Lab, School of Metallurgy and Materials, University of Birmingham, Birmingham, United Kingdom. · School of Electronic Engineering and Computer Science, Queen Mary University of London, London, United Kingdom. · Intelligent Systems Engineering Dept., Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington, IN, USA

Abstract

This paper presents an evaluation of a goal-free probabilistic framework for human intent inference during robotic manipulation. We deploy the Global User Intent Dual-phase Estimation for Robots (GUIDER) on data collected from a robotic arm to test the manipulation phase across various assistance scenarios, including making tea and fetching medicine. To support operation, we add online probability updates, workspace limits, support-plane filtering, and a grasping mode that prioritizes feasible grasp regions, all of which are tested on the recorded data while preserving its original temporal conditions. Across 20 manipulation steps in three scenarios, GUIDER estimated human intent within the correct grasp-candidate set in all cases and achieved a time to confident prediction of 3.7 s, a remaining time before first grasp of 49.6 s, a prediction stability of 96.4%, and a runtime of 4.857/4.474 s (mean/median) per perceptual phase of intent.

Figures & tables

Panel of Figure 3 — Fig. 4 : Robot performing tasks under the three scenarios

Stage	Mean / Median [s]
2D $\rightarrow$ 3D point cloud reconstruction	0.671 / 0.745
RANSAC plane fitting	0.421 / 0.354
HDBSCAN clustering	0.863 / 0.813
U 2 -Net saliency detection	0.370 / 0.404
FastSAM filtering	2.007 / 1.894
Saliency probability fusion	0.002839 / 0.002732

TABLE I : Main stages timing of the manipulation phase of the GUIDER framework. Reported values are mean/median runtime (Across 20 intent detections)

Fig. 5 : Initial and confident-prediction target distances across all real-world trials. Blue bars indicate the distance to the target at TCP, red markers indicate the initial distance at the start of GUIDER calculations, and the annotated probabilities are normalized by the maximum object probability at each respective time.

Scenario	Trial	Step	TCP	RTCP	Prediction stability	Target within predicted items	Human time to first grasp	Runtime
Making Tea	Trial 1	Tea bag	4.1s	40.9s	100.0%	Yes	45s	5.52s
Making Tea	Trial 1	Water container	4.7s	51.3s	100.0%	Yes	56s	5.21s
Making Tea	Trial 2	Tea bag	0.0s	37.0s	99.8%	Yes	37s	4.60s
Making Tea	Trial 2	Water container	0.3s	94.7s	95.2%	Yes	95s	3.44s
Making Tea	Trial 3	Tea bag	0.0s	41.0s	67.4%	Yes	41s	5.49s
Making Tea	Trial 3	Water container	5.7s	85.3s	100.0%	Yes	91s	4.91s

TABLE II : Per-trial results across the three scenarios. Making Tea and Fetching Medicine are reported step by step in each trial. For Finding Belongings, the single trial is reported object-by-object across the saved target objects. TCP is the time from the GUIDER evolving probability phase start to the first sustained correct prediction. RTCP is the remaining time from that prediction to the operator’s first grasp action. Runtime reports the calculation time of the manipulation perception stage of GUIDER.

Human-Robot Interaction

Human-robot interaction studies how people and robots communicate, coordinate, and safely work or live together.

See all 452 papers

GazeVLA: Learning Human Intention for Robotic Manipulation
Apr 24, 2026Chengyang Li, Kaiyi Xiong, Yuan Xu +3Long-Horizon Robotic Manipulation Robot Skill Learning
Shanghai Jiao Tong University · Peking University · ShanghaiTech University +1
Uncertainty-Aware Action Segmentation for Human-to-Robot Assembly Teleoperation
Jun 6, 2026Fnu Heman, Yixuan Wang, Kolin Xu +7Temporal Action Segmentation Robot Teleoperation
Mechanical Engineering, University of California, Riverside, CA, USA. · Electrical and Computer Engineering, University of Miami, FL, USA.
Cross-Embodiment Affordance Transfer for Robotic Manipulation
Oct 13, 2026Sawera Yaseen, Serge Thill, Rekha RajaAffordance-Based Robotic Manipulation Robotic Affordance Inference
School of Artificial Intelligence, Donders Centre for Cognition, Department of Human-Centred Intelligent Systems, Radboud University, Nijmegen, The Netherlands.

Apr 24, 2026J/K move · Enter open · S save

GazeVLA: Learning Human Intention for Robotic Manipulation

Chengyang Li, Kaiyi Xiong, Yuan Xu, Lei Qian+2

Shanghai Jiao Tong University · Peking University · ShanghaiTech University+1

Long-Horizon Robotic Manipulation Robot Skill Learning Cross-Embodiment Robot Learning Robotic Manipulation Robot Imitation Learning

Open paper AskPDF arXiv

Embodied foundation models have achieved significant breakthroughs in robotic manipulation, yet they still depend heavily on large-scale robot demonstrations. Although recent works have explored leveraging human data to alleviate this dependency, effectively extracting transferable knowledge remains a significant challenge due to the inherent embodiment gap between human and robot. We argue that the intention underlying human actions can serve as a powerful intermediate representation for bridging this gap. In this paper, we introduce a novel framework that explicitly learns and transfers human intention to facilitate robotic manipulation. Specifically, we model intention through gaze, as it naturally precedes physical actions and serves as an observable proxy for human intent. Our model is first pretrained on a large-scale egocentric human dataset to capture human intention and its synergy with action, followed by finetuning on a small set of robot and human data. During inference, the model adopts a Chain-of-Thought reasoning paradigm, sequentially predicting intention before executing the action. Extensive evaluations in simulation and real-world settings, across long-horizon and fine-grained tasks, and under few-shot and robustness benchmarks, show that our method consistently outperforms strong baselines, generalizes better, and achieves state-of-the-art performance. Project page: https://gazevla.github.io .

Abstract

Human-Robot Interaction

Explore similar work

GazeVLA: Learning Human Intention for Robotic Manipulation

Uncertainty-Aware Action Segmentation for Human-to-Robot Assembly Teleoperation

Cross-Embodiment Affordance Transfer for Robotic Manipulation