Intuitive Human-Robot Interaction: Development and Evaluation of a Gesture-Based User Interface for Object Selection

Bijan Kavousian; Oliver Petrovic; Werner Herfs

cs.ROApr 7, 2026

Intuitive Human-Robot Interaction: Development and Evaluation of a Gesture-Based User Interface for Object Selection

Authors: Bijan Kavousian, Oliver Petrovic, Werner Herfs

Organizations: Laboratory for Machine Tools and Production Engineering WZL of RWTH Aachen, Steinbachstraße 25, D-52072 Aachen

Abstract

Gestures are a natural form of communication between humans and can also be leveraged for human-robot interaction. This work presents a gesture-based user interface for object selection using pointing and click gestures. An experiment with 20 participants evaluates accuracy and selection time, demonstrating the potential for efficient collaboration.

Explore similar work

Glance-Say: Multimodal Human-Robot Collaboration and Intent Recognition via Sticky Glance
Mar 6, 2026Yuzhi Lai, Shenghai Yuan, Peizheng Li +2Robotic Control Human-AI Interaction
University of Tuebingen, Geschwister-Scholl-Platz, 72074 Germany · Nanyang Technological University, 50 Nanyang Avenue, Singapore 639798
HumanoidToolBench: Benchmarking Humanoid Tool Use from Selection to Mobile Execution
Oct 1, 2026Kyochul Jang, Seohyeon Park, Ohchul Kwon +9Mobile Manipulation Robotic Tool Use
Seoul National University · University of Massachusetts Amherst · Google Research
Point & Grasp: Flexible Selection of Out-of-Reach Objects Through Probabilistic Cue Integration
Apr 24, 2026Xuejing Luo, Hee-Seung Moon, Christian Holz +1
Department of Information and Communication Engineering, School of Electrical Engineering Aalto University Helsinki, Finland ELLIS Institute Helsinki, Finland · School of Computer Science and Engineering Chung-Ang University Seoul, Republic of Korea · ETH Zürich Zurich, Switzerland

Mar 6, 2026J/K move · Enter open · S save

Glance-Say: Multimodal Human-Robot Collaboration and Intent Recognition via Sticky Glance

Yuzhi Lai, Shenghai Yuan, Peizheng Li, Benjamin Kiefer+1

University of Tuebingen, Geschwister-Scholl-Platz, 72074 Germany · Nanyang Technological University, 50 Nanyang Avenue, Singapore 639798

Robotic Control Human-AI Interaction Human-Robot Collaboration Assistive Robotics

Open paper AskPDF arXiv

Gaze and speech are promising interaction modalities for individuals with motor impairments, yet robust intent recognition in multi-object environments remains challenging due to micro-saccades, semantic ambiguity, and viewpoint changes. This paper presents a multimodal interaction framework for assistive robotic manipulation. We propose a sticky-glance algorithm that stabilizes gaze-based intent by jointly accumulating geometric distance and directional evidence, enabling robust real-time target selection and switching. We further introduce Glance-Say, a gaze-speech interaction paradigm in which gaze specifies objects and speech specifies actions, together with a continuous shared-control scheme that provides high-readiness robot motion and human-in-the-loop feedback. Experiments demonstrate a tracking rate of 0.92 for moving targets, selection accuracy of 0.97 for static targets, and reduced task duration. These results indicate improved robustness, efficiency, and usability over representative interaction paradigms.

Abstract

Explore similar work

Glance-Say: Multimodal Human-Robot Collaboration and Intent Recognition via Sticky Glance

HumanoidToolBench: Benchmarking Humanoid Tool Use from Selection to Mobile Execution

Point & Grasp: Flexible Selection of Out-of-Reach Objects Through Probabilistic Cue Integration