cs.RONov 12, 2026

HEIR: Harness Egocentric Intent for Human-Robot Interactions

Authors: Yichi Zhang, Bo Wang, Zhenghan Zhu, Yutong Han, Jiayin Zou, Gao Kejun, Chang Liu, Shaoxuan Xie, 

Organizations: Tsinghua University · Beijing Academy of Artificial Intelligence · Fudan University · Yangu (Beijing) Robotics Intelligent Technology Co., Ltd.

Abstract

For robots to assist, serve, and collaborate with humans, they must not only interpret human requests conveyed through language and multimodal intentional cues, but also execute long-horizon tasks that interleave navigation and manipulation. In this paper, we introduce HEIR, a resource for studying how to Harness Egocentric Intent for Human-Robot Interaction. HEIR pairs human speech, measured gaze, and egocentric video with robot observations and teleoperated navigation-manipulation trajectories, while each session preserves successive episodes, with each request fulfilled through multiple execution subtasks. The captured data, together with human-authored annotations, support the study of intent grounding, execution planning, and task completion within a unified interaction setting. To demonstrate the value of HEIR, we conduct extensive offline experiments to evaluate how models benefit from additional human-centric context across these tasks, together with real-robot experiments that assess execution task completion. HEIR thus provides a valuable resource for studying user-centric robot interaction in naturalistic household environments. Our demo and dataset are available online.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. EDITH: Teaching Robot to Follow Nonverbal Human Intent from Egocentric Vision and Gaze

    Jun 9, 2026Dongjun Lee, Juheon Choi, Dong Kyu Shin +2Robotic ControlLanguage-Conditioned Robot Control

  2. GazeVLA: Learning Human Intention for Robotic Manipulation

    Apr 24, 2026Chengyang Li, Kaiyi Xiong, Yuan Xu +3Long-Horizon Robotic ManipulationRobot Skill Learning

  3. Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities

    Sep 7, 2026Raphael Lorenzo-Louis, Bertrand Luvison, Serena IvaldiHuman Behavior PredictionHuman-Robot Interaction