HEIR: Harness Egocentric Intent for Human-Robot Interactions
Authors: Yichi Zhang, Bo Wang, Zhenghan Zhu, Yutong Han, Jiayin Zou, Gao Kejun, Chang Liu, Shaoxuan Xie, Chi Zhang, Shi Haoyang, Zongxuan Lai, Sen Cui, Guocai Yao, Miao Liu
Organizations: Tsinghua University · Beijing Academy of Artificial Intelligence · Fudan University · Yangu (Beijing) Robotics Intelligent Technology Co., Ltd.
For robots to assist, serve, and collaborate with humans, they must not only interpret human requests conveyed through language and multimodal intentional cues, but also execute long-horizon tasks that interleave navigation and manipulation. In this paper, we introduce HEIR, a resource for studying how to Harness Egocentric Intent for Human-Robot Interaction. HEIR pairs human speech, measured gaze, and egocentric video with robot observations and teleoperated navigation-manipulation trajectories, while each session preserves successive episodes, with each request fulfilled through multiple execution subtasks. The captured data, together with human-authored annotations, support the study of intent grounding, execution planning, and task completion within a unified interaction setting. To demonstrate the value of HEIR, we conduct extensive offline experiments to evaluate how models benefit from additional human-centric context across these tasks, together with real-robot experiments that assess execution task completion. HEIR thus provides a valuable resource for studying user-centric robot interaction in naturalistic household environments. Our demo and dataset are available online.
Figures & tables
Figure 1: Example long-horizon HRI session in HEIR . Each recorded session contains successive episodes, each initiated by a natural human request grounded in the user’s egocentric view and gaze. Each request is mapped to a clarified task specification and fulfilled through one or more navigation and manipulation subtasks, while the top-down trajectory visualizes the corresponding spatial execution and state transitions across episodes.
Work
Real- world
H&R views
Gaze
Speech
Intent annot.
Long horizon
Navi.& Mani.
# of Subtasks
Gaze2Act [ 69 ]
✓
✓
✓
✓
✓
×
×
653
HABIT [ 50 ]
✓
✓
×
×
✓
✓
×
38,412
HARMONIC [ 36 ]
✓
×
✓
×
×
×
×
480
NatSGLD [ 48 ]
×
✓
×
✓
✓
×
×
1,143 ∗
RoboVQA [ 47 ]
✓
×
×
×
✓
✓
✓
61,153
RoboCasa365 [ 35 ]
×
×
×
×
×
✓
✓
55,000 ∗
Table 1: Comparison of representative robot datasets and systems. H&R denotes paired human and robot views. Units and granularity vary. HEIR counts navigation/manipulation segments. Gaze2Act/HARMONIC use protocol-mapped demos/trials. HABIT counts robot subtasks computed from released subtask annotations, RoboVQA robot-only medium-horizon annotations, and TEACh EDH segments. ∗ marks original-unit exceptions: NatSGLD commands with paired demonstrations, RoboCasa365 teleoperated demonstrations, WatchAct evaluation instances, and EDITH interaction demonstrations.
Figure 2: Overview of the HEIR data components . (a) Hardware setup combining human egocentric sensing with gaze and audio, together with robot-side sensory signals and teleoperated execution. (b) Each continuous session contains multiple episodes initiated by natural human requests, with each episode composed of navigation and manipulation subtasks. (c) Human annotators first create temporally grounded task annotations from the recorded interactions, which are then enriched and refined with VLM assistance into structured instructions and subtask descriptions.
Figure 3: Statistics and composition of HEIR . Our dataset contains 300 sessions across 20 task categories, totaling 38.56 hours, 1,836 episodes, and 6,416 subtasks. We further report interaction composition, train/test splits, and participant statistics.
Grounding
Planning
Visibility
Input condition
Slot F1 ↑
Loc. Acc. ↑
Plan EM ↑
RV F1 ↑
Textual Instruction only
59.52
20.47
30.27
N/A
Robot View only
71.22
54.86
63.51
60.33
Human + Robot
74.85
71.13
72.43
57.71
Human + Robot + center prior
74.91
71.39
75.14
61.61
Table 2: Input ablations for Gemma-4-31B. ↑ denotes higher is better. Bold indicates the best score across input conditions for each model. RV is N/A without robot views.
Grounding
Planning
Visibility
History
Slot F1 ↑
Loc. Acc. ↑
Plan EM ↑
RV F1 ↑
Ground-truth history
81.25
80.31
82.70
64.88
No history
76.69
77.17
76.22
63.13
Predicted history
77.89
79.27
81.08
63.51
Table 3: Intent understanding results for Gemma-4-31B under different history conditions. ↑ denotes higher is better. Bold indicates the best score excluding ground-truth history. The no-history row is a separate run from the full-input row of Tab. 2 ; see Appendix D.2 .
Planning
Completion
Model
Setting
Acc ↑
Macro-F1 ↑
Next Type ↑
Core ↑
Completion F1 ↑
Cosmos3-Nano
Pre-trained
33.07
21.03
0.00
0.00
0.00
[ 37 ]
Fine-tuned
68.90
70.81
54.87
44.25
32.32
Qwen3.5-4B
Pre-trained
46.85
45.71
49.56
32.74
26.61
[ 41 ]
Fine-tuned
83.46
84.33
75.22
61.06
36.85
Gemma-4-E4B-it
Pre-trained
38.58
33.48
23.01
19.47
11.30
Table 4: Planner results before and after fine-tuning on HEIR. All planners receive the annotated intent. All scores are percentages; ↑ denotes higher is better. Bold indicates the better score within each model.
Action prediction
Stopping
Model
Weights
Accuracy ↑
Macro-F1 ↑
STOP Recall † ↑
STOP F1 † ↑
NaVid
Original
48.74
45.38
53.49
30.95
Fine-tuned
86.59
86.15
80.72
60.05
StreamVLN
Original
46.84
39.65
19.39
20.34
Fine-tuned
86.36
86.20
75.49
67.02
InternVLA-N1
Original
49.08
45.41
58.28
31.48
Table 5: Offline VLN action prediction before and after fine-tuning with English subgoal instructions. All scores are percentages; ↑ denotes higher is better. Bold denotes the better weight setting within each model block. Original denotes the released model without project LoRA; fine-tuned checkpoints are selected on validation. † denotes annotation-bounded STOP proxy scores.
First output
Action chunk
Model
Joint MSE ↓ (rad 2 )
EEF Pos. ↓ (cm)
EEF Rot. ↓ ( ∘ )
Joint MSE ↓ (rad 2 )
EEF Pos. ↓ (cm)
EEF Rot. ↓ ( ∘ )
DreamZero
0.002809
1.489
3.455
0.013215
2.604
5.244
π0 -FAST
0.001644
0.808
1.093
0.009090
1.466
1.858
π0
0.000588
0.396
0.653
0.003688
0.937
1.499
π0.5
0.000468
0.373
0.652
0.003398
0.911
1.519
FastWAM
0.000374
0.369
0.673
0.002921
0.806
1.327
Table 6: Offline VLA/WAM action prediction across model variants. All models are fine-tuned using ground-truth subgoal descriptions as instructions. Joint MSE is reported in rad 2 , end-effector position error in cm, and end-effector rotation error in degrees (lower is better).
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Stream
Source
Recorded information
Human scene video
Pupil Labs Neon
1600×1200 RGB video; nominally 30 FPS
Human speech
Neon microphone
Audio synchronized with human video
Human gaze
Neon eye tracking
Timestamped measured gaze, overlaid on scene video
Robot head video
Two head cameras
1280×720 RGB video per camera; 30 FPS
Robot wrist video
Two wrist cameras
640×360 RGB video per camera; 30 FPS
Robot state–action
R1 Lite logs
Arm, gripper, torso, and base states; recorded actions in Parquet
Appendix
Table 7: Human and robot data streams used for annotation. Video specifications refer to the recorded streams; gaze is matched to scene frames by timestamp.
Subtask / interval (s)
Subgoal
Completion condition
S001: Manipulation [184.20,191.57)
Straighten the lowered torso while keeping both grippers empty and the base stationary.
Torso straightened; both grippers empty; base remains at the counter.
S002: Navigation [191.60,205.47)
Return from the counter to the seated human with both grippers empty.
Base stopped facing the seated human; both grippers empty.
S003: Manipulation [205.50,238.50)
Lower the torso; receive the stapler with the left gripper, then the tape roll with the right; retract both arms and straighten the torso.
Stapler held on the left and tape on the right; both arms retracted; torso straightened.
S004: Navigation [238.50,262.43)
Navigate to the kitchen counter and white box while carrying the stapler in the left gripper and the tape roll in the right.
Base stopped at the counter; white box visible; both objects retained in their respective grippers.
S005: Manipulation [262.47,283.70)
Lower the torso; place the stapler into the white box and retract the left arm, then place the tape roll into the same box and retract the right arm.
Both objects inside the box; both grippers empty and arms retracted; torso remains lowered.
Appendix
Table 8: Subtask annotations for episode E004. Types and intervals are human annotated; subgoals and completion conditions are VLM-enriched descriptions, condensed here from the annotated record. Intervals are in seconds on the synchronized session timeline.
Figure 4: Trajectory comparison between MapAnything–wheel-base fusion and HTC VIVE Tracker measurements.
Figure 5: Real-robot execution pipeline of HEIR. For the request “Bring me that folder.”, the intent reasoner infers a structured intent, and the closed-loop planner then dispatches subtasks to the navigation and manipulation backends until Done .
Setting
NaVid
StreamVLN
InternVLA-N1
Initial action adaptation
3 epochs; 10−4
3 epochs; 10−4
3 epochs; 10−4
Selected action checkpoint
Epoch 2
Epoch 3
Epoch 3
Joint action/visibility adaptation
3 epochs; 2×10−5
Not applied
3 epochs; 2×10−5
Selected joint checkpoint
Epoch 3
—
Epoch 3
Termination-aware adaptation
1 epoch; 10−5
1 epoch; 10−5
1 epoch; 10−5
Adapter used for testing
Joint epoch 3
Action epoch 3
Termination-aware epoch 1
Appendix
Table 9: VLN adaptation stages and selected checkpoints. Epoch counts refer to individual stages; each later stage starts from the selected checkpoint of the preceding applicable stage.
Model
Batch
Chunk
Optimizer
Peak LR
Weight decay
π0
960
50
AdamW
2.5×10−5
10−10
π0.5
1344
50
AdamW
2.5×10−5
10−10
π0 -FAST
208
32
AdamW
2.5×10−5
10−10
FastWAM
2816
32
AdamW
2.5×10−5
0.01
DreamZero
32
24
AdamW
10−5
10−5
Lingbot-VLA2
256
50
Muon/AdamW
10−5
0
Appendix
Table 10: VLA/WAM fine-tuning configuration. All models train for 5 epochs from pretrained weights. Batch denotes the effective global batch; chunk denotes the native action sequence length.
Grounding
Planning
Visibility
Input condition
Slot F1 ↑
Loc. Acc. ↑
Plan EM ↑
RV F1 ↑
Qwen3.8-27B
Textual Instruction only
60.18
22.05
29.19
N/A
Robot View only
71.83
54.33
54.05
61.23
Human + Robot
74.24
62.47
64.86
60.68
Appendix
Table 11: Input ablations for Qwen3.8-27B, Muse-Glimmer-30B, and InternVL3.5-38B. All scores are scaled by 100; ↑ denotes higher is better. Bold indicates the best score across input conditions for each model. RV is N/A without robot views.
Grounding
Planning
Visibility
History
Slot F1 ↑
Loc. Acc. ↑
Plan EM ↑
RV F1 ↑
Qwen3.8-27B
Ground-truth history
81.25
79.53
76.76
65.19
No history
76.85
73.23
68.92
63.25
Predicted history
78.43
76.38
69.19
64.36
Appendix
Table 12: Intent understanding results for Qwen3.8-27B, Muse-Glimmer-30B, and InternVL3.5-38B under different history conditions. All scores are scaled by 100; ↑ denotes higher is better. Bold indicates the best score within each model, excluding ground-truth history. The no-history rows are separate runs from the full-input rows of Tab. 11 ; see Appendix D.2 .
Planning
Completion
Intent source
Next Type ↑
Core ↑
Completion F1 ↑
Ground-truth intent
89.00
65.00
42.79
Predicted intent (full)
87.00
56.00
42.73
Predicted intent (robot-only)
84.00
47.00
39.51
No intent (instruction + robot view)
83.00
40.00
39.85
Appendix
Table 13: Initial-subtask planning with different intent sources. Qwen3.5-4B planners are evaluated at the first decision, with no active subtask or completed-subtask history. All scores are percentages; ↑ denotes higher is better. Bold indicates the best score among the non-oracle conditions.