Gaze2HOI: Gaze-Aware Intention Disambiguation for Egocentric Hand-Object Interaction Forecasting
Authors: Chuanyang Ma, Chenye Wang, Yixuan Wang, Huiling Zhou, Xingqun Qi, Muyi Sun, Man Zhang, Caifeng Shan, Zhenan Sun
Organizations: Beijing University of Posts and Telecommunications, Beijing, China · The Hong Kong University of Science and Technology, Hong Kong, China · North China Electric Power University, Beijing, China · the Suzhou Campus, Nanjing University, Suzhou, China · the Institute of Automation, Chinese Academy of Sciences, Beijing, China
Anticipating how users will interact with their surroundings is essential for proactive systems that provide timely assistance in AR, VR, and embodied applications. Prior work typically forecasts hand-object interactions directly from historical observations, leaving intention and contact geometry implicit. However, similar observation histories can precede semantically or geometrically distinct futures. Inspired by behavioral evidence that gaze can provide early cues for underlying intention, we propose Gaze2HOI, a gaze-aware framework that formulates hand-object interaction forecasting as a hierarchical problem comprising intention inference, contact grounding, and coupled hand-object motion generation. Specifically, Gaze2HOI first weights gaze cues by fixation strength and recency, and uses them to modulate the visual representations of a vision-language model (VLM) through Temporal Gaze Saliency Modulation (TGSM), thereby reducing semantic ambiguity in intention inference. Subsequently, an Anticipatory Contact Grounding (ACG) module maps these structured intentions into explicit contact, support, and placement fields, effectively mitigating geometric ambiguity. Driven by these explicit conditions, a diffusion-based model jointly generates the 6-DoF object trajectories and future articulated motion of both hands. To evaluate Gaze2HOI, we further introduce EgoAIM, a phase-aware benchmark that organizes egocentric hand-object interactions into four phases defined by evolving hand-object relations and provides fine-grained intention annotations. Extensive experiments demonstrate that Gaze2HOI achieves state-of-the-art performance in both intention inference and motion forecasting, even approaching oracle-level end-to-end performance in specific interaction phases. Project page: https://gaze2hoi.github.io/.
Figures & tables
Fig. 1: Egocentric hand-object interaction forecasting. Gaze2HOI forecasts coordinated interactions involving both hands and multiple objects across four phases: Establish Contact , Move , Manipulate , and Put Down . L and R denote the left and right hands, respectively.
Fig. 2: Hierarchical intention disambiguation in Gaze2HOI. Gaze-aware intention inference reduces semantic ambiguity about the intended object and action. Anticipatory Contact Grounding then combines the inferred intentions with object geometry and interaction history to localize likely contact regions, reducing geometric ambiguity. The resulting semantic and geometric cues guide coordinated hand-object motion forecasting.
Fig. 3: Illustration of EgoAIM annotations. (Top) A 2-second window pairing egocentric observations with future hand-object interaction. (Middle) Alignment between 2D egocentric observations and 3D geometric annotations. (Bottom) Structured future intentions annotated separately for each hand.
Fig. 4: Overview of Gaze2HOI. Hierarchical inference from egocentric observations to future hand-object interactions: (a) TGSM modulates VLM visual features with gaze cues to infer structured per-hand intentions. (b) ACG grounds inferred intentions into explicit 3D contact, support, and placement fields. (c) Conditional motion diffusion leverages semantic and geometric conditions to jointly forecast articulated hand motion and 6-DoF object trajectories.
Fig. 5: Qualitative comparison of hand-object interaction forecasting on EgoAIM. From top to bottom, the rows correspond to Establish Contact , Move , Manipulate , and Put Down . Observed inputs and ground-truth (GT) futures are shown alongside predictions from representative baselines and Gaze2HOI.
Fig. 6: Illustration of gaze injection strategies. Each panel shows the image and corresponding prompt for the same egocentric observation. The TGSM overlay is for visualization only; gaze cues are injected through visual feature modulation.
Method
Active-hand EM (%) ↑
Median Point Error (px) ↓
Sample EM@100px (%) ↑
No Gaze
81.76
51.2
72.67
Gaze Text
83.95
38.0
77.44
Gaze Marker
86.20
39.2
80.85
Gaze Trace
85.05
40.2
78.35
Gaze Spotlight
83.22
50.7
76.00
TGSM (ours)
87.84
35.2
82.29
TABLE III: Comparison of gaze injection strategies.
Model
Gaze Injection
Sample EM@100px (%) ↑
GPT-5.4
Gaze Marker
39.21
Gemini 3.1 Pro
Gaze Marker
25.66
Qwen2-VL-2B
TGSM
68.81
Qwen2-VL-7B
TGSM
77.97
Qwen3-VL-2B
TGSM
69.04
TABLE IV: Intention inference performance across VLMs.
Fig. 7: Visualization of geometric disambiguation via ACG. ACG predicts future contact regions using contact priors and observed interactions. Warmer colors indicate higher relative contact intensity within each map.
Intention
H-ADE ↓
O-ADE ↓
Reach-SR ↑
Place-Hit ↑
None
6.49
1.05
56.91
18.94
Predicted
4.16 −35.9%
0.72 −31.4%
92.43 +35.5pp
44.93 +26.0pp
Oracle
3.65 −43.8%
0.55 −47.6%
96.22 +39.3pp
95.59 +76.7pp
TABLE V: Motion forecasting across intention conditions.
Setting
H-ADE ↓
O-ADE ↓
Reach-SR ↑
Place-Hit ↑
w/o TGSM
4.32
0.79
91.61
33.48
w/o Gaze
4.65
0.87
81.25
26.87
w/o ACG
4.35
0.78
88.82
33.48
Full Model
4.16
0.72
92.43
44.93
TABLE VI: End-to-end ablation study of Gaze2HOI.
Fig. 8: Failure cases of Gaze2HOI. Examples of (a) purpose confusion, (b) contact-part confusion, and (c) hand assignment errors, with predicted intentions and the corresponding future interactions shown alongside ground truth.
Hand-object interaction (HOI) is a fundamental human behavior with broad applications in AR/VR, digital humans, and embodied interaction. Existing methods typically require predefined object geometry, object trajectories, or task-specific conditions, limiting their use with natural real-world inputs. To address this, we study a more practical problem of synthesizing 3D hand-object interaction sequences from a single RGB photograph and an open-vocabulary language instruction, and introduce PhotoHOI. PhotoHOI first uses a vision-language model to parse the input image and instruction into a structured task specification, including the interaction object, target region, and spatial relation. It then recovers a compact task-relevant 3D scene and plans a smooth collision-aware object trajectory based on the recovered object states, support relations, and surrounding scene geometry. To synthesize hand motion that generalizes to real-world photographs and unseen objects, it learns transferable task-conditioned contact and contact-conditioned grasp priors from large-scale affordance and HOI data. The grasp is further refined in a learned latent space, constraining the optimization to a plausible hand-pose manifold. Experiments on GRAB and H2O demonstrate improved contact quality and reduced penetration over representative baselines. Results on real-world photographs further demonstrate higher task success and scene consistency, together with generalization to unseen objects and open-vocabulary instructions.
Forecasting future 3D hand pose sequences from egocentric video is essential for understanding human intention and enabling embodied applications such as AR/VR assistance and human-robot interaction. However, this task remains a highly challenging problem because egocentric hand motion is driven by complex human intent, exhibits highly dexterous articulations, and is observed under drastic viewpoint shifts induced by ego-motion. In this work, we introduce EggHand, a foundation-model-based framework for egocentric hand pose forecasting that unifies multimodal semantic reasoning with dynamic motion modeling. Our approach couples an action decoder from a Vision-Language-Action (VLA) model, which captures the structured temporal dynamics of hand motion, with an egocentric video-text encoder that provides viewpoint-aware contextual information learned from large-scale first-person video. Together, these components overcome the brittleness of generic visual encoders under ego-motion and enable joint reasoning over motion, context, and high-level intent-without relying on body pose or external tracking. Experiments on the EgoExo4D dataset show that EggHand sets a new state of the art in forecasting accuracy, remains robust under severe ego-motion, and further enables controllable prediction via language-based task prompts. Project page: https://jyoun9.github.io/EggHand
Human object interaction (HOI), gaze pattern, and their anticipation are intricately linked, providing valuable insights into cognitive processes, intentions, and behavior. However, most existing models handle gaze and actions separately, missing both their interdependence and the advantages of a unified solution. This paper presents a novel unified framework, SAGE (Synchronized Action-GazE), which integrates simultaneous recognition and anticipation of both HOI and human gaze into a single unified end-to-end trainable model. Our approach leverages a transformer-based architecture and incorporates gaze data into spatiotemporal attention mechanisms to simultaneously predict current and future human actions and gaze behavior. We explore this bidirectional relationship between gaze and actions under different scenarios, whether requiring a close-up, detailed view (egocentric) or a wider, more contextual view (exocentric), making our framework versatile for various applications. Additionally, due to lack of datasets for comprehensive analysis of both HOI and gaze in exocentric videos, we establish a new benchmark Exo-Cook to facilitate further research in this domain. Experiments on three benchmark datasets: VidHOI, EGTEA Gaze+, and Exo-Cook show that jointly modeling gaze and actions across current and future frames achieves consistently strong results, often surpassing specialized state-of-the-art models tailored to individual tasks. By unifying actions and attention in a comprehensive way, our work lays the groundwork for more intuitive human-machine interaction.
Chenyi Kuang, Nakul Agarwal
Honda Research Institute, USA · Rensselaer Polytechnic Institute