Leveraging Gaze and Set-of-Mark in VLLMs for Human-Object Interaction Anticipation from Egocentric Videos

Daniele Materia; Francesco Ragusa; Giovanni Maria Farinella

cs.CVApr 4, 2026

Leveraging Gaze and Set-of-Mark in VLLMs for Human-Object Interaction Anticipation from Egocentric Videos

Authors: Daniele Materia, Francesco Ragusa, Giovanni Maria Farinella

Organizations: Department of Mathematics and Computer Science – University of Catania, Italy · Next Vision s.r.l. – Spinoff of the University of Catania, Italy

Abstract

The ability to anticipate human-object interactions is highly desirable in an intelligent assistive system in order to guide users during daily life activities and understand their short and long-term goals. Creating systems with such capabilities requires to approach several complex challenges. This work addresses the problem of human-object interaction anticipation in Egocentric Vision using Vision Large Language Models (VLLMs). We tackle key limitations in existing approaches by improving visual grounding capabilities through Set-of-Mark prompting and understanding user intent via the trajectory formed by the user's most recent gaze fixations. To effectively capture the temporal dynamics immediately preceding the interaction, we further introduce a novel inverse exponential sampling strategy for input video frames. Experiments conducted on the egocentric dataset HD-EPIC demonstrate that our method surpasses state-of-the-art approaches for the considered task, showing its model-agnostic nature.

Figures & tables

Figure 1: Conceptual scheme of the VQA Interaction Anticipation task.

Figure 2: The proposed architecture for the human-object interaction anticipation task. It is composed of 4 main modules: 1) Set-of-Mark module, 2) Gaze module, 3) Sampling module, and 4) VLLM module.

Panel of Figure 3 — Figure 3: The figure illustrates how the input frame is processed by the SoM and Gaze modules. The input RGB frame (a) is first processed by the SoM module (b) and the Gaze module (c). Their outputs are then fused (d) to obtain a visual representation that incorporates both spatial and intention-related information.

Figure 4: Visualization of our inverse exponential sampling strategy with $n=10$ . Setting $\lambda=0$ (left) results in a uniform distribution, while $\lambda>0$ (right) concentrates the sampled frames (the blue dots) on the instants immediately preceding the interaction. The blue dots represent the $n-1$ probabilistically sampled frames; the final frame, which is always part of the input, is omitted for clarity.

Approach	Accuracy (%)
K-Net [ 36 ]	15.7
T-CoT [ 42 ]	15.8
LLaVA-OneVision 7B [ 16 ]	20.4
Gemini 1.5 Pro (HD-EPIC baseline) [ 25 ]	21.0
Qwen2.5VL-Ricoh [ 45 ]	22.0
Ours (with LLaVA-OneVision 7B)	27.2

Table 1: Results for the Interaction Anticipation task. The best results are reported in bold , whereas the second best results are underlined .

Panel of Figure 5 — Figure 5: Qualitative example showing the positive impact of incorporating gaze information. (a) Classic VLLM, (b) proposed approach considering VLLM and SoM modules, (c) proposed approach considering VLLM and Gaze modules, (d) proposed approach considering VLLM, SoM and Gaze modules.

Panel of Figure 6 — Figure 6: A challenging scenario highlighting the positive effect of the synergy between Set-of-Mark and Gaze .

Strategy	$\lambda=0$	$\lambda=\tfrac{1}{100}$	$\lambda=\tfrac{1}{50}$	$\lambda=\tfrac{1}{25}$	$\lambda=\tfrac{1}{10}$	$\lambda=1$
VLLM only	0.184	0.201	0.218	0.229	0.249	0.248
SoM	0.188	0.223	0.234	0.231	0.248	0.258
Gaze	0.203	0.210	0.220	0.230	0.252	0.256
SoM + Gaze	0.201	0.222	0.243	0.249	0.272	0.261

Table 2: Impact of SoM, Gaze, and Sampling modules when adopting LLaVA-OneVision as the VLLM, with sample size

n=15

frames.

Sample size	$\lambda=0$	$\lambda=\tfrac{1}{100}$	$\lambda=\tfrac{1}{50}$	$\lambda=\tfrac{1}{25}$	$\lambda=\tfrac{1}{10}$	$\lambda=1$
$n=5$	0.196	0.225	0.228	0.240	0.250	0.235
$n=10$	0.214	0.228	0.250	0.254	0.259	0.240
$n=15$	0.201	0.222	0.243	0.249	0.272	0.261
$n=20$	0.222	0.217	0.216	0.244	0.252	0.251
$n=25$	0.212	0.215	0.222	0.235	0.241	0.251
$n=30$	0.210	0.207	0.212	0.223	0.237	0.240

Table 3: Impact of the chosen sample size

n

on LLaVA-OneVision’s performance. The strategy used here is SoM + Gaze .

Strategy	1 fps	2 fps	4 fps	8 fps	16 fps
VLLM only	0.205	0.254	0.256	0.226	0.218
SoM	0.207	0.220	0.242	0.202	0.181
Gaze	0.193	0.236	0.264	0.256	0.228
SoM + Gaze	0.153	0.275	0.266	0.273	0.271

Table 4: Performance of Gemini 2.0 Flash across varying input frame rates (fps).

Human-Object Interaction

Human-object interaction studies how people perceive, manipulate, and coordinate with objects in images, videos, and embodied settings.

See all 112 papers

Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI
Aug 19, 2026Mohammad Zamani, Fatemeh ZiaeetabarHand-Object Interaction Detection Egocentric Video Understanding
Department of Computer Science, School of Mathematics, Statistics and Computer Science, College of Science, University of Tehran, Tehran, Iran
From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision
Jan 10, 2025Dominick Reilly, Manish Kumar Govind, Le Xue +1Video Understanding VLM Adaptation
University of North Carolina at Charlotte · Elorian AI
EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning
Nov 23, 2025Yogesh Kulkarni, Pooyan FazliEgocentric Video Understanding Video Reasoning
Arizona State University

Aug 19, 2026J/K move · Enter open · S save

Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI

Mohammad Zamani, Fatemeh Ziaeetabar

Department of Computer Science, School of Mathematics, Statistics and Computer Science, College of Science, University of Tehran, Tehran, Iran

Hand-Object Interaction Detection Egocentric Video Understanding Embodied AI Temporal Video Understanding Human-Object Interaction VLMs for Robotics

Open paper AskPDF arXiv

Egocentric video captures activities from the wearer's perspective, providing a direct view of human attention, hand--object interaction, and goal-directed behavior. This perspective is increasingly important for wearable intelligence, assistive systems, human--robot interaction, and embodied AI, yet it introduces challenges including ego-motion, occlusion, small active objects, viewpoint-dependent appearance, and long-range temporal dependencies. Vision--language models (VLMs) offer a promising foundation for addressing these challenges by linking visual observations with semantic knowledge and natural-language supervision. This survey presents a critical review of VLMs for egocentric video understanding, tracing the progression from conventional recognition architectures to multimodal foundation models and embodied systems. We organize the literature around tasks, datasets, hand--object interaction understanding, temporal reasoning, frame and clip selection, multimodal representation learning, prompting, semantic alignment, and model adaptation. Particular attention is given to graph-based and object-centric reasoning as mechanisms for modeling relations among hands, objects, actions, and scene context over time. We further examine how first-person perception and multimodal foundation models support wearable assistance, robot skill learning, human-to-robot transfer, and embodied decision making. Across the reviewed literature, a consistent limitation emerges: current models recognize visible objects more reliably than evolving interactions, actions, and user intent, especially over long activities. We therefore identify temporally grounded reasoning, interaction-aware supervision, efficient long-video processing, multimodal fusion, graph-enhanced representations, cross-domain generalization, privacy, and trustworthy evaluation as key priorities for deployable embodied intelligence.

Abstract

Human-Object Interaction

Explore similar work

Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI

From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision

EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning