Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI

Mohammad Zamani; Fatemeh Ziaeetabar

cs.CVAug 19, 2026

Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI

Authors: Mohammad Zamani, Fatemeh Ziaeetabar

Organizations: Department of Computer Science, School of Mathematics, Statistics and Computer Science, College of Science, University of Tehran, Tehran, Iran

Abstract

Egocentric video captures activities from the wearer's perspective, providing a direct view of human attention, hand--object interaction, and goal-directed behavior. This perspective is increasingly important for wearable intelligence, assistive systems, human--robot interaction, and embodied AI, yet it introduces challenges including ego-motion, occlusion, small active objects, viewpoint-dependent appearance, and long-range temporal dependencies. Vision--language models (VLMs) offer a promising foundation for addressing these challenges by linking visual observations with semantic knowledge and natural-language supervision. This survey presents a critical review of VLMs for egocentric video understanding, tracing the progression from conventional recognition architectures to multimodal foundation models and embodied systems. We organize the literature around tasks, datasets, hand--object interaction understanding, temporal reasoning, frame and clip selection, multimodal representation learning, prompting, semantic alignment, and model adaptation. Particular attention is given to graph-based and object-centric reasoning as mechanisms for modeling relations among hands, objects, actions, and scene context over time. We further examine how first-person perception and multimodal foundation models support wearable assistance, robot skill learning, human-to-robot transfer, and embodied decision making. Across the reviewed literature, a consistent limitation emerges: current models recognize visible objects more reliably than evolving interactions, actions, and user intent, especially over long activities. We therefore identify temporally grounded reasoning, interaction-aware supervision, efficient long-video processing, multimodal fusion, graph-enhanced representations, cross-domain generalization, privacy, and trustworthy evaluation as key priorities for deployable embodied intelligence.

Figures & tables

Survey	Venue/Yr	Ego	Data/tasks	VLMs	HOI	Spat.–temp.	Graph/rel.	Prompt/align.	Embodied
Rodin et al. [ 105 ]	CVIU ’21	✓	$\sim$	$\times$	$\sim$	$\sim$	$\times$	$\times$	$\sim$
Bandini & Zariffa [ 5 ]	TPAMI ’23	✓	$\sim$	$\times$	✓	$\times$	$\times$	$\times$	$\sim$
Plizzari et al. [ 94 ]	IJCV ’24	✓	✓	$\sim$	$\sim$	$\sim$	$\times$	$\times$	✓
Li et al. [ 64 ]	MIR ’26	✓	✓	$\sim$	✓	$\sim$	$\times$	$\times$	$\sim$
Zhang et al. [ 160 ]	TPAMI ’24	$\times$	$\sim$	✓	$\times$	$\times$	$\times$	✓	$\times$
Li et al. [ 60 ]	Neurocomp. ’24	$\times$	$\sim$	$\sim$	$\sim$	✓	✓	$\times$	$\times$

Table 1: Coverage of representative recent surveys versus this one (Section 1.5 ). ✓ = covered in depth,

\sim

= partial/incidental,

\times

= not covered. “Graph/rel.” denotes scene-graph, hand–object-interaction-graph, or relational structure used as an organizing lens—the axis on which this survey is distinctive.

Panel of Figure 1 — Figure 1: Roadmap of this survey. The paper is organized as a single trajectory from hand–object interaction to embodied AI. Foundations and datasets (Sec. 2 – 3 ) supply the substrate; the model lineage advances from classical and transformer architectures to egocentric vision-language models (Sec. 4 – 5 ); interaction and reasoning form the conceptual core, spanning hand–object interaction, spatiotemporal reasoning, graph-based reasoning, and prompting and semantic alignment (Sec. 6 – 9 ); and these converge on applications toward embodied intelligence (Sec. 10 ). Open challenges (Sec. 11 ) cut across every stage, shown by the dashed feedback links.

Task	What it is	Key difficulty
Action recognition [ 17 ]	Classify a trimmed clip as a (verb, noun) action	Long-tailed compositions; blur; small active objects
Action anticipation [ 33 , 35 ]	Predict future action(s) before they occur	Inherent uncertainty; many plausible futures
Video captioning [ 56 , 149 , 161 ]	Generate language narrations of events	Grounding fine HOI; long-video coherence
Video retrieval [ 141 , 3 ]	Cross-modal text $\leftrightarrow$ video matching	Many-to-many relevance; long-horizon queries
HOI understanding [ 112 , 14 ]	Detect hands/objects, contact, and state change	Interaction (verb) vs. mere object recognition

Table 2: The five canonical egocentric video-understanding tasks (Section 2.2 ): what each is and its defining first-person difficulty.

Dimension	EPIC-KITCHENS-100	Ego4D	H2O	Charades-Ego	Ego-Exo4D
Scale	100 h	3,670 h	571,645 frames	68.8 h	1,286 h
Participants	37	931	4	112	740
Geographic spread	4 cities	9 countries	3 envs. (lab)	worldwide homes	13 cities
Naturalism	unscripted	unscripted	lab	scripted	unscripted
Viewpoint	ego	ego	ego + static	paired ego-exo	simultaneous ego-exo
Domain	kitchen	daily life	tabletop HOI	household	skilled activity

Table 3: Comparison of major egocentric video datasets along dimensions relevant to vision-language modeling. Ego4D scale refers to the full video collection, while annotations and auxiliary modalities cover subsets; H2O is reported in frames. Figures are version-dependent for Ego4D and Ego-Exo4D (see Section 3.4 ).

Panel of Figure 2 — Figure 2: Evolution of video understanding models (Section 4 ). Three eras, read left to right, each inheriting representational machinery from the preceding era while changing its temporal modeling and supervision strategy. Classical models hand-engineer temporal structure through spatiotemporal convolutions, recurrent aggregation, and parallel appearance–motion streams (Section 4.1 ). Transformer models replace those inductive biases with factorized space-time attention, made trainable at video scale by masked pretraining (Section 4.2 ). Vision-language models replace the fixed label space itself with natural-language supervision, a lineage running CLIP $\to$ VideoCLIP $\to$ Frozen-in-Time $\to$ EgoVLP. Red boxes mark the gap each era left open and the next set out to close; the horizontal dashed links mark what each era inherited rather than replaced—image pretraining and convolutional backbones carried into the transformer era, and transformer space-time encoders carried into egocentric video-language pretraining.

Generation	Representative models	Key innovation	Problem solved	Left unsolved for egocentric
CNN / 3D-CNN	C3D [ 127 ] , I3D [ 12 ] , SlowFast [ 26 ] , TSM [ 68 ]	Spatiotemporal convolutions; inflate 2D pretraining into 3D	Learns motion from pixels with mature image priors	Fixed short clips miss long-horizon, goal-directed activity
CNN–RNN	LRCN [ 20 ] , ConvLSTM [ 113 ] , LSTA [ 119 ]	Per-frame CNN + recurrent temporal aggregation	Variable-length modeling; tracks hands/gaze over time	Weak appearance–motion separation; closed vocabulary
Two-stream	Two-Stream [ 116 ] , TSN [ 134 ] , EPIC-Fusion [ 51 ]	Parallel appearance + motion (and audio) pathways	Explicit motion and multimodal cues for fine actions	Optical flow costly/brittle under ego-motion; no language grounding

Table 4: The three classical (pre-transformer) video-understanding families: what each generation contributed and the gap it left for egocentric vision-language understanding (Section 4.1 ).

Figure 3: The CLIP framework. (Left) Contrastive pretraining jointly aligns an image encoder and a text encoder over batches of image-text pairs; (right) zero-shot prediction constructs a classifier from natural-language prompts. Illustration inspired by [ 102 ] .

Figure 4: The EgoVLP framework: the EgoClip pretraining set curated from Ego4D, a space-time video-language model trained with the EgoNCE objective, and the EgoMCQ development benchmark, with transfer to egocentric downstream tasks. Illustration inspired by [ 69 ] .

Figure 5: The EgoHOIBench diagnostic. A model must select the correct caption from candidates that differ by only a single verb or noun, scoring a trial correct only when both are right; egocentric video-language models exhibit a large verb–noun accuracy gap, revealing only superficial hand–object interaction understanding. Illustration inspired by [ 143 ] .

Model	Category	Backbone / data	Key idea	Reported result
EgoVLP [ 69 ]	Retrieval	TimeSformer; EgoClip (3.8M)	EgoNCE action-aware positives and scene-aware negatives	59.4% nDCG on EK-100 MIR; 90.7%/57.2% on inter-/intra-video EgoMCQ
EgoVLPv2 [ 98 ]	Retrieval	TimeSformer-B + RoBERTa; EgoClip	Gated cross-modal fusion inside the video and language backbones	Consistent gains on EgoTaskQA, CharadesEgo, and EK-100 MIR
HelpingHands [ 157 ]	Retrieval	Dual encoder; Ego4D	Object-aware decoder predicts hand and manipulated-object positions	Strong zero-shot performance on EK-100 MIR and EGTEA
LaViLa [ 161 ]	Captioning	TimeSformer; Ego4D narrations	LLM-based Narrator and Rephraser generate dense and diverse narrations	$+10.1\%$ on EGTEA cls.; $+5.9\%$ on EK-100 MIR
Vid2Seq [ 149 ]	Captioning	Language model + time tokens; YT-Temporal-1B	Time tokens jointly predict event boundaries and captions	State-of-the-art dense captioning performance; not egocentric-specific
EgoHOD [ 93 ]	Action-centric	EgoVideo + motion adapter; Ego4D	HOI-dynamics narrations generated using a detector and an LLM	$+6.3\%$ on EK-100 MIR; $+16.3\%$ on EGTEA in zero-shot evaluation

Table 5: Comparison of representative egocentric vision-language models and relevant multimodal or embodied extensions discussed in Section 5.2 . “EK-100 MIR” denotes EPIC-KITCHENS-100 multi-instance retrieval; “cls.” denotes classification.

Figure 6: The 100DOH hand-object detector. For each hand it jointly predicts a bounding box, side (left/right), contact state (none, self, other-person, portable, or non-portable), and a link to the box of the object in contact, recovering the active object among many distractors. Illustration inspired by [ 112 ] .

Method / Resource	Category	Output / representation	Egocentric focus	Key contribution
100DOH [ 112 ]	Detection	Hand box, side, contact state, in-contact object	Internet-scale	Joint hand + active-object contact detection
Hands23 [ 14 ]	Detection	Pixel segments, grasp, contact vocabulary	In-the-wild	Richer 2D parsing (touch/hold/use)
MANO [ 108 ]	Parametric model	Low-dimensional pose + shape mesh	—	Near-universal hand output representation
HandOccNet [ 88 ]	Reconstruction	MANO mesh	Occlusion-heavy	Feature injection into occluded regions
HaMeR [ 90 ]	Reconstruction	3D hand mesh	In-the-wild	ViT-scale, robust across viewpoints
WildHands [ 97 ]	Reconstruction	3D hand pose	Everyday ego	FOV cues; best 3D pose on ARCTIC ego split

Table 6: Comparison of representative hand–object interaction methods and resources, spanning hand recovery, manipulated-object reasoning, and interaction datasets (Section 6 ).

Figure 7: Egocentric Action Scene Graph (EASG). An interaction is represented as a temporally evolving graph rooted at the camera-wearer node. Verb (action) nodes connect to direct objects, while the manipulated object is linked via active edges to the acting hands. Peripheral objects provide contextual relations. The graph is grounded by bounding boxes on the pre-condition, point-of-no-return (PNR), and post-condition frames, capturing both spatial hand–object relations and their evolution over time. Illustration inspired by [ 106 ] .

Figure 8: Interaction-aware temporal modeling in egocentric video. Hand–object interaction tracklets extracted from long videos are organized into a structured active memory indexed by interaction events and locations. This representation supports efficient temporal reasoning—including sequencing, concurrency, temporal grounding, and retrieval—without reprocessing the full video. Interaction-aware representations and graph-guided frame selection further improve long-horizon reasoning by using hand–object structure as the scaffold for temporal understanding. Illustration inspired by [ 37 ] .

Method	Alignment strategy	Target weakness / finding
EgoVLP [ 69 ]	EgoNCE mines ego-aware positives/negatives	Naive InfoNCE ignores shared verb/noun semantics
EgoVLPv2 [ 98 ]	Cross-modal fusion in the backbone	Late fusion limits video–text interaction
LaViLa [ 161 ]	LLM re-narration (Narrator + Rephraser)	Sparse, unevenly timed Ego4D narrations
EgoNCE++ [ 143 ]	HOI-aware hard-negative captions	Object-over-action (verb) bias
EgoHOD [ 93 ]	HOI-dynamics narrations + motion adapter	Condensed narrations lack fine detail
Wray et al. [ 141 ]	Semantic (many-to-many) relevance evaluation	One-caption-one-video assumption is false

Table 7: Egocentric video–text semantic alignment approaches (Section 9.2 ) and the weakness each targets.

Challenge	Diagnosed failure	Evidence	Promising direction
Temporal reasoning (Sec. 11.1 )	Models exploit spatial/appearance shortcuts rather than order	Text-only 31.3% and single-frame $\sim$ 51% on EgoSchema; single-frame collapses to 9.1% on EgoTempo [ 95 ]	TGPO lifts a 3B model to 49.7% on EgoSchema [ 146 ] ; EgoVLM explores complementary GRPO-based policy optimization [ 131 ]
Interaction understanding (Sec. 11.2 )	Verb–noun asymmetry: nouns recognized, verbs missed	LaViLa scores 74.33% on nouns vs. 46.61% on verbs (EgoHOIBench) [ 143 ]	Generate interaction supervision: EgoNCE++ ( $+34.02$ percentage points in LaViLa++ verb accuracy) and EgoHOD [ 143 , 93 ]
Graph-enhanced VLMs (Sec. 11.3 )	Dense tokens store no explicit relation; symbol–vector gap	Layout-only STLT still recovers the verb without appearance [ 101 ]	Hypergraph-into-LLM, graph-guided self-training, scene-graph forecasting [ 86 , 99 , 92 ]
Efficient frame sampling (Sec. 11.4 )	Uniform sampling wastes budget on uninformative frames	Frame selection substantially affects long-video VLM accuracy and efficiency [ 152 , 153 , 155 ]	Query- and structure-aware selection: SeViLA, Frame-Voyager, FocusGraph [ 152 , 153 , 155 ]
Multimodal learning (Sec. 11.5 )	Video-only models ignore IMU, gaze, audio, and SLAM	IMU signals provide motion cues when visual frames are degraded by ego-motion [ 121 ]	Adaptive fusion: ImageBind, EgoDistill, EgoLife [ 34 , 121 , 150 ]
Embodied deployment (Sec. 11.6 )	Embodiment and visual domain gaps; on-device compute limits	Original EgoSchema evaluation: tested models $<$ 33% vs. $\sim$ 76% human [ 79 ] ; wearable deployment additionally imposes strict compute constraints [ 22 ]	Egocentric world models; privacy and trust benchmarks [ 129 , 2 , 111 ]

Table 8: Summary of open challenges and future directions (Section 11 ). Each row pairs a diagnosed failure with its supporting evidence and the most promising line of attack.

Explore similar work

Jan 10, 2025

From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision

Vision Language Models (VLMs) have achieved strong performance across a wide range of video understanding tasks. However, their viewpoint-invariant training limits their ability to infer egocentric properties, such as human-object interactions, from exocentric video observations. This limitation is particularly critical for applications such as Activities of Daily Living (ADL) monitoring, where understanding egocentric properties is essential but deploying wearable egocentric cameras is often impractical. We propose Ego2ExoVLM, a framework that enables VLMs to infer egocentric properties directly from exocentric videos by leveraging time-synchronized ego-exo video pairs during training. Our key insight is to treat the egocentric viewpoint as privileged supervision, providing rich interaction signals that are available only during training. Ego2ExoVLM consists of two complementary components: Ego2Exo Sequence Distillation, which transfers egocentric reasoning through a language-level sequence distillation objective, and Ego Adaptive Visual Tokens, which encourage the model to surface interaction-relevant cues within exocentric visual representations. To evaluate this capability, we introduce Ego-in-Exo Perception, a benchmark for assessing the understanding of egocentric properties from exocentric videos. We evaluate Ego2ExoVLM on 10 tasks spanning Ego-in-Exo Perception and existing ADL benchmarks, achieving state-of-the-art performance on the ADL-X benchmark suite and consistently outperforming strong baselines on our proposed benchmark. All code, models, and data will be released at https://github.com/dominickrei/EgoExo4ADL.

Dominick Reilly, Manish Kumar Govind, Le Xue +1

University of North Carolina at Charlotte · Elorian AI

Video Understanding VLM Adaptation Egocentric Video Understanding Learning Using Privileged Information

Ask PDF

Nov 23, 2025

EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning

Egocentric video understanding requires procedural reasoning under partial observability and continuously shifting viewpoints. Current multimodal large language models (MLLMs) struggle with this setting, often generating plausible but visually inconsistent or weakly grounded responses. We introduce

\textbf{EgoVITA}

, a framework that decomposes egocentric video reasoning into a structured

\textit{plan-then-verify}

process. The model first generates an

\textbf{egocentric plan}

: a causal sequence of anticipated actions from a first-person perspective. This plan is then evaluated by an

\textbf{exocentric verification}

stage that uses third-person reasoning over the same video to verify its spatiotemporal and logical consistency, without exocentric video input. This decomposition enables cross-perspective feedback without requiring paired ego-exo supervision. To train this reasoning process, we adopt Group Relative Policy Optimization (GRPO) with two dense reward signals: one that grounds anticipated actions in subsequent visual observations and another that reinforces consistent third-person verification.

\textbf{EgoVITA}

achieves state-of-the-art performance on egocentric reasoning benchmarks, outperforming Qwen2.5-VL-7B by

\mathbf{+7.7}

on EgoBlind and

\mathbf{+4.4}

on EgoOrient, while maintaining strong generalization on exocentric video tasks with only

52k

training samples.

Yogesh Kulkarni, Pooyan Fazli

Arizona State University

Egocentric Video Understanding Video Reasoning Visually Grounded Reasoning LLM Reasoning

Ask PDF

Oct 13, 2026

EgoOmniWorld: Unifying Egocentric Understanding via Ego-Action-World Interactions

Egocentric video understanding spans diverse capabilities, modalities, and evaluation formats, yet these are often studied in isolation despite their tight coupling in real-world interaction. We propose the Ego-Action-World interaction loop as a unifying principle: the wearer perceives the world, forms goals, acts, changes the world, and uses feedback to guide subsequent behavior. Based on this view, we introduce EgoOmniWorld, a benchmark organized around four complementary interaction structures: Node captures ego, action, and world states; Edge captures directed relations among them; Chain captures linked dependencies; and Loop captures feedback-closed interactions. We develop an interaction-first, evidence-grounded data pipeline that extracts structured interactions from egocentric videos, grounds them in visual and auditory evidence, and generates questions according to their interaction structure. Required modalities and single- or multi-turn formats follow naturally from the supporting evidence and dependencies. EgoOmniWorld contains 24,166 QA groups, including 20,284 training and 3,882 test samples. Experiments on representative open- and closed-source multimodal large language models reveal substantial and uneven limitations across interaction structures. Models trained on EgoOmniWorld improve in-domain interaction understanding and generalize to existing egocentric benchmarks. EgoOmniWorld provides a unified foundation for evaluating and advancing egocentric video understanding. Data, code, and models will be released.

Songruo Chen, Yuqian Fu, Yu Li +7

Fudan University · KAUST · Universiti Malaya +4

Egocentric Video Understanding Egocentric Video QA Multimodal QA Audio-Visual Understanding

Ask PDF