cs.CVAug 19, 2026
SaveVision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI
Organizations: Department of Computer Science, School of Mathematics, Statistics and Computer Science, College of Science, University of Tehran, Tehran, Iran
Abstract
Egocentric video captures activities from the wearer's perspective, providing a direct view of human attention, hand--object interaction, and goal-directed behavior. This perspective is increasingly important for wearable intelligence, assistive systems, human--robot interaction, and embodied AI, yet it introduces challenges including ego-motion, occlusion, small active objects, viewpoint-dependent appearance, and long-range temporal dependencies. Vision--language models (VLMs) offer a promising foundation for addressing these challenges by linking visual observations with semantic knowledge and natural-language supervision. This survey presents a critical review of VLMs for egocentric video understanding, tracing the progression from conventional recognition architectures to multimodal foundation models and embodied systems. We organize the literature around tasks, datasets, hand--object interaction understanding, temporal reasoning, frame and clip selection, multimodal representation learning, prompting, semantic alignment, and model adaptation. Particular attention is given to graph-based and object-centric reasoning as mechanisms for modeling relations among hands, objects, actions, and scene context over time. We further examine how first-person perception and multimodal foundation models support wearable assistance, robot skill learning, human-to-robot transfer, and embodied decision making. Across the reviewed literature, a consistent limitation emerges: current models recognize visible objects more reliably than evolving interactions, actions, and user intent, especially over long activities. We therefore identify temporally grounded reasoning, interaction-aware supervision, efficient long-video processing, multimodal fusion, graph-enhanced representations, cross-domain generalization, privacy, and trustworthy evaluation as key priorities for deployable embodied intelligence.
Figures & tables
| Survey | Venue/Yr | Ego | Data/tasks | VLMs | HOI | Spat.–temp. | Graph/rel. | Prompt/align. | Embodied |
|---|---|---|---|---|---|---|---|---|---|
| Rodin et al. [ 105 ] | CVIU ’21 | ✓ | |||||||
| Bandini & Zariffa [ 5 ] | TPAMI ’23 | ✓ | ✓ | ||||||
| Plizzari et al. [ 94 ] | IJCV ’24 | ✓ | ✓ | ✓ | |||||
| Li et al. [ 64 ] | MIR ’26 | ✓ | ✓ | ✓ | |||||
| Zhang et al. [ 160 ] | TPAMI ’24 | ✓ | ✓ | ||||||
| Li et al. [ 60 ] | Neurocomp. ’24 | ✓ | ✓ |
Table 1: Coverage of representative recent surveys versus this one (Section 1.5 ). ✓ = covered in depth, = partial/incidental, = not covered. “Graph/rel.” denotes scene-graph, hand–object-interaction-graph, or relational structure used as an organizing lens—the axis on which this survey is distinctive.
Figure 1: Roadmap of this survey. The paper is organized as a single trajectory from hand–object interaction to embodied AI. Foundations and datasets (Sec. 2 – 3 ) supply the substrate; the model lineage advances from classical and transformer architectures to egocentric vision-language models (Sec. 4 – 5 ); interaction and reasoning form the conceptual core, spanning hand–object interaction, spatiotemporal reasoning, graph-based reasoning, and prompting and semantic alignment (Sec. 6 – 9 ); and these converge on applications toward embodied intelligence (Sec. 10 ). Open challenges (Sec. 11 ) cut across every stage, shown by the dashed feedback links.
| Task | What it is | Key difficulty |
|---|---|---|
| Action recognition [ 17 ] | Classify a trimmed clip as a (verb, noun) action | Long-tailed compositions; blur; small active objects |
| Action anticipation [ 33 , 35 ] | Predict future action(s) before they occur | Inherent uncertainty; many plausible futures |
| Video captioning [ 56 , 149 , 161 ] | Generate language narrations of events | Grounding fine HOI; long-video coherence |
| Video retrieval [ 141 , 3 ] | Cross-modal text video matching | Many-to-many relevance; long-horizon queries |
| HOI understanding [ 112 , 14 ] | Detect hands/objects, contact, and state change | Interaction (verb) vs. mere object recognition |
Table 2: The five canonical egocentric video-understanding tasks (Section 2.2 ): what each is and its defining first-person difficulty.
| Dimension | EPIC-KITCHENS-100 | Ego4D | H2O | Charades-Ego | Ego-Exo4D |
|---|---|---|---|---|---|
| Scale | 100 h | 3,670 h | 571,645 frames | 68.8 h | 1,286 h |
| Participants | 37 | 931 | 4 | 112 | 740 |
| Geographic spread | 4 cities | 9 countries | 3 envs. (lab) | worldwide homes | 13 cities |
| Naturalism | unscripted | unscripted | lab | scripted | unscripted |
| Viewpoint | ego | ego | ego + static | paired ego-exo | simultaneous ego-exo |
| Domain | kitchen | daily life | tabletop HOI | household | skilled activity |
Table 3: Comparison of major egocentric video datasets along dimensions relevant to vision-language modeling. Ego4D scale refers to the full video collection, while annotations and auxiliary modalities cover subsets; H2O is reported in frames. Figures are version-dependent for Ego4D and Ego-Exo4D (see Section 3.4 ).
Figure 2: Evolution of video understanding models (Section 4 ). Three eras, read left to right, each inheriting representational machinery from the preceding era while changing its temporal modeling and supervision strategy. Classical models hand-engineer temporal structure through spatiotemporal convolutions, recurrent aggregation, and parallel appearance–motion streams (Section 4.1 ). Transformer models replace those inductive biases with factorized space-time attention, made trainable at video scale by masked pretraining (Section 4.2 ). Vision-language models replace the fixed label space itself with natural-language supervision, a lineage running CLIP VideoCLIP Frozen-in-Time EgoVLP. Red boxes mark the gap each era left open and the next set out to close; the horizontal dashed links mark what each era inherited rather than replaced—image pretraining and convolutional backbones carried into the transformer era, and transformer space-time encoders carried into egocentric video-language pretraining.
| Generation | Representative models | Key innovation | Problem solved | Left unsolved for egocentric |
|---|---|---|---|---|
| CNN / 3D-CNN | C3D [ 127 ] , I3D [ 12 ] , SlowFast [ 26 ] , TSM [ 68 ] | Spatiotemporal convolutions; inflate 2D pretraining into 3D | Learns motion from pixels with mature image priors | Fixed short clips miss long-horizon, goal-directed activity |
| CNN–RNN | LRCN [ 20 ] , ConvLSTM [ 113 ] , LSTA [ 119 ] | Per-frame CNN + recurrent temporal aggregation | Variable-length modeling; tracks hands/gaze over time | Weak appearance–motion separation; closed vocabulary |
| Two-stream | Two-Stream [ 116 ] , TSN [ 134 ] , EPIC-Fusion [ 51 ] | Parallel appearance + motion (and audio) pathways | Explicit motion and multimodal cues for fine actions | Optical flow costly/brittle under ego-motion; no language grounding |
Table 4: The three classical (pre-transformer) video-understanding families: what each generation contributed and the gap it left for egocentric vision-language understanding (Section 4.1 ).
Figure 3: The CLIP framework. (Left) Contrastive pretraining jointly aligns an image encoder and a text encoder over batches of image-text pairs; (right) zero-shot prediction constructs a classifier from natural-language prompts. Illustration inspired by [ 102 ] .
Figure 4: The EgoVLP framework: the EgoClip pretraining set curated from Ego4D, a space-time video-language model trained with the EgoNCE objective, and the EgoMCQ development benchmark, with transfer to egocentric downstream tasks. Illustration inspired by [ 69 ] .
Figure 5: The EgoHOIBench diagnostic. A model must select the correct caption from candidates that differ by only a single verb or noun, scoring a trial correct only when both are right; egocentric video-language models exhibit a large verb–noun accuracy gap, revealing only superficial hand–object interaction understanding. Illustration inspired by [ 143 ] .
| Model | Category | Backbone / data | Key idea | Reported result |
|---|---|---|---|---|
| EgoVLP [ 69 ] | Retrieval | TimeSformer; EgoClip (3.8M) | EgoNCE action-aware positives and scene-aware negatives | 59.4% nDCG on EK-100 MIR; 90.7%/57.2% on inter-/intra-video EgoMCQ |
| EgoVLPv2 [ 98 ] | Retrieval | TimeSformer-B + RoBERTa; EgoClip | Gated cross-modal fusion inside the video and language backbones | Consistent gains on EgoTaskQA, CharadesEgo, and EK-100 MIR |
| HelpingHands [ 157 ] | Retrieval | Dual encoder; Ego4D | Object-aware decoder predicts hand and manipulated-object positions | Strong zero-shot performance on EK-100 MIR and EGTEA |
| LaViLa [ 161 ] | Captioning | TimeSformer; Ego4D narrations | LLM-based Narrator and Rephraser generate dense and diverse narrations | on EGTEA cls.; on EK-100 MIR |
| Vid2Seq [ 149 ] | Captioning | Language model + time tokens; YT-Temporal-1B | Time tokens jointly predict event boundaries and captions | State-of-the-art dense captioning performance; not egocentric-specific |
| EgoHOD [ 93 ] | Action-centric | EgoVideo + motion adapter; Ego4D | HOI-dynamics narrations generated using a detector and an LLM | on EK-100 MIR; on EGTEA in zero-shot evaluation |
Table 5: Comparison of representative egocentric vision-language models and relevant multimodal or embodied extensions discussed in Section 5.2 . “EK-100 MIR” denotes EPIC-KITCHENS-100 multi-instance retrieval; “cls.” denotes classification.
Figure 6: The 100DOH hand-object detector. For each hand it jointly predicts a bounding box, side (left/right), contact state (none, self, other-person, portable, or non-portable), and a link to the box of the object in contact, recovering the active object among many distractors. Illustration inspired by [ 112 ] .
| Method / Resource | Category | Output / representation | Egocentric focus | Key contribution |
|---|---|---|---|---|
| 100DOH [ 112 ] | Detection | Hand box, side, contact state, in-contact object | Internet-scale | Joint hand + active-object contact detection |
| Hands23 [ 14 ] | Detection | Pixel segments, grasp, contact vocabulary | In-the-wild | Richer 2D parsing (touch/hold/use) |
| MANO [ 108 ] | Parametric model | Low-dimensional pose + shape mesh | — | Near-universal hand output representation |
| HandOccNet [ 88 ] | Reconstruction | MANO mesh | Occlusion-heavy | Feature injection into occluded regions |
| HaMeR [ 90 ] | Reconstruction | 3D hand mesh | In-the-wild | ViT-scale, robust across viewpoints |
| WildHands [ 97 ] | Reconstruction | 3D hand pose | Everyday ego | FOV cues; best 3D pose on ARCTIC ego split |
Table 6: Comparison of representative hand–object interaction methods and resources, spanning hand recovery, manipulated-object reasoning, and interaction datasets (Section 6 ).
Figure 7: Egocentric Action Scene Graph (EASG). An interaction is represented as a temporally evolving graph rooted at the camera-wearer node. Verb (action) nodes connect to direct objects, while the manipulated object is linked via active edges to the acting hands. Peripheral objects provide contextual relations. The graph is grounded by bounding boxes on the pre-condition, point-of-no-return (PNR), and post-condition frames, capturing both spatial hand–object relations and their evolution over time. Illustration inspired by [ 106 ] .
Figure 8: Interaction-aware temporal modeling in egocentric video. Hand–object interaction tracklets extracted from long videos are organized into a structured active memory indexed by interaction events and locations. This representation supports efficient temporal reasoning—including sequencing, concurrency, temporal grounding, and retrieval—without reprocessing the full video. Interaction-aware representations and graph-guided frame selection further improve long-horizon reasoning by using hand–object structure as the scaffold for temporal understanding. Illustration inspired by [ 37 ] .
| Method | Alignment strategy | Target weakness / finding |
|---|---|---|
| EgoVLP [ 69 ] | EgoNCE mines ego-aware positives/negatives | Naive InfoNCE ignores shared verb/noun semantics |
| EgoVLPv2 [ 98 ] | Cross-modal fusion in the backbone | Late fusion limits video–text interaction |
| LaViLa [ 161 ] | LLM re-narration (Narrator + Rephraser) | Sparse, unevenly timed Ego4D narrations |
| EgoNCE++ [ 143 ] | HOI-aware hard-negative captions | Object-over-action (verb) bias |
| EgoHOD [ 93 ] | HOI-dynamics narrations + motion adapter | Condensed narrations lack fine detail |
| Wray et al. [ 141 ] | Semantic (many-to-many) relevance evaluation | One-caption-one-video assumption is false |
Table 7: Egocentric video–text semantic alignment approaches (Section 9.2 ) and the weakness each targets.
| Challenge | Diagnosed failure | Evidence | Promising direction |
|---|---|---|---|
| Temporal reasoning (Sec. 11.1 ) | Models exploit spatial/appearance shortcuts rather than order | Text-only 31.3% and single-frame 51% on EgoSchema; single-frame collapses to 9.1% on EgoTempo [ 95 ] | TGPO lifts a 3B model to 49.7% on EgoSchema [ 146 ] ; EgoVLM explores complementary GRPO-based policy optimization [ 131 ] |
| Interaction understanding (Sec. 11.2 ) | Verb–noun asymmetry: nouns recognized, verbs missed | LaViLa scores 74.33% on nouns vs. 46.61% on verbs (EgoHOIBench) [ 143 ] | Generate interaction supervision: EgoNCE++ ( percentage points in LaViLa++ verb accuracy) and EgoHOD [ 143 , 93 ] |
| Graph-enhanced VLMs (Sec. 11.3 ) | Dense tokens store no explicit relation; symbol–vector gap | Layout-only STLT still recovers the verb without appearance [ 101 ] | Hypergraph-into-LLM, graph-guided self-training, scene-graph forecasting [ 86 , 99 , 92 ] |
| Efficient frame sampling (Sec. 11.4 ) | Uniform sampling wastes budget on uninformative frames | Frame selection substantially affects long-video VLM accuracy and efficiency [ 152 , 153 , 155 ] | Query- and structure-aware selection: SeViLA, Frame-Voyager, FocusGraph [ 152 , 153 , 155 ] |
| Multimodal learning (Sec. 11.5 ) | Video-only models ignore IMU, gaze, audio, and SLAM | IMU signals provide motion cues when visual frames are degraded by ego-motion [ 121 ] | Adaptive fusion: ImageBind, EgoDistill, EgoLife [ 34 , 121 , 150 ] |
| Embodied deployment (Sec. 11.6 ) | Embodiment and visual domain gaps; on-device compute limits | Original EgoSchema evaluation: tested models 33% vs. 76% human [ 79 ] ; wearable deployment additionally imposes strict compute constraints [ 22 ] | Egocentric world models; privacy and trust benchmarks [ 129 , 2 , 111 ] |
Table 8: Summary of open challenges and future directions (Section 11 ). Each row pairs a diagnosed failure with its supporting evidence and the most promising line of attack.