Egocentric Action Recognition with Objects and Vision-Language Affordances
Authors: Raúl Boix-Aldeguer, Manuel Benavent-Lledo, David Ortiz-Perez, Jose Garcia-Rodriguez
Organizations: Department of Computer Technology, University of Alicante, Alicante, Spain · Visual Analysis and Perception Lab, Aalborg University, Denmark
Action recognition, and more specifically egocentric action recognition, is a core task in computer vision, with applications such as monitoring the daily life activities of elderly people and task assistance. Recent methods rely on visual modalities such as RGB video or depth. However, in egocentric video, an action is largely defined by its objects, which these methods rarely model. We propose a method that combines a frozen SlowFast video expert with an attention-based object expert by late fusion. Each object is represented with its affordances, that is, what can be done with it, generated zero-shot by a vision-language model. Since datasets such as MECCANO annotate only the objects involved in each action, evaluating with these boxes tells the model which objects are active. We therefore train on annotated boxes and evaluate the method with detections from the official active-object detector, and with annotated boxes as an upper bound. Without affordances, our fusion reaches 68.4% Top-1 with annotated boxes but 49.0% with detections. About half of this drop comes from non-active detections. To remove them, the same vision-language model generates an interaction affordance that decides whether a hand is manipulating each detected object. On MECCANO, objects raise the Top-1 of the video expert from 44.9% to 49.0%, and the two affordances raise it to 49.9%, above SlowFast and object-aware RGB baselines. Code will be released upon acceptance.
Figures & tables
Fig. 1: The interaction affordance removes an idle object. The VLM answers that no hand manipulates the booklet (red, dashed), and the filter removes it. The video-only SlowFast baseline and the fusion with all detected objects predict check booklet , whereas our method predicts the correct action, take bolt . Bars show the three most probable actions, averaged over five seeds for our method.
Fig. 2: Overview of the method at test time. A frozen SlowFast expert gives the video probabilities pv . An object detector finds the objects in the candidate frames of the segment. For each detection, a VLM answers whether a hand is manipulating the object, and the interaction filter removes the detections answered with no (red, dashed). Then K frames are selected, and each remaining object becomes a token of crop, class name, position, and affordance features. The affordance is generated by a separate prompt to the same VLM. The object expert applies self-attention to the tokens, averages them, and an MLP gives the probabilities po . Finally, late fusion combines both experts with the weight α . During training, the annotated boxes replace the detector, and the filter is not used.
Fig. 3: Prompts of the two VLM queries for the gray bar of Fig. 2 . Both queries receive the frame with the detection in a red box and its crop. The object affordance (center) returns a comma-separated list, which is encoded with DistilBERT. The interaction affordance (right) describes the hands and answers whether a hand manipulates the object; detections answered no are removed. Prompts used on MECCANO. The list of object classes is shortened.
Method
Extra input
Top-1
Top-5
Macro-F1
SlowFast [ 22 ]
–
45.16
73.75
18.91 †
HOCL+OSL [ 28 ]
–
44.81
77.01
16.70
UCF Swin3D-B [ 13 ] ‡
–
50.80
82.36
–
SlowFast [ 22 ]
D+G
49.66
77.82
–
G3Ego [ 11 ]
G+H+VLM+OD
46.48
82.04
21.34
Video expert
–
44.93
74.18
19.43
TABLE I: Comparison with published results on MECCANO. Extra input : what each method uses at inference besides RGB video.
Fig. 4: Decomposition of the Top-1 drop from annotated boxes to detections. Each orange bar is the loss of one step that replaces annotated boxes with detections. The blue bar is the gain of affordances and the interaction filter (Sec. IV-D ). The dashed line marks the accuracy without non-active detections, the accuracy of a perfect interaction filter.
Token parts
Annotated boxes
Detections
Full (crop + class + position)
63.28
36.39
Class only
63.71
33.83*
Crop only
52.81*
36.39
Position only
33.88*
26.11*
Crop + position (no class)
50.00*
35.27*
TABLE II: Top-1 of the object expert alone with partial object tokens (mean pooling, no affordances).
Aff.
Filter
Top-1
Top-5
Macro-F1
Δ Top-1
–
–
49.04 ± 0.68
81.34
22.69
–
✓
–
49.53 ± 0.49
82.08
23.36
+0.50 [+0.11, +0.93] *
–
✓
49.54 ± 0.95
81.74
23.64
+0.50 [+0.04, +0.92] *
✓
✓
49.87 ± 0.47
82.54
24.06
+0.84 [+0.30, +1.34] *
TABLE III: Ablation of the affordances and the interaction filter (late fusion, detections).
Fig. 5: Affordances generated by the VLM for a detected screwdriver with the four inputs: the crop (left) and the frame with the detection in a red box (right). The text-only answer depends only on the class name and is the same for every screwdriver. With the crop, the answer describes the screwdriver in use, whereas the frame alone gives a short answer.
VLM input
No
P / R
Removed
Non-active
Active
Validation (500 detections)
Frame
11.0
60.0 / 14.1
8.1
2.3
+ crop
11.0
58.2 / 13.7
7.3
2.6
+ hand description (used)
14.2
56.3 / 17.1
9.4
2.6
Test (26,874 detections)
TABLE IV: Prompts of the interaction affordance.
VLM input
Annotated boxes
Detections
Text only
− 0.06
+0.35*
Text + crop
+0.55*
+0.41*
Text + frame
− 0.72*
+0.02
Text + frame + crop (used)
− 0.54
+0.50*
TABLE V: Top-1 difference of each affordance input to Video + objects, without interaction filter.
Detections
Annotated
Model
Top-1
Top-5
Macro-F1
Top-1
Video expert
44.93
74.18
19.43
44.93
Object expert, mean
36.39 ± 0.38
72.00
11.20
63.28 ± 0.33
Object expert, attn.
37.75 ± 0.55
74.69
12.84
64.27 ± 0.70
Learned: sum, mean
48.13 ± 0.58
80.19
21.64
58.44 ± 0.25
Learned: concat., mean
47.29 ± 0.36
77.03
21.02
55.26 ± 0.32
TABLE VI: Fusion strategies without affordances.
Frames
K
Detections
Annotated
Maximize transitions (used)
4
49.04
68.38
Fixed offsets
4
− 0.45
− 4.55*
Uniform annotated
4
+0.72*
+2.05*
High variability
4
− 0.06
− 2.98*
Progressive discovery
4
+0.33
− 1.01*
Maximize transitions
6
+0.22
+2.06*
TABLE VII: Frame selection and number of frames K (late fusion, no affordances).
Fig. 6: Qualitative results on the test set. Top: annotated boxes (blue). Middle: all detections (orange). Bottom: detections after the interaction filter (green: kept; red, dashed: removed), with the VLM’s answer in parentheses. Predictions are the majority over five seeds. (a) The filter removes the idle booklet, and our method corrects the prediction. (b) The VLM answers no for the bolt being taken, and the filter removes the only active object. (c) The VLM answers yes for every detection near the hands, and the non-active screwdriver remains.
Action reasoning in egocentric video requires capturing fine-grained transitions of hand-object interactions, a task where general-purpose Vision-Language Models (VLMs) often struggle when operating directly on raw pixels. We propose to decouple visual perception from symbolic reasoning by converting videos into Temporal Action Graphs. In a multi-stage prompting pipeline, we first generate dense natural language narratives over short temporal windows as a semantic bottleneck, then formalize them into structured, open-vocabulary graph representations. On the EGTEA and Epic-Kitchens-100 datasets, the symbolic representation unlocks efficient in-context learning: few-shot graph demonstrations yield substantial accuracy gains over zero-shot frame-based inference. Even in the zero-shot setting, graph-based reasoning remains competitive with pixel-based inference despite potential pretraining contamination favoring the latter. Across 11 open-weight VLMs from 6 model families ranging from 2B to 235B parameters, our findings indicate that current VLMs are more effective as symbolic reasoners than as direct visual observers. By projecting video into the language domain, we provide a scalable, fine-tuning-free alternative to end-to-end approaches that better leverages these models' latent reasoning strengths. The code will be made public.
Bessie Dominguez-Dager, Francisco Gomez-Donoso, Miguel Cazorla +3
The EPIC-KITCHENS-100 Action Detection challenge evaluates whether a model can localize the start and end of each action in long untrimmed egocentric videos and assign the corresponding verb--noun action label. In this report, we formulate our submission as EgoAction (Egocentric Action Composition with Reliability-Aware Temporal Fusion), a unified decoupled detection and fusion pipeline. The pipeline uses EPIC-finetuned VideoMAE-L features, trains separate noun and verb temporal detectors with causal temporal modeling, composes action hypotheses from top noun--verb pairs, and introduces a confidence-adaptive boundary fusion rule at post-processing time. The key observation is that verb and noun streams often fail differently: verb scores are sensitive to motion transitions, whereas noun scores are sensitive to hand-object visibility and object clutter. A fixed arithmetic mean of their predicted boundaries can therefore amplify localization errors when one stream degenerates. We replace this hard-coded mean with Dynamic Weighted Fusion (DWF), which normalizes the maximum noun and verb classification confidences into proposal-wise boundary weights and linearly combines the two intervals. This lightweight tensor-only operator shifts boundary authority toward the more reliable stream while preserving the decoupled action scoring mechanism. Together with sliding-window inference, top-K noun--verb action composition, and class-wise Soft-NMS, EgoAction provides a compact and reproducible system for egocentric temporal action detection.
Zhiheng Fu, Zixu Li, Zhiwei Chen +4
1Shandong University · 2Harbin Institute of Technology (Shenzhen)
Learning manipulation skills from human videos is promising for scalable robot learning. However, the embodiment mismatch between humans and robots makes this challenging. One promising solution is to learn object-centric actionable affordances that are embodiment-agnostic. In this work, we propose a framework that leverages egocentric human videos with state-of-the-art 3D Structure-from-Motion and hand mesh reconstruction to extract actionable affordances such as visual, grasp, and trajectory affordances that explicitly encode where to interact, how to grasp, and how to move. We construct EgoAffordance, a large-scale dataset comprising 204K episodes with 5.6M visual affordances and 11.6M grasp and trajectory affordances. Building on this, we introduce VLAff, a large vision-language model-based unified foundation model that learns cross-modal correlations across all actionable affordances. Given a visual observation and instruction, VLAff generates visual affordance heatmaps, grasp poses, and trajectories, which are then converted into directly executable actions by utilizing 3D scene information. Through extensive experiments, we demonstrate that VLAff not only achieves state-of-the-art performance on visual affordance prediction, but can also be effectively applied to real robot applications such as zero-shot manipulation and affordance-guided robot learning.