Egocentric Action Recognition with Objects and Vision-Language Affordances
Authors: Raúl Boix-Aldeguer, Manuel Benavent-Lledo, David Ortiz-Perez, Jose Garcia-Rodriguez
Organizations: Department of Computer Technology, University of Alicante, Alicante, Spain · Visual Analysis and Perception Lab, Aalborg University, Denmark
Action recognition, and more specifically egocentric action recognition, is a core task in computer vision, with applications such as monitoring the daily life activities of elderly people and task assistance. Recent methods rely on visual modalities such as RGB video or depth. However, in egocentric video, an action is largely defined by its objects, which these methods rarely model. We propose a method that combines a frozen SlowFast video expert with an attention-based object expert by late fusion. Each object is represented with its affordances, that is, what can be done with it, generated zero-shot by a vision-language model. Since datasets such as MECCANO annotate only the objects involved in each action, evaluating with these boxes tells the model which objects are active. We therefore train on annotated boxes and evaluate the method with detections from the official active-object detector, and with annotated boxes as an upper bound. Without affordances, our fusion reaches 68.4% Top-1 with annotated boxes but 49.0% with detections. About half of this drop comes from non-active detections. To remove them, the same vision-language model generates an interaction affordance that decides whether a hand is manipulating each detected object. On MECCANO, objects raise the Top-1 of the video expert from 44.9% to 49.0%, and the two affordances raise it to 49.9%, above SlowFast and object-aware RGB baselines. Code will be released upon acceptance.
Figures & tables
Fig. 1: The interaction affordance removes an idle object. The VLM answers that no hand manipulates the booklet (red, dashed), and the filter removes it. The video-only SlowFast baseline and the fusion with all detected objects predict check booklet , whereas our method predicts the correct action, take bolt . Bars show the three most probable actions, averaged over five seeds for our method.
Fig. 2: Overview of the method at test time. A frozen SlowFast expert gives the video probabilities pv . An object detector finds the objects in the candidate frames of the segment. For each detection, a VLM answers whether a hand is manipulating the object, and the interaction filter removes the detections answered with no (red, dashed). Then K frames are selected, and each remaining object becomes a token of crop, class name, position, and affordance features. The affordance is generated by a separate prompt to the same VLM. The object expert applies self-attention to the tokens, averages them, and an MLP gives the probabilities po . Finally, late fusion combines both experts with the weight α . During training, the annotated boxes replace the detector, and the filter is not used.
Fig. 3: Prompts of the two VLM queries for the gray bar of Fig. 2 . Both queries receive the frame with the detection in a red box and its crop. The object affordance (center) returns a comma-separated list, which is encoded with DistilBERT. The interaction affordance (right) describes the hands and answers whether a hand manipulates the object; detections answered no are removed. Prompts used on MECCANO. The list of object classes is shortened.
Method
Extra input
Top-1
Top-5
Macro-F1
SlowFast [ 22 ]
–
45.16
73.75
18.91 †
HOCL+OSL [ 28 ]
–
44.81
77.01
16.70
UCF Swin3D-B [ 13 ] ‡
–
50.80
82.36
–
SlowFast [ 22 ]
D+G
49.66
77.82
–
G3Ego [ 11 ]
G+H+VLM+OD
46.48
82.04
21.34
Video expert
–
44.93
74.18
19.43
TABLE I: Comparison with published results on MECCANO. Extra input : what each method uses at inference besides RGB video.
Fig. 4: Decomposition of the Top-1 drop from annotated boxes to detections. Each orange bar is the loss of one step that replaces annotated boxes with detections. The blue bar is the gain of affordances and the interaction filter (Sec. IV-D ). The dashed line marks the accuracy without non-active detections, the accuracy of a perfect interaction filter.
Token parts
Annotated boxes
Detections
Full (crop + class + position)
63.28
36.39
Class only
63.71
33.83*
Crop only
52.81*
36.39
Position only
33.88*
26.11*
Crop + position (no class)
50.00*
35.27*
TABLE II: Top-1 of the object expert alone with partial object tokens (mean pooling, no affordances).
Aff.
Filter
Top-1
Top-5
Macro-F1
Δ Top-1
–
–
49.04 ± 0.68
81.34
22.69
–
✓
–
49.53 ± 0.49
82.08
23.36
+0.50 [+0.11, +0.93] *
–
✓
49.54 ± 0.95
81.74
23.64
+0.50 [+0.04, +0.92] *
✓
✓
49.87 ± 0.47
82.54
24.06
+0.84 [+0.30, +1.34] *
TABLE III: Ablation of the affordances and the interaction filter (late fusion, detections).
Fig. 5: Affordances generated by the VLM for a detected screwdriver with the four inputs: the crop (left) and the frame with the detection in a red box (right). The text-only answer depends only on the class name and is the same for every screwdriver. With the crop, the answer describes the screwdriver in use, whereas the frame alone gives a short answer.
VLM input
No
P / R
Removed
Non-active
Active
Validation (500 detections)
Frame
11.0
60.0 / 14.1
8.1
2.3
+ crop
11.0
58.2 / 13.7
7.3
2.6
+ hand description (used)
14.2
56.3 / 17.1
9.4
2.6
Test (26,874 detections)
TABLE IV: Prompts of the interaction affordance.
VLM input
Annotated boxes
Detections
Text only
− 0.06
+0.35*
Text + crop
+0.55*
+0.41*
Text + frame
− 0.72*
+0.02
Text + frame + crop (used)
− 0.54
+0.50*
TABLE V: Top-1 difference of each affordance input to Video + objects, without interaction filter.
Detections
Annotated
Model
Top-1
Top-5
Macro-F1
Top-1
Video expert
44.93
74.18
19.43
44.93
Object expert, mean
36.39 ± 0.38
72.00
11.20
63.28 ± 0.33
Object expert, attn.
37.75 ± 0.55
74.69
12.84
64.27 ± 0.70
Learned: sum, mean
48.13 ± 0.58
80.19
21.64
58.44 ± 0.25
Learned: concat., mean
47.29 ± 0.36
77.03
21.02
55.26 ± 0.32
TABLE VI: Fusion strategies without affordances.
Frames
K
Detections
Annotated
Maximize transitions (used)
4
49.04
68.38
Fixed offsets
4
− 0.45
− 4.55*
Uniform annotated
4
+0.72*
+2.05*
High variability
4
− 0.06
− 2.98*
Progressive discovery
4
+0.33
− 1.01*
Maximize transitions
6
+0.22
+2.06*
TABLE VII: Frame selection and number of frames K (late fusion, no affordances).
Fig. 6: Qualitative results on the test set. Top: annotated boxes (blue). Middle: all detections (orange). Bottom: detections after the interaction filter (green: kept; red, dashed: removed), with the VLM’s answer in parentheses. Predictions are the majority over five seeds. (a) The filter removes the idle booklet, and our method corrects the prediction. (b) The VLM answers no for the bolt being taken, and the filter removes the only active object. (c) The VLM answers yes for every detection near the hands, and the non-active screwdriver remains.