Integrating Affordances and Attention models for Short-Term Object Interaction Anticipation

Lorenzo Mur Labadia; Ruben Martinez-Cantin; Jose J. Guerrero; Giovanni M. Farinella; Antonino Furnari

Fig. 1: (a) Our approach takes as input an image-video pair. (b) The input is processed by our novel STAformer++, and end-to-end short term anticipation model based on transformers which predicts object bounding boxes, the associated verb/noun probabilities, time-to-contact estimates and confidence scores. (c) The model learns to predict environment noun and verb affordances (

p_{\text{aff}}(n\lvert\mathcal{V^{\prime}})

and

p_{\text{aff}}(v\lvert\mathcal{V^{\prime}})

in a dynamic and flexible way during training. This representation are used to refine later the predicted noun/verbs to obtain the final predictions (e).

Fig. 2: STAformer architecture. DINO-v2 and TimeSformer extract 2D and 3D features form the image-video input. (a) Frame-guided temporal pooling attention spatially aligns video to image features. (b) Dual image-video attention enriches 2D features with temporal dynamics and 3D features with fine-grained image details. Image and video representations are joined to obtain a global class token (c) and a feature pyramid (d), from which we obtain the STA predictions (e).

Fig. 3: STAformer++ architecture. The Swin-T backbone extracts hierarchical multi-scale 2D feature maps from the high-resolution image, while the EgoVideo backbone extracts spatio-temporal 3D features. a) We compute per-scale Frame-guided temporal pooling, and then resize the pooled video tokens to the respective image map. b) The two feature maps are summed to obtain the fused feature pyramid

P_{T}

. c) The DETR Encoder enhances the features and applies the Mixed Query Selection to initialize the positional part of the object queries

\rho_{m}

, while the content parts are kept as learnable parameters. d) The DETR Decoder incorporates the refined image-video features to the object queries. We accelerate the convergence using a Contrastive DeNoising (CDN) part with positive and negative samples as proposed in [ 60 ] . e) The STA prediction head applies independent MLP layers to obtain the final predictions

(\hat{b}_{m},\hat{n}_{m},\hat{v}_{m},\hat{\delta}_{m},\hat{s}_{m})

.

Fig. 4: Environment affordances in forecasting. a) We build an affordance database by linking training videos according to their visual similarity, obtaining activity-centric zones with affordances values

V_{{\color[rgb]{0,0,0}j}}^{AFF}

and respective video

\mathcal{Z}_{j}^{\mathcal{V}}

, text

\mathcal{Z}_{j}^{\mathcal{T}}

descriptors. b) Our first approach matches the input encoded video

\Phi^{\mathcal{V}}(\mathcal{V}^{\prime})

to the affordance database by selecting the K nearest neighbors in terms of the cosine similarity with the visual

\mathcal{Z}^{\mathcal{V}}

and text

\mathcal{Z}^{\mathcal{V}}

zone descriptors. The affordance probability

p_{AFF}

is obtained by weighting the counts of nouns present in the top-2K nearest zones (

\star

) according to the respective similarity

\mathcal{S}

. This will be late-fused with the predictions made by the end-to-end model. Example for K=2. c) In our second methodology, an attention mechanism (

Q^{AFF},K^{AFF}

) learns to associate a novel video

\mathcal{V^{\prime}}

with all the potential zone candidates

Z_{j}

in the affordance database. This dynamically obtains the noun

\mathcal{N}_{AFF}

and verb

\mathcal{A}_{AFF}

affordance distributions, which are summed to the DETR predicted nouns

n_{m}

and verb

v_{m}

logits during model training. The final binary class probabilities

p(n)_{m}

,

p(v)_{m}

are obtained after a Sigmoid layer.

Fig. 5: Refinement of confidence scores based on the interaction hotspots. The interaction hotspot model observes frames, hands, and objects and forecasts a map encoding the probability of the interaction in each pixel. STA confidence scores are re-weighted based on the probability values at the bounding box coordinate centers, reducing confidence in false positive predictions falling far from the interaction hotspot.

Table 6

	T	B	B+N	B+V	B+ $\delta$	B+N+V	B+N+ $\delta$	B+V+ $\delta$	All
Slowfast	v1	40.50	24.50	0.34	8.16	0.34	5.00	0.06	0.06
Slowfast (w/Transformer)	v1	40.50	24.50	8.20	7.50	8.20	4.50	1.30	0.73
AVT	v1	40.50	24.50	8.45	7.12	8.45	4.39	1.15	0.71
ANACTO	v1	40.50	24.50	8.90	7.47	8.90	4.55	1.54	0.91
MeMVIT	v1	40.50	24.50	10.05	9.27	10.04	4.95	2.11	1.34
GANO (w/o guided attention)	v1	40.50	24.50	7.10	9.01	7.10	4.20	1.22	0.75

TABLE III: Results in AP metric on the validation split of Ego4D-STA. Best results in bold. Relative gain is with respect to second best . T denotes training data.

Model	N	N + V	N + $\delta$	All
FRCNN+SF. [ 22 ]	20.45	6.78	6.17	2.45
FRCNN+Feat. [ 69 ]	20.45	4.81	4.40	1.31
InternVideo [ 23 ]	24.60	9.18	7.64	3.40
Transfusion [ 25 ]	24.69	9.97	7.33	3.44
StillFast [ 26 ]	19.51	9.95	6.45	3.49
STAformer	24.39	12.49	7.54	4.03

TABLE IV: Results in mAP on the test split of Ego4D-STA of models trained on the v1 training split.

Model	N	N + V	N + $\delta$	All
StillFast [ 26 ]	25.06	13.29	9.14	5.12
GANO v2 [ 28 ]	25.67	13.60	9.02	5.16
Language NAO	30.43	13.45	10.38	5.18
EgoVideo	31.08	16.18	12.41	7.21
STAformer	30.61	16.67	10.06	5.62
STAformer & AFF (fixed)	32.39	17.38	10.26	5.70

TABLE V: Results in mAP on the test split of Ego4D-STA of models trained on the v2 training split.

Model	N	N + V	N + $\delta$	All
StillFast [ 26 ]	21.24	12.41	6.22	3.28
STAformer	25.25	17.17	9.10	6.13
STAformer & AFF (fixed)	28.37	18.95	9.29	6.60
STAformer++	44.96	24.67	14.01	7.87
STAformer++&AFF(learned)	45.34	25.82	14.06	8.67
Gain (rel $\%$ )	+59.5	+36.20	+51.6	+31.5

TABLE VI: Results in mAP on the validation split of EPIC-Kitchens. Best results in bold. Relative gain is with respect to second best .

Exp.	Image Encoder	Video Encoder	Temporal pooling	2D-3D Fusion	Detection Head	N	N + V	N + $\delta$	All
[ 26 ]	R50	X3D	Mean	Sum	Fast-RCNN	16.21	7.52	4.94	2.48
A1	DINOv2	-	-	-	Fast-RCNN	17.48	8.64	5.20	2.52
A2	Swin-T	-	-	-	DETR	27.69	9.57	5.43	2.71
A3	Swin-T	-	-	-	DETR	28.77	11.04	6.12	2.85
A4	DINOv2	-	-	-	DETR	29.33	11.65	6.46	2.98
B1	DINOv2	DINOv2	Mean	Sum	Fast-RCNN	15.82	7.65	4.11	2.19

TABLE VII: Ablation study of the architectural components of STA-former on the validation split of Ego4D-v1. Encoder frozen Encoder fine-tuned. Overall best results in bold. Best result of each part is underlined. For fair comparison, we fine-tune 3 blocks in the image encoders and 4 blocks in the video encoders and the video comprises 0.5 sec. Configurations using the Faster R-CNN head correspond to STAformer, whereas configurations using the DETR head are instantiations of STAformer++

Fig. 6: STAformer++ performance evolution according to the amount of video seen. We report the mAP N, mAP N+V, mAP N+

\delta

, mAP Overall on the validation split of Ego4D-STA v1.

Fig. 7: STAformer performance evolution according to the amount of video seen. We report the mAP N, mAP N+V, mAP N+

\delta

, mAP Overall on the validation split of Ego4D-STA v1.

Fig. 8: Comparative between the fixed affordance distribution inferred from top-K environment zones (left) vs. learning the cross-zone similarity (right). We also visualize the closest environments in terms of the visual

\mathcal{K}^{\mathcal{V}}

and narrative

\mathcal{K}^{\mathcal{T}}

cosine similarity. We show in orange the STA ground-truth label, showing that the affordances distribution effectively captures the STA action. Learning the cross-zone similarity enables a more flexible affordance distribution representation, as we do not rely on a fixed number of zones.

		N	N+V	N + TTC	All
-	-	16.20	7.47	4.94	2.48
Env. Aff.	Count-based priors	16.44	7.84	4.50	2.39
	Inverse-freq.Prior	15.22	6.98	4.40	2.28
	Ego-Topo [ 51 ]	14.92	6.45	4.01	2.14
	Fixed Weighted (Ours)	18.44	8.46	5.47	2.85
Int. Hot	Center Prior	14.44	6.86	3.90	2.05

TABLE VIII: Comparative of the affordances priors effect on Stillfast. Results in mAP on the validation split of Ego4D v1. Best results in bold. Relative gain is respect with the base model .

		N	N+V	N + TTC	All
-	-	21.71	10.75	7.24	3.53
Env. Aff.	Count-based Prior	21.96	10.98	6.80	3.56
	Inverse-freq. Prior	22.40	10.27	6.02	2.84
	Ego-Topo [ 51 ]	17.21	8.45	5.32	2.64
	Fixed Weighted (Ours)	23.55	11.75	7.55	3.74
Int. Hot	Center Prior	17.70	8.82	5.22	2.62

TABLE IX: Comparative of the affordances priors effect on STAformer. Results in mAP on the validation split of Ego4D v1. Best results in bold. Relative gain is respect with the base model .

		N	N+V	N + TTC	All
-	-	32.07	15.00	8.53	4.31
Env. Aff.	Count-based Prior	31.39	13.68	8.71	4.42
	Inverse-freq. Prior	31.67	13.42	8.48	4.17
	Ego-Topo [ 51 ]	28.39	12.02	7.72	3.78
	Fixed Weighted	31.00	14.17	8.69	3.84
	Learned	33.21	15.94	8.98	4.66

TABLE X: Comparative of the affordances priors effect on STAformer++. Results in mAP on the validation split of Ego4D v1. Best results in bold. Relative gain is respect with the base model .

Fig. 9: Dual image-video attention maps, qualitative results. Top to bottom: STAformer final predictions, attention map of pooled TimeSformer tokens (queries) on DINOv2 image tokens (keys and values), attention of DINOv2 image tokens (queries) on pooled TimeSformer video tokens (keys and values), attention map of pooled EgoVideo tokens (queries) on intermediate Swin-T image tokens (keys and values) and attention of intermediate Swim-T image tokens (queries) on pooled EgoVideo features. Video tokens attend fine-grained object information from the high-resolution image; image features focus on objects which are important for future interactions.

Fig. 10: Ego4D Qualitative results Top to bottom: ground truth, STAformer predictions and STAformer++ predictions in Ego4D v2 validation split. We visualize the top-5 detections by the model. It is appreciated how the STAformer++ detections capture better the contour of the object, and that the whole model achieves a better understanding of the potential interactions in the video.

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Integrating Affordances and Attention models for Short-Term Object Interaction Anticipation

Abstract

Figures & tables

Next-Action Prediction

VISTA: Technical Report for the Ego4D Short-Term Object Interaction Anticipation at EgoVis 2026

FROST-STA: Frozen Dense Features for the Ego4D Short-Term Object Interaction Anticipation

RAAP: Retrieval-Augmented Affordance Prediction with Cross-Image Action Alignment

Abstract

Next-Action Prediction

Explore similar work

VISTA: Technical Report for the Ego4D Short-Term Object Interaction Anticipation at EgoVis 2026

FROST-STA: Frozen Dense Features for the Ego4D Short-Term Object Interaction Anticipation

RAAP: Retrieval-Augmented Affordance Prediction with Cross-Image Action Alignment