Integrating Affordances and Attention models for Short-Term Object Interaction Anticipation

Lorenzo Mur Labadia; Ruben Martinez-Cantin; Jose J. Guerrero; Giovanni M. Farinella; Antonino Furnari

cs.CVFeb 16, 2026

Integrating Affordances and Attention models for Short-Term Object Interaction Anticipation

Authors: Lorenzo Mur Labadia, Ruben Martinez-Cantin, Jose J. Guerrero, Giovanni M. Farinella, Antonino Furnari

Organizations: Aragon Institute for Engineering Research (I3A), University of Zaragoza, Spain · Department of Computer Science, University of Catania, Italy

Abstract

Short Term object-interaction Anticipation consists in detecting the location of the next active objects, the noun and verb categories of the interaction, as well as the time to contact from the observation of egocentric video. This ability is fundamental for wearable assistants to understand user goals and provide timely assistance, or to enable human-robot interaction. In this work, we present a method to improve the performance of STA predictions. Our contributions are two-fold: 1 We propose STAformer and STAformer plus plus, two novel attention-based architectures integrating frame-guided temporal pooling, dual image-video attention, and multiscale feature fusion to support STA predictions from an image-input video pair; 2 We introduce two novel modules to ground STA predictions on human behavior by modeling affordances. First, we integrate an environment affordance model which acts as a persistent memory of interactions that can take place in a given physical scene. We explore how to integrate environment affordances via simple late fusion and with an approach which adaptively learns how to best fuse affordances with end-to-end predictions. Second, we predict interaction hotspots from the observation of hands and object trajectories, increasing confidence in STA predictions localized around the hotspot. Our results show significant improvements on Overall Top-5 mAP, with gain up to +23p.p on Ego4D and +31p.p on a novel set of curated EPIC-Kitchens STA labels. We released the code, annotations, and pre-extracted affordances on Ego4D and EPIC-Kitchens to encourage future research in this area.

Figures & tables

Fig. 1: (a) Our approach takes as input an image-video pair. (b) The input is processed by our novel STAformer++, and end-to-end short term anticipation model based on transformers which predicts object bounding boxes, the associated verb/noun probabilities, time-to-contact estimates and confidence scores. (c) The model learns to predict environment noun and verb affordances ( $p_{\text{aff}}(n\lvert\mathcal{V^{\prime}})$ and $p_{\text{aff}}(v\lvert\mathcal{V^{\prime}})$ in a dynamic and flexible way during training. This representation are used to refine later the predicted noun/verbs to obtain the final predictions (e).

Fig. 2: STAformer architecture. DINO-v2 and TimeSformer extract 2D and 3D features form the image-video input. (a) Frame-guided temporal pooling attention spatially aligns video to image features. (b) Dual image-video attention enriches 2D features with temporal dynamics and 3D features with fine-grained image details. Image and video representations are joined to obtain a global class token (c) and a feature pyramid (d), from which we obtain the STA predictions (e).

Fig. 3: STAformer++ architecture. The Swin-T backbone extracts hierarchical multi-scale 2D feature maps from the high-resolution image, while the EgoVideo backbone extracts spatio-temporal 3D features. a) We compute per-scale Frame-guided temporal pooling, and then resize the pooled video tokens to the respective image map. b) The two feature maps are summed to obtain the fused feature pyramid $P_{T}$ . c) The DETR Encoder enhances the features and applies the Mixed Query Selection to initialize the positional part of the object queries $\rho_{m}$ , while the content parts are kept as learnable parameters. d) The DETR Decoder incorporates the refined image-video features to the object queries. We accelerate the convergence using a Contrastive DeNoising (CDN) part with positive and negative samples as proposed in [ 60 ] . e) The STA prediction head applies independent MLP layers to obtain the final predictions $(\hat{b}_{m},\hat{n}_{m},\hat{v}_{m},\hat{\delta}_{m},\hat{s}_{m})$ .

Fig. 4: Environment affordances in forecasting. a) We build an affordance database by linking training videos according to their visual similarity, obtaining activity-centric zones with affordances values $V_{{\color[rgb]{0,0,0}j}}^{AFF}$ and respective video $\mathcal{Z}_{j}^{\mathcal{V}}$ , text $\mathcal{Z}_{j}^{\mathcal{T}}$ descriptors. b) Our first approach matches the input encoded video $\Phi^{\mathcal{V}}(\mathcal{V}^{\prime})$ to the affordance database by selecting the K nearest neighbors in terms of the cosine similarity with the visual $\mathcal{Z}^{\mathcal{V}}$ and text $\mathcal{Z}^{\mathcal{V}}$ zone descriptors. The affordance probability $p_{AFF}$ is obtained by weighting the counts of nouns present in the top-2K nearest zones ( $\star$ ) according to the respective similarity $\mathcal{S}$ . This will be late-fused with the predictions made by the end-to-end model. Example for K=2. c) In our second methodology, an attention mechanism ( $Q^{AFF},K^{AFF}$ ) learns to associate a novel video $\mathcal{V^{\prime}}$ with all the potential zone candidates $Z_{j}$ in the affordance database. This dynamically obtains the noun $\mathcal{N}_{AFF}$ and verb $\mathcal{A}_{AFF}$ affordance distributions, which are summed to the DETR predicted nouns $n_{m}$ and verb $v_{m}$ logits during model training. The final binary class probabilities $p(n)_{m}$ , $p(v)_{m}$ are obtained after a Sigmoid layer.

Fig. 5: Refinement of confidence scores based on the interaction hotspots. The interaction hotspot model observes frames, hands, and objects and forecasts a map encoding the probability of the interaction in each pixel. STA confidence scores are re-weighted based on the probability values at the bounding box coordinate centers, reducing confidence in false positive predictions falling far from the interaction hotspot.

Table 6

	T	B	B+N	B+V	B+ $\delta$	B+N+V	B+N+ $\delta$	B+V+ $\delta$	All
Slowfast	v1	40.50	24.50	0.34	8.16	0.34	5.00	0.06	0.06
Slowfast (w/Transformer)	v1	40.50	24.50	8.20	7.50	8.20	4.50	1.30	0.73
AVT	v1	40.50	24.50	8.45	7.12	8.45	4.39	1.15	0.71
ANACTO	v1	40.50	24.50	8.90	7.47	8.90	4.55	1.54	0.91
MeMVIT	v1	40.50	24.50	10.05	9.27	10.04	4.95	2.11	1.34
GANO (w/o guided attention)	v1	40.50	24.50	7.10	9.01	7.10	4.20	1.22	0.75

TABLE III: Results in AP metric on the validation split of Ego4D-STA. Best results in bold. Relative gain is with respect to second best . T denotes training data.

Model	N	N + V	N + $\delta$	All
FRCNN+SF. [ 22 ]	20.45	6.78	6.17	2.45
FRCNN+Feat. [ 69 ]	20.45	4.81	4.40	1.31
InternVideo [ 23 ]	24.60	9.18	7.64	3.40
Transfusion [ 25 ]	24.69	9.97	7.33	3.44
StillFast [ 26 ]	19.51	9.95	6.45	3.49
STAformer	24.39	12.49	7.54	4.03

TABLE IV: Results in mAP on the test split of Ego4D-STA of models trained on the v1 training split.

Model	N	N + V	N + $\delta$	All
StillFast [ 26 ]	25.06	13.29	9.14	5.12
GANO v2 [ 28 ]	25.67	13.60	9.02	5.16
Language NAO	30.43	13.45	10.38	5.18
EgoVideo	31.08	16.18	12.41	7.21
STAformer	30.61	16.67	10.06	5.62
STAformer & AFF (fixed)	32.39	17.38	10.26	5.70

TABLE V: Results in mAP on the test split of Ego4D-STA of models trained on the v2 training split.

Model	N	N + V	N + $\delta$	All
StillFast [ 26 ]	21.24	12.41	6.22	3.28
STAformer	25.25	17.17	9.10	6.13
STAformer & AFF (fixed)	28.37	18.95	9.29	6.60
STAformer++	44.96	24.67	14.01	7.87
STAformer++&AFF(learned)	45.34	25.82	14.06	8.67
Gain (rel $\%$ )	+59.5	+36.20	+51.6	+31.5

TABLE VI: Results in mAP on the validation split of EPIC-Kitchens. Best results in bold. Relative gain is with respect to second best .

Exp.	Image Encoder	Video Encoder	Temporal pooling	2D-3D Fusion	Detection Head	N	N + V	N + $\delta$	All
[ 26 ]	R50	X3D	Mean	Sum	Fast-RCNN	16.21	7.52	4.94	2.48
A1	DINOv2	-	-	-	Fast-RCNN	17.48	8.64	5.20	2.52
A2	Swin-T	-	-	-	DETR	27.69	9.57	5.43	2.71
A3	Swin-T	-	-	-	DETR	28.77	11.04	6.12	2.85
A4	DINOv2	-	-	-	DETR	29.33	11.65	6.46	2.98
B1	DINOv2	DINOv2	Mean	Sum	Fast-RCNN	15.82	7.65	4.11	2.19

TABLE VII: Ablation study of the architectural components of STA-former on the validation split of Ego4D-v1. Encoder frozen Encoder fine-tuned. Overall best results in bold. Best result of each part is underlined. For fair comparison, we fine-tune 3 blocks in the image encoders and 4 blocks in the video encoders and the video comprises 0.5 sec. Configurations using the Faster R-CNN head correspond to STAformer, whereas configurations using the DETR head are instantiations of STAformer++

Panel of Figure 12 — Fig. 6: STAformer++ performance evolution according to the amount of video seen. We report the mAP N, mAP N+V, mAP N+ $\delta$ , mAP Overall on the validation split of Ego4D-STA v1.

Panel of Figure 13 — Fig. 7: STAformer performance evolution according to the amount of video seen. We report the mAP N, mAP N+V, mAP N+ $\delta$ , mAP Overall on the validation split of Ego4D-STA v1.

Panel of Figure 14 — Fig. 8: Comparative between the fixed affordance distribution inferred from top-K environment zones (left) vs. learning the cross-zone similarity (right). We also visualize the closest environments in terms of the visual $\mathcal{K}^{\mathcal{V}}$ and narrative $\mathcal{K}^{\mathcal{T}}$ cosine similarity. We show in orange the STA ground-truth label, showing that the affordances distribution effectively captures the STA action. Learning the cross-zone similarity enables a more flexible affordance distribution representation, as we do not rely on a fixed number of zones.

		N	N+V	N + TTC	All
-	-	16.20	7.47	4.94	2.48
Env. Aff.	Count-based priors	16.44	7.84	4.50	2.39
	Inverse-freq.Prior	15.22	6.98	4.40	2.28
	Ego-Topo [ 51 ]	14.92	6.45	4.01	2.14
	Fixed Weighted (Ours)	18.44	8.46	5.47	2.85
Int. Hot	Center Prior	14.44	6.86	3.90	2.05

TABLE VIII: Comparative of the affordances priors effect on Stillfast. Results in mAP on the validation split of Ego4D v1. Best results in bold. Relative gain is respect with the base model .

		N	N+V	N + TTC	All
-	-	21.71	10.75	7.24	3.53
Env. Aff.	Count-based Prior	21.96	10.98	6.80	3.56
	Inverse-freq. Prior	22.40	10.27	6.02	2.84
	Ego-Topo [ 51 ]	17.21	8.45	5.32	2.64
	Fixed Weighted (Ours)	23.55	11.75	7.55	3.74
Int. Hot	Center Prior	17.70	8.82	5.22	2.62

TABLE IX: Comparative of the affordances priors effect on STAformer. Results in mAP on the validation split of Ego4D v1. Best results in bold. Relative gain is respect with the base model .

		N	N+V	N + TTC	All
-	-	32.07	15.00	8.53	4.31
Env. Aff.	Count-based Prior	31.39	13.68	8.71	4.42
	Inverse-freq. Prior	31.67	13.42	8.48	4.17
	Ego-Topo [ 51 ]	28.39	12.02	7.72	3.78
	Fixed Weighted	31.00	14.17	8.69	3.84
	Learned	33.21	15.94	8.98	4.66

TABLE X: Comparative of the affordances priors effect on STAformer++. Results in mAP on the validation split of Ego4D v1. Best results in bold. Relative gain is respect with the base model .

Panel of Figure 18 — Fig. 9: Dual image-video attention maps, qualitative results. Top to bottom: STAformer final predictions, attention map of pooled TimeSformer tokens (queries) on DINOv2 image tokens (keys and values), attention of DINOv2 image tokens (queries) on pooled TimeSformer video tokens (keys and values), attention map of pooled EgoVideo tokens (queries) on intermediate Swin-T image tokens (keys and values) and attention of intermediate Swim-T image tokens (queries) on pooled EgoVideo features. Video tokens attend fine-grained object information from the high-resolution image; image features focus on objects which are important for future interactions.

Panel of Figure 19 — Fig. 10: Ego4D Qualitative results Top to bottom: ground truth, STAformer predictions and STAformer++ predictions in Ego4D v2 validation split. We visualize the top-5 detections by the model. It is appreciated how the STAformer++ detections capture better the contour of the object, and that the whole model achieves a better understanding of the potential interactions in the video.

Next-Action Prediction

Next-action prediction estimates what a person, process, or agent will do next from observations and prior events.

See all 34 papers

May 20, 2026

VISTA: Technical Report for the Ego4D Short-Term Object Interaction Anticipation at EgoVis 2026

We propose VISTA, a V-JEPA Integrated StillFast Temporal Anticipator for the Ego4D Short-Term Object Interaction Anticipation (STA) Challenge at EgoVis 2026. Given an egocentric video timestamp, the task requires anticipating the next human-object interaction, including the future active object's bounding box, noun category, verb category, time-to-contact, and confidence score. VISTA follows a StillFast-style design that combines object-centric spatial detection with short-horizon temporal context. Specifically, a COCO-pretrained Faster R-CNN ResNet-50 FPN detector generates object proposals from the last observed high-resolution frame, while a frozen V-JEPA 2.1 temporal branch extracts clip-level egocentric context from the observed video. The temporal representation is injected into the detection pathway through feature modulation and ROI-level context fusion. The fused proposal features are then passed to multi-head STA predictors for box refinement, noun classification, verb classification, time-to-contact regression, and interaction confidence estimation. For the final submission, we further ensemble complementary predictions to improve robustness. Experimental results on the official challenge server show that VISTA achieves first place in the EgoVis 2026 Ego4D STA Challenge. Our code will be released at https://github.com/CorrineQiu/VISTA.

Qiaohui Chu, Haoyu Zhang, Yisen Feng +4

Harbin Institute of Technology (Shenzhen) · Pengcheng Laboratory · Shandong Jianzhu University

Egocentric Video Understanding

Ask PDF

May 30, 2026

FROST-STA: Frozen Dense Features for the Ego4D Short-Term Object Interaction Anticipation

Short-term anticipation in egocentric video requires more than recognizing the current scene: a system must infer which object the camera wearer will contact, which action will follow, and how soon the contact will happen. This report describes FROST-STA, our submission to the Ego4D Short-Term Object Interaction Anticipation (STA) Challenge at EgoVis 2026. For each query time, the model produces a ranked set of structured hypotheses containing an active-object box, noun label, verb label, time-to-contact (TTC), and confidence. FROST-STA builds on the V-JEPA 2.1 STA evaluation protocol, but adapts it to the challenge by using object-centric decoding, multi-head prediction, and a submission-oriented training and ensembling recipe. We keep the V-JEPA 2.1 ViT-G backbone fixed and extract two dense token streams: video tokens from a short clip resized to 384 pixels before the query, and image tokens from the last observed high-resolution frame. A compact alignment module, consisting of an attentive probe and frame-guided temporal pooling, maps the clip representation onto the spatial reference of the final frame before fusing it with image features. The fused maps are decoded by Faster R-CNN-style STA heads that estimate box offsets, nouns, verbs, TTC values, and interaction quality. For the final leaderboard entry, we train for 25 epochs with the official training split plus additional permitted validation annotations, and combine predictions across eight heads and checkpoints from epochs 15-25. FROST-STA obtains 5.13 Overall Top-5 mAP on the official test server, ranking second in the challenge and showing that frozen dense image-video features can serve as a strong basis for object-level interaction forecasting.

Chaoyang Wang, Lexuan Xu

Beihang University

Egocentric Video Understanding Temporal Video Understanding

Ask PDF

Mar 31, 2026

RAAP: Retrieval-Augmented Affordance Prediction with Cross-Image Action Alignment

Understanding object affordances is essential for enabling robots to perform purposeful and fine-grained interactions in diverse and unstructured environments. However, existing approaches either rely on retrieval, which is fragile due to sparsity and coverage gaps, or on large-scale models, which frequently mislocalize contact points and mispredict post-contact actions when applied to unseen categories, thereby hindering robust generalization. We introduce Retrieval-Augmented Affordance Prediction (RAAP), a framework that unifies affordance retrieval with alignment-based learning. By decoupling static contact localization and dynamic action direction, RAAP transfers contact points via dense correspondence and predicts action directions through a retrieval-augmented alignment model that consolidates multiple references with dual-weighted attention. Trained on compact subsets of DROID and HOI4D with as few as tens of samples per task, RAAP achieves consistent performance across unseen objects and categories, and enables zero-shot robotic manipulation in both simulation and the real world. Project website: https://github.com/SEU-VIPGroup/RAAP.

Qiyuan Zhuang, He-Yang Xu, Yijun Wang +3

School of Computer Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, Nanjing 211189, China · Southeast University-Monash University Joint Graduate School, Southeast University, Suzhou 215123, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China

Robotic Affordance Inference Affordance-Based Robotic Manipulation Generalization in Robotic Manipulation Zero-Shot Robotic Manipulation Data-Efficient Robot Learning

Ask PDF

Abstract

Next-Action Prediction

Explore similar work

VISTA: Technical Report for the Ego4D Short-Term Object Interaction Anticipation at EgoVis 2026

FROST-STA: Frozen Dense Features for the Ego4D Short-Term Object Interaction Anticipation

RAAP: Retrieval-Augmented Affordance Prediction with Cross-Image Action Alignment