Action-Guided Attention for Video Action Anticipation

Tsung-Ming Tai; Sofia Casarin; Andrea Pilzer; Werner Nutt; Oswald Lanz

cs.CVMar 2, 2026

Action-Guided Attention for Video Action Anticipation

Authors: Tsung-Ming Tai, Sofia Casarin, Andrea Pilzer, Werner Nutt, Oswald Lanz

Organizations: Free University of Bozen-Bolzano · NVIDIA

Abstract

Anticipating future actions in videos is challenging, as the observed frames provide only evidence of past activities, requiring the inference of latent intentions to predict upcoming actions. Existing transformer-based approaches, which rely on dot-product attention over pixel representations, often lack the high-level semantics necessary to model video sequences for effective action anticipation. As a result, these methods tend to overfit to explicit visual cues present in the past frames, limiting their ability to capture underlying intentions and degrading generalization to unseen samples. To address this, we propose Action-Guided Attention (AGA), an attention mechanism that explicitly leverages predicted action sequences as queries and keys to guide sequence modeling. Our approach fosters the attention module to emphasize relevant moments from the past based on the upcoming activity and combine this information with the current frame embedding via a dedicated gating function. The design of AGA enables post-training analysis of the knowledge discovered from the training set. Experiments on the widely adopted EPIC-Kitchens-100 benchmark demonstrate that AGA generalizes well from validation to unseen test sets. Post-training analysis can further examine the action dependencies captured by the model and the counterfactual evidence it has internalized, offering transparent and interpretable insights into its anticipative predictions.

Figures & tables

Figure 1: Architecture Overview. The model consists of two modules. The Action-Guided Attention uses the most recent $S$ action predictions as keys, the exponential moving average (EMA) of all predicted actions as the query, and $S$ frame embeddings as values to generate a history context $\tilde{h}_{t}$ . The Adaptive Gating then integrates this history context with the current frame embedding $e_{t}$ to produce a fused representation, which is mapped to the new prediction $\hat{y}_{t}$ .

Methods	Modality	Overall Classes			Unseen Classes			Tail Classes
Methods	Modality	Action	Verb	Noun	Action	Verb	Noun	Action	Verb	Noun
RULSTM ( Furnari and Farinella, 2020 )	RGB, Flow, Obj	11.2	25.3	26.7	9.7	19.4	26.9	7.9	17.6	16.0
AVT+ ( Girdhar and Grauman, 2021 )	RGB, Obj	12.6	25.6	28.8	8.8	20.9	22.3	10.1	19.0	22.0
AVT++ ( Girdhar and Grauman, 2021 )	RGB, Flow, Obj	16.7	26.7	32.3	12.9	21.0	27.6	13.8	19.3	24.0
AFFT-TSN+ ( Zhong et al., 2023 )	RGB, Flow, Obj, Audio	13.4	19.4	28.3	9.9	14.0	24.2	10.9	12.0	19.5
AFFT-Swin+ ( Zhong et al., 2023 )	RGB, Flow, Obj, Audio	14.9	20.7	31.8	12.1	16.2	27.7	11.8	13.4	23.8

Table 1: EK100 Test Results . Evaluation on a unseen test set via the official challenge server.

Methods	Backbone	Overall Classes			Unseen Classes			Tail Classes
Methods	Backbone	Action	Verb	Noun	Action	Verb	Noun	Action	Verb	Noun
RULSTM ( Furnari and Farinella, 2020 )	TSN	13.3	27.5	29.0	-	-	-	-	-	-
AVT-h ( Girdhar and Grauman, 2021 )	TSN	13.6	27.2	30.7	-	-	-	-	-	-
AVT-h ( Girdhar and Grauman, 2021 )	irCSN152	12.8	25.5	28.1	-	-	-	-	-	-
AVT-h ( Girdhar and Grauman, 2021 )	AVT-b	14.9	30.2	31.7	-	-	-	-	-	-
AFFT ( Zhong et al., 2023 )	TSN	16.4	21.3	32.7	13.6	24.1	25.5	14.3	13.2	25.8

Table 2: EK100 Validation Results . Comparison of methods on the validation set using only RGB input. Methods highlighted in gray use the same TSN backbone weights.

Methods	Backbone	Top-1	Top-5	MT5R
Methods	Backbone	Acc	Acc	MT5R
RULSTM ( Furnari and Farinella, 2020 )	TSN	13.1	30.8	12.5
TempAgg ( Sener et al., 2020 )	TSN	12.3	28.5	13.1
ImagineRNN ( Wu et al., 2020 )	TSN	13.7	31.6	-
SRL ( Qi et al., 2021 )	TSN	-	31.7	13.2
AVT-h ( Girdhar and Grauman, 2021 )	TSN	13.1	28.1	13.5

Table 3: EK55 Validation Results . Top-1/Top-5 action accuracy and MT5R at

\tau_{a}=1s

.Methods highlighted in gray use the same TSN backbone weights.

Methods	Top-1 Acc			Mean Top-1 Recall
Methods	Action	Verb	Noun	Action	Verb	Noun
I3D-Res50 ( Carreira and Zisserman, 2017 )	34.8	48.0	42.1	23.2	31.3	30.0
FHOI ( Liu et al., 2020 )	36.6	49.0	45.5	32.5	32.7	25.3
TSN-AVT-h ( Girdhar and Grauman, 2021 )	39.8	51.7	50.3	28.3	41.2	41.4
AVT ( Girdhar and Grauman, 2021 )	43.0	54.9	52.2	35.2	49.9	48.3
TSN-AFFT ( Zhong et al., 2023 )	42.5	53.4	50.4	35.2	42.4	44.5

Table 4: EGTEA Gaze+ Validation Results .Top-1 accuracies and Mean Top-1 Recall at

\tau_{a}=0.5s

. Methods highlighted in gray use the same TSN backbone weights.

Table 6

Figure 2: Forward Analysis identifies which past actions the model attends to when predicting its next action in response to a query. This analysis was conducted on the model trained with EK100.

Figure 3: Backward Analysis. The figure compares the top-5 original predicted actions with the counterfactual supportive actions optimized toward the target action take pan . Each column represents a timestep; the final column shows the anticipated output. Suppressed actions are highlighted in red and promoted actions appear in green. This example is drawn from the EK100 validation set.

Figure 4: Visualization of Adaptive Gating Ratio. The gating values, displayed alongside the action sequence, demonstrate context-aware behavior. In background regions (black), the gate retains historical context; in action regions (colored), it prioritizes current visual evidence.

$f_{x}$	$E_{Q}$ , $E_{K}$	$E_{V}$	FFN	MLP in Adaptive Gating	MLP in classifier
LayerNorm	LayerNorm	LayerNorm	Linaer (2048, 1024)	Linear (4096, 512)	LayerNorm
ReLU	ReLU	ReLU	GELU	ReLU	ReLU
Dropout	Dropout	Dropout	Dropout	Dropout	Dropout
Linear ( $d_{backbone}$ , 2048)	Linear (2048, 512)	Linear (2048, 2048)	Linear (1024, 2048)	Linear (512, 2048)	Linear(2048, $N_{c}$ )
ScaleNorm

Table 7: Modeling components in AGA.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Queue Size	AGA (MT5R)
S=4	18.4
S=16	18.8
S=30	18.0

Appendix

Table 8: Ablation study with different queue sizes on EK100.

Table 12

Table 13

Panel of Figure 5 — Figure 5: Robustness analysis of AGA against the error occurred in frame prediction.

Methods	Train On	Inference On	Overall Classes			Unseen Classes			Tail Classes
Methods	Train On	Inference On	Action	Verb	Noun	Action	Verb	Noun	Action	Verb	Noun
GT	GT	GT	16.9	30.6	36.7	14.6	34.4	26.5	16.1	25.3	32.8
GT	Self-Pred	GT	17.1	31.9	36.0	14.1	32.9	26.3	16.6	26.8	32.2
Self-Pred (Top-1 One-Hot)	Self-Pred (Top-1 One-Hot)	Self-Pred (Top-1 One-Hot)	17.2	31.8	37.7	14.5	35.6	27.7	16.3	26.6	34.2
Self-Pred (Top-1 One-Hot)	Self-Pred	Self-Pred (Top-1 One-Hot)	17.5	33.4	36.4	16.3	32.6	27.7	16.5	28.4	32.5
AGA	Self-Pred	Self-Pred	18.8	32.5	38.7	16.3	34.4	28.5	18.4	27.4	35.0

Appendix

Table 13: Comparison between different action-guidance signals for

Q

and

K

Figure 6: Four video clips from the EPIC-Kitchens-100 validation set, illustrating correct predictions. Each clip demonstrates the last eight frames along with the top-5 action anticipations at $\tau_{a}=1s$ , with the ground truth highlighted in bold green.

Figure 7: Four video clips from the EPIC-Kitchens-100 validation set, showcasing incorrect predictions. Each clip demonstrates the last eight frames and the top-5 action anticipations at $\tau_{a}=1s$ , with the ground truth revealed in the final frame. The correct verb and noun are highlighted in bold green.

Figure 8: Four video clips from the EPIC-Kitchens-55 validation set, illustrating correct predictions. Each clip displays the last eight frames and the top-5 action predictions. The ground truth is highlighted in bold green in the latest frame, occurring at $\tau_{a}=1s$ .

Figure 9: Four video clips from the EPIC-Kitchens-55 validation set, showcasing incorrect predictions. Each clip displays the last eight frames and the top-5 action predictions. The ground truth is highlighted in bold green in the latest frame, occurring at $\tau_{a}=1s$ .

Figure 10: Four video clips from the EGTEA Gaze+ validation set, highlighting correct predictions. Each clip displays the last eight frames and the top-5 action predictions. The ground truth is highlighted in bold green in the latest frame, occurring at $\tau_{a}=0.5s$ .

Figure 11: Four video clips from the EGTEA Gaze+ validation set, showcasing incorrect predictions. Each clip displays the last eight frames and the top-5 action predictions. The ground truth is highlighted in bold green in the latest frame, occurring at $\tau_{a}=0.5s$ .

Temporal Video Understanding

Temporal video understanding studies how models track actions, events, and changes over time to answer questions, locate moments, and predict what happens next.

See all 371 papers

Language-Augmented Video Action Anticipation: Design Fundamentals, Benchmarks, and Open Challenges
Sep 15, 2026Mahsa Mohammadi, Zeyu Fu, Sareh Rowlands
Department of Computer Science, University of Exeter, Harrison Building, Streatham Campus, North Park Road, Exeter, EX4 4QF, Devon, United Kingdom
TAP-JEPA: Frozen Future-Latent Probing and Two-Stage Score Fusion for EPIC-KITCHENS-100 Action Anticipation
May 30, 2026Chaoyang Wang, Lexuan XuNext-Action Prediction Egocentric Video Understanding
Beihang University
Action-Dynamics Modeling and Cross-Temporal Interaction for Online Action Understanding
Oct 12, 2025Xinyu Yang, Zheheng Jiang, Feixiang Zhou +5Temporal Action Detection Next-Action Prediction
School of Computing and Mathematical Sciences, University of Leicester, United Kingdom. · School of Eye and Vision Sciences, University of Liverpool, United Kingdom. · School of Information Science and Engineering, University of Jinan, China. +1

Sep 15, 2026J/K move · Enter open · S save

Language-Augmented Video Action Anticipation: Design Fundamentals, Benchmarks, and Open Challenges

Mahsa Mohammadi, Zeyu Fu, Sareh Rowlands

Department of Computer Science, University of Exeter, Harrison Building, Streatham Campus, North Park Road, Exeter, EX4 4QF, Devon, United Kingdom

Open paper AskPDF arXiv

Action anticipation predicts future human actions from partial video under incomplete context and temporal uncertainty. Recent systems introduce large language models (LLMs), vision-language models (VLMs), or language-derived semantics at different stages, but reported gains are difficult to interpret when task formulation, visual pretraining, supervision, decoder design, and evaluation code change simultaneously. The central contribution of this review is an evidence-aware design map that crosses task regime with the point at which language-derived information intervenes. We characterise task regimes along six axes. These axes organise the literature into five broad task families: single-action, sequence, object-interaction, cross-view, and planning-oriented settings. C1-C3 locate interventions in context construction, goal/intention modelling, and future decoding, while C4 is treated as an adjacent, emerging grounding/executability extension. Unlike a generic processing pipeline, the map links each intervention to an appropriate counterfactual, failure diagnosis, and permissible evidence claim. Supporting contributions include a protocol-level audit of Ego4D-LTA and EPIC-KITCHENS-100, a multidimensional evidence profile, and the Backbone-Aware Comparison and Ablation Protocol (BCAP). The unresolved EK-100 record is treated as a reporting-comparability case study and is not used as a leaderboard. Evidence for LLM benefits, goal ambiguity, and horizon effects is therefore formulated as testable hypotheses requiring matched validation, not as causal conclusions. The accompanying package contains the coded evidence, source locators, protocol metadata, and versioned catalogue used in the review.

Abstract

Temporal Video Understanding

Explore similar work

Language-Augmented Video Action Anticipation: Design Fundamentals, Benchmarks, and Open Challenges

TAP-JEPA: Frozen Future-Latent Probing and Two-Stage Score Fusion for EPIC-KITCHENS-100 Action Anticipation

Action-Dynamics Modeling and Cross-Temporal Interaction for Online Action Understanding