Action-Dynamics Modeling and Cross-Temporal Interaction for Online Action Understanding

Xinyu Yang; Zheheng Jiang; Feixiang Zhou; Yi-hang Zhu; Na Lyu; Nan Xing; Qi Wang; Huiyu Zhou

cs.CVOct 12, 2025

Action-Dynamics Modeling and Cross-Temporal Interaction for Online Action Understanding

Authors: Xinyu Yang, Zheheng Jiang, Feixiang Zhou, Yi-hang Zhu, Na Lyu, Nan Xing, Qi Wang, Huiyu Zhou

Organizations: School of Computing and Mathematical Sciences, University of Leicester, United Kingdom. · School of Eye and Vision Sciences, University of Liverpool, United Kingdom. · School of Information Science and Engineering, University of Jinan, China. · School of Automation and Information Engineering, Xi’an University of Technology, China.

Abstract

Online action understanding, encompassing action detection and anticipation, is central to numerous practical applications. However, untrimmed videos contain substantial redundant and noisy historical information, and the complementary interactions among historical, current, and future-oriented representations remain underexplored. To address these challenges, we propose the State-Specific Model (SSM), a unified framework for online action detection and anticipation. Its Representative State-Based Memory Compression module selects representative anchor frames from the observed memory and aggregates anchor-conditioned context into a compact set of representative memory states, thereby reducing memory redundancy. The subsequent Action Pattern Learning module learns multi-dimensional relations among these states and performs graph reasoning over the resulting directed representative-state graph to model action dynamics and derive a future-oriented cue. The Cross-Temporal Interaction module then enables this cue to interact with historical and current cues, refining the representations used for simultaneous online action detection and anticipation. Extensive experiments on EPIC-Kitchens-100, THUMOS'14, TVSeries, and the introduced Parkinson's Disease Mouse Behaviour dataset demonstrate competitive performance and generalizability across diverse online action understanding settings. These results underscore the value of graph-based action-dynamics modelling and cross-temporal interaction for online action understanding.

Figures & tables

Fig. 1: Overview of the proposed State-Specific Model. (a) Representative State-Based Memory Compression. Observed video features are compressed into representative, anchor-conditioned memory states. (b) Action Pattern Learning. The representative states serve as graph nodes, while their learned multi-dimensional relations jointly encode action dynamics. A subsequent Gated Graph Convolutional Network (Gated GCN) propagates and aggregates the relational information to generate a future-oriented graph cue. (c) Cross-Temporal Interaction. Features interact across different time domains to refine current and future cues, supporting action detection and anticipation.

Fig. 2: CTI models the interaction between the future-oriented graph cue and past-present cues. This process further refines the temporal representations, supporting online action detection and anticipation.

Fig. 3: Ablation study on (a) memory sequence length; (b) number of clusters; (c) fully shared (FS) classifier vs. unshared (US) classifier; (d) scaling parameter $\delta$ ; (e) balancing hyperparameter $\lambda_{1}$ ; and (f) balancing hyperparameter $\lambda_{2}$ . All these experiments are on the THUMOS’14 test set and mAP is adopted as the evaluation metric.

Temporal weighting	Detection	Anticipation
None (vanilla attention)	69.8	59.3
Gaussian	71.8	62.6
Exponential	70.9	61.3
Cauchy	71.1	60.7

TABLE I: Ablation of temporal weighting kernels on the THUMOS’14 test set (mAP, %).

Setting	$\lambda_{2}$	Detection	Anticipation
Without future-distribution consistency loss	0	59.9	42.1
Full model	0.7	71.8	62.6

TABLE II: Ablation study of the future-distribution consistency loss on the THUMOS’14 test set (mAP, %). Setting

\lambda_{2}=0

removes

L_{\mathrm{fdc}}

Fig. 4: t-SNE visualization of action-feature discriminability under different representative anchor selection strategies. Each point denotes one video frame, and different colours represent the action categories in THUMOS’14.

Method	Selected Frame Number
Method	2	3	4	5	6	7	8
Uniform Sampling	44.1	50.3	54.9	56.8	55.2	54.1	52.9
SRS	42.0	46.6	50.1	54.3	56.2	55.6	52.2
K-means	41.9	47.4	52.0	49.3	48.7	48.0	47.4
K-medoids	42.1	48.2	50.6	53.7	52.9	49.3	48.8
GMM	49.9	52.2	57.1	60.4	58.9	57.2	56.8

TABLE III: Ablation study of representative anchor selection strategies (mAP, %). SRS denotes stratified random sampling.

Anchor criterion	Detection	Anticipation
Cosine distance	71.0	61.3
Mahalanobis distance	69.6	60.7
Maximum posterior	70.5	61.4
Euclidean distance	71.8	62.6

TABLE IV: Ablation of the within-component anchor selection criterion on the THUMOS’14 test set. Results are reported in mAP (%).

Feature/anchor stage	State construction	Detection	Anticipation
Uniform sampling	Anchor only	51.9	39.6
Uniform sampling	TWA	54.9	40.7
GMM	Anchor only	53.5	39.9
GMM	TWA	57.1	43.0
ProPos–GMM	Anchor only	67.9	54.1
ProPos–GMM	Vanilla attention	69.8	59.3

TABLE V: Controlled decomposition of representative-anchor selection and memory-state construction on the THUMOS’14 test set (mAP, %). Anchor only denotes the setting without TWA.

Method	Sup.	FAR	AC	Det.	Ant.
			$\uparrow$	$\uparrow$	$\uparrow$
GMM	No	68.5	55.2	57.1	43.0
ProPos–GMM	No	75.8	70.3	71.8	62.6
ProPos–Act. Proto.	Yes	89.4	67.5	68.1	55.3
ProPos–Act.+BG Proto.	Yes	79.8	66.2	69.5	57.7

TABLE VI: Comparison of anchor selection methods on THUMOS’14 with

K=4

. Sup. indicates whether training labels are used to construct fixed prototypes. FAR and AC denote foreground anchor ratio and action coverage, respectively, and are reported in percentage. Test annotations are used only to calculate FAR and AC after anchor selection. Det. and Ant. are reported in mAP (%).

Relation Modelling Configuration	Detection	Anticipation
Mean State Pooling	33.7	12.1
Nonlinear State Pooling (MLP)	41.4	26.5
Shared-edge Gated GCN	52.3	41.9
APL with Multi-dimensional Edges	71.8	62.6

TABLE VII: Ablation study of relational modelling configurations in APL on the THUMOS’14 test set (mAP, %).

Representation	Current-Action	Future-Action
	mAP (%)	mAP (%)
Present cue $\mathbf{F}_{c}$	50.9	8.6
RSMC state mean $\overline{\mathbf{S}}$	26.1	15.3
Future-oriented graph cue $\mathbf{F}_{g}$	37.7	43.9
CTI-refined future representation $\mathbf{F}^{\prime}_{g}$	41.0	50.8

TABLE VIII: Linear probing of current- and future-action information on THUMOS’14 (mAP, %). For each frozen representation, two independent linear classifiers with identical architectures are trained on the training split using the current-action label and the future-action label at

\tau=1

s, respectively, and evaluated on the test split.

Fig. 5: Ablation study of inter-state edge strategies. All variants use the same representative states produced by RSMC and differ only in their graph relation encoding. Dyn, MD, Sim, TA and TD denote dynamic change, multi-dimensional edge, similarity edge, temporal adjacency edge and temporal direction edge, respectively.

No.	Present-repr. update	Future-repr. update	Det.	Ant.
(1)	None	None	49.4	44.3
(2)	$\mathbf{F}_{p}\!\rightarrow\!\mathbf{F}^{\prime}_{c}$	None	63.5	44.9
(3)	None	$\mathbf{F}_{p}\!\rightarrow\!\mathbf{F}^{\prime}_{g}$	49.6	52.0
(4)	None	$\mathbf{F}_{c}\!\rightarrow\!\mathbf{F}^{\prime}_{g}$	49.9	55.3
(5)	$\mathbf{F}_{g}\!\rightarrow\!\mathbf{F}^{\prime}_{c}$	None	59.6	45.4
(6)	$\{\mathbf{F}_{p},\mathbf{F}_{g}\}\!\rightarrow\!\mathbf{F}^{\prime}_{c}$	$\{\mathbf{F}_{p},\mathbf{F}^{\prime}_{c}\}\!\rightarrow\!\mathbf{F}^{\prime}_{g}$	71.8	62.6

TABLE IX: Ablation study of feature-refinement paths in CTI on the THUMOS’14 test set (mAP, %). Here,

\mathbf{X}\!\rightarrow\!\mathbf{Y}

indicates that cue

\mathbf{X}

is used as contextual information to refine target representation

\mathbf{Y}

through cross-attention, while

\{\mathbf{X}_{1},\mathbf{X}_{2}\}\!\rightarrow\!\mathbf{Y}

indicates that both cues are used jointly.

\mathbf{F}_{p}

\mathbf{F}_{c}

, and

\mathbf{F}_{g}

denote the past cue, present cue, and future-oriented graph cue, respectively; a prime denotes a refined representation. “None” indicates that the corresponding cross-temporal update is not applied.

Method	Modality	Verb	Noun	Action
RULSTM [ 39 ]	RGB	27.5	29.0	13.3
AVT [ 11 ]	RGB	30.2	31.7	14.9
TeSTra [ 10 ]	RGB	26.8	36.2	17.0
MeMViT [ 40 ]	RGB	32.8	33.2	15.1
MAT [ 38 ]	RGB	32.7	39.7	18.8
S-GEAR $-$ 2B [ 41 ]	RGB	32.7	37.9	19.6

TABLE X: Comparison with prior work on EPIC-Kitchens-100 for action anticipation.

Method	THUMOS’14		TVSeries
Method	Kinetics	ANet	Kinetics	ANet
TRN [ 13 ]	62.1	47.2	86.2	83.7
OadTR [ 19 ]	65.2	58.3	87.2	85.41
Colar [ 20 ]	66.9	59.4	88.1	86.0
LSTR [ 8 ]	69.5	65.3	89.1	88.1
GateHub [ 9 ]	70.7	69.1	89.6	88.4

TABLE XI: Online action detection performance on THUMOS’14 and TVSeries.

Dataset	Method	Pretrained	Anticipation Time Gap								Avg.
Dataset	Method	Pretrained	0.25s	0.50s	0.75s	1.00s	1.25s	1.50s	1.75s	2.00s	Avg.
THUMOS 14	TRN [ 13 ]	ANet	45.1	42.4	40.7	39.1	37.7	36.4	35.3	34.3	38.9
	LSTR [ 8 ]	ANet	-	-	-	-	-	-	-	-	50.1
	TeSTra [ 10 ]	ANet	64.7	61.8	58.7	55.7	53.2	51.1	49.2	47.8	55.3
	OadTR [ 19 ]	ANet	49.3	48.1	46.8	45.3	43.9	42.4	41.1	40.1	45.9
	HCM [ 7 ]	ANet	62.4	59.4	56.3	53.3	50.5	48.3	46.4	44.8	52.8

TABLE XII: Action anticipation results on THUMOS’14 and TVSeries with different anticipation horizons. mAP is reported for THUMOS’14 and mcAP for TVSeries. “ANet” and “Kinetics” denote features pretrained on ActivityNet v1.3 and Kinetics-400, respectively.

Fig. 6: Verb, noun, and action anticipation under different cluster numbers in our representative state-based compression framework.

Fig. 7: Qualitative anticipation results for SSM and MAT. Incorrect anticipations are marked in red.

Fig. 8: Example-level verb, noun, and action anticipation results under different cluster numbers $K$ . Finer state granularity often recovers object detail, whereas excessive states do not consistently improve verb or action anticipation.

Temporal Action Detection

Temporal action detection identifies action categories and their start and end times in untrimmed video.

See all 41 papers

Mar 2, 2026

Action-Guided Attention for Video Action Anticipation

Anticipating future actions in videos is challenging, as the observed frames provide only evidence of past activities, requiring the inference of latent intentions to predict upcoming actions. Existing transformer-based approaches, which rely on dot-product attention over pixel representations, often lack the high-level semantics necessary to model video sequences for effective action anticipation. As a result, these methods tend to overfit to explicit visual cues present in the past frames, limiting their ability to capture underlying intentions and degrading generalization to unseen samples. To address this, we propose Action-Guided Attention (AGA), an attention mechanism that explicitly leverages predicted action sequences as queries and keys to guide sequence modeling. Our approach fosters the attention module to emphasize relevant moments from the past based on the upcoming activity and combine this information with the current frame embedding via a dedicated gating function. The design of AGA enables post-training analysis of the knowledge discovered from the training set. Experiments on the widely adopted EPIC-Kitchens-100 benchmark demonstrate that AGA generalizes well from validation to unseen test sets. Post-training analysis can further examine the action dependencies captured by the model and the counterfactual evidence it has internalized, offering transparent and interpretable insights into its anticipative predictions.

Tsung-Ming Tai, Sofia Casarin, Andrea Pilzer +2

Free University of Bozen-Bolzano · NVIDIA

Temporal Video Understanding Attention Mechanisms

Ask PDF

Sep 30, 2026

Learning Skills from Historical Action Trajectories: Action Experience Dictionary for World Action Models

World Action Models (WAMs) couple visual dynamics prediction with action generation, yet they do not explicitly support the reuse of action experience across manipulation tasks. Furthermore, existing WAMs struggle to capture underlying cross-task semantic relationships that could guide target action prediction, as redundant background elements interfere with the extraction of key visual information. To address these challenges, we develop a novel Action Experience Dictionary (AED) that encodes historical physical action trajectories into shared action embeddings to support skill reuse and model cross-task relationships. Specifically, we first aggregate historical actions to align with visual observations and retrieve action embeddings from the AED using a pretrained action tokenizer. Subsequently, we visually condition the pooled embeddings through cross-attention and prepend them to noisy action tokens, providing interaction context and action intent for prediction. To model action-related motion and reduce reliance on irrelevant background cues, we introduce a motion-aware transition loss that supervises visual feature change prediction over random temporal intervals. Experiments on simulation benchmarks and in real-world cross-embodiment settings verify the effectiveness of our AED. The project code is available at https://github.com/JiahuaDong/AED .

Qi Lyu, Jiahua Dong, Hao Shen +9

Shenyang Institute of Automation, Chinese Academy of Sciences · Mohamed bin Zayed University of Artificial Intelligence · Anhui University +3

Robot Skill Learning Action-Conditioned World Models Cross-Embodiment Robot Learning World Action Models Trajectory Representation Learning

Ask PDF

Mar 6, 2026

Hierarchical Latent Action Model

Latent Action Models (LAMs) enable learning from actionless data for applications ranging from robotic control to interactive world models. However, existing LAMs typically focus on short-horizon frame transitions and capture low-level motion while overlooking longer-term temporal structure. In contrast, actionless videos often contain temporally extended and high-level skills. We present HiLAM, a hierarchical latent action model that discovers latent skills by modeling long-term temporal information. To capture these dependencies across long horizons, we utilize a pretrained LAM as a low-level extractor. This architecture aggregates latent action sequences, which contain the underlying dynamic patterns of the video, into high-level latent skills. Our experiments demonstrate that HiLAM improves over the baseline and exhibits robust dynamic skill discovery.

Hanjung Kim, Lerrel Pinto, Seon Joo Kim

Yonsei University · New York University

Latent Action Learning Agent Skill Learning Hierarchical Representation Learning

Ask PDF

Abstract

Temporal Action Detection

Explore similar work

Action-Guided Attention for Video Action Anticipation

Learning Skills from Historical Action Trajectories: Action Experience Dictionary for World Action Models

Hierarchical Latent Action Model