In video action detection, modeling the temporal dependencies in continuous video streams is crucial. Previous methods based on deep neural networks can perform frame-level video recognition tasks. However, these methods struggle to model dynamic temporal patterns, facing challenges such as redundant frame information, difficulty balancing global and local features, and insufficient fusion of visual and textual features. To address these issues, we propose a video temporal attention module based on the Multi-Scale Latent Temporal Compression Attention, which aggregates continuous temporal sequences into feature representations across diverse temporal scales while efficiently reducing redundancy. We also introduce a Video-Text Cross Attention Fusion module that leverages cross-attention at both latent and standard feature levels, achieving fine-grained fusion between video and textual features to enhance semantic alignment and feature interaction. Experimental results across multiple video recognition benchmarks demonstrate that our proposed methods achieve superior performance compared to existing methods, validating the effectiveness and generalization capability.
Figures & tables
Figure 1: (a) MLTC-Attention models multi-scale temporal dependencies via compress-then-attend in latent space, and VTCA-Fusion enforces hierarchical video-text alignment at both standard and latent feature levels. (b) Performance–efficiency on video anomaly detection (blue, left axis) and video action detection (orange, right axis): our method (red) attains the best accuracy at the highest inference speed (FPS) with the fewest FLOPs on both tasks.
Figure 2: Overview of the overall architecture of our proposed video action detection framework. Our method integrates the VTCA-Fusion module and MLTC-Attention Blocks to process temporal and categorical features, with the resulting predictions feeding into various downstream tasks.
Method
Vision Encoder
UCF-Crime
XD-Violence
AUC(%)
AP(%)
LAVAD [ 63 ]
ImageBind [ 13 ]
80.28
62.01
DMU † [ 67 ]
I3D [ 3 ]
86.14
81.46
AnomalyCLIP [ 62 ]
ViT-B16 [ 11 ]
86.30
77.40
VERA [ 60 ]
InternVL2 [ 8 ]
86.55
70.54
UMIL [ 29 ]
ViT-B32 [ 11 ]
86.75
–
Table 1: Comparison of frame-level performance between our Method and Previous methods on the UCF-Crime and XD-Violence benchmarks. † : results reproduced by us with the listed vision encoder.
Table 4Table 5
Figure 3: Qualitative results of our proposed methods on Video Anomaly Detection tasks.
Figure 4: Qualitative results of our proposed methods on temporal action detection tasks.
Figure 5: Visualization of internal representation category-wise temporal feature.
Temporal human action detection aims to identify and localize action segments within untrimmed videos, serving as a pivotal task in video understanding. Despite the progress achieved by prior architectures like CNN and Transformer models, these continue to struggle with feature redundancy and degraded global dependency modeling capabilities when applied to long video sequences. These limitations severely constrain their scalability in real-world video analysis. State Space Models (SSMs) offer a promising alternative with linear long-term modeling and robust global temporal reasoning capabilities. Rethinking the application of SSMs in temporal modeling, this research constructs a novel framework for video human action detection. Specifically, we introduce the Efficient Spatial-Temporal Focal (ESTF) Adapter into the pre-trained layers. This module integrates the advantages of our proposed Temporal Boundary-aware SSM(TB-SSM) for temporal feature modeling with efficient processing of spatial features. We perform comprehensive and quantitative analyses across multiple benchmarks, comparing our proposed method against previous SSM-based and other structural methods. Extensive experiments demonstrate that our improved strategy significantly enhances both localization performance and robustness, validating the effectiveness of our proposed method.
Yicheng Qiu, Keiji Yanai
Department of Informatics The University of Electro-Communications Chofu, Tokyo, Japan
Zero-Shot Temporal Action Detection (ZSTAD) aims to lo- calize and recognize action instances from unseen action categories in untrimmed videos. Although existing meth- ods have shown effectiveness by advancing architectural text-video alignment, they still struggle with capturing se- mantic distinctions between action classes, resulting in text- irrelevant predictions. To address this issue, we propose a Text-Foreground Concentrated Alignment for zero-shot temporal action DEtector (TF-CADE) that explicitly aligns textual information with action-relevant foreground regions. Specifically, we introduce Action Concentrate Aggregation (ACA), which extracts action concentrate scores to aggregate temporally informative video segments into a foreground- weighted video embedding. This foreground concentrated alignment enhances the semantic consistency between text and video features and improves inter-class discriminabil- ity. In addition, a Certainty-based Confidence Re-weighting (CCR) strategy refines per-snippet confidence scores by lever- aging foreground-aware similarity, effectively suppressing irrelevant action classes during inference. Extensive evalua- tions show that our TF-CADE not only achieves state-of-the- art performance under in-distribution settings but also excels in cross-dataset generalization to unseen action classes.
Yearang Lee, Ho-Joong Kim, Seong-Whan Lee
Dept. of Artificial Intelligence, Korea University, Seoul, Korea
Spatio-temporal action detection in videos requires jointly localizing actors in space and identifying action boundaries over time. A common challenge is constructing temporally stable action tubes, as frame-level detectors often suffer from jitter, fragmentation, and imprecise temporal localization. Many recent approaches address this by introducing heavy spatio-temporal transformers or optical-flow-based pipelines, leading to high computational cost and limited scalability. We propose TubeLite, a lightweight framework for spatio-temporal action detection that focuses on stable tube construction and boundary-aware temporal modeling. TubeLite represents each actor as a tube, defined as a sequence of bounding boxes associated with a single actor over time, and explicitly enforces temporal consistency at both the spatial and semantic levels. The method combines low-jitter actor detection, Gaussian-weighted actor feature extraction, efficient short-term temporal propagation, and a boundary-focused temporal prediction head, while avoiding optical flow and large-scale temporal attention. Despite its compact design, TubeLite achieves strong video-level localization performance. It improves [email protected] by 4.5 and 7.1 percentage points over the best compared method on the MultiSports and UCF101-24 datasets, respectively, with substantially fewer parameters and floating-point operations than transformer-based alternatives, demonstrating that effective spatio-temporal action detection can be obtained through principled, lightweight temporal modeling.
Ali Soltaninezhad, Melissa Cote, Alejandro Rico Espinosa +2
Electrical and Computer Engineering, University of Victoria, Victoria, BC, Canada