In video action detection, modeling the temporal dependencies in continuous video streams is crucial. Previous methods based on deep neural networks can perform frame-level video recognition tasks. However, these methods struggle to model dynamic temporal patterns, facing challenges such as redundant frame information, difficulty balancing global and local features, and insufficient fusion of visual and textual features. To address these issues, we propose a video temporal attention module based on the Multi-Scale Latent Temporal Compression Attention, which aggregates continuous temporal sequences into feature representations across diverse temporal scales while efficiently reducing redundancy. We also introduce a Video-Text Cross Attention Fusion module that leverages cross-attention at both latent and standard feature levels, achieving fine-grained fusion between video and textual features to enhance semantic alignment and feature interaction. Experimental results across multiple video recognition benchmarks demonstrate that our proposed methods achieve superior performance compared to existing methods, validating the effectiveness and generalization capability.
Figures & tables
Figure 1: (a) MLTC-Attention models multi-scale temporal dependencies via compress-then-attend in latent space, and VTCA-Fusion enforces hierarchical video-text alignment at both standard and latent feature levels. (b) Performance–efficiency on video anomaly detection (blue, left axis) and video action detection (orange, right axis): our method (red) attains the best accuracy at the highest inference speed (FPS) with the fewest FLOPs on both tasks.
Figure 2: Overview of the overall architecture of our proposed video action detection framework. Our method integrates the VTCA-Fusion module and MLTC-Attention Blocks to process temporal and categorical features, with the resulting predictions feeding into various downstream tasks.
Method
Vision Encoder
UCF-Crime
XD-Violence
AUC(%)
AP(%)
LAVAD [ 63 ]
ImageBind [ 13 ]
80.28
62.01
DMU † [ 67 ]
I3D [ 3 ]
86.14
81.46
AnomalyCLIP [ 62 ]
ViT-B16 [ 11 ]
86.30
77.40
VERA [ 60 ]
InternVL2 [ 8 ]
86.55
70.54
UMIL [ 29 ]
ViT-B32 [ 11 ]
86.75
–
Table 1: Comparison of frame-level performance between our method and previous methods on the UCF-Crime and XD-Violence benchmarks. † : results reproduced by us with the listed vision encoder.
Table 4Table 5
Figure 3: Qualitative results of our proposed methods on Video Anomaly Detection tasks.
Figure 4: Qualitative results of our proposed methods on temporal action detection tasks.
Figure 5: Visualization of internal representation category-wise temporal feature.