Temporal Action Detection

Ask about this topic

Overview

Temporal action detection studies how to find and classify action intervals in long, untrimmed videos, supporting tasks such as sports event spotting, surgical phase monitoring, and online alerts. Common approaches use temporal convolutional networks, transformers, or state-space models to combine local motion cues with longer-range context, then predict boundaries and rank candidate events. Research also addresses weak or point-level labels, streaming and efficient inference, and zero-shot or open-vocabulary detection using language and vision-language models; THUMOS14 and ActivityNet-1.3 are common benchmarks. A central challenge is balancing accurate, coherent boundaries and generalization to unseen settings against the cost of dense annotation and computation, while ensuring evaluation reflects event-level performance rather than frame scores alone.

Latest papers

CardsList

41 papers6 papers in the last four weeks, up 50% on the four weeks before. 0.0% of all new papers.

May 4, 2026

TemPose-TF-ASF: Two-Stage Bidirectional Stroke Context Fusion for Badminton Stroke Classification

Accurate badminton stroke prediction is crucial for fine-grained sports analysis and tactical decision support. However, existing methods struggle to model rich temporal context. This paper introduces TemPose-TF-ASF (Adjacent-Stroke Fusion), a context-aware extension of TemPose. It enhances stroke recognition by incorporating stroke-type information from both preceding and subsequent strokes. A two-stage training and inference strategy is adopted. Preliminary predictions from the baseline model are reused as estimated temporal context. These predictions guide the joint optimization of the ASF module and the classifier. By explicitly modeling bidirectional temporal stroke dependencies, the proposed method can be seamlessly integrated into existing state-of-the-art models. Experiments on a large-scale badminton match dataset show consistent improvements over the baseline and its variants in terms of Accuracy and Macro-F1. Moreover, integrating ASF into other advanced methods yields notable performance gains. These results demonstrate strong transferability and generalization capability.

Tzu-Yu Liu, Duan-Shin Lee

Institute of Communications Engineering National Tsing Hua University, Hsinchu City, Taiwan · Department of Computer Science National Tsing Hua University, Hsinchu City, Taiwan

Sports Video AnalysisVideo Action RecognitionTemporal Action DetectionSequence ModelingTemporal Video Understanding
AskPDF
Apr 20, 2026

Denoise and Align: Diffusion-Driven Foreground Knowledge Prompting for Open-Vocabulary Temporal Action Detection

Open-Vocabulary Temporal Action Detection (OV-TAD) aims to localize and classify action segments of unseen categories in untrimmed videos, where effective alignment between action semantics and video representations is critical for accurate detection. However, existing methods struggle to mitigate the semantic imbalance between concise, abstract action labels and rich, complex video contents, inevitably introducing semantic noise and misleading cross-modal alignment. To address this challenge, we propose DFAlign, the first framework that leverages diffusion-based denoising to generate foreground knowledge for the guidance of action-video alignment. Following the 'conditioning, denoising and aligning' manner, we first introduce the Semantic-Unify Conditioning (SUC) module, which unifies action-shared and action-specific semantics as conditions for diffusion denoising. Then, the Background-Suppress Denoising (BSD) module generates foreground knowledge by progressively removing background redundancy from videos through denoising process. This foreground knowledge serves as effective intermediate semantic anchor between video and text representations, mitigating the semantic gap and enhancing the discriminability of action-relevant segments. Furthermore, we introduce the Foreground-Prompt Alignment (FPA) module to inject extracted foreground knowledge as prompt tokens into text representations, guiding model's attention towards action-relevant segments and enabling precise cross-modal alignment. Extensive experiments demonstrate that our method achieves state-of-the-art performance on two OV-TAD benchmarks. The code repository is provided as follows: https://anonymous.4open.science/r/Code-2114/.

Sa Zhu, Wanqian Zhang, Lin Wang +3

Institute of Information Engineering, Chinese Academy of Sciences, School of Cyber Security, University of Chinese Academy of Sciences, State Key Laboratory of Cyberspace Security Defense, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · Hangzhou Dianzi University, Hangzhou, China +1

Cross-Modal AlignmentTemporal Action DetectionTemporal Video Understanding
AskPDF
Apr 20, 2026

LiquidTAD: Efficient Temporal Action Detection via Parallel Liquid-Inspired Temporal Relaxation

Temporal Action Detection (TAD) requires precise localization of action boundaries within long, untrimmed video sequences. While current high-performing methods achieve strong accuracy, they are often characterized by excessive parameter counts, substantial computational overhead, and a reliance on specialized operators that hinder deployment across diverse hardware platforms. This paper presents LiquidTAD, a framework that distills the exponential relaxation prior of liquid neural dynamics into a parallel temporal operator, rather than reproducing full Liquid Neural Network (LNN) dynamics. By introducing a Parallel Liquid-inspired Relaxation mechanism, sequential ODE solving is avoided through a fully vectorized, non-recursive formulation built entirely upon standard neural operations, enabling hardware-agnostic deployment with linear complexity with respect to the temporal length. A complementary Hierarchical Decay-Rate Sharing Strategy further adapts this relaxation prior across feature pyramid levels, stabilizing optimization and implicitly compensating for temporal compression in deeper layers. Experimental evaluations on THUMOS-14 and ActivityNet-1.3 demonstrate that LiquidTAD achieves accuracy competitive with strong baselines while substantially lowering the model footprint. Specifically, on THUMOS-14, LiquidTAD achieves 69.46% average mAP with only 10.82M parameters and 27.17G FLOPs, reducing the parameter count by over 60% compared with ActionFormer.

Zepeng Sun, Naichuan Zheng, Hailun Xia +3

Liquid Neural NetworksHuman Activity RecognitionTemporal Action DetectionTemporal Video UnderstandingTemporal Action Localization
AskPDF
Apr 10, 2026

From Frames to Events: Rethinking Evaluation in Human-Centric Video Anomaly Detection

Pose-based Video Anomaly Detection (VAD) has gained significant attention for its privacy-preserving nature and robustness to environmental variations. However, traditional frame-level evaluations treat video as a collection of isolated frames, fundamentally misaligned with how anomalies manifest and are acted upon in the real world. In operational surveillance systems, what matters is not the flagging of individual frames, but the reliable detection, localization, and reporting of a coherent anomalous event, a contiguous temporal episode with an identifiable onset and duration. Frame-level metrics are blind to this distinction, and as a result, they systematically overestimate model performance for any deployment that requires actionable, event-level alerts. In this work, we propose a shift toward an event-centric perspective in VAD. We first audit widely used VAD benchmarks, including SHT[19], CHAD[6], NWPUC[4], and HuVAD[25], to characterize their event structure. We then introduce two strategies for temporal event localization: a score-refinement pipeline with hierarchical Gaussian smoothing and adaptive binarization, and an end-to-end Dual-Branch Model that directly generates event-level detections. Finally, we establish the first event-based evaluation standard for VAD by adapting Temporal Action Localization metrics, including tIoU-based event matching and multi-threshold F1 evaluation. Our results quantify a substantial performance gap: while all SoTA models achieve frame-level AUC-ROC exceeding 52% on the NWPUC[4], their event-level localization precision falls below 10% even at a minimal tIoU=0.2, with an average event-level F1 of only 0.11 across all thresholds. The code base for this work is available at https://github.com/TeCSAR-UNCC/EventCentric-VAD.

Narges Rashvand, Shanle Yao, Armin Danesh Pazho +2

University of North Carolina at Charlotte, Charlotte, NC, USA

Reference-Based Video Anomaly DetectionVideo Anomaly DetectionTemporal Action Detection
AskPDF
Apr 10, 2026

Efficient Spatial-Temporal Focal Adapter with SSM for Temporal Action Detection

Temporal human action detection aims to identify and localize action segments within untrimmed videos, serving as a pivotal task in video understanding. Despite the progress achieved by prior architectures like CNN and Transformer models, these continue to struggle with feature redundancy and degraded global dependency modeling capabilities when applied to long video sequences. These limitations severely constrain their scalability in real-world video analysis. State Space Models (SSMs) offer a promising alternative with linear long-term modeling and robust global temporal reasoning capabilities. Rethinking the application of SSMs in temporal modeling, this research constructs a novel framework for video human action detection. Specifically, we introduce the Efficient Spatial-Temporal Focal (ESTF) Adapter into the pre-trained layers. This module integrates the advantages of our proposed Temporal Boundary-aware SSM(TB-SSM) for temporal feature modeling with efficient processing of spatial features. We perform comprehensive and quantitative analyses across multiple benchmarks, comparing our proposed method against previous SSM-based and other structural methods. Extensive experiments demonstrate that our improved strategy significantly enhances both localization performance and robustness, validating the effectiveness of our proposed method.

Yicheng Qiu, Keiji Yanai

Department of Informatics The University of Electro-Communications Chofu, Tokyo, Japan

Temporal Action DetectionTemporal Action LocalizationTemporal Video UnderstandingVisual SSMs
AskPDF
Mar 31, 2026

From Skeletons to Semantics: Design and Deployment of a Hybrid Edge-Based Action Detection System for Public Safety

Public spaces such as transport hubs, city centres, and event venues require timely and reliable detection of potentially violent behaviour to support public safety. While automated video analysis has made significant progress, practical deployment remains constrained by latency, privacy, and resource limitations, particularly under edge-computing conditions. This paper presents the design and demonstrator-based deployment of a hybrid edge-based action detection system that combines skeleton-based motion analysis with vision-language models for semantic scene interpretation. Skeleton-based processing enables continuous, privacy-aware monitoring with low computational overhead, while vision-language models provide contextual understanding and zero-shot reasoning capabilities for complex and previously unseen situations. Rather than proposing new recognition models, the contribution focuses on a system-level comparison of both paradigms under realistic edge constraints. The system is implemented on a GPU-enabled edge device and evaluated with respect to latency, resource usage, and operational trade-offs using a demonstrator-based setup. The results highlight the complementary strengths and limitations of motioncentric and semantic approaches and motivate a hybrid architecture that selectively augments fast skeletonbased detection with higher-level semantic reasoning. The presented system provides a practical foundation for privacy-aware, real-time video analysis in public safety applications.

Ganen Sethupathy, Lalit Dumka, Jan Schagen

Sopra Steria Germany, Hamburg, Germany · Sopra Steria India, Noida, India

Video SurveillanceTemporal Action DetectionSkeleton-Based Action RecognitionEdge ComputingVision-Language Models
AskPDF
Mar 25, 2026

Decompose and Transfer: CoT-Prompting Enhanced Alignment for Open-Vocabulary Temporal Action Detection

Open-Vocabulary Temporal Action Detection (OV-TAD) aims to classify and localize action segments in untrimmed videos for unseen categories. Previous methods rely solely on global alignment between label-level semantics and visual features, which is insufficient to transfer temporal consistent visual knowledge from seen to unseen classes. To address this, we propose a Phase-wise Decomposition and Alignment (PDA) framework, which enables fine-grained action pattern learning for effective prior knowledge transfer. Specifically, we first introduce the CoT-Prompting Semantic Decomposition (CSD) module, which leverages the chain-of-thought (CoT) reasoning ability of large language models to automatically decompose action labels into coherent phase-level descriptions, emulating human cognitive processes. Then, Text-infused Foreground Filtering (TIF) module is introduced to adaptively filter action-relevant segments for each phase leveraging phase-wise semantic cues, producing semantically aligned visual representations. Furthermore, we propose the Adaptive Phase-wise Alignment (APA) module to perform phase-level visual-textual matching, and adaptively aggregates alignment results across phases for final prediction. This adaptive phase-wise alignment facilitates the capture of transferable action patterns and significantly enhances generalization to unseen actions. Extensive experiments on two OV-TAD benchmarks demonstrated the superiority of the proposed method.

Sa Zhu, Wanqian Zhang, Lin Wang +4

Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · State Key Laboratory of Cyberspace Security Defense +2

Temporal Action DetectionTemporal Video UnderstandingVision-Language Alignment
AskPDF
Mar 25, 2026

CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning

Online Action Detection (OAD) systems face two primary challenges: high computational cost and insufficient modeling of discriminative temporal dynamics against background motion. Adding optical flow could provides strong motion cues but it incurs significant computational overhead. We propose CAKE, a OAD Flow-based distillation framework to transfer motion knowledge into RGB models. We propose Dynamic Motion Adapter (DMA) to suppress static background noise and emphasize pixel changes, effectively approximating optical flow without explicit computation. The framework also integrates a Floating Contrastive Learning strategy to distinguish informative motion dynamics from temporal background. Various experiments conducted on the TVSeries, THUMOS'14, Kinetics-400 datasets show effectiveness of our model. CAKE achieves a standout mAP compared with SOTA while using the same backbone. Our model operates at over 72 FPS on a single CPU, making it highly suitable for resource-constrained systems.

Hieu Hoang, Dung Trung Tran, Hong Nguyen +1

Viettel Group, Vietnam · School of Electrical and Electronics Engineering, Hanoi University of Science and Technology, Vietnam · Signal Analysis and Interpretation Lab, University of Southern California, United States

Temporal Action DetectionHuman Activity RecognitionKnowledge Distillation
AskPDF
Feb 5, 2026

Exploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localization

Point-supervised Temporal Action Localization (PTAL) adopts a lightly frame-annotated paradigm (i.e., labeling only a single frame per action instance) to train a model to effectively locate action instances within untrimmed videos. Most existing approaches design the task head of models with only a point-supervised snippet-level classification, without explicit modeling of understanding temporal relationships among frames of an action. However, understanding the temporal relationships of frames is crucial because it can help a model understand how an action is defined and therefore benefits localizing the full frames of an action. To this end, in this paper, we design a multi-task learning framework that fully utilizes point supervision to boost the model's temporal understanding capability for action localization. Specifically, we design three self-supervised temporal understanding tasks: (i) Action Completion, (ii) Action Order Understanding, and (iii) Action Regularity Understanding. These tasks help a model understand the temporal consistency of actions across videos. To the best of our knowledge, this is the first attempt to explicitly explore temporal consistency for point supervision action localization. Extensive experimental results on four benchmark datasets demonstrate the effectiveness of the proposed method compared to several state-of-the-art approaches.

Yunchuan Ma, Laiyun Qing, Guorong Li +3

University of Chinese Academy of Sciences, Beijing, 100190, China · Macquarie University

Weakly Supervised LearningTemporal ConsistencyTemporal Action DetectionTemporal Video UnderstandingTemporal Action Localization
AskPDF
Nov 17, 2025

MGCA-Net: Multi-Grained Category-Aware Network for Open-Vocabulary Temporal Action Localization

Open-Vocabulary Temporal Action Localization (OV-TAL) aims to recognize and localize instances of any desired action categories in videos without explicitly curating training data for all categories. Existing methods mostly recognize action categories at a single granularity, which degrades the recognition accuracy of both base and novel action categories. To address these issues, we propose a Multi-Grained Category-Aware Network (MGCA-Net) comprising a localizer, an action presence predictor, a conventional classifier, and a coarse-to-fine classifier. Specifically, the localizer localizes category-agnostic action proposals. For these action proposals, the action presence predictor estimates the probability that they belong to an action instance. At the same time, the conventional classifier predicts the probability of each action proposal over base action categories at the snippet granularity. Novel action categories are recognized by the coarse-to-fine classifier, which first identifies action presence at the video granularity. Finally, it assigns each action proposal to one category from the coarse categories at the proposal granularity. Through coarse-to-fine category awareness for novel actions and the conventional classifier's awareness of base actions, multi-grained category awareness is achieved, effectively enhancing localization performance. Comprehensive evaluations on the THUMOS'14 and ActivityNet-1.3 benchmarks demonstrate that our method achieves state-of-the-art performance. Furthermore, our MGCA-Net achieves state-of-the-art results under the Zero-Shot Temporal Action Localization setting.

Zhenying Fang, Richang Hong

School of Computer Science and Information Engineering, Hefei University of Technology, 188065, China

Video Action RecognitionTemporal Action DetectionTemporal Video UnderstandingTemporal Action Localization
AskPDF
Oct 12, 2025

Action-Dynamics Modeling and Cross-Temporal Interaction for Online Action Understanding

Online action understanding, encompassing action detection and anticipation, is central to numerous practical applications. However, untrimmed videos contain substantial redundant and noisy historical information, and the complementary interactions among historical, current, and future-oriented representations remain underexplored. To address these challenges, we propose the State-Specific Model (SSM), a unified framework for online action detection and anticipation. Its Representative State-Based Memory Compression module selects representative anchor frames from the observed memory and aggregates anchor-conditioned context into a compact set of representative memory states, thereby reducing memory redundancy. The subsequent Action Pattern Learning module learns multi-dimensional relations among these states and performs graph reasoning over the resulting directed representative-state graph to model action dynamics and derive a future-oriented cue. The Cross-Temporal Interaction module then enables this cue to interact with historical and current cues, refining the representations used for simultaneous online action detection and anticipation. Extensive experiments on EPIC-Kitchens-100, THUMOS'14, TVSeries, and the introduced Parkinson's Disease Mouse Behaviour dataset demonstrate competitive performance and generalizability across diverse online action understanding settings. These results underscore the value of graph-based action-dynamics modelling and cross-temporal interaction for online action understanding.

Xinyu Yang, Zheheng Jiang, Feixiang Zhou +5

School of Computing and Mathematical Sciences, University of Leicester, United Kingdom. · School of Eye and Vision Sciences, University of Liverpool, United Kingdom. · School of Information Science and Engineering, University of Jinan, China. +1

Temporal Action DetectionNext-Action PredictionMemory-Augmented Video Understanding
AskPDF