Temporal action detection in long untrimmed videos suffers from inaccurate localization and low-quality proposals, especially with sparse annotations and ambiguous boundaries. For open-ended video understanding and LLM-driven event analysis, reliable proposals connect temporal localization, event annotation, and semantic understanding. However, conventional temporal action detection (TAD) pipelines lack explicit modeling of proposal localization quality, cross-representation support, and inter-proposal relations. We propose Hybrid Rank-Calibration of Action Proposals (HyrCap), a proposal-centric query-based TAD framework for temporal event understanding. Hybrid combines complementary video features with different localization cues, while Rank-Calibration adjusts proposal ranking confidence to reflect localization reliability and candidate competition. HyrCap treats dense query-based predictions as an over-complete temporal event hypothesis space. Cross-Representation Consensus Calibration, Localization Quality Estimation, and Proposal Redundancy Suppression improve proposal ranking reliability, localization quality, and candidate discrimination. Inspired by LLMs' capabilities in fine-grained semantic recognition, event description, and reasoning, we introduce HyrCap for Semantic Action Understanding, using calibrated proposals for standard TAD and as inputs to LLM-based semantic understanding. We define a staged evaluation protocol to separately measure event localization, segment-level semantic understanding, and combined event understanding performance. Experiments on public benchmarks show that HyrCap achieves state-of-the-art performance across multiple metrics and provides a finer-grained paradigm and evaluation protocol for query-based event segmentation and annotation.
Figures & tables
Figure 2: Visualization of different features under TAD. Intern. denotes InternVideo2 features.
Figure 3: Overview of HyrCap. (a) Query-based event candidate generation with InternVideo2 and V-JEPA2 representations. (b) HyrCap calibrates and refines dense proposals through cross-representation consensus calibration, localization quality estimation, proposal redundancy suppression, and post-processing. (c) Conventional one-stage loc-cls detection protocol. (d) Two-stage loc-first protocol for staged semantic understanding. Loc denotes localization.
Training Type
Head Type
Method
Feature
THUMOS14
ActivityNet v1.3
0.3
0.4
0.5
0.6
0.7
Avg.
0.5
0.75
0.95
Avg.
Full
-
AFSD
I3D [ 14 ]
67.3
62.4
55.5
43.7
31.1
52.0
52.4
35.3
6.5
34.4
TALLFormer
Swin-B [ 27 ]
76.0
-
63.2
-
34.5
59.2
54.1
36.2
7.9
35.6
ViT-TAD
ViT-B [ 28 ]
85.1
80.9
74.2
61.8
45.4
69.5
55.8
38.5
8.8
37.4
AdaTAD
VideoMAEv2-g [ 19 ]
89.7
86.7
80.9
71.0
56.1
76.9
61.7
43.4
10.9
41.9
Head only
Anchor -free
TriDet
I3D [ 14 ] / R(2+1)D [ 17 ]
83.6
80.1
72.9
62.4
47.4
69.3
54.7
38.0
8.4
36.8
Table 1: Comparison with state-of-the-art methods on THUMOS14 [ 1 ] and ActivityNet v1.3 [ 2 ] . For TE -TAD + Ours, we additionally apply our methods on TE-TAD.
Head Type
Method
Feature
mAP
0.5
0.75
0.95
Avg.
Anchor-based
SSN [ 29 ]
I3D [ 14 ]
28.8
18.8
5.3
19.0
G-TAD [ 30 ]
I3D [ 14 ]
41.1
27.6
8.3
27.5
BMN [ 4 ]
SlowFast [ 31 ]
52.5
36.4
10.4
35.8
TCANet [ 32 ]
SlowFast [ 31 ]
54.1
37.2
11.3
36.8
Anchor-free
TALLFormer [ 33 ]
Swin-B [ 27 ]
55.0
36.1
11.8
36.5
Table 2: Performance comparison with state-of-the-art methods on HACS-Segment.
Understanding causal event relationships and achieving fine-grained temporal grounding in videos remain challenging for vision-language models (VLMs). We propose TEMPURA (Temporal Event Masked Prediction and Understanding for Reasoning in Action), a two-stage training framework that enhances the video temporal understanding of VLMs. Inspired by infilling techniques in language modeling, TEMPURA first performs masked event prediction, learning to reconstruct missing events and generate step-by-step causal explanations from dense event annotations. It then learns video segmentation and dense captioning, decomposing videos into non-overlapping events with detailed, timestamp-aligned descriptions. We train TEMPURA on VER, our large-scale dataset of 500K videos annotated with temporally aligned event descriptions and structured reasoning steps. Experiments on video temporal grounding and highlight detection benchmarks show that TEMPURA substantially improves strong base VLMs across model families and scales, confirming that combining event-level reasoning with fine-grained temporal segmentation is an effective recipe for video temporal understanding.
Jen-Hao Cheng, Yi-Hao Peng, Huapeng Zhou +11
University of Washington · Carnegie Mellon University · National Yang Ming Chiao Tung University +1
Reasoning temporal localization (RTL) requires a model to generate an answer that itself contains the time interval supporting it, so high-level reasoning and precise temporal grounding must be produced jointly in a single response. To tackle this challenging task, we propose the first event-centric video chain-of-thought framework, dubbed EventCoT. EventCoT first performs event-centric tokenization of the input video to convert it into compact event tokens, enabling efficient identification of question-relevant events. It then reasons within the identified events to generate the answer, grounding the time interval via embedding matching that aligns placeholder tokens with visual embeddings. EventCoT achieves state-of-the-art results on ActivityNet-RTL for reasoning temporal localization while using substantially fewer visual tokens than previous work. To verify its general performance, we further evaluate EventCoT on the grounded video question answering benchmark ReXTime, where it attains strong zero-shot results.
Youngkil Song, Yoonjae Baek, Dongwon Kim +3
1Pohang University of Science and Technology, Pohang, South Korea · 2Korea Advanced Institute of Science and Technology, Daejeon, South Korea · 3Handong Global University, Pohang, South Korea
Temporal Action Detection (TAD) requires precise localization of action boundaries within long, untrimmed video sequences. While current high-performing methods achieve strong accuracy, they are often characterized by excessive parameter counts, substantial computational overhead, and a reliance on specialized operators that hinder deployment across diverse hardware platforms. This paper presents LiquidTAD, a framework that distills the exponential relaxation prior of liquid neural dynamics into a parallel temporal operator, rather than reproducing full Liquid Neural Network (LNN) dynamics. By introducing a Parallel Liquid-inspired Relaxation mechanism, sequential ODE solving is avoided through a fully vectorized, non-recursive formulation built entirely upon standard neural operations, enabling hardware-agnostic deployment with linear complexity with respect to the temporal length. A complementary Hierarchical Decay-Rate Sharing Strategy further adapts this relaxation prior across feature pyramid levels, stabilizing optimization and implicitly compensating for temporal compression in deeper layers. Experimental evaluations on THUMOS-14 and ActivityNet-1.3 demonstrate that LiquidTAD achieves accuracy competitive with strong baselines while substantially lowering the model footprint. Specifically, on THUMOS-14, LiquidTAD achieves 69.46% average mAP with only 10.82M parameters and 27.17G FLOPs, reducing the parameter count by over 60% compared with ActionFormer.