Temporal action detection in long untrimmed videos suffers from inaccurate localization and low-quality proposals, especially with sparse annotations and ambiguous boundaries. For open-ended video understanding and LLM-driven event analysis, reliable proposals connect temporal localization, event annotation, and semantic understanding. However, conventional temporal action detection (TAD) pipelines lack explicit modeling of proposal localization quality, cross-representation support, and inter-proposal relations. We propose Hybrid Rank-Calibration of Action Proposals (HyrCap), a proposal-centric query-based TAD framework for temporal event understanding. Hybrid combines complementary video features with different localization cues, while Rank-Calibration adjusts proposal ranking confidence to reflect localization reliability and candidate competition. HyrCap treats dense query-based predictions as an over-complete temporal event hypothesis space. Cross-Representation Consensus Calibration, Localization Quality Estimation, and Proposal Redundancy Suppression improve proposal ranking reliability, localization quality, and candidate discrimination. Inspired by LLMs' capabilities in fine-grained semantic recognition, event description, and reasoning, we introduce HyrCap for Semantic Action Understanding, using calibrated proposals for standard TAD and as inputs to LLM-based semantic understanding. We define a staged evaluation protocol to separately measure event localization, segment-level semantic understanding, and combined event understanding performance. Experiments on public benchmarks show that HyrCap achieves state-of-the-art performance across multiple metrics and provides a finer-grained paradigm and evaluation protocol for query-based event segmentation and annotation.
Figures & tables
Figure 2: Visualization of different features under TAD. Intern. denotes InternVideo2 features.
Figure 3: Overview of HyrCap. (a) Query-based event candidate generation with InternVideo2 and V-JEPA2 representations. (b) HyrCap calibrates and refines dense proposals through cross-representation consensus calibration, localization quality estimation, proposal redundancy suppression, and post-processing. (c) Conventional one-stage loc-cls detection protocol. (d) Two-stage loc-first protocol for staged semantic understanding. Loc denotes localization.
Training Type
Head Type
Method
Feature
THUMOS14
ActivityNet v1.3
0.3
0.4
0.5
0.6
0.7
Avg.
0.5
0.75
0.95
Avg.
Full
-
AFSD
I3D [ 14 ]
67.3
62.4
55.5
43.7
31.1
52.0
52.4
35.3
6.5
34.4
TALLFormer
Swin-B [ 27 ]
76.0
-
63.2
-
34.5
59.2
54.1
36.2
7.9
35.6
ViT-TAD
ViT-B [ 28 ]
85.1
80.9
74.2
61.8
45.4
69.5
55.8
38.5
8.8
37.4
AdaTAD
VideoMAEv2-g [ 19 ]
89.7
86.7
80.9
71.0
56.1
76.9
61.7
43.4
10.9
41.9
Head only
Anchor -free
TriDet
I3D [ 14 ] / R(2+1)D [ 17 ]
83.6
80.1
72.9
62.4
47.4
69.3
54.7
38.0
8.4
36.8
Table 1: Comparison with state-of-the-art methods on THUMOS14 [ 1 ] and ActivityNet v1.3 [ 2 ] . For TE -TAD + Ours, we additionally apply our methods on TE-TAD.
Head Type
Method
Feature
mAP
0.5
0.75
0.95
Avg.
Anchor-based
SSN [ 29 ]
I3D [ 14 ]
28.8
18.8
5.3
19.0
G-TAD [ 30 ]
I3D [ 14 ]
41.1
27.6
8.3
27.5
BMN [ 4 ]
SlowFast [ 31 ]
52.5
36.4
10.4
35.8
TCANet [ 32 ]
SlowFast [ 31 ]
54.1
37.2
11.3
36.8
Anchor-free
TALLFormer [ 33 ]
Swin-B [ 27 ]
55.0
36.1
11.8
36.5
Table 2: Performance comparison with state-of-the-art methods on HACS-Segment.
1Pohang University of Science and Technology, Pohang, South Korea · 2Korea Advanced Institute of Science and Technology, Daejeon, South Korea · 3Handong Global University, Pohang, South Korea