Unsupervised temporal action localization (UTAL) aims to identify action categories and localize their temporal boundaries without annotation supervision. Existing methods typically derive supervision for both tasks solely from RGB and optical-flow representations, often resulting in noisy category assignments and inaccurate temporal boundaries. In this work, we reveal an overlooked property of video codecs: group-of-pictures quantization patterns preserve category-discriminative dynamics, while motion-vector and residual variations exhibit pronounced responses around action transitions. Based on this insight, we propose CoDy, a codec dynamics-guided framework that exploits the codec as complementary information to the visual view for UTAL. CoDy integrates a codec-enhanced view with RGB and flow through confidence-aware consensus clustering, producing reliable soft pseudo-labels. It further converts codec dynamics into a class-agnostic boundary prior, which supervises visual boundary learning and guides proposal initialization and iterative refinement. The codec branches are only used during the training stage and introduce no extra codec-side computation at inference. Extensive experiments on THUMOS14 and ActivityNet v1.3 demonstrate that CoDy consistently outperforms existing UTAL methods and achieves state-of-the-art localization performance. These results establish codec signals as effective annotation-free priors for jointly inferring which actions occur and when they occur.
Figures & tables
Figure 1: The codec view’s predictive dynamics provide complementary priors to correct errors in the visual view.
Figure 2: Empirical results of codec-prior dynamics for category discrimination and boundary sensitivity detection.
Figure 3: Overview of the proposed CoDy. CoDy executes from two views of visual and complementary codec view and is composed of P-MIL-based baseline, codec-enhanced pseudo-label generation and codec-based boundary prior learning.
Sup.
Method
mAP@IoU (%)
AVG
0.1
0.2
0.3
0.4
0.5
0.6
0.7
(0.1:0.5)
(0.3:0.7)
(0.1:0.7)
Weakly
CoLA [ 27 ]
66.2
59.5
51.5
41.9
32.2
22.0
13.1
50.3
32.1
40.9
DGCNN [ 17 ]
66.3
59.9
52.3
43.2
32.8
22.1
13.1
50.9
32.7
41.3
AUMN [ 12 ]
66.2
61.9
54.9
44.4
33.3
20.5
9.0
52.1
32.4
41.5
CSCL [ 9 ]
68.0
61.8
52.7
43.3
33.4
21.8
12.3
51.8
32.7
41.9
FAC-Net [ 26 ]
67.6
62.1
52.6
44.3
33.4
22.5
12.7
52.0
33.1
42.2
Table 1: Comparison results on the THUMOS14. The best and second-best results are in bold and underlined .
Method
mAP@IoU (%)
0.5
0.75
0.95
AVG
STPN [ 14 ]
23.9
14.9
3.8
15.0
TSCN [ 19 ]
25.9
16.2
3.8
16.3
WSAL-BM + UGCT [ 24 ]
30.1
19.2
3.9
18.5
TSCN + UGCT [ 24 ]
30.3
19.4
4.3
18.7
STPN + UGCT [ 24 ]
32.5
20.3
4.5
19.8
Table 2: Comparison results on the ActivityNet v1.3. The best and second-best results are in bold and underlined .
Main components
mAP@IoU (%)
Classification
Localization
0.3
0.5
0.7
AVG
44.8
30.8
8.5
35.7
✓
48.8
34.9
11.0
38.4
✓
49.9
35.2
12.1
39.5
✓
✓
53.6
37.5
14.8
43.1
Table 3: Ablation study of key components on THUMOS14.
Figure 4: Ablation study of codec for proposal source and pseudo-label quality.
Codec Feature
mAP@IoU (%)
0.3
0.5
0.7
AVG
Classification
Residual-only
48.9
33.6
10.1
38.9
MV-only
49.6
35.5
11.1
40.9
Localization
Residual-only
50.6
35.7
12.4
41.8
MV-only
49.5
34.6
11.8
39.8
CoDy(Ours)
MV+Residual
53.6
37.5
14.8
43.1
Table 4: Ablation study of motion vector and residual features on classification and localization performance.
Open-Vocabulary Temporal Action Localization (OV-TAL) aims to recognize and localize instances of any desired action categories in videos without explicitly curating training data for all categories. Existing methods mostly recognize action categories at a single granularity, which degrades the recognition accuracy of both base and novel action categories. To address these issues, we propose a Multi-Grained Category-Aware Network (MGCA-Net) comprising a localizer, an action presence predictor, a conventional classifier, and a coarse-to-fine classifier. Specifically, the localizer localizes category-agnostic action proposals. For these action proposals, the action presence predictor estimates the probability that they belong to an action instance. At the same time, the conventional classifier predicts the probability of each action proposal over base action categories at the snippet granularity. Novel action categories are recognized by the coarse-to-fine classifier, which first identifies action presence at the video granularity. Finally, it assigns each action proposal to one category from the coarse categories at the proposal granularity. Through coarse-to-fine category awareness for novel actions and the conventional classifier's awareness of base actions, multi-grained category awareness is achieved, effectively enhancing localization performance. Comprehensive evaluations on the THUMOS'14 and ActivityNet-1.3 benchmarks demonstrate that our method achieves state-of-the-art performance. Furthermore, our MGCA-Net achieves state-of-the-art results under the Zero-Shot Temporal Action Localization setting.
Zhenying Fang, Richang Hong
School of Computer Science and Information Engineering, Hefei University of Technology, 188065, China
Temporal Action Localization (TAL) typically relies on segment annotations or offline access to full videos, limiting scalability and online use. We introduce Point-Supervised Online TAL (POTAL), which localizes actions in streaming videos using only one temporal point per instance. To solve POTAL, we propose OnPoint, an offline-to-online multi-level distillation framework that transfers knowledge from a point-supervised offline teacher to an online student via (i) pseudo-segment instance distillation, (ii) class-activation sequence distillation, and (iii) anticipatory window-level distillation. We further improve robustness by incorporating the original point labels into student training and by refining anchor decoding with actionness-guided attention calibration. Experiments on five datasets show OnPoint consistently outperforms strong baselines, establishing a solid foundation for POTAL.
Sakib Reza, Gauri Jagatap, Mohsen Moghaddam +2
Northeastern University, Boston, MA 02115, USA · Georgia Institute of Technology, Atlanta, GA 30332, USA · Dolby Laboratories +1
Point-supervised Temporal Action Localization (PTAL) adopts a lightly frame-annotated paradigm (i.e., labeling only a single frame per action instance) to train a model to effectively locate action instances within untrimmed videos. Most existing approaches design the task head of models with only a point-supervised snippet-level classification, without explicit modeling of understanding temporal relationships among frames of an action. However, understanding the temporal relationships of frames is crucial because it can help a model understand how an action is defined and therefore benefits localizing the full frames of an action. To this end, in this paper, we design a multi-task learning framework that fully utilizes point supervision to boost the model's temporal understanding capability for action localization. Specifically, we design three self-supervised temporal understanding tasks: (i) Action Completion, (ii) Action Order Understanding, and (iii) Action Regularity Understanding. These tasks help a model understand the temporal consistency of actions across videos. To the best of our knowledge, this is the first attempt to explicitly explore temporal consistency for point supervision action localization. Extensive experimental results on four benchmark datasets demonstrate the effectiveness of the proposed method compared to several state-of-the-art approaches.
Yunchuan Ma, Laiyun Qing, Guorong Li +3
University of Chinese Academy of Sciences, Beijing, 100190, China · Macquarie University