Unsupervised temporal action localization (UTAL) aims to identify action categories and localize their temporal boundaries without annotation supervision. Existing methods typically derive supervision for both tasks solely from RGB and optical-flow representations, often resulting in noisy category assignments and inaccurate temporal boundaries. In this work, we reveal an overlooked property of video codecs: group-of-pictures quantization patterns preserve category-discriminative dynamics, while motion-vector and residual variations exhibit pronounced responses around action transitions. Based on this insight, we propose CoDy, a codec dynamics-guided framework that exploits the codec as complementary information to the visual view for UTAL. CoDy integrates a codec-enhanced view with RGB and flow through confidence-aware consensus clustering, producing reliable soft pseudo-labels. It further converts codec dynamics into a class-agnostic boundary prior, which supervises visual boundary learning and guides proposal initialization and iterative refinement. The codec branches are only used during the training stage and introduce no extra codec-side computation at inference. Extensive experiments on THUMOS14 and ActivityNet v1.3 demonstrate that CoDy consistently outperforms existing UTAL methods and achieves state-of-the-art localization performance. These results establish codec signals as effective annotation-free priors for jointly inferring which actions occur and when they occur.
Figures & tables
Figure 1: The codec view’s predictive dynamics provide complementary priors to correct errors in the visual view.
Figure 2: Empirical results of codec-prior dynamics for category discrimination and boundary sensitivity detection.
Figure 3: Overview of the proposed CoDy. CoDy executes from two views of visual and complementary codec view and is composed of P-MIL-based baseline, codec-enhanced pseudo-label generation and codec-based boundary prior learning.
Sup.
Method
mAP@IoU (%)
AVG
0.1
0.2
0.3
0.4
0.5
0.6
0.7
(0.1:0.5)
(0.3:0.7)
(0.1:0.7)
Weakly
CoLA [ 27 ]
66.2
59.5
51.5
41.9
32.2
22.0
13.1
50.3
32.1
40.9
DGCNN [ 17 ]
66.3
59.9
52.3
43.2
32.8
22.1
13.1
50.9
32.7
41.3
AUMN [ 12 ]
66.2
61.9
54.9
44.4
33.3
20.5
9.0
52.1
32.4
41.5
CSCL [ 9 ]
68.0
61.8
52.7
43.3
33.4
21.8
12.3
51.8
32.7
41.9
FAC-Net [ 26 ]
67.6
62.1
52.6
44.3
33.4
22.5
12.7
52.0
33.1
42.2
Table 1: Comparison results on the THUMOS14. The best and second-best results are in bold and underlined .
Method
mAP@IoU (%)
0.5
0.75
0.95
AVG
STPN [ 14 ]
23.9
14.9
3.8
15.0
TSCN [ 19 ]
25.9
16.2
3.8
16.3
WSAL-BM + UGCT [ 24 ]
30.1
19.2
3.9
18.5
TSCN + UGCT [ 24 ]
30.3
19.4
4.3
18.7
STPN + UGCT [ 24 ]
32.5
20.3
4.5
19.8
Table 2: Comparison results on the ActivityNet v1.3. The best and second-best results are in bold and underlined .
Main components
mAP@IoU (%)
Classification
Localization
0.3
0.5
0.7
AVG
44.8
30.8
8.5
35.7
✓
48.8
34.9
11.0
38.4
✓
49.9
35.2
12.1
39.5
✓
✓
53.6
37.5
14.8
43.1
Table 3: Ablation study of key components on THUMOS14.
Figure 4: Ablation study of codec for proposal source and pseudo-label quality.
Codec Feature
mAP@IoU (%)
0.3
0.5
0.7
AVG
Classification
Residual-only
48.9
33.6
10.1
38.9
MV-only
49.6
35.5
11.1
40.9
Localization
Residual-only
50.6
35.7
12.4
41.8
MV-only
49.5
34.6
11.8
39.8
CoDy(Ours)
MV+Residual
53.6
37.5
14.8
43.1
Table 4: Ablation study of motion vector and residual features on classification and localization performance.