DeformStruct: Deformable Structure Modeling for Point-Supervised Temporal Action Localization
Authors: Mengnan Liu, Sanping Zhou, Kun Xia, Yonghao Dong, Jie Wang, Gang Hua
Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · School of Software Engineering, Xi’an Jiaotong University, Xi’an 710049, China · MoE Key Laboratory of Brain-Inspired Intelligent Perception and Cognition, University of Science and Technology of China, Hefei 230027, China · Amazon Alexa AI, Bellevue, WA 98004 USA
Point-supervised Temporal Action Localization (PTAL) aims to localize action instances using only a single annotated timestamp for each instance. Such sparse supervision offers limited guidance for modeling the structural organization of an action instance. Existing PTAL methods primarily infer action regions from classification responses, leaving the structural organization of multiple action-related parts within each instance insufficiently modeled. To address this problem, we propose DeformStruct, an instance-level deformable structure modeling framework for PTAL. DeformStruct progressively constructs the structure of each action instance through three stages. First, Structural Anchor Mining discovers multiple anchors from annotated points and action-responsive regions, providing structural nodes grounded in the current instance. Second, Deformable Part Alignment associates latent structural parts with these anchors through deformation-aware matching, allowing their temporal supports to adapt to instance-dependent variations. Third, Relational Structural Organization explicitly models the relative temporal relations among the aligned parts, forming a relational representation that characterizes how different action-related parts are organized together. Extensive experiments on THUMOS14, ActivityNet-v1.3, GTEA, and BEOID demonstrate consistent improvements in point-supervised temporal localization and validate the effectiveness of deformable structural modeling.
Figures & tables
Fig. 1: Motivation of deformable structural modeling for PTAL. A complete action instance may contain multiple temporally distributed action-related stages, whose locations, durations, and relative configurations vary across instances. Existing holistic modeling tends to overlook such regional evidence, temporal variation, and structural relations. Our DeformStruct instead models the action as a deformable structure composed of multiple action-related parts, adaptively aligning them to instance-specific temporal evidence and capturing their structural relations.
Fig. 2: Overview of the proposed DeformStruct framework. Given pre-extracted video features and point annotations, a Transformer Encoder first produces the encoded feature sequence XE . DeformStruct progressively models each action instance through three stages: Structural Anchor Mining discovers multiple action-related anchors, Deformable Part Alignment adaptively aligns latent parts with these anchors to accommodate temporal variations, and Relational Structural Organization models the relative temporal relations among the aligned parts to characterize their structural configuration. Finally, the Transformer Decoder predicts candidate action segments, and the framework is jointly optimized with localization and structural objectives.
Supervision
Method
mAP@tIoU (%)
AVG
0.3
0.4
0.5
0.6
0.7
[0.1:0.1:0.5]
[0.3:0.1:0.7]
Fully supervised
BMN [ 1 ] (ICCV’19)
56.0
47.4
38.8
29.7
20.5
–
38.5
G-TAD [ 19 ] (CVPR’20)
54.5
47.6
40.2
30.8
23.4
–
39.3
TCANet [ 62 ] (CVPR’21)
60.6
53.2
44.6
36.8
26.7
–
44.4
React [ 3 ] (ECCV’22)
69.2
65.0
57.1
47.8
35.6
–
55.0
ASL [ 63 ] (ICCV’23)
83.1
79.0
71.7
59.7
45.8
–
67.9
TABLE I: Comparison with state-of-the-art methods on THUMOS14 under different supervision settings. Results of DeformStruct are reported as mean ± standard deviation over three runs.
Fig. 3: Visualization of the Part-Anchor alignment score V . The rows represent the learned deformable parts, and the columns correspond to the candidate anchors mined from the video. The color intensity indicates the matching score, where darker colors represent higher compatibility. The red boxes highlight the optimal assignment results obtained by the Hungarian algorithm. This matrix demonstrates how our model jointly considers appearance similarity and temporal deformation to align parts with anchors.
Fig. 4: Qualitative visualization of the DPA module on THUMOS14. The colored markers indicate the locations of anchors matched with latent action parts. The bars at the bottom indicate the ground-truth segments. Notably, the mined anchors align well with the ground-truth action intervals and capture complementary action-relevant regions distributed over time.
Method
mAP@tIoU (%)
AVG
0.5
0.75
0.95
[0.5:0.05:0.95]
SF-Net [ 5 ]
–
–
–
–
Ju et al. [ 51 ]
–
–
–
–
LACP [ 6 ]
40.4
24.6
5.7
25.1
HR-Pro [ 7 ]
42.8
27.2
8.0
27.1
RCTSI [ 8 ]
–
–
–
–
TABLE II: Comparison with the state-of-the-art point-supervised methods on ActivityNet-v1.3. AVG represents the mean average precision (mAP) at thresholds [0.5:0.05:0.95] .
Method
mAP@tIoU (%)
AVG
0.3
0.5
0.7
[0.1:0.1:0.7]
SF-Net [ 5 ]
37.9
19.3
11.9
31.0
Ju et al. [ 51 ]
38.3
21.9
18.1
33.7
LACP [ 6 ]
55.7
33.9
20.8
43.5
HR-Pro [ 7 ]
61.1
37.3
17.5
47.3
RCTSI [ 8 ]
60.9
39.5
16.6
49.0
TABLE III: Comparison with state-of-the-art point-supervised methods on GTEA. AVG represents the mean average precision (mAP) at thresholds [0.1:0.1:0.7] .
Method
mAP@tIoU (%)
AVG
0.3
0.5
0.7
[0.1:0.1:0.7]
SF-Net [ 5 ]
40.6
16.7
3.5
30.9
Ju et al. [ 51 ]
46.8
20.9
5.8
34.9
LACP [ 6 ]
61.4
42.7
25.1
51.8
HR-Pro [ 7 ]
72.1
55.3
26.1
59.4
RCTSI [ 8 ]
73.0
51.1
23.8
59.6
TABLE IV: Comparison with the state-of-the-art point-supervised methods on BEOID. AVG represents the mean average precision (mAP) at thresholds [0.1:0.1:0.7] .
Fig. 5: Visualization of instance-adaptive part deformation. For each action instance, hollow and solid circles denote the matched anchor positions and the final aligned part positions, respectively, both normalized within the ground-truth action interval. Arrows indicate the deformation from anchors to aligned parts, where Δ=raligned−ranchor . Different latent parts exhibit instance-dependent shifts in both direction and magnitude.
Setting
QR
DPA
RSO
mAP@tIoU (%)
AVG
0.3
0.5
0.7
[0.3:0.1:0.7]
Pseudo-label Generation
NPE
–
–
–
4.5
1.9
0.4
2.2
FW
–
–
–
35.9
13.9
2.4
16.4
GW
–
–
–
12.2
5.4
1.3
5.9
SF-Net
–
–
–
54.3
31.4
10.7
31.6
TABLE V: Comparison of different pseudo-label generation strategies and component contributions under the same DETR architecture on THUMOS14.
Loss
Weight
mAP@tIoU (%)
AVG
0.3
0.5
0.7
[0.3:0.1:0.7]
λpart
0.1
73.7
51.9
20.4
49.2
0.5
75.2
56.1
25.5
53.4
1
76.4
57.9
28.0
54.8
2
75.5
56.0
24.0
53.3
λalign
0.1
75.1
57.3
26.0
53.9
TABLE VI: Sensitivity analysis of the loss weights on THUMOS14. We evaluate the effects of λpart , λalign , and λrel on localization performance.
Lpart
Lalign
Lrel
mAP@tIoU (%)
AVG
0.3
0.5
0.7
[0.3:0.1:0.7]
✓
✓
75.6
57.4
27.8
54.3
✓
✓
75.0
57.1
26.8
53.1
✓
✓
75.1
56.9
27.1
53.5
✓
✓
✓
76.4
57.9
28.0
54.8
TABLE VII: Ablation study of different structural consistency objectives on THUMOS14.
Fig. 6: Qualitative visualization of relational structures learned by RSO. From left to right: aligned-part coordinates and learned structural coordinates, their mean-centered configurations, and pairwise coordinate differences. These examples show that RSO preserves part ordering and approximately captures relative spacing despite absolute coordinate offsets, supporting its ability to learn relational structure.
Variant
mAP@tIoU (%)
AVG
0.3
0.5
0.7
[0.3:0.1:0.7]
DPA
75.6
57.4
27.8
54.3
+ Absolute Position
75.8
57.7
26.6
53.8
+ Distance Relation
76.1
55.2
22.6
52.2
+ Randomized Relation
74.1
54.1
21.5
51.1
+ RSO
76.4
57.9
28.0
54.8
TABLE VIII: Ablation study of different relational formulations.
Relation
Param.
MAE ↓
Order Acc. ↑
ρ↑
AVG
Pairwise Rel.
197,121
0.1294
55.7
0.201
53.5
Shared Coord.
66,049
0.1226
69.5
0.397
54.8
TABLE IX: Comparison of relational representation strategies on THUMOS14. MAE measures pairwise relation error, while Order Acc. and Spearman ρ evaluate relative temporal structure.
Semantic Consistency
Temporal Diversity
Same: 0.595 ↑
Coll.: 1.57% ↓
Diff.: 0.459 ↓
CR0.05 : 24.38% ↓
Gap: +0.136 ↑
Dist.: 0.373 ↑
TABLE X: Semantic consistency and temporal diversity of latent parts.
Anchor Category
Fraction (%)
Current action instance
90.45
Other same-class instance
5.45
Other different-class instance
1.11
Background
2.99
Target action class
95.90
Other classes / background
4.10
TABLE XI: Anchor Consistency Analysis.
Anchor
Deformation
mAP@tIoU (%)
AVG [0.3:0.1:0.7]
0.3
0.5
0.7
Mining
–
73.9
54.1
23.8
52.2
Mining
✓
75.6
57.4
27.8
54.3
Uniform
✓
68.4
50.9
21.8
47.8
Random
✓
68.3
47.1
19.1
45.4
TABLE XII: Ablation study of anchor construction and deformation modeling in DPA.
Method
Inside ↑
Pos. Disp. ↓
Gap Disp. ↓
Span Disp. ↓
DPA
87.6
0.1842
0.1425
0.1693
DPA + RSO
90.8
0.1517
0.1098
0.1326
TABLE XIII: Cross-instance consistency of GT-normalized part configurations on THUMOS14.
Fig. 7: Qualitative comparison of our proposed method with QROT [ 14 ] and HR-Pro [ 7 ] on THUMOS14 [ 10 ] . We provide two test videos with action classes of GolfSwing and HighJump, respectively. The first row of each video is the input video, whereas the following three rows show the predictions from HR-Pro, QROT and Ours. The bottom row shows the ground-truth intervals. It can be seen that our detection results show higher IoUs with the ground truth.
Method
Params.
FLOPs
Train Runtime
Memory
(M)
(G)
(ms)
(MB)
QROT
18.717
10.04
30.46
797.53
DeformStruct w/o RSO
18.930
10.04
31.10
798.04
DeformStruct
18.996
10.04
33.53
798.29
TABLE XIV: Training-time efficiency comparison.
Setting
mAP@tIoU (%)
AVG
0.3
0.5
0.7
[0.3:0.1:0.7]
Dataset-distribution
76.4
57.9
28.0
54.8
Random Point
71.1
56.7
28.3
53.4
Center-biased Point
71.5
61.6
30.7
56.5
Discriminative-frame Point
76.0
58.3
26.3
54.7
TABLE XV: Robustness analysis under different point annotation distributions. All variants use DeformStruct and differ only in the point-annotation distribution.
Point-supervised Temporal Action Localization (PTAL) adopts a lightly frame-annotated paradigm (i.e., labeling only a single frame per action instance) to train a model to effectively locate action instances within untrimmed videos. Most existing approaches design the task head of models with only a point-supervised snippet-level classification, without explicit modeling of understanding temporal relationships among frames of an action. However, understanding the temporal relationships of frames is crucial because it can help a model understand how an action is defined and therefore benefits localizing the full frames of an action. To this end, in this paper, we design a multi-task learning framework that fully utilizes point supervision to boost the model's temporal understanding capability for action localization. Specifically, we design three self-supervised temporal understanding tasks: (i) Action Completion, (ii) Action Order Understanding, and (iii) Action Regularity Understanding. These tasks help a model understand the temporal consistency of actions across videos. To the best of our knowledge, this is the first attempt to explicitly explore temporal consistency for point supervision action localization. Extensive experimental results on four benchmark datasets demonstrate the effectiveness of the proposed method compared to several state-of-the-art approaches.
Yunchuan Ma, Laiyun Qing, Guorong Li +3
University of Chinese Academy of Sciences, Beijing, 100190, China · Macquarie University
Recently, point-supervised temporal action localization has attracted increasing attention due to its low annotation cost (only one frame per action instance) and competitive detection performance. However, since point annotations are less complete than proposal annotations, existing methods mainly rely on multiple instance learning for snippet-level classification, which leads to a performance gap compared to fully supervised models. To bridge this gap, we propose P2P-Former, a novel point-to-proposal Transformer-based framework that effectively connects point-level supervision with proposal-level learning for temporal action localization. In this work, we identify two key challenges in transitioning from point-level supervision to full supervision: (1) how to generate reliable proposals as pseudo ground-truth, and (2) how to effectively integrate point-level cues into the fully supervised paradigm. To address the first challenge, we propose a Point-aware Proposal Generation (PPG) strategy that refines the confidence of proposals produced by a base locator using a point-aware importance estimation function, and then performs optimal matching between point annotations and candidate proposals to select reliable proposals as pseudo ground-truth. To tackle the second challenge, we introduce a Point-guided Semantic Enhancement (PSE) module that aggregates prototypes from point-annotated features and injects them into snippet features through Transformer-based interactions. Moreover, we design a Point-to-Proposal Constraint Loss (PPCL) that enforces each predicted proposal to include the annotated point, thereby encouraging more accurate localization. Extensive experiments on the THUMOS14 and ActivityNet-1.3 datasets demonstrate that our method outperforms several existing state-of-the-art approaches.
Yunchuan Ma, Laiyun Qing, Sai Xu +2
University of Chinese Academy of Sciences, Beijing,100190, China
Point-supervised temporal action localization (PTAL) aims to recognize and localize action instances in untrimmed videos using only point-level annotations. Existing pseudo-label-based methods have made substantial progress in PTAL. However, confidence-based initialization can overlook well-localized proposals. Meanwhile, existing methods either keep pseudo-labels fixed during detector training or update them without fully exploiting alternative boundary evidence from the original proposal pool. These limitations constrain supervision quality, thereby hindering detection performance. To learn high-quality pseudo-labels, we propose a framework that integrates prior-guided initialization with consensus-guided refinement during detector training. Our framework comprises two modules: Prior-Guided Initialization (PGI) and Consensus-Guided Refinement (CGR). PGI combines proposal confidence with point-derived priors to establish instance-specific reliability for initial pseudo-label selection and candidate retention. CGR continually uses the retained original candidates to construct boundary targets, combining their consensus with evolving detector predictions to refine pseudo-labels throughout training. Experiments on four benchmarks demonstrate that our method achieves competitive detection performance compared with existing methods. In particular, our method achieves 68.5% average mAP over tIoU thresholds 0.1--0.7 on THUMOS14, surpassing PseR by 3.7 percentage points. Code is available at https://github.com/zhenyingfang/LHQP