DeformStruct: Deformable Structure Modeling for Point-Supervised Temporal Action Localization

Mengnan Liu; Sanping Zhou; Kun Xia; Yonghao Dong; Jie Wang; Gang Hua

cs.CVOct 23, 2026

DeformStruct: Deformable Structure Modeling for Point-Supervised Temporal Action Localization

Authors: Mengnan Liu, Sanping Zhou, Kun Xia, Yonghao Dong, Jie Wang, Gang Hua

Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · School of Software Engineering, Xi’an Jiaotong University, Xi’an 710049, China · MoE Key Laboratory of Brain-Inspired Intelligent Perception and Cognition, University of Science and Technology of China, Hefei 230027, China ·

Abstract

Point-supervised Temporal Action Localization (PTAL) aims to localize action instances using only a single annotated timestamp for each instance. Such sparse supervision offers limited guidance for modeling the structural organization of an action instance. Existing PTAL methods primarily infer action regions from classification responses, leaving the structural organization of multiple action-related parts within each instance insufficiently modeled. To address this problem, we propose DeformStruct, an instance-level deformable structure modeling framework for PTAL. DeformStruct progressively constructs the structure of each action instance through three stages. First, Structural Anchor Mining discovers multiple anchors from annotated points and action-responsive regions, providing structural nodes grounded in the current instance. Second, Deformable Part Alignment associates latent structural parts with these anchors through deformation-aware matching, allowing their temporal supports to adapt to instance-dependent variations. Third, Relational Structural Organization explicitly models the relative temporal relations among the aligned parts, forming a relational representation that characterizes how different action-related parts are organized together. Extensive experiments on THUMOS14, ActivityNet-v1.3, GTEA, and BEOID demonstrate consistent improvements in point-supervised temporal localization and validate the effectiveness of deformable structural modeling.

Figures & tables

Fig. 1: Motivation of deformable structural modeling for PTAL. A complete action instance may contain multiple temporally distributed action-related stages, whose locations, durations, and relative configurations vary across instances. Existing holistic modeling tends to overlook such regional evidence, temporal variation, and structural relations. Our DeformStruct instead models the action as a deformable structure composed of multiple action-related parts, adaptively aligning them to instance-specific temporal evidence and capturing their structural relations.

Fig. 2: Overview of the proposed DeformStruct framework. Given pre-extracted video features and point annotations, a Transformer Encoder first produces the encoded feature sequence $\mathbf{X}_{E}$ . DeformStruct progressively models each action instance through three stages: Structural Anchor Mining discovers multiple action-related anchors, Deformable Part Alignment adaptively aligns latent parts with these anchors to accommodate temporal variations, and Relational Structural Organization models the relative temporal relations among the aligned parts to characterize their structural configuration. Finally, the Transformer Decoder predicts candidate action segments, and the framework is jointly optimized with localization and structural objectives.

Supervision	Method	mAP@tIoU (%)					AVG
Supervision	Method	0.3	0.4	0.5	0.6	0.7	[0.1:0.1:0.5]	[0.3:0.1:0.7]
Fully supervised	BMN [ 1 ] (ICCV’19)	56.0	47.4	38.8	29.7	20.5	–	38.5
	G-TAD [ 19 ] (CVPR’20)	54.5	47.6	40.2	30.8	23.4	–	39.3
	TCANet [ 62 ] (CVPR’21)	60.6	53.2	44.6	36.8	26.7	–	44.4
	React [ 3 ] (ECCV’22)	69.2	65.0	57.1	47.8	35.6	–	55.0
	ASL [ 63 ] (ICCV’23)	83.1	79.0	71.7	59.7	45.8	–	67.9

TABLE I: Comparison with state-of-the-art methods on THUMOS14 under different supervision settings. Results of DeformStruct are reported as mean

\pm

standard deviation over three runs.

Fig. 3: Visualization of the Part-Anchor alignment score $V$ . The rows represent the learned deformable parts, and the columns correspond to the candidate anchors mined from the video. The color intensity indicates the matching score, where darker colors represent higher compatibility. The red boxes highlight the optimal assignment results obtained by the Hungarian algorithm. This matrix demonstrates how our model jointly considers appearance similarity and temporal deformation to align parts with anchors.

Fig. 4: Qualitative visualization of the DPA module on THUMOS14. The colored markers indicate the locations of anchors matched with latent action parts. The bars at the bottom indicate the ground-truth segments. Notably, the mined anchors align well with the ground-truth action intervals and capture complementary action-relevant regions distributed over time.

Method	mAP@tIoU (%)			AVG
Method	0.5	0.75	0.95	[0.5:0.05:0.95]
SF-Net [ 5 ]	–	–	–	–
Ju et al. [ 51 ]	–	–	–	–
LACP [ 6 ]	40.4	24.6	5.7	25.1
HR-Pro [ 7 ]	42.8	27.2	8.0	27.1
RCTSI [ 8 ]	–	–	–	–

TABLE II: Comparison with the state-of-the-art point-supervised methods on ActivityNet-v1.3. AVG represents the mean average precision (mAP) at thresholds

[0.5:0.05:0.95]

Method	mAP@tIoU (%)			AVG
Method	0.3	0.5	0.7	[0.1:0.1:0.7]
SF-Net [ 5 ]	37.9	19.3	11.9	31.0
Ju et al. [ 51 ]	38.3	21.9	18.1	33.7
LACP [ 6 ]	55.7	33.9	20.8	43.5
HR-Pro [ 7 ]	61.1	37.3	17.5	47.3
RCTSI [ 8 ]	60.9	39.5	16.6	49.0

TABLE III: Comparison with state-of-the-art point-supervised methods on GTEA. AVG represents the mean average precision (mAP) at thresholds

[0.1:0.1:0.7]

Method	mAP@tIoU (%)			AVG
Method	0.3	0.5	0.7	[0.1:0.1:0.7]
SF-Net [ 5 ]	40.6	16.7	3.5	30.9
Ju et al. [ 51 ]	46.8	20.9	5.8	34.9
LACP [ 6 ]	61.4	42.7	25.1	51.8
HR-Pro [ 7 ]	72.1	55.3	26.1	59.4
RCTSI [ 8 ]	73.0	51.1	23.8	59.6

TABLE IV: Comparison with the state-of-the-art point-supervised methods on BEOID. AVG represents the mean average precision (mAP) at thresholds

[0.1:0.1:0.7]

Fig. 5: Visualization of instance-adaptive part deformation. For each action instance, hollow and solid circles denote the matched anchor positions and the final aligned part positions, respectively, both normalized within the ground-truth action interval. Arrows indicate the deformation from anchors to aligned parts, where

\Delta=r_{\mathrm{aligned}}-r_{\mathrm{anchor}}

. Different latent parts exhibit instance-dependent shifts in both direction and magnitude.

Setting	QR	DPA	RSO	mAP@tIoU (%)			AVG
Setting	QR	DPA	RSO	0.3	0.5	0.7	[0.3:0.1:0.7]
Pseudo-label Generation
NPE	–	–	–	4.5	1.9	0.4	2.2
FW	–	–	–	35.9	13.9	2.4	16.4
GW	–	–	–	12.2	5.4	1.3	5.9
SF-Net	–	–	–	54.3	31.4	10.7	31.6

TABLE V: Comparison of different pseudo-label generation strategies and component contributions under the same DETR architecture on THUMOS14.

Loss	Weight	mAP@tIoU (%)			AVG
Loss	Weight	0.3	0.5	0.7	[0.3:0.1:0.7]
$\lambda_{\text{part}}$	0.1	73.7	51.9	20.4	49.2
	0.5	75.2	56.1	25.5	53.4
	1	76.4	57.9	28.0	54.8
	2	75.5	56.0	24.0	53.3
$\lambda_{\text{align}}$	0.1	75.1	57.3	26.0	53.9

TABLE VI: Sensitivity analysis of the loss weights on THUMOS14. We evaluate the effects of

\lambda_{\text{part}}

\lambda_{\text{align}}

, and

\lambda_{\text{rel}}

on localization performance.

$\mathcal{L}_{\mathrm{part}}$	$\mathcal{L}_{\mathrm{align}}$	$\mathcal{L}_{\mathrm{rel}}$	mAP@tIoU (%)			AVG
$\mathcal{L}_{\mathrm{part}}$	$\mathcal{L}_{\mathrm{align}}$	$\mathcal{L}_{\mathrm{rel}}$	0.3	0.5	0.7	[0.3:0.1:0.7]
✓	✓		75.6	57.4	27.8	54.3
	✓	✓	75.0	57.1	26.8	53.1
✓		✓	75.1	56.9	27.1	53.5
✓	✓	✓	76.4	57.9	28.0	54.8

TABLE VII: Ablation study of different structural consistency objectives on THUMOS14.

Fig. 6: Qualitative visualization of relational structures learned by RSO. From left to right: aligned-part coordinates and learned structural coordinates, their mean-centered configurations, and pairwise coordinate differences. These examples show that RSO preserves part ordering and approximately captures relative spacing despite absolute coordinate offsets, supporting its ability to learn relational structure.

Variant	mAP@tIoU (%)			AVG
Variant	0.3	0.5	0.7	[0.3:0.1:0.7]
DPA	75.6	57.4	27.8	54.3
+ Absolute Position	75.8	57.7	26.6	53.8
+ Distance Relation	76.1	55.2	22.6	52.2
+ Randomized Relation	74.1	54.1	21.5	51.1
+ RSO	76.4	57.9	28.0	54.8

TABLE VIII: Ablation study of different relational formulations.

Relation	Param.	MAE $\downarrow$	Order Acc. $\uparrow$	$\rho$ $\uparrow$	AVG
Pairwise Rel.	197,121	0.1294	55.7	0.201	53.5
Shared Coord.	66,049	0.1226	69.5	0.397	54.8

TABLE IX: Comparison of relational representation strategies on THUMOS14. MAE measures pairwise relation error, while Order Acc. and Spearman

\rho

evaluate relative temporal structure.

Semantic Consistency	Temporal Diversity
Same: 0.595 $\uparrow$	Coll.: 1.57% $\downarrow$
Diff.: 0.459 $\downarrow$	$CR_{0.05}$ : 24.38% $\downarrow$
Gap: +0.136 $\uparrow$	Dist.: 0.373 $\uparrow$

TABLE X: Semantic consistency and temporal diversity of latent parts.

Anchor Category	Fraction (%)
Current action instance	90.45
Other same-class instance	5.45
Other different-class instance	1.11
Background	2.99
Target action class	95.90
Other classes / background	4.10

TABLE XI: Anchor Consistency Analysis.

Anchor	Deformation	mAP@tIoU (%)			AVG $[0.3{:}0.1{:}0.7]$
Anchor	Deformation	0.3	0.5	0.7	AVG $[0.3{:}0.1{:}0.7]$
Mining	–	73.9	54.1	23.8	52.2
Mining	✓	75.6	57.4	27.8	54.3
Uniform	✓	68.4	50.9	21.8	47.8
Random	✓	68.3	47.1	19.1	45.4

TABLE XII: Ablation study of anchor construction and deformation modeling in DPA.

Method	Inside $\uparrow$	Pos. Disp. $\downarrow$	Gap Disp. $\downarrow$	Span Disp. $\downarrow$
DPA	87.6	0.1842	0.1425	0.1693
DPA + RSO	90.8	0.1517	0.1098	0.1326

TABLE XIII: Cross-instance consistency of GT-normalized part configurations on THUMOS14.

Fig. 7: Qualitative comparison of our proposed method with QROT [ 14 ] and HR-Pro [ 7 ] on THUMOS14 [ 10 ] . We provide two test videos with action classes of GolfSwing and HighJump, respectively. The first row of each video is the input video, whereas the following three rows show the predictions from HR-Pro, QROT and Ours. The bottom row shows the ground-truth intervals. It can be seen that our detection results show higher IoUs with the ground truth.

Method	Params.	FLOPs	Train Runtime	Memory
	(M)	(G)	(ms)	(MB)
QROT	18.717	10.04	30.46	797.53
DeformStruct w/o RSO	18.930	10.04	31.10	798.04
DeformStruct	18.996	10.04	33.53	798.29

TABLE XIV: Training-time efficiency comparison.

Setting	mAP@tIoU (%)			AVG
Setting	0.3	0.5	0.7	[0.3:0.1:0.7]
Dataset-distribution	76.4	57.9	28.0	54.8
Random Point	71.1	56.7	28.3	53.4
Center-biased Point	71.5	61.6	30.7	56.5
Discriminative-frame Point	76.0	58.3	26.3	54.7

TABLE XV: Robustness analysis under different point annotation distributions. All variants use DeformStruct and differ only in the point-annotation distribution.

TABLE XVI: Hyperparameter sensitivity analysis.

Temporal Action Localization

Temporal action localization finds the start and end times of actions in untrimmed videos.

See all 32 papers

Exploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localization
Feb 5, 2026Yunchuan Ma, Laiyun Qing, Guorong Li +3Weakly Supervised Learning Temporal Consistency
University of Chinese Academy of Sciences, Beijing, 100190, China · Macquarie University
P2P-Former: Bridging the Supervision Gap Between Point and Proposal in Temporal Action Localization
Oct 25, 2026Yunchuan Ma, Laiyun Qing, Sai Xu +2Temporal Action Localization Temporal Video Understanding
University of Chinese Academy of Sciences, Beijing,100190, China
Learning High-Quality Pseudo-Labels for Point-Supervised Temporal Action Localization
Oct 26, 2026Zhenying FangTemporal Action Localization Temporal Video Understanding
Independent Researcher

Feb 5, 2026J/K move · Enter open · S save

Exploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localization

Yunchuan Ma, Laiyun Qing, Guorong Li, Yuqing Liu+2

University of Chinese Academy of Sciences, Beijing, 100190, China · Macquarie University

Weakly Supervised Learning Temporal Consistency Temporal Action Detection Temporal Video Understanding Temporal Action Localization

Open paper AskPDF arXiv

Point-supervised Temporal Action Localization (PTAL) adopts a lightly frame-annotated paradigm (i.e., labeling only a single frame per action instance) to train a model to effectively locate action instances within untrimmed videos. Most existing approaches design the task head of models with only a point-supervised snippet-level classification, without explicit modeling of understanding temporal relationships among frames of an action. However, understanding the temporal relationships of frames is crucial because it can help a model understand how an action is defined and therefore benefits localizing the full frames of an action. To this end, in this paper, we design a multi-task learning framework that fully utilizes point supervision to boost the model's temporal understanding capability for action localization. Specifically, we design three self-supervised temporal understanding tasks: (i) Action Completion, (ii) Action Order Understanding, and (iii) Action Regularity Understanding. These tasks help a model understand the temporal consistency of actions across videos. To the best of our knowledge, this is the first attempt to explicitly explore temporal consistency for point supervision action localization. Extensive experimental results on four benchmark datasets demonstrate the effectiveness of the proposed method compared to several state-of-the-art approaches.

Abstract

Temporal Action Localization

Explore similar work

Exploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localization

P2P-Former: Bridging the Supervision Gap Between Point and Proposal in Temporal Action Localization

Learning High-Quality Pseudo-Labels for Point-Supervised Temporal Action Localization