P2P-Former: Bridging the Supervision Gap Between Point and Proposal in Temporal Action Localization

Yunchuan Ma; Laiyun Qing; Sai Xu; Guorong Li; Qingming Huang

cs.CVOct 25, 2026

P2P-Former: Bridging the Supervision Gap Between Point and Proposal in Temporal Action Localization

Authors: Yunchuan Ma, Laiyun Qing, Sai Xu, Guorong Li, Qingming Huang

Organizations: University of Chinese Academy of Sciences, Beijing,100190, China

Abstract

Recently, point-supervised temporal action localization has attracted increasing attention due to its low annotation cost (only one frame per action instance) and competitive detection performance. However, since point annotations are less complete than proposal annotations, existing methods mainly rely on multiple instance learning for snippet-level classification, which leads to a performance gap compared to fully supervised models. To bridge this gap, we propose P2P-Former, a novel point-to-proposal Transformer-based framework that effectively connects point-level supervision with proposal-level learning for temporal action localization. In this work, we identify two key challenges in transitioning from point-level supervision to full supervision: (1) how to generate reliable proposals as pseudo ground-truth, and (2) how to effectively integrate point-level cues into the fully supervised paradigm. To address the first challenge, we propose a Point-aware Proposal Generation (PPG) strategy that refines the confidence of proposals produced by a base locator using a point-aware importance estimation function, and then performs optimal matching between point annotations and candidate proposals to select reliable proposals as pseudo ground-truth. To tackle the second challenge, we introduce a Point-guided Semantic Enhancement (PSE) module that aggregates prototypes from point-annotated features and injects them into snippet features through Transformer-based interactions. Moreover, we design a Point-to-Proposal Constraint Loss (PPCL) that enforces each predicted proposal to include the annotated point, thereby encouraging more accurate localization. Extensive experiments on the THUMOS14 and ActivityNet-1.3 datasets demonstrate that our method outperforms several existing state-of-the-art approaches.

Figures & tables

Figure 1: Comparison. (a) Existing methods rely on sparse point-level annotations for classification and derive proposals by thresholding class activation sequences, without explicitly learning temporal localization. (b) P2P-Former discovers high-quality action instances as pseudo labels, enabling joint classification and boundary regression for more complete action localization.

Figure 2: Overview of the proposed P2P-Former framework. A base action locator first generates coarse proposals, which are then evaluated using point-aware scoring and matched with point annotations via the Hungarian algorithm to obtain reliable pseudo labels. Point-labeled features are clustered to construct category prototypes, which are injected into a prototype-guided semantic enhancement block to enrich video tokens. The enhanced tokens are processed by a Transformer encoder and FPN, followed by classification and regression heads for temporal action localization. The model is trained with $\mathcal{L}_{cls}$ , $\mathcal{L}_{reg}$ , and the point-to-proposal constraint $\mathcal{L}_{p2p}$ .

Supervision	Method	mAP@IoU (%)							AVG	AVG	AVG
Supervision	Method	0.1	0.2	0.3	0.4	0.5	0.6	0.7	(0.1:0.5)	(0.3:0.7)	(0.1:0.7)
Fully supervised	TCANet [ 25 ] (CVPR’21)	-	-	60.6	53.2	44.6	36.8	26.7	-	44.3	-
	GCM [ 34 ] (TPAMI’21)	72.5	70.9	66.5	60.8	51.9	-	-	64.5	-	-
	VSGN [ 37 ] (ICCV’21)	-	-	66.7	60.4	52.4	41.0	30.4	-	50.2	-
	AFSD [ 14 ] (CVPR’21)	-	-	67.3	62.4	55.5	43.7	31.1	-	52.0	-
Weakly supervised	P-MIL [ 26 ] (CVPR’23)	71.8	67.5	58.9	49.0	40.0	27.1	15.1	57.4	38.0	47.0

Table 1: Comparison with the state-of-the-art point-supervised methods on THUMOS14. We further compare our method with both fully-supervised and weakly-supervised approaches. For fairness, we adopt the same point annotations as used in [ 11 ] .

Method	mAP@IoU (%)			AVG
Method	0.5	0.75	0.95	AVG
LACP [ 11 ] (ICCV’21)	40.4	24.6	5.7	25.1
CRRC-Net [ 5 ] (TIP’22)	39.8	24.1	5.9	24.0
HR-Pro [ 36 ] (AAAI’24)	42.8	27.2	8.0	27.1
SMBD [ 18 ] (ECCV’24)	43.2	27.8	8.3	27.5
SNPR [ 12 ] (TIP’24)	41.3	30.9	4.8	26.5

Table 2: Comparison with existing point-supervised methods on ActivityNet-1.3.

Row Index	Setup				AVG mAP@IoU(%)
Row Index	Baseline	PPG	PSE	PPCL	[0.1:0.5]	[0.3:0.7]	[0.1:0.7]
1	✓	✗	✗	✗	73.2	54.7	62.6
2	✓	✓	✗	✗	74.4	56.6	64.1
3	✓	✓	✓	✗	75.3	56.9	64.6
4	✓	✓	✗	✓	74.6	57.3	64.7
5	✓	✓	✓	✓	76.0	58.6	65.8

Table 3: Ablation results for the three proposed components on THUMOS14.

Method	mAP@IoU(%)
Method	0.1	0.3	0.5	0.7	AVG
LACP ∗	75.3	64.3	45.1	20.6	52.4
LACP + Ours	86.7	76.7	59.3	31.5	64.7
HR-Pro ${}_{1}^{*}$	82.0	69.4	48.2	21.9	56.2
HR-Pro 1 + Ours	85.2	78.9	61.4	32.2	65.8

Table 4: Performance of P2P-Former using different PTAL methods as base locators. HR-Pro 1 denotes the snippet-level learning model of HR-Pro, and ∗ indicates our reproduced results.

$L$	mAP@IoU(%)				AVG
$L$	0.1	0.3	0.5	0.7	[0.1:0.7]
$1$	85.1	78.2	61.0	31.9	65.4
$2$	85.2	78.9	61.4	32.2	65.8
$3$	85.0	78.3	60.8	31.9	65.3
$4$	84.7	78.0	60.3	31.7	65.1

Table 5: Analysis of the number of clusters constructed for each category.

$\lambda$	mAP@IoU(%)				AVG
$\lambda$	0.1	0.3	0.5	0.7	[0.1:0.7]
$0.5$	85.1	78.3	60.8	31.9	65.3
$1$	85.2	78.9	61.4	32.2	65.8
$1.5$	85.2	78.7	61.1	31.6	65.5
$2$	84.8	77.8	60.2	31.2	65.0

Table 6: Analysis of the balance coefficient

\lambda

in the loss function.

Method	mAP (%)	Params (M)	FLOPs (G)	Time (ms)
SF-Net ∗	40.2	16.8	1.7	1.0
LACP ∗	52.4	12.6	1.3	1.2
HR-Pro ∗	60.2	49.4	8.2	14.1
P2P-Former	65.8	46.0	84.4	49.3

Table 7: Performance and efficiency comparison on THUMOS14 with a fixed input feature length of

T

=100. ∗ denotes our reproduced results.

Figure 3: Qualitative comparison between our proposed method, HR-Pro [ 36 ] , and LACP [ 11 ] on ActivityNet-1.3. We provide four cases of temporal action localization from different action classes, including “River tubing”, “Applying sunscreen”, “Paintball”, and “Scuba diving”. Prediction errors are highlighted with red dashed boxes. The dark blue region represents the overlap between different predicted instances. The IoUs between our detection results and the ground truths are notably higher.

Figure 4: False Positive Profiling on THUMOS14 using the diagnostic tool proposed by [ 1 ] . The left panel shows the analysis of the base locator results, whereas the right panel presents the analysis of our P2P-Former.

Figure 5: Visualization of the optimal matching between labeled points and candidate proposals. The successfully matched proposals are selected as pseudo ground-truth, while unmatched proposals are discarded. Ground-truth intervals are provided only for reference.

Figure 6: Visualization of the clustering results for six randomly selected categories from the THUMOS14 dataset. The point-annotated features of each category are grouped into two clusters, yielding two representative sub-prototypes per category.

Temporal Action Localization

Temporal action localization finds the start and end times of actions in untrimmed videos.

See all 32 papers

Feb 5, 2026

Exploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localization

Point-supervised Temporal Action Localization (PTAL) adopts a lightly frame-annotated paradigm (i.e., labeling only a single frame per action instance) to train a model to effectively locate action instances within untrimmed videos. Most existing approaches design the task head of models with only a point-supervised snippet-level classification, without explicit modeling of understanding temporal relationships among frames of an action. However, understanding the temporal relationships of frames is crucial because it can help a model understand how an action is defined and therefore benefits localizing the full frames of an action. To this end, in this paper, we design a multi-task learning framework that fully utilizes point supervision to boost the model's temporal understanding capability for action localization. Specifically, we design three self-supervised temporal understanding tasks: (i) Action Completion, (ii) Action Order Understanding, and (iii) Action Regularity Understanding. These tasks help a model understand the temporal consistency of actions across videos. To the best of our knowledge, this is the first attempt to explicitly explore temporal consistency for point supervision action localization. Extensive experimental results on four benchmark datasets demonstrate the effectiveness of the proposed method compared to several state-of-the-art approaches.

Yunchuan Ma, Laiyun Qing, Guorong Li +3

University of Chinese Academy of Sciences, Beijing, 100190, China · Macquarie University

Weakly Supervised Learning Temporal Consistency Temporal Action Detection Temporal Video Understanding Temporal Action Localization

Ask PDF

Oct 26, 2026

Learning High-Quality Pseudo-Labels for Point-Supervised Temporal Action Localization

Point-supervised temporal action localization (PTAL) aims to recognize and localize action instances in untrimmed videos using only point-level annotations. Existing pseudo-label-based methods have made substantial progress in PTAL. However, confidence-based initialization can overlook well-localized proposals. Meanwhile, existing methods either keep pseudo-labels fixed during detector training or update them without fully exploiting alternative boundary evidence from the original proposal pool. These limitations constrain supervision quality, thereby hindering detection performance. To learn high-quality pseudo-labels, we propose a framework that integrates prior-guided initialization with consensus-guided refinement during detector training. Our framework comprises two modules: Prior-Guided Initialization (PGI) and Consensus-Guided Refinement (CGR). PGI combines proposal confidence with point-derived priors to establish instance-specific reliability for initial pseudo-label selection and candidate retention. CGR continually uses the retained original candidates to construct boundary targets, combining their consensus with evolving detector predictions to refine pseudo-labels throughout training. Experiments on four benchmarks demonstrate that our method achieves competitive detection performance compared with existing methods. In particular, our method achieves 68.5% average mAP over tIoU thresholds 0.1--0.7 on THUMOS14, surpassing PseR by 3.7 percentage points. Code is available at https://github.com/zhenyingfang/LHQP

Zhenying Fang

Independent Researcher

Temporal Action Localization Temporal Video Understanding Pseudo-Labeling

Ask PDF

Oct 23, 2026

DeformStruct: Deformable Structure Modeling for Point-Supervised Temporal Action Localization

Point-supervised Temporal Action Localization (PTAL) aims to localize action instances using only a single annotated timestamp for each instance. Such sparse supervision offers limited guidance for modeling the structural organization of an action instance. Existing PTAL methods primarily infer action regions from classification responses, leaving the structural organization of multiple action-related parts within each instance insufficiently modeled. To address this problem, we propose DeformStruct, an instance-level deformable structure modeling framework for PTAL. DeformStruct progressively constructs the structure of each action instance through three stages. First, Structural Anchor Mining discovers multiple anchors from annotated points and action-responsive regions, providing structural nodes grounded in the current instance. Second, Deformable Part Alignment associates latent structural parts with these anchors through deformation-aware matching, allowing their temporal supports to adapt to instance-dependent variations. Third, Relational Structural Organization explicitly models the relative temporal relations among the aligned parts, forming a relational representation that characterizes how different action-related parts are organized together. Extensive experiments on THUMOS14, ActivityNet-v1.3, GTEA, and BEOID demonstrate consistent improvements in point-supervised temporal localization and validate the effectiveness of deformable structural modeling.

Mengnan Liu, Sanping Zhou, Kun Xia +3

State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · School of Software Engineering, Xi’an Jiaotong University, Xi’an 710049, China +2

Temporal Action Localization Temporal Video Understanding

Ask PDF

Abstract

Temporal Action Localization

Explore similar work

Exploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localization

Learning High-Quality Pseudo-Labels for Point-Supervised Temporal Action Localization

DeformStruct: Deformable Structure Modeling for Point-Supervised Temporal Action Localization