P2P-Former: Bridging the Supervision Gap Between Point and Proposal in Temporal Action Localization

Yunchuan Ma; Laiyun Qing; Sai Xu; Guorong Li; Qingming Huang

cs.CVOct 25, 2026

P2P-Former: Bridging the Supervision Gap Between Point and Proposal in Temporal Action Localization

Authors: Yunchuan Ma, Laiyun Qing, Sai Xu, Guorong Li, Qingming Huang

Organizations: University of Chinese Academy of Sciences, Beijing,100190, China

Abstract

Recently, point-supervised temporal action localization has attracted increasing attention due to its low annotation cost (only one frame per action instance) and competitive detection performance. However, since point annotations are less complete than proposal annotations, existing methods mainly rely on multiple instance learning for snippet-level classification, which leads to a performance gap compared to fully supervised models. To bridge this gap, we propose P2P-Former, a novel point-to-proposal Transformer-based framework that effectively connects point-level supervision with proposal-level learning for temporal action localization. In this work, we identify two key challenges in transitioning from point-level supervision to full supervision: (1) how to generate reliable proposals as pseudo ground-truth, and (2) how to effectively integrate point-level cues into the fully supervised paradigm. To address the first challenge, we propose a Point-aware Proposal Generation (PPG) strategy that refines the confidence of proposals produced by a base locator using a point-aware importance estimation function, and then performs optimal matching between point annotations and candidate proposals to select reliable proposals as pseudo ground-truth. To tackle the second challenge, we introduce a Point-guided Semantic Enhancement (PSE) module that aggregates prototypes from point-annotated features and injects them into snippet features through Transformer-based interactions. Moreover, we design a Point-to-Proposal Constraint Loss (PPCL) that enforces each predicted proposal to include the annotated point, thereby encouraging more accurate localization. Extensive experiments on the THUMOS14 and ActivityNet-1.3 datasets demonstrate that our method outperforms several existing state-of-the-art approaches.

Figures & tables

Figure 1: Comparison. (a) Existing methods rely on sparse point-level annotations for classification and derive proposals by thresholding class activation sequences, without explicitly learning temporal localization. (b) P2P-Former discovers high-quality action instances as pseudo labels, enabling joint classification and boundary regression for more complete action localization.

Figure 2: Overview of the proposed P2P-Former framework. A base action locator first generates coarse proposals, which are then evaluated using point-aware scoring and matched with point annotations via the Hungarian algorithm to obtain reliable pseudo labels. Point-labeled features are clustered to construct category prototypes, which are injected into a prototype-guided semantic enhancement block to enrich video tokens. The enhanced tokens are processed by a Transformer encoder and FPN, followed by classification and regression heads for temporal action localization. The model is trained with $\mathcal{L}_{cls}$ , $\mathcal{L}_{reg}$ , and the point-to-proposal constraint $\mathcal{L}_{p2p}$ .

Supervision	Method	mAP@IoU (%)							AVG	AVG	AVG
Supervision	Method	0.1	0.2	0.3	0.4	0.5	0.6	0.7	(0.1:0.5)	(0.3:0.7)	(0.1:0.7)
Fully supervised	TCANet [ 25 ] (CVPR’21)	-	-	60.6	53.2	44.6	36.8	26.7	-	44.3	-
	GCM [ 34 ] (TPAMI’21)	72.5	70.9	66.5	60.8	51.9	-	-	64.5	-	-
	VSGN [ 37 ] (ICCV’21)	-	-	66.7	60.4	52.4	41.0	30.4	-	50.2	-
	AFSD [ 14 ] (CVPR’21)	-	-	67.3	62.4	55.5	43.7	31.1	-	52.0	-
Weakly supervised	P-MIL [ 26 ] (CVPR’23)	71.8	67.5	58.9	49.0	40.0	27.1	15.1	57.4	38.0	47.0

Table 1: Comparison with the state-of-the-art point-supervised methods on THUMOS14. We further compare our method with both fully-supervised and weakly-supervised approaches. For fairness, we adopt the same point annotations as used in [ 11 ] .

Method	mAP@IoU (%)			AVG
Method	0.5	0.75	0.95	AVG
LACP [ 11 ] (ICCV’21)	40.4	24.6	5.7	25.1
CRRC-Net [ 5 ] (TIP’22)	39.8	24.1	5.9	24.0
HR-Pro [ 36 ] (AAAI’24)	42.8	27.2	8.0	27.1
SMBD [ 18 ] (ECCV’24)	43.2	27.8	8.3	27.5
SNPR [ 12 ] (TIP’24)	41.3	30.9	4.8	26.5

Table 2: Comparison with existing point-supervised methods on ActivityNet-1.3.

Row Index	Setup				AVG mAP@IoU(%)
Row Index	Baseline	PPG	PSE	PPCL	[0.1:0.5]	[0.3:0.7]	[0.1:0.7]
1	✓	✗	✗	✗	73.2	54.7	62.6
2	✓	✓	✗	✗	74.4	56.6	64.1
3	✓	✓	✓	✗	75.3	56.9	64.6
4	✓	✓	✗	✓	74.6	57.3	64.7
5	✓	✓	✓	✓	76.0	58.6	65.8

Table 3: Ablation results for the three proposed components on THUMOS14.

Method	mAP@IoU(%)
Method	0.1	0.3	0.5	0.7	AVG
LACP ∗	75.3	64.3	45.1	20.6	52.4
LACP + Ours	86.7	76.7	59.3	31.5	64.7
HR-Pro ${}_{1}^{*}$	82.0	69.4	48.2	21.9	56.2
HR-Pro 1 + Ours	85.2	78.9	61.4	32.2	65.8

Table 4: Performance of P2P-Former using different PTAL methods as base locators. HR-Pro 1 denotes the snippet-level learning model of HR-Pro, and ∗ indicates our reproduced results.

$L$	mAP@IoU(%)				AVG
$L$	0.1	0.3	0.5	0.7	[0.1:0.7]
$1$	85.1	78.2	61.0	31.9	65.4
$2$	85.2	78.9	61.4	32.2	65.8
$3$	85.0	78.3	60.8	31.9	65.3
$4$	84.7	78.0	60.3	31.7	65.1

Table 5: Analysis of the number of clusters constructed for each category.

$\lambda$	mAP@IoU(%)				AVG
$\lambda$	0.1	0.3	0.5	0.7	[0.1:0.7]
$0.5$	85.1	78.3	60.8	31.9	65.3
$1$	85.2	78.9	61.4	32.2	65.8
$1.5$	85.2	78.7	61.1	31.6	65.5
$2$	84.8	77.8	60.2	31.2	65.0

Table 6: Analysis of the balance coefficient

\lambda

in the loss function.

Method	mAP (%)	Params (M)	FLOPs (G)	Time (ms)
SF-Net ∗	40.2	16.8	1.7	1.0
LACP ∗	52.4	12.6	1.3	1.2
HR-Pro ∗	60.2	49.4	8.2	14.1
P2P-Former	65.8	46.0	84.4	49.3

Table 7: Performance and efficiency comparison on THUMOS14 with a fixed input feature length of

T

=100. ∗ denotes our reproduced results.

Figure 3: Qualitative comparison between our proposed method, HR-Pro [ 36 ] , and LACP [ 11 ] on ActivityNet-1.3. We provide four cases of temporal action localization from different action classes, including “River tubing”, “Applying sunscreen”, “Paintball”, and “Scuba diving”. Prediction errors are highlighted with red dashed boxes. The dark blue region represents the overlap between different predicted instances. The IoUs between our detection results and the ground truths are notably higher.

Figure 4: False Positive Profiling on THUMOS14 using the diagnostic tool proposed by [ 1 ] . The left panel shows the analysis of the base locator results, whereas the right panel presents the analysis of our P2P-Former.

Figure 5: Visualization of the optimal matching between labeled points and candidate proposals. The successfully matched proposals are selected as pseudo ground-truth, while unmatched proposals are discarded. Ground-truth intervals are provided only for reference.

Figure 6: Visualization of the clustering results for six randomly selected categories from the THUMOS14 dataset. The point-annotated features of each category are grouped into two clusters, yielding two representative sub-prototypes per category.

Temporal Action Localization

Temporal action localization finds the start and end times of actions in untrimmed videos.

See all 32 papers

Exploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localization
Feb 5, 2026Yunchuan Ma, Laiyun Qing, Guorong Li +3Weakly Supervised Learning Temporal Consistency
University of Chinese Academy of Sciences, Beijing, 100190, China · Macquarie University
Learning High-Quality Pseudo-Labels for Point-Supervised Temporal Action Localization
Oct 26, 2026Zhenying FangTemporal Action Localization Temporal Video Understanding
Independent Researcher
DeformStruct: Deformable Structure Modeling for Point-Supervised Temporal Action Localization
Oct 23, 2026Mengnan Liu, Sanping Zhou, Kun Xia +3Temporal Action Localization Temporal Video Understanding
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · School of Software Engineering, Xi’an Jiaotong University, Xi’an 710049, China +2

Feb 5, 2026J/K move · Enter open · S save

Exploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localization

Yunchuan Ma, Laiyun Qing, Guorong Li, Yuqing Liu+2

University of Chinese Academy of Sciences, Beijing, 100190, China · Macquarie University

Weakly Supervised Learning Temporal Consistency Temporal Action Detection Temporal Video Understanding Temporal Action Localization

Open paper AskPDF arXiv

Point-supervised Temporal Action Localization (PTAL) adopts a lightly frame-annotated paradigm (i.e., labeling only a single frame per action instance) to train a model to effectively locate action instances within untrimmed videos. Most existing approaches design the task head of models with only a point-supervised snippet-level classification, without explicit modeling of understanding temporal relationships among frames of an action. However, understanding the temporal relationships of frames is crucial because it can help a model understand how an action is defined and therefore benefits localizing the full frames of an action. To this end, in this paper, we design a multi-task learning framework that fully utilizes point supervision to boost the model's temporal understanding capability for action localization. Specifically, we design three self-supervised temporal understanding tasks: (i) Action Completion, (ii) Action Order Understanding, and (iii) Action Regularity Understanding. These tasks help a model understand the temporal consistency of actions across videos. To the best of our knowledge, this is the first attempt to explicitly explore temporal consistency for point supervision action localization. Extensive experimental results on four benchmark datasets demonstrate the effectiveness of the proposed method compared to several state-of-the-art approaches.

Abstract

Temporal Action Localization

Explore similar work

Exploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localization

Learning High-Quality Pseudo-Labels for Point-Supervised Temporal Action Localization

DeformStruct: Deformable Structure Modeling for Point-Supervised Temporal Action Localization