Learning High-Quality Pseudo-Labels for Point-Supervised Temporal Action Localization

Zhenying Fang

cs.CVOct 26, 2026

Learning High-Quality Pseudo-Labels for Point-Supervised Temporal Action Localization

Authors: Zhenying Fang

Organizations: Independent Researcher

Abstract

Point-supervised temporal action localization (PTAL) aims to recognize and localize action instances in untrimmed videos using only point-level annotations. Existing pseudo-label-based methods have made substantial progress in PTAL. However, confidence-based initialization can overlook well-localized proposals. Meanwhile, existing methods either keep pseudo-labels fixed during detector training or update them without fully exploiting alternative boundary evidence from the original proposal pool. These limitations constrain supervision quality, thereby hindering detection performance. To learn high-quality pseudo-labels, we propose a framework that integrates prior-guided initialization with consensus-guided refinement during detector training. Our framework comprises two modules: Prior-Guided Initialization (PGI) and Consensus-Guided Refinement (CGR). PGI combines proposal confidence with point-derived priors to establish instance-specific reliability for initial pseudo-label selection and candidate retention. CGR continually uses the retained original candidates to construct boundary targets, combining their consensus with evolving detector predictions to refine pseudo-labels throughout training. Experiments on four benchmarks demonstrate that our method achieves competitive detection performance compared with existing methods. In particular, our method achieves 68.5% average mAP over tIoU thresholds 0.1--0.7 on THUMOS14, surpassing PseR by 3.7 percentage points. Code is available at https://github.com/zhenyingfang/LHQP

Figures & tables

Figure 1: Proposal quality and pseudo-label evolution. (a) Mean tIoU of top-score and oracle (maximum ground-truth tIoU) proposals among same-class candidates containing each annotated point. (b) Fraction of pseudo-labels with tIoU

\geq 0.7

for fixed PGI and PGI+CGR at the final update, evaluated on the same instances. Brackets indicate differences in percentage points.

Figure 2: Overview of proposal-guided pseudo-label learning. (a) Two-stage framework: candidate proposal generation and pseudo-label initialization, followed by detector training with iterative refinement. (b) PGI combines confidence, geometric, and conflict priors to select initial pseudo-labels and retain top- $K$ candidates. (c) CGR constructs gated boundary targets from retained-proposal consensus and detector predictions, then refines pseudo-labels through consistency-controlled momentum updates.

Supervision	Method	mAP@IoU(%)							AVG (0.1:0.5)	AVG (0.3:0.7)	AVG (0.1:0.7)
Supervision	Method	0.1	0.2	0.3	0.4	0.5	0.6	0.7	AVG (0.1:0.5)	AVG (0.3:0.7)	AVG (0.1:0.7)
Frame-level	ActionFormer ECCV’22 [ 47 ]	-	-	82.1	77.8	71.0	59.4	43.9	-	66.8	-
	TriDet CVPR’23 [ 35 ]	-	-	83.6	80.1	72.9	62.4	47.4	-	69.3	-
	MDVLM-TAL NeurIPS’26 [ 37 ]	-	-	84.6	80.8	74.4	63.5	60.1	-	72.7	-
Video-level	P-MIL CVPR’23 [ 32 ]	71.8	67.5	58.9	49.0	40.0	27.1	15.1	57.4	38.0	47.0
	PivoTAL CVPR’23 [ 33 ]	74.1	69.6	61.7	52.1	42.8	30.6	16.7	60.1	40.8	49.6

Table 1: Comparison with state-of-the-art methods on THUMOS14. Results are reported as mAP (%) at different tIoU thresholds.

Supervision	Method	mAP@tIoU (%)
Supervision	Method	0.5	0.75	0.95	AVG[0.5:0.95]
Video-level	PivoTAL CVPR’23	45.1	28.2	5.0	28.1
	ISSF AAAI’24	39.4	25.8	6.4	25.8
	PVLR MM’24	43.6	27.4	6.5	27.4
	PseudoFormer CVPR’25	46.9	28.6	6.5	29.0
	CRC TPAMI’25	42.0	26.2	5.4	26.2

Table 2: Comparison with state-of-the-art methods on ActivityNet1.3. Results are reported as mAP (%) at different tIoU thresholds.

Method	mAP@tIoU (%)
Method	0.1	0.3	0.5	0.7	AVG[0.1:0.7]
SF-Net ECCV’20	58.0	37.9	19.3	11.9	31.0
Ju et al. ICCV’21	59.7	38.3	21.9	18.1	33.7
LACP ICCV’21	63.9	55.7	33.9	20.8	43.5
Yin et al. BMVC’23	63.1	52.1	37.3	22.2	45.1
HR-Pro AAAI’24	72.6	61.1	37.3	17.5	47.3

Table 3: Comparison with state-of-the-art methods on GTEA. Results are reported as mAP (%) at different tIoU thresholds.

Method	mAP@tIoU (%)
Method	0.1	0.3	0.5	0.7	AVG[0.1:0.7]
SF-Net ECCV’20	62.9	40.6	16.7	3.5	30.9
Ju et al. ICCV’21	63.2	46.8	20.9	5.8	34.9
LACP ICCV’21	76.9	61.4	42.7	25.1	51.8
Yin et al. BMVC’23	78.2	65.3	45.1	26.6	54.2
HR-Pro AAAI’24	78.5	72.1	55.3	26.1	59.4

Table 4: Comparison with state-of-the-art methods on BEOID. Results are reported as mAP (%) at different tIoU thresholds.

Figure 3: Generalization across proposal generators and detectors on THUMOS14. Baselines train matched detectors with each generator’s original pseudo-labels; Ours adds PGI and CGR. AVG mAP is evaluated over tIoU thresholds 0.3–0.7. Brackets show gains in percentage points.

PGI	CGR	THUMOS14	ActivityNet1.3	GTEA	BEOID
–	–	64.1	29.1	56.5	69.5
$\checkmark$	–	67.0	30.0	57.0	72.2
$\checkmark$	$\checkmark$	68.5	30.9	62.2	74.2

Table 5: Component ablation of PGI and CGR. Results are average mAP (%) over tIoU thresholds 0.1–0.7 for THUMOS14, GTEA, and BEOID, and 0.5–0.95 for ActivityNet1.3. A checkmark indicates an enabled module.

Figure 4: Pseudo-label refinement dynamics on THUMOS14. (a) Mean pseudo-label tIoU. (b) Fraction of pseudo-labels with tIoU

\geq 0.7

. Fixed PGI, detector-only refinement, and full CGR share the same initialization and are evaluated against official ground truth on the same instances. Epoch 0 denotes initialization; markers indicate refinement checkpoints. Final values are annotated.

Confidence	Geometric	Conflict	Mean	tIoU $\geq 0.7$
Confidence	Geometric	Conflict	tIoU	(%)
$\checkmark$	–	–	0.571	39.9
$\checkmark$	$\checkmark$	–	0.601	50.9
$\checkmark$	–	$\checkmark$	0.651	47.0
$\checkmark$	$\checkmark$	$\checkmark$	0.678	55.1

Table 6: Ablation of PGI priors on THUMOS14. Initial pseudo-label quality is measured by mean tIoU and the fraction of labels with tIoU

\geq 0.7

Consistency	$\tau_{\mathrm{con}}^{k}$	AVG mAP	Mean	tIoU $\geq 0.7$
control	$\tau_{\mathrm{con}}^{k}$	(%)	tIoU	(%)
None	$0$	64.3	0.674	56.4
Fixed	$0.5$	67.4	0.684	58.4
Progressive	$1\!\rightarrow\!0$	68.5	0.708	62.4

Table 7: Ablation of consistency control in CGR on THUMOS14. AVG mAP (%) is computed over tIoU thresholds 0.1–0.7. Pseudo-label quality is evaluated against official ground truth at the final update.

Boundary evidence	AVG mAP	Mean	tIoU $\geq 0.7$
Boundary evidence	(%)	tIoU	(%)
Detector-only	67.4	0.691	58.5
Initial boundary	68.0	0.699	61.5
Proposal consensus	68.5	0.708	62.4

Table 8: Contribution of proposal consensus on THUMOS14. AVG mAP (%) is computed over tIoU thresholds 0.1–0.7 at the final training epoch. Mean tIoU and the high-quality label fraction are evaluated against official GT at the final update. The highlighted row denotes full CGR.

Figure 5: Sensitivity to the number of retained proposals

K

on THUMOS14, measured by AVG mAP (%) over tIoU thresholds 0.1–0.7. The

K=1

result is the initial-boundary variant in Table 8 . The orange diamond marks the default

K=10

$\tau_{\mathrm{iou}}$	THUMOS14		ActivityNet1.3
$\tau_{\mathrm{iou}}$	[email protected]	AVG	[email protected]	AVG
0.1	35.5	68.1	7.0	30.8
0.3	35.1	68.5	7.1	30.9
0.5	35.5	68.5	7.2	30.9
0.7	33.7	67.8	7.0	30.7

Table 9: Sensitivity to the consensus fusion threshold

\tau_{\mathrm{iou}}

. All results are final-epoch mAP (%). AVG denotes the average over tIoU thresholds 0.1–0.7 on THUMOS14 and 0.5–0.95 on ActivityNet1.3. The highlighted row uses the default threshold of 0.5.

Figure 6: Qualitative comparison of pseudo-labels on THUMOS14 training clips. Each panel tracks two highlighted instances across top-score selection, PGI initialization, and the final CGR update. Top-score selection uses the highest-confidence same-class proposal containing each annotated point. Dashed lines mark annotated points, and numbers inside pseudo-label intervals denote tIoU with official GT. Instance-level tIoUs are rounded to one decimal place; row means are computed before rounding.

Figure 7: Final detection results on THUMOS14 test videos. GT and predicted intervals are shown for CleanAndJerk and BasketballDunk. All saved predictions with confidence $\geq 0.5$ are displayed. Numbers inside prediction intervals denote tIoU with the corresponding official GT, rounded to one decimal place.

Temporal Action Localization

Temporal action localization finds the start and end times of actions in untrimmed videos.

See all 32 papers

Exploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localization
Feb 5, 2026Yunchuan Ma, Laiyun Qing, Guorong Li +3Weakly Supervised Learning Temporal Consistency
University of Chinese Academy of Sciences, Beijing, 100190, China · Macquarie University
P2P-Former: Bridging the Supervision Gap Between Point and Proposal in Temporal Action Localization
Oct 25, 2026Yunchuan Ma, Laiyun Qing, Sai Xu +2Temporal Action Localization Temporal Video Understanding
University of Chinese Academy of Sciences, Beijing,100190, China
OnPoint: Offline-to-Online Multi-Level Distillation for Point-Supervised Online Temporal Action Localization
Jul 1, 2026Sakib Reza, Gauri Jagatap, Mohsen Moghaddam +2Knowledge Distillation Temporal Video Understanding
Northeastern University, Boston, MA 02115, USA · Georgia Institute of Technology, Atlanta, GA 30332, USA · Dolby Laboratories +1

Feb 5, 2026J/K move · Enter open · S save

Exploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localization

Yunchuan Ma, Laiyun Qing, Guorong Li, Yuqing Liu+2

University of Chinese Academy of Sciences, Beijing, 100190, China · Macquarie University

Weakly Supervised Learning Temporal Consistency Temporal Action Detection Temporal Video Understanding Temporal Action Localization

Open paper AskPDF arXiv

Point-supervised Temporal Action Localization (PTAL) adopts a lightly frame-annotated paradigm (i.e., labeling only a single frame per action instance) to train a model to effectively locate action instances within untrimmed videos. Most existing approaches design the task head of models with only a point-supervised snippet-level classification, without explicit modeling of understanding temporal relationships among frames of an action. However, understanding the temporal relationships of frames is crucial because it can help a model understand how an action is defined and therefore benefits localizing the full frames of an action. To this end, in this paper, we design a multi-task learning framework that fully utilizes point supervision to boost the model's temporal understanding capability for action localization. Specifically, we design three self-supervised temporal understanding tasks: (i) Action Completion, (ii) Action Order Understanding, and (iii) Action Regularity Understanding. These tasks help a model understand the temporal consistency of actions across videos. To the best of our knowledge, this is the first attempt to explicitly explore temporal consistency for point supervision action localization. Extensive experimental results on four benchmark datasets demonstrate the effectiveness of the proposed method compared to several state-of-the-art approaches.

Abstract

Temporal Action Localization

Explore similar work

Exploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localization

P2P-Former: Bridging the Supervision Gap Between Point and Proposal in Temporal Action Localization

OnPoint: Offline-to-Online Multi-Level Distillation for Point-Supervised Online Temporal Action Localization