Point-Supervised Skeleton-Based Human Action Segmentation

Hongsong Wang; Yiqin Shen; Pengbo Yan; Jie Gui; Yang Zhang

cs.CVMar 6, 2026

Point-Supervised Skeleton-Based Human Action Segmentation

Authors: Hongsong Wang, Yiqin Shen, Pengbo Yan, Jie Gui, Yang Zhang

Organizations: School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · School of Cyber Science and Engineering, Southeast University, Nanjing 210096, China · Southeast University - Monash University Joint Graduate School, SuZhou 215123, China · School of Computer Science and Software Engineering, Shenzhen University, Shenzhen 518060, China

Abstract

Skeleton-based temporal action segmentation is a fundamental yet challenging task, playing a crucial role in enabling intelligent systems to perceive and respond to human activities. While fully-supervised methods achieve satisfactory performance, they require costly frame-level annotations and are sensitive to ambiguous action boundaries. To address these issues, we introduce a point-supervised framework for skeleton-based action segmentation, where only a single frame per action segment is labeled. We leverage multimodal skeleton data, including joint, bone, and motion information, encoded via a pretrained unified model to extract rich feature representations. To generate reliable pseudo-labels, we propose a novel prototype similarity method and integrate it with two existing methods: energy function and constrained K-Medoids clustering. Multimodal pseudo-label integration is proposed to enhance the reliability of the pseudo-label and guide the model training. We establish new benchmarks on PKU-MMD (X-Sub and X-View), MCFS-22, and MCFS-130, and implement baselines for point-supervised skeleton-based human action segmentation. Extensive experiments show that our method achieves competitive performance, even surpassing some fully-supervised methods while significantly reducing annotation effort.

Figures & tables

Figure 1: Comparison between point-supervised annotation and fully-supervised annotation for skeleton-based action segmentation.

Figure 2: Overall framework of point-supervised skeleton-based temporal action segmentation via multimodal pseudo-label generation and integration.

Supervision	Dataset	PKU-MMD (X-sub)					PKU-MMD (X-view)
Supervision	Metric	Acc	Edit	F1@10	F1@25	F1@50	Acc	Edit	F1@10	F1@25	F1@50
Fully- Supervised	MS-TCN [ 4 ]	65.5	-	-	-	46.3	58.2	56.6	58.6	53.6	39.4
	MS-GCN [ 5 ]	68.5	-	-	-	51.6	65.3	58.1	61.3	56.7	44.1
	CTC [ 33 ]	69.2	-	69.9	66.4	53.8	-	-	-	-	-
	DeST-TCN [ 14 ]	67.6	66.3	71.7	68.0	55.5	62.4	58.2	63.2	59.2	47.6
	DeST-Former [ 14 ]	70.3	69.3	74.5	71.0	58.7	67.3	64.7	69.3	65.6	52.0

Table 1: Results of skeleton-based temporal action segmentation on the PKU-MMD.

{\dagger}

denotes the method with results reproduced by ours.

Supervision	Dataset	MCFS-22					MCFS-130
Supervision	Metric	Acc	Edit	F1@10	F1@25	F1@50	Acc	Edit	F1@10	F1@25	F1@50
Fully- Supervised	MS-TCN [ 4 ]	75.6	74.2	74.3	69.7	59.5	65.7	54.5	56.4	52.2	42.5
	MS-GCN [ 5 ]	75.5	72.6	75.7	70.5	57.9	64.9	52.6	52.4	48.8	39.1
	DeST-TCN [ 14 ]	78.7	82.3	86.6	83.5	73.2	70.5	73.8	74.0	70.7	61.8
	DeST-Former [ 14 ]	80.4	85.2	87.4	84.5	75.0	71.4	75.8	75.8	72.2	63.0
	LaSA [ 9 ]	80.8	86.7	89.3	86.2	76.3	72.6	79.3	79.3	75.8	66.6

Table 2: Skeleton-based temporal action segmentation on the MCFS-22 and MCFS-130.

Dataset	PKU-MMD (X-sub)					PKU-MMD (X-view)					MCFS-22					MCFS-130
Method	Acc	Edit	F1@10	F1@25	F1@50	Acc	Edit	F1@10	F1@25	F1@50	Acc	Edit	F1@10	F1@25	F1@50	Acc	Edit	F1@10	F1@25	F1@50
MS-TCN [ 4 ]	61.6	67.0	69.6	62.5	44.2	67.1	68.8	73.5	67.2	49.5	69.1	67.5	67.0	61.4	46.5	59.1	53.2	53.8	48.9	36.9
MS-TCN++ [ 12 ]	61.7	65.6	67.8	61.1	44.2	66.3	67.3	71.5	65.2	48.0	69.1	67.5	67.0	61.4	46.5	59.1	53.2	53.8	48.9	36.9

Table 3: The influence of segmentation models for point-supervised skeleton-based action segmentation on four benchmarks.

Table 6

Prototype Similarity	K-Medoids Clustering	Energy Function	Acc	Edit	F1@10	F1@25	F1@50
J+JF	J+JF	J+JF	62.2	65.7	69.0	61.6	43.7
B+BF	B+BF	B+BF	62.8	66.1	69.3	62.7	44.7
M+MF	M+MF	M+MF	61.3	64.8	67.5	61.5	42.7
J+JF	B+BF	M+MF	61.6	67.0	69.6	62.5	44.2

Table 6: Influence of multimodal input on pseudo-label generation methods.

Figure 3: Comparison of results of different pseudo-label generation methods on the PKU-MMD (X-sub) dataset.

Figure 4: Comparison of action segmentation results of different point-supervised methods on the PKU-MMD (X-sub) dataset.

Temporal Action Segmentation

Temporal action segmentation labels when actions begin and end in continuous video, enabling fine-grained analysis of human and robotic activity.

See all 49 papers

Generative Data Augmentation for Skeleton Action Recognition
Apr 16, 2026Xu Dong, Wanqing Li, Anthony Adeyemi-Ejeye +1Skeleton-Based Action Recognition Synthetic Data Augmentation
Innovative Media Lab, University of Surrey, Guildford, UK · Advanced Multimedia Research Lab, University of Wollongong, Wollongong, Australia
Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantization
Apr 16, 2026Umer Ahmed, Syed Ahmed Mahmood, Fawad Javed Fateh +3Temporal Action Segmentation Vector Quantization
Retrocausal, Inc. Redmond, WA
LaDy: Lagrangian-Dynamic Informed Network for Skeleton-based Action Segmentation via Spatial-Temporal Modulation
Mar 25, 2026Haoyu Ji, Xueting Liu, Yu Gao +5Temporal Action Segmentation Boundary-Aware Segmentation
Harbin Institute of Technology, Shenzhen · Southern University of Science and Technology · Southeast University

Apr 16, 2026J/K move · Enter open · S save

Generative Data Augmentation for Skeleton Action Recognition

Xu Dong, Wanqing Li, Anthony Adeyemi-Ejeye, Andrew Gilbert

Innovative Media Lab, University of Surrey, Guildford, UK · Advanced Multimedia Research Lab, University of Wollongong, Wollongong, Australia

Skeleton-Based Action Recognition Synthetic Data Augmentation Human Activity Recognition Human Motion Generation

Open paper AskPDF arXiv

Skeleton-based human action recognition is a powerful approach for understanding human behaviour from pose data, but collecting large-scale, diverse, and well-annotated 3D skeleton datasets is both expensive and labor-intensive. To address this challenge, we propose a conditional generative pipeline for data augmentation in skeleton action recognition. Our method learns the distribution of real skeleton sequences under the constraint of action labels, enabling the synthesis of diverse and high-fidelity data. Even with limited training samples, it can effectively generate skeleton sequences and achieve competitive recognition performance in low-data scenarios, demonstrating strong generalisation in downstream tasks. Specifically, we introduce a Transformer-based encoder-decoder architecture, combined with a generative refinement module and a dropout mechanism, to balance fidelity and diversity during sampling. Experiments on HumanAct12 and the refined NTU-RGBD (NTU-VIBE) dataset show that our approach consistently improves the accuracy of multiple skeleton-based action recognition models, validating its effectiveness in both few-shot and full-data settings. The source code can be found at here.

Abstract

Temporal Action Segmentation

Explore similar work

Generative Data Augmentation for Skeleton Action Recognition

Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantization

LaDy: Lagrangian-Dynamic Informed Network for Skeleton-based Action Segmentation via Spatial-Temporal Modulation