Spectral Scalpel: Amplifying Adjacent Action Discrepancy via Frequency-Selective Filtering for Skeleton-Based Action Segmentation
Authors: Haoyu Ji, Bowen Chen, Zhihao Yang, Wenze Huang, Yu Gao, Xueting Liu, Weihong Ren, Zhiyong Wang, Honghai Liu
Organizations: Harbin Institute of Technology, Shenzhen · Shenzhen H&T Intelligent Control Co., Ltd. · Southern University of Science and Technology · Southeast University
Skeleton-based Temporal Action Segmentation (STAS) seeks to densely segment and classify diverse actions within long, untrimmed skeletal motion sequences. However, existing STAS methodologies face challenges of limited inter-class discriminability and blurred segmentation boundaries, primarily due to insufficient distinction of spatio-temporal patterns between adjacent actions. To address these limitations, we propose Spectral Scalpel, a frequency-selective filtering framework aimed at suppressing shared frequency components between adjacent distinct actions while amplifying their action-specific frequencies, thereby enhancing inter-action discrepancies and sharpening transition boundaries. Specifically, Spectral Scalpel employs adaptive multi-scale spectral filters as scalpels to edit frequency spectra, coupled with a discrepancy loss between adjacent actions serving as the surgical objective. This design amplifies representational disparities between neighboring actions, effectively mitigating boundary localization ambiguities and inter-class confusion. Furthermore, complementing long-term temporal modeling, we introduce a frequency-aware channel mixer to strengthen channel evolution by aggregating spectra across channels. This work presents a novel paradigm for STAS that extends conventional spatio-temporal modeling by incorporating frequency-domain analysis. Extensive experiments on five public datasets demonstrate that Spectral Scalpel achieves state-of-the-art performance. Code is available at https://github.com/HaoyuJi/SpecScalpel.
Figures & tables
Figure 1: Effect of a designed frequency filter in amplifying adjacent action discrepancy. The synthetic single-joint motion sequence consists of two concatenated action segments, each idealized as a sum of cosine waves with both shared and action-specific frequency components. After FFT, a designed filter amplifies action-specific frequencies and suppresses shared ones. The filtered sequence shows enhanced action discriminability and a clearer boundary.
Figure 2: Overview of Spectral Scalpel. After spatial modeling, joint features are transformed into the frequency domain. A discrepancy-guided filter, optimized by adjacent action discrepancy loss, enhances action-specific spectra and suppresses shared ones, improving discriminability and boundary clarity.
Figure 3: Overview of the Spectral Scalpel framework. It comprises four sequential stages: (1) Spatial Modeling with multi-scale, dual-branch dynamic GCNs; (2) Frequency Modeling via the Multi-scale Adaptive Spectral Filter (MASF); (3) Temporal Modeling through Linear Transformers, Frequency-Aware Channel Mixers (FACM), and adaptive fusion; (4) Prediction Refinement, which generates and refines class and boundary predictions. The model is optimized by supervised losses computed from filtered features, final representations, classification predictions, and boundary predictions.
Figure 4: Illustration of the three core components. (a) Multi-scale Adaptive Spectral Filter (MASF) enhances frequency discriminability using multi-head filters at multiple scales, combined with dual-branch dynamic-static channel-wise fusion. (b) Adjacent Action Discrepancy Loss (AADL) guides frequency dynamics by maximizing amplitude spectrum discrepancies between adjacent segments. (c) Frequency-Aware Channel Mixer (FACM) models frequency-domain channel interactions via real-imaginary decomposition and shared point-wise convolutions.
Dataset
PKU-MMD v2 (X-view)
PKU-MMD v2 (X-sub)
Metric
FLOPs ↓
Param. ↓
Acc ↑
Edit ↑
F1@{10,25,50} ↑
Acc ↑
Edit ↑
F1@{10,25,50} ↑
VTAS
MS-TCN [ 15 ]
3.27G
0.54M
58.2
56.6
58.6
53.6
39.4
65.5
-
-
-
46.3
MS-TCN++ [ 33 ]
5.37G
0.90M
59.1
59.4
61.6
56.1
42.0
66.0
66.7
69.6
65.1
51.5
ASFormer [ 58 ]
6.34G
1.04M
64.8
62.6
65.1
60.0
45.8
68.3
68.1
71.9
68.2
54.5
ASRF [ 24 ]
7.15G
1.20M
60.4
59.3
62.5
58.0
46.1
67.7
67.1
72.1
68.3
56.8
LTContext [ 1 ]
3.71G
0.56M
62.3
59.6
62.9
57.8
42.6
68.8
68.2
72.1
68.5
55.4
Table 1: Comparison with the latest results on PKU-MMD v2 (X-view) and PKU-MMD v2 (X-sub). Bold and underline indicate the best and second-best results in each column. FLOPs denote the computational complexity of the model when the input is X∈R12×6000×25 on the PKU-MMD dataset.
Dataset
MCFS-130
TCG-15
LARa
Metric
Acc ↑
Edit ↑
F1@{10, 25, 50} ↑
Acc ↑
Edit ↑
F1@{10, 25, 50} ↑
Acc ↑
Edit ↑
F1@{10, 25, 50} ↑
VTAS
MS-TCN [ 15 ]
65.7
54.5
56.4
52.2
42.5
85.5
67.7
70.0
67.3
59.1
65.8
-
-
-
39.6
MS-TCN++ [ 33 ]
65.5
59.8
60.1
55.7
46.1
86.0
71.0
74.6
71.7
62.9
71.7
58.6
60.1
58.6
47.0
ASFormer [ 58 ]
67.5
69.1
68.3
64.0
55.1
82.3
69.1
71.3
67.7
57.6
72.2
62.2
66.1
61.9
49.2
ASRF [ 24 ]
65.6
65.6
66.7
62.3
51.9
83.9
65.3
71.5
69.9
62.4
71.9
63.0
68.3
65.3
53.2
LTContext [ 1 ]
72.2
74.2
74.6
70.0
59.6
86.9
75.3
79.1
75.9
67.4
71.4
62.5
68.1
64.3
52.1
Table 2: Comparison with the latest results on MCFS-130, TCG-15, and LARa. Bold and underline indicate the best and second-best results in each column.
Figure 5: Qualitative results of action segmentation on the PKU-MMD v2 (X-sub, X-view), LARa and MCFS-130 datasets. Different colors represent distinct action classes. Red boxes highlight segmentation errors in other methods compared to Spectral Scalpel.
Figure 6: Visualization of the representation space on the PKU-MMD v2 datasets. Each point represents a feature of an action segment, which is colored according to its class label. As observed, compared to others, Spectral Scalpel engenders a more distinctly structured semantic feature space.
Components
Acc
Edit
F1@{10, 25, 50}
Baseline
73.6
73.0
78.2
74.6
64.3
Baseline+MASF
74.5
73.7
78.7
75.4
65.6
Baseline+AADL
74.8
73.9
79.0
75.5
65.5
Baseline+FACM
74.0
73.2
78.0
74.8
65.7
Baseline+MASF+AADL
75.1
73.9
79.3
76.3
66.0
Baseline+MASF+FACM
74.8
74.1
79.5
76.5
66.2
Table 3: Ablation study of MASF, AADL, and FACM modules on PKU-MMD (X-sub) dataset.
Figure 7: Comparison of unfiltered (top) and filtered (bottom) frame-wise average features. X-axis denotes time, y-axis shows per-frame mean value, and colors indicate different actions.
Dataset
PKU-MMD v2 (X-sub)
PKU-MMD v2 (X-view)
MCFS-130
LARa
Metric
SC ↑
CH ↑
DB ↓
SC ↑
CH ↑
DB ↓
SC ↑
CH ↑
DB ↓
SC ↑
CH ↑
DB ↓
DeST [ 34 ]
0.19
63.5
1.59
0.08
62.6
2.07
0.01
36.6
2.22
0.14
722.4
1.74
LaSA [ 27 ]
0.25
79.7
1.31
0.16
78.3
1.62
0.05
42.9
2.07
0.24
1245.1
1.18
SpecScalpel (Ours)
0.30
88.6
1.24
0.29
128.9
1.26
0.07
49.3
1.94
0.25
1189.7
1.12
Table 4: Comparison of Mean Clustering Metrics (SC, CH, DB) on the PKU-MMD v2 (X-sub, X-view), LARa and MCFS-130 datasets. SC denotes the Silhouette Coefficient Index, CH is the Calinski–Harabasz Index, and DB indicates the Davies–Bouldin Index.
Figure 8: Per-class frame-wise F1 score differences between our SpecScalpel and the state-of-the-art LaSA method on the PKU-MMD v2 (X-sub and X-view), LARa, and MCFS-130 datasets. Blue bars represent action classes where SpecScalpel achieves higher F1 scores, whereas green bars correspond to minor declines. The results indicate that SpecScalpel consistently enhances performance across the majority of action categories, with only limited exceptions.
Noise
0.3x Gaussian Noise
0.5x Gaussian Noise
30% Joint Occlusion
Metric
Acc
Edit
F1@{10, 25, 50}
Acc
Edit
F1@{10, 25, 50}
Acc
Edit
F1@{10, 25, 50}
DeST
66.8
65.2
71.0
67.4
54.3
60.5
58.6
64.1
59.7
45.4
67.3
66.5
71.7
67.8
53.5
-4.94%
-5.95%
-4.64%
-5.12%
-7.48%
-13.98%
-15.44%
-13.93%
-15.93%
-22.59%
-4.24%
-4.11%
-3.71%
-4.57%
-8.80%
LaSA
71.3
71.5
77.3
72.9
60.8
65.5
64.7
69.9
65.3
51.6
70.2
70.4
75.4
71.5
58.5
-3.00%
-2.60%
-1.29%
-2.54%
-4.40%
-10.91%
-11.92%
-10.67%
-12.70%
-18.84%
-4.50%
-4.11%
-3.73%
-4.41%
-8.01%
SpecScalpel
74.7
74.1
79.3
76.2
66.0
73.9
73.2
78.3
75.6
64.8
72.7
72.0
77.4
74.2
62.3
Table 5: Robustness evaluation under random Gaussian noise and joint occlusion on PKU-MMD v2 (X-sub) dataset. For each method, the first row reports the absolute performance, and the second row indicates the relative performance drop compared with the clean setting.
Noisy Labels
Acc
Edit
F1@{10, 25, 50}
DeST
65.9
61.8
68.5
63.2
48.9
-6.21%
-10.76%
-8.09%
-10.94%
-16.78%
LaSA
68.1
65.1
70.8
65.7
51.2
-7.37%
-11.37%
-9.64%
-12.17%
-19.47%
Baseline
67.1
64.9
71.4
65.3
51.3
-8.78%
-11.04%
-8.71%
-12.40%
-20.21%
Table 6: Robustness evaluation against noisy training boundaries on the PKU-MMD v2 (X-sub) dataset. For each method, the first row reports the absolute performance under random boundary jitter ( δ∼U(−25,25) frames), and the second row indicates the relative performance drop compared to the clean training setting.
Temporal Rescaling
0.7x Temporal Rescaling
1.5x Temporal Rescaling
Metric
Acc
Edit
F1@{10, 25, 50}
Acc
Edit
F1@{10, 25, 50}
DeST
69.6
66.1
73.1
70.3
57.6
62.5
61.0
65.4
60.6
48.1
-1.00%
-4.67%
-1.88%
-0.97%
-1.83%
-11.16%
-11.98%
-12.20%
-14.68%
-18.11%
LaSA
72.9
71.3
77.2
74.2
62.4
64.3
64.3
67.9
63.4
50.3
-0.82%
-2.86%
-1.40%
-0.80%
-1.89%
-12.52%
-12.47%
-13.24%
-15.24%
-20.88%
Baseline
72.6
71.0
77.0
73.3
62.2
64.0
64.0
67.2
61.8
50.4
Table 7: Robustness evaluation against temporal rescaling on the PKU-MMD v2 (X-sub) dataset. Models trained on the original temporal scale are evaluated on sequences rescaled by 0.7x and 1.5x. For each method, the first row reports the absolute performance, and the second row indicates the relative performance drop compared to the clean setting.
Position of Frequency Module
Acc
Edit
F1@{10, 25, 50}
At the Very Beginning
74.8
74.3
78.8
75.9
65.6
After Spatial Modeling
75.4
74.5
79.7
76.8
66.6
After Spatial Modeling
70.6
70.0
75.4
71.9
60.6
Table 8: Ablation study on the position of the frequency module on PKU-MMD v2 (X-sub) dataset.
FFT or Not
Acc
Edit
F1@{10, 25, 50}
Both use FFT
75.4
74.5
79.7
76.8
66.6
MASF w/o FFT
75.1
73.4
79.1
76.2
65.5
AADL w/o FFT
74.7
73.6
78.9
75.9
65.4
Both w/o FFT
74.2
73.2
78.5
75.4
64.9
Table 9: Ablation study on the necessity of frequency-domain operations on PKU-MMD v2 (X-sub) dataset.
FFT or Not
Acc ↑
Edit ↑
F1@{10, 25, 50} ↑
FLOPs ↓
Param. ↓
Filter-Shared + Channel-Weight
75.4
74.5
79.7
76.8
66.6
0.01G
0.006M
Filter-Shared + Temporal-Weight
74.7
74.2
78.7
76.0
65.7
0.045G
0.010M
Filter-Shared + Both Weights
75.2
74.3
78.9
76.0
66.0
0.045G
0.261M
Filter-Specific + Temporal-Weight
75.5
74.7
79.5
76.8
66.6
0.045G
0.337M
Filter-Specific + Both Weights
75.6
74.9
79.5
77.0
66.8
0.045G
0.588M
Table 10: Ablation study on filter dimension and weighting strategy in MASF on PKU-MMD v2 (X-sub) dataset.
Dynamic
Static
Average
Acc
Edit
F1@{10, 25, 50}
✓
✗
✗
75.2
74.0
79.4
76.5
66.2
✗
✓
✗
74.6
73.9
78.8
75.9
66.4
✗
✗
✓
75.2
74.2
79.4
76.5
66.3
✓
✗
✓
75.4
74.2
78.6
76.2
66.3
✗
✓
✓
75.3
74.4
79.5
76.4
66.4
✓
✓
✗
75.4
74.5
79.7
76.8
66.6
Table 11: Effect of weighted fusion methods on multi-scale filtered features on PKU-MMD v2 (X-sub) dataset.
Feature Domain
Weight Generator
Acc
Edit
F1@{10, 25, 50}
Temporal
SE Block
74.8
73.4
78.4
75.6
65.6
Temporal
Equation (3)
75.4
74.5
79.7
76.8
66.6
Frequency
SE Block
74.8
74.3
78.6
75.5
65.6
Frequency
Equation (3)
75.3
74.2
79.8
76.8
66.4
Table 12: Effect of feature domain and weight generation in dynamic weighting on PKU-MMD v2 (X-sub) dataset.
Weight Granularity
Initialization
Acc
Edit
F1@{10, 25, 50}
(M, C)
Random
75.4
74.6
79.6
76.6
66.6
(M, C)
Zero
75.4
74.5
79.7
76.8
66.6
(M, 1)
Random
75.4
74.3
79.6
76.3
66.3
(M, 1)
Zero
75.3
74.4
79.6
76.4
66.4
Table 13: Effect of granularity and initialization in static weighting on PKU-MMD v2 (X-sub) dataset.
Fusion Method
Acc
Edit
F1@{10, 25, 50}
Average Fusion
75.4
74.5
79.7
76.8
66.6
Learnable Scalar Weight
75.4
74.2
79.6
76.7
66.3
Channel-wise Weight
75.4
74.4
79.3
76.3
66.4
Channel Concat + Conv
75.5
74.6
79.7
76.5
66.7
Table 14: Effect of fusion strategy for dynamic and static branch features on PKU-MMD v2 (X-sub) dataset.
Number of filter heads
Acc
Edit
F1@{10, 25, 50}
2
75.4
74.4
79.5
76.6
66.4
4
75.4
74.5
79.7
76.8
66.6
6
75.7
74.7
80.0
77.2
66.7
8
75.8
74.9
80.1
77.4
66.7
Table 15: Effect of different numbers of filter heads M on PKU-MMD v2 (X-sub) dataset.
Maximum Filter Length
Acc
Edit
F1@{10, 25, 50}
32
75.3
74.2
79.5
76.7
66.5
64
75.4
74.5
79.7
76.8
66.6
128
75.7
74.3
79.8
76.8
66.8
256
75.8
74.7
79.8
77.0
66.9
Table 16: Impact of maximum filter length Rmax in spectral filtering on PKU-MMD v2 (X-sub) dataset.
Multi-scale strategies
Acc
Edit
F1@{10, 25, 50}
Single-scale
75.3
74.5
79.5
76.6
66.4
Equation (2)
75.4
74.5
79.7
76.8
66.6
Rm=2m+b
75.4
74.6
79.6
76.7
66.5
Table 17: Comparison of multi-scale strategies with fixed maximum filter length 64 on PKU-MMD v2 (X-sub) dataset.
Discrepancy Calculation Scope
Acc
Edit
F1@{10, 25, 50}
AADL
75.4
74.5
79.7
76.8
66.6
IADL
75.5
74.6
79.7
76.9
66.6
BADL
75.7
74.5
79.9
77.1
66.7
Table 18: Impact of action discrepancy scope in adjacent action discrepancy loss on PKU-MMD v2 (X-sub) dataset. AADL is Adjacent Action Discrepancy Loss, IADL is Instance-scope Action Discrepancy Loss, and BADL is Batch-scope Action Discrepancy Loss.
Spectral Representation
Acc
Edit
F1@{10, 25, 50}
Amplitude Only
75.4
74.5
79.7
76.8
66.6
Complex
75.5
74.4
79.6
76.8
66.6
Table 19: Ablation on spectral representations (Amplitude vs. Complex) for computing the AADL on the PKU-MMD v2 (X-sub) dataset.
Discrepancy Metric
Loss Form
Acc
Edit
F1@{10, 25, 50}
Normalized L1
-log(tanh( ⋅ ))
75.4
74.5
79.7
76.8
66.6
Normalized L2
-log(tanh( ⋅ ))
75.2
74.0
79.2
76.2
66.3
L-infinity
-log(tanh( ⋅ ))
74.8
73.4
78.6
75.4
64.8
Normalized L1
-log(B-sigmoid( ⋅ ))
75.2
74.3
79.6
76.4
66.5
Normalized L1
1 - tanh( ⋅ )
75.5
74.3
79.4
76.5
66.4
Normalized L1
1 - B-sigmoid( ⋅ )
75.3
74.4
79.6
76.7
66.6
Table 20: Effect of different AADL computation strategies on PKU-MMD v2 (X-sub) dataset.
Sampling Strategies
Acc
Edit
F1@{10, 25, 50}
Linear Interpolation
75.4
74.5
79.7
76.8
66.6
Nearest Neighbor
75.6
73.6
79.4
76.3
65.9
AdaptiveAvgPool
75.3
74.3
79.7
76.7
66.3
Table 21: Impact of the sampling strategies for action spectral features on PKU-MMD v2 (X-sub) dataset.
Sampling Length
Acc
Edit
F1@{10, 25, 50}
16
75.3
73.6
78.9
76.1
66.2
32
75.4
74.5
79.7
76.8
66.6
48
75.5
74.6
79.8
76.6
66.5
64
75.6
74.0
79.1
76.5
66.6
Table 22: Effect of fixed sampling length Sf on action spectral features on PKU-MMD v2 (X-sub) dataset.
Scaling Factor α
Acc
Edit
F1@{10, 25, 50}
50
75.0
74.6
79.3
76.1
66.4
100
75.4
74.5
79.7
76.8
66.6
200
75.3
73.9
79.2
76.3
66.0
400
75.0
74.3
79.6
76.7
65.9
Table 23: Impact of discrepancy scaling factor α in adjacent action discrepancy loss on PKU-MMD v2 (X-sub) dataset.
Weight λ3
Acc
Edit
F1@{10, 25, 50}
0.1
75.4
74.4
79.6
76.7
66.3
1.0
75.4
74.5
79.7
76.8
66.6
5.0
75.5
74.5
79.6
76.7
66.7
10.0
75.3
74.6
79.7
76.8
66.6
Table 24: Effects of the weight λ3 for adjacent action discrepancy loss on PKU-MMD v2 (X-sub) dataset.
FFT
PConv
Depth
Acc
Edit
F1@{10, 25, 50}
✗
✗
-
75.1
73.9
79.3
76.3
66.0
✓
✗
-
75.2
74.2
79.7
76.6
66.3
✗
✓
2
75.1
74.1
79.3
76.4
66.1
✓
✓
1
75.3
74.4
79.4
76.2
66.4
✓
✓
2
75.4
74.5
79.7
76.8
66.6
✓
✓
3
75.5
74.5
79.7
76.7
66.7
Table 25: Impact of frequency-aware channel mixer and stacking depth on PKU-MMD v2 (X-sub) dataset.
Conv Strategy
Acc
Edit
F1@{10, 25, 50}
Real-only
74.5
73.5
78.7
75.9
64.5
Imaginary-only
74.7
73.5
77.9
74.7
63.5
Shared Weights
75.4
74.5
79.7
76.8
66.6
Independent Weights
74.8
74.4
78.9
76.1
65.8
Table 26: Ablation on complex feature processing in frequency-aware channel mixer on PKU-MMD v2 (X-sub) dataset.
Post-Processing
Acc
Edit
F1@{10, 25, 50}
Identity
75.4
74.5
79.7
76.8
66.6
ReLU
74.2
73.4
78.6
75.5
64.9
BatchNorm
75.1
74.3
78.9
76.3
65.4
Dropout(0.3)
75.6
74.5
79.6
76.5
66.5
Table 27: Ablation on post-processing operations for frequency-aware mixed features on PKU-MMD v2 (X-sub) dataset.
Kernel Size
Acc
Edit
F1@{10, 25, 50}
1
75.4
74.5
79.7
76.8
66.6
3
74.5
73.9
78.1
75.3
64.6
5
73.8
74.2
78.5
75.7
64.6
7
73.8
74.4
78.9
75.4
64.5
Table 28: Impact of kernel size choices in frequency-aware channel mixer on PKU-MMD v2 (X-sub) dataset.
Normalization Mode
Acc
Edit
F1@{10, 25, 50}
Forward (normalize by 1/S )
75.5
74.6
79.8
76.7
66.6
Backward (no normalization)
75.4
74.5
79.7
76.8
66.6
Ortho (normalize by 1/S )
75.3
74.6
79.5
76.4
66.5
Table 29: Comparing FFT normalization strategies in frequency-aware channel mixer on PKU-MMD v2 (X-sub) dataset.
block counts
Acc
Edit
F1@{10, 25, 50}
6
72.1
71.0
76.7
73.4
62.1
8
74.2
73.2
78.5
75.1
64.5
10
75.4
74.5
79.7
76.8
66.6
12
75.7
75.1
80.0
77.0
66.6
14
76.0
75.0
79.6
77.1
66.8
Table 30: Ablation on temporal block depth in temporal modeling on PKU-MMD v2 (X-sub) dataset.
Class stages
Boundary stages
Acc
Edit
F1@{10, 25, 50}
1
1
75.7
74.1
78.2
75.3
66.2
1
2
75.4
74.5
79.7
76.8
66.6
2
2
75.3
74.5
79.6
76.9
66.7
2
3
74.8
74.0
79.6
75.9
66.0
3
3
74.3
74.5
79.5
76.2
65.3
Table 31: Ablation on the stage numbers of classification/boundary refinement branches on PKU-MMD v2 (X-sub) dataset.
Skeleton-based Temporal Action Segmentation (STAS) aims to densely parse untrimmed skeletal sequences into frame-level action categories. However, existing methods, while proficient at capturing spatio-temporal kinematics, neglect the underlying physical dynamics that govern human motion. This oversight limits inter-class discriminability between actions with similar kinematics but distinct dynamic intents, and hinders precise boundary localization where dynamic force profiles shift. To address these, we propose the Lagrangian-Dynamic Informed Network (LaDy), a framework integrating principles of Lagrangian dynamics into the segmentation process. Specifically, LaDy first computes generalized coordinates from joint positions and then estimates Lagrangian terms under physical constraints to explicitly synthesize the generalized forces. To further ensure physical coherence, our Energy Consistency Loss enforces the work-energy theorem, aligning kinetic energy change with the work done by the net force. The learned dynamics then drive a Spatio-Temporal Modulation module: Spatially, generalized forces are fused with spatial representations to provide more discriminative semantics. Temporally, salient dynamic signals are constructed for temporal gating, thereby significantly enhancing boundary awareness. Experiments on challenging datasets show that LaDy achieves state-of-the-art performance, validating the integration of physical dynamics for action segmentation. Code is available at https://github.com/HaoyuJi/LaDy.
Haoyu Ji, Xueting Liu, Yu Gao +5
Harbin Institute of Technology, Shenzhen · Southern University of Science and Technology · Southeast University
We propose a novel hierarchical spatiotemporal vector quantization framework for unsupervised skeleton-based temporal action segmentation. We first introduce a hierarchical approach, which includes two consecutive levels of vector quantization. Specifically, the lower level associates skeletons with fine-grained subactions, while the higher level further aggregates subactions into action-level representations. Our hierarchical approach outperforms the non-hierarchical baseline, while primarily exploiting spatial cues by reconstructing input skeletons. Next, we extend our approach by leveraging both spatial and temporal information, yielding a hierarchical spatiotemporal vector quantization scheme. In particular, our hierarchical spatiotemporal approach performs multi-level clustering, while simultaneously recovering input skeletons and their corresponding timestamps. Lastly, extensive experiments on multiple benchmarks, including HuGaDB, LARa, and BABEL, demonstrate that our approach establishes a new state-of-the-art performance and reduces segment length bias in unsupervised skeleton-based temporal action segmentation.
Umer Ahmed, Syed Ahmed Mahmood, Fawad Javed Fateh +3
Unsupervised skeleton-based temporal action segmentation is a crucial task for understanding human behavior in long untrimmed sequences. Recent approaches often rely on discrete quantization to discover action boundaries from motion representations. However, when spatial masking is introduced for representation learning, it can introduce representation ambiguity, while discrete quantization further amplifies small fluctuations in the latent space. The interaction between these two factors often leads to unstable code switching and severe temporal jitter near action boundaries.To address these limitations, we propose a novel Mask-aware Action Spatiotemporal Quantization (MASQ) framework. Our framework decouples the conflicting tasks of spatial feature inference and temporal smoothing.In the spatial dimension, we introduce a Joint-Level Structured Dropout (JLSD) mechanism that masks the entire temporal trajectory of selected joints, to encourage the model to learn discriminative inter-joint coordination patterns. In the temporal dimension, we design a mask-aware velocity loss that enforces motion consistency only on visible joints, that prevents gradient conflicts caused by masked signals and stabilizing temporal predictions. Extensive experiments on three widely used skeleton datasets, including HuGaDB, LARa, and BABEL, demonstrate that the proposed MASQ framework significantly outperforms existing state-of-the-art unsupervised methods. In particular, our model establishes a comprehensive and substantial leading advantage in the Mean over Frames accuracy.
Xinyao Qin, Linxiang Peng, Youbao Ye +2
University of Science and Technology of China, China