Spectral Scalpel: Amplifying Adjacent Action Discrepancy via Frequency-Selective Filtering for Skeleton-Based Action Segmentation

Haoyu Ji; Bowen Chen; Zhihao Yang; Wenze Huang; Yu Gao; Xueting Liu; Weihong Ren; Zhiyong Wang

cs.CVMar 25, 2026

Spectral Scalpel: Amplifying Adjacent Action Discrepancy via Frequency-Selective Filtering for Skeleton-Based Action Segmentation

Authors: Haoyu Ji, Bowen Chen, Zhihao Yang, Wenze Huang, Yu Gao, Xueting Liu, Weihong Ren, Zhiyong Wang,

Organizations: Harbin Institute of Technology, Shenzhen · Shenzhen H&T Intelligent Control Co., Ltd. · Southern University of Science and Technology · Southeast University

Abstract

Skeleton-based Temporal Action Segmentation (STAS) seeks to densely segment and classify diverse actions within long, untrimmed skeletal motion sequences. However, existing STAS methodologies face challenges of limited inter-class discriminability and blurred segmentation boundaries, primarily due to insufficient distinction of spatio-temporal patterns between adjacent actions. To address these limitations, we propose Spectral Scalpel, a frequency-selective filtering framework aimed at suppressing shared frequency components between adjacent distinct actions while amplifying their action-specific frequencies, thereby enhancing inter-action discrepancies and sharpening transition boundaries. Specifically, Spectral Scalpel employs adaptive multi-scale spectral filters as scalpels to edit frequency spectra, coupled with a discrepancy loss between adjacent actions serving as the surgical objective. This design amplifies representational disparities between neighboring actions, effectively mitigating boundary localization ambiguities and inter-class confusion. Furthermore, complementing long-term temporal modeling, we introduce a frequency-aware channel mixer to strengthen channel evolution by aggregating spectra across channels. This work presents a novel paradigm for STAS that extends conventional spatio-temporal modeling by incorporating frequency-domain analysis. Extensive experiments on five public datasets demonstrate that Spectral Scalpel achieves state-of-the-art performance. Code is available at https://github.com/HaoyuJi/SpecScalpel.

Figures & tables

Figure 1: Effect of a designed frequency filter in amplifying adjacent action discrepancy. The synthetic single-joint motion sequence consists of two concatenated action segments, each idealized as a sum of cosine waves with both shared and action-specific frequency components. After FFT, a designed filter amplifies action-specific frequencies and suppresses shared ones. The filtered sequence shows enhanced action discriminability and a clearer boundary.

Figure 2: Overview of Spectral Scalpel. After spatial modeling, joint features are transformed into the frequency domain. A discrepancy-guided filter, optimized by adjacent action discrepancy loss, enhances action-specific spectra and suppresses shared ones, improving discriminability and boundary clarity.

Figure 3: Overview of the Spectral Scalpel framework. It comprises four sequential stages: (1) Spatial Modeling with multi-scale, dual-branch dynamic GCNs; (2) Frequency Modeling via the Multi-scale Adaptive Spectral Filter (MASF); (3) Temporal Modeling through Linear Transformers, Frequency-Aware Channel Mixers (FACM), and adaptive fusion; (4) Prediction Refinement, which generates and refines class and boundary predictions. The model is optimized by supervised losses computed from filtered features, final representations, classification predictions, and boundary predictions.

Figure 4: Illustration of the three core components. (a) Multi-scale Adaptive Spectral Filter (MASF) enhances frequency discriminability using multi-head filters at multiple scales, combined with dual-branch dynamic-static channel-wise fusion. (b) Adjacent Action Discrepancy Loss (AADL) guides frequency dynamics by maximizing amplitude spectrum discrepancies between adjacent segments. (c) Frequency-Aware Channel Mixer (FACM) models frequency-domain channel interactions via real-imaginary decomposition and shared point-wise convolutions.

	Dataset			PKU-MMD v2 (X-view)					PKU-MMD v2 (X-sub)
	Metric	FLOPs $\downarrow$	Param. $\downarrow$	Acc $\uparrow$	Edit $\uparrow$	F1@{10,25,50} $\uparrow$			Acc $\uparrow$	Edit $\uparrow$	F1@{10,25,50} $\uparrow$
VTAS	MS-TCN [ 15 ]	3.27G	0.54M	58.2	56.6	58.6	53.6	39.4	65.5	-	-	-	46.3
	MS-TCN++ [ 33 ]	5.37G	0.90M	59.1	59.4	61.6	56.1	42.0	66.0	66.7	69.6	65.1	51.5
	ASFormer [ 58 ]	6.34G	1.04M	64.8	62.6	65.1	60.0	45.8	68.3	68.1	71.9	68.2	54.5
	ASRF [ 24 ]	7.15G	1.20M	60.4	59.3	62.5	58.0	46.1	67.7	67.1	72.1	68.3	56.8
	LTContext [ 1 ]	3.71G	0.56M	62.3	59.6	62.9	57.8	42.6	68.8	68.2	72.1	68.5	55.4

Table 1: Comparison with the latest results on PKU-MMD v2 (X-view) and PKU-MMD v2 (X-sub). Bold and underline indicate the best and second-best results in each column. FLOPs denote the computational complexity of the model when the input is

X\in\mathbb{R}^{12\times 6000\times 25}

on the PKU-MMD dataset.

	Dataset	MCFS-130					TCG-15					LARa
	Metric	Acc $\uparrow$	Edit $\uparrow$	F1@{10, 25, 50} $\uparrow$			Acc $\uparrow$	Edit $\uparrow$	F1@{10, 25, 50} $\uparrow$			Acc $\uparrow$	Edit $\uparrow$	F1@{10, 25, 50} $\uparrow$
VTAS	MS-TCN [ 15 ]	65.7	54.5	56.4	52.2	42.5	85.5	67.7	70.0	67.3	59.1	65.8	-	-	-	39.6
	MS-TCN++ [ 33 ]	65.5	59.8	60.1	55.7	46.1	86.0	71.0	74.6	71.7	62.9	71.7	58.6	60.1	58.6	47.0
	ASFormer [ 58 ]	67.5	69.1	68.3	64.0	55.1	82.3	69.1	71.3	67.7	57.6	72.2	62.2	66.1	61.9	49.2
	ASRF [ 24 ]	65.6	65.6	66.7	62.3	51.9	83.9	65.3	71.5	69.9	62.4	71.9	63.0	68.3	65.3	53.2
	LTContext [ 1 ]	72.2	74.2	74.6	70.0	59.6	86.9	75.3	79.1	75.9	67.4	71.4	62.5	68.1	64.3	52.1

Table 2: Comparison with the latest results on MCFS-130, TCG-15, and LARa. Bold and underline indicate the best and second-best results in each column.

Figure 5: Qualitative results of action segmentation on the PKU-MMD v2 (X-sub, X-view), LARa and MCFS-130 datasets. Different colors represent distinct action classes. Red boxes highlight segmentation errors in other methods compared to Spectral Scalpel.

Figure 6: Visualization of the representation space on the PKU-MMD v2 datasets. Each point represents a feature of an action segment, which is colored according to its class label. As observed, compared to others, Spectral Scalpel engenders a more distinctly structured semantic feature space.

Components	Acc	Edit	F1@{10, 25, 50}
Baseline	73.6	73.0	78.2	74.6	64.3
Baseline+MASF	74.5	73.7	78.7	75.4	65.6
Baseline+AADL	74.8	73.9	79.0	75.5	65.5
Baseline+FACM	74.0	73.2	78.0	74.8	65.7
Baseline+MASF+AADL	75.1	73.9	79.3	76.3	66.0
Baseline+MASF+FACM	74.8	74.1	79.5	76.5	66.2

Table 3: Ablation study of MASF, AADL, and FACM modules on PKU-MMD (X-sub) dataset.

Figure 7: Comparison of unfiltered (top) and filtered (bottom) frame-wise average features. X-axis denotes time, y-axis shows per-frame mean value, and colors indicate different actions.

Dataset	PKU-MMD v2 (X-sub)			PKU-MMD v2 (X-view)			MCFS-130			LARa
Metric	SC $\uparrow$	CH $\uparrow$	DB $\downarrow$	SC $\uparrow$	CH $\uparrow$	DB $\downarrow$	SC $\uparrow$	CH $\uparrow$	DB $\downarrow$	SC $\uparrow$	CH $\uparrow$	DB $\downarrow$
DeST [ 34 ]	0.19	63.5	1.59	0.08	62.6	2.07	0.01	36.6	2.22	0.14	722.4	1.74
LaSA [ 27 ]	0.25	79.7	1.31	0.16	78.3	1.62	0.05	42.9	2.07	0.24	1245.1	1.18
SpecScalpel (Ours)	0.30	88.6	1.24	0.29	128.9	1.26	0.07	49.3	1.94	0.25	1189.7	1.12

Table 4: Comparison of Mean Clustering Metrics (SC, CH, DB) on the PKU-MMD v2 (X-sub, X-view), LARa and MCFS-130 datasets. SC denotes the Silhouette Coefficient Index, CH is the Calinski–Harabasz Index, and DB indicates the Davies–Bouldin Index.

Figure 8: Per-class frame-wise F1 score differences between our SpecScalpel and the state-of-the-art LaSA method on the PKU-MMD v2 (X-sub and X-view), LARa, and MCFS-130 datasets. Blue bars represent action classes where SpecScalpel achieves higher F1 scores, whereas green bars correspond to minor declines. The results indicate that SpecScalpel consistently enhances performance across the majority of action categories, with only limited exceptions.

Noise	0.3x Gaussian Noise					0.5x Gaussian Noise					30% Joint Occlusion
Metric	Acc	Edit	F1@{10, 25, 50}			Acc	Edit	F1@{10, 25, 50}			Acc	Edit	F1@{10, 25, 50}
DeST	66.8	65.2	71.0	67.4	54.3	60.5	58.6	64.1	59.7	45.4	67.3	66.5	71.7	67.8	53.5
	-4.94%	-5.95%	-4.64%	-5.12%	-7.48%	-13.98%	-15.44%	-13.93%	-15.93%	-22.59%	-4.24%	-4.11%	-3.71%	-4.57%	-8.80%
LaSA	71.3	71.5	77.3	72.9	60.8	65.5	64.7	69.9	65.3	51.6	70.2	70.4	75.4	71.5	58.5
	-3.00%	-2.60%	-1.29%	-2.54%	-4.40%	-10.91%	-11.92%	-10.67%	-12.70%	-18.84%	-4.50%	-4.11%	-3.73%	-4.41%	-8.01%
SpecScalpel	74.7	74.1	79.3	76.2	66.0	73.9	73.2	78.3	75.6	64.8	72.7	72.0	77.4	74.2	62.3

Table 5: Robustness evaluation under random Gaussian noise and joint occlusion on PKU-MMD v2 (X-sub) dataset. For each method, the first row reports the absolute performance, and the second row indicates the relative performance drop compared with the clean setting.

Noisy Labels	Acc	Edit	F1@{10, 25, 50}
DeST	65.9	61.8	68.5	63.2	48.9
	-6.21%	-10.76%	-8.09%	-10.94%	-16.78%
LaSA	68.1	65.1	70.8	65.7	51.2
	-7.37%	-11.37%	-9.64%	-12.17%	-19.47%
Baseline	67.1	64.9	71.4	65.3	51.3
	-8.78%	-11.04%	-8.71%	-12.40%	-20.21%

Table 6: Robustness evaluation against noisy training boundaries on the PKU-MMD v2 (X-sub) dataset. For each method, the first row reports the absolute performance under random boundary jitter (

\delta\sim\mathcal{U}(-25,25)

frames), and the second row indicates the relative performance drop compared to the clean training setting.

Temporal Rescaling	0.7x Temporal Rescaling					1.5x Temporal Rescaling
Metric	Acc	Edit	F1@{10, 25, 50}			Acc	Edit	F1@{10, 25, 50}
DeST	69.6	66.1	73.1	70.3	57.6	62.5	61.0	65.4	60.6	48.1
	-1.00%	-4.67%	-1.88%	-0.97%	-1.83%	-11.16%	-11.98%	-12.20%	-14.68%	-18.11%
LaSA	72.9	71.3	77.2	74.2	62.4	64.3	64.3	67.9	63.4	50.3
	-0.82%	-2.86%	-1.40%	-0.80%	-1.89%	-12.52%	-12.47%	-13.24%	-15.24%	-20.88%
Baseline	72.6	71.0	77.0	73.3	62.2	64.0	64.0	67.2	61.8	50.4

Table 7: Robustness evaluation against temporal rescaling on the PKU-MMD v2 (X-sub) dataset. Models trained on the original temporal scale are evaluated on sequences rescaled by 0.7x and 1.5x. For each method, the first row reports the absolute performance, and the second row indicates the relative performance drop compared to the clean setting.

Position of Frequency Module	Acc	Edit	F1@{10, 25, 50}
At the Very Beginning	74.8	74.3	78.8	75.9	65.6
After Spatial Modeling	75.4	74.5	79.7	76.8	66.6
After Spatial Modeling	70.6	70.0	75.4	71.9	60.6

Table 8: Ablation study on the position of the frequency module on PKU-MMD v2 (X-sub) dataset.

FFT or Not	Acc	Edit	F1@{10, 25, 50}
Both use FFT	75.4	74.5	79.7	76.8	66.6
MASF w/o FFT	75.1	73.4	79.1	76.2	65.5
AADL w/o FFT	74.7	73.6	78.9	75.9	65.4
Both w/o FFT	74.2	73.2	78.5	75.4	64.9

Table 9: Ablation study on the necessity of frequency-domain operations on PKU-MMD v2 (X-sub) dataset.

FFT or Not	Acc $\uparrow$	Edit $\uparrow$	F1@{10, 25, 50} $\uparrow$			FLOPs $\downarrow$	Param. $\downarrow$
Filter-Shared + Channel-Weight	75.4	74.5	79.7	76.8	66.6	0.01G	0.006M
Filter-Shared + Temporal-Weight	74.7	74.2	78.7	76.0	65.7	0.045G	0.010M
Filter-Shared + Both Weights	75.2	74.3	78.9	76.0	66.0	0.045G	0.261M
Filter-Specific + Temporal-Weight	75.5	74.7	79.5	76.8	66.6	0.045G	0.337M
Filter-Specific + Both Weights	75.6	74.9	79.5	77.0	66.8	0.045G	0.588M

Table 10: Ablation study on filter dimension and weighting strategy in MASF on PKU-MMD v2 (X-sub) dataset.

Dynamic	Static	Average	Acc	Edit	F1@{10, 25, 50}
✓	✗	✗	75.2	74.0	79.4	76.5	66.2
✗	✓	✗	74.6	73.9	78.8	75.9	66.4
✗	✗	✓	75.2	74.2	79.4	76.5	66.3
✓	✗	✓	75.4	74.2	78.6	76.2	66.3
✗	✓	✓	75.3	74.4	79.5	76.4	66.4
✓	✓	✗	75.4	74.5	79.7	76.8	66.6

Table 11: Effect of weighted fusion methods on multi-scale filtered features on PKU-MMD v2 (X-sub) dataset.

Feature Domain	Weight Generator	Acc	Edit	F1@{10, 25, 50}
Temporal	SE Block	74.8	73.4	78.4	75.6	65.6
Temporal	Equation (3)	75.4	74.5	79.7	76.8	66.6
Frequency	SE Block	74.8	74.3	78.6	75.5	65.6
Frequency	Equation (3)	75.3	74.2	79.8	76.8	66.4

Table 12: Effect of feature domain and weight generation in dynamic weighting on PKU-MMD v2 (X-sub) dataset.

Weight Granularity	Initialization	Acc	Edit	F1@{10, 25, 50}
(M, C)	Random	75.4	74.6	79.6	76.6	66.6
(M, C)	Zero	75.4	74.5	79.7	76.8	66.6
(M, 1)	Random	75.4	74.3	79.6	76.3	66.3
(M, 1)	Zero	75.3	74.4	79.6	76.4	66.4

Table 13: Effect of granularity and initialization in static weighting on PKU-MMD v2 (X-sub) dataset.

Fusion Method	Acc	Edit	F1@{10, 25, 50}
Average Fusion	75.4	74.5	79.7	76.8	66.6
Learnable Scalar Weight	75.4	74.2	79.6	76.7	66.3
Channel-wise Weight	75.4	74.4	79.3	76.3	66.4
Channel Concat + Conv	75.5	74.6	79.7	76.5	66.7

Table 14: Effect of fusion strategy for dynamic and static branch features on PKU-MMD v2 (X-sub) dataset.

Number of filter heads	Acc	Edit	F1@{10, 25, 50}
2	75.4	74.4	79.5	76.6	66.4
4	75.4	74.5	79.7	76.8	66.6
6	75.7	74.7	80.0	77.2	66.7
8	75.8	74.9	80.1	77.4	66.7

Table 15: Effect of different numbers of filter heads

M

on PKU-MMD v2 (X-sub) dataset.

Maximum Filter Length	Acc	Edit	F1@{10, 25, 50}
32	75.3	74.2	79.5	76.7	66.5
64	75.4	74.5	79.7	76.8	66.6
128	75.7	74.3	79.8	76.8	66.8
256	75.8	74.7	79.8	77.0	66.9

Table 16: Impact of maximum filter length

R_{max}

in spectral filtering on PKU-MMD v2 (X-sub) dataset.

Multi-scale strategies	Acc	Edit	F1@{10, 25, 50}
Single-scale	75.3	74.5	79.5	76.6	66.4
Equation (2)	75.4	74.5	79.7	76.8	66.6
$R_{m}=2^{m+b}$	75.4	74.6	79.6	76.7	66.5

Table 17: Comparison of multi-scale strategies with fixed maximum filter length 64 on PKU-MMD v2 (X-sub) dataset.

Discrepancy Calculation Scope	Acc	Edit	F1@{10, 25, 50}
AADL	75.4	74.5	79.7	76.8	66.6
IADL	75.5	74.6	79.7	76.9	66.6
BADL	75.7	74.5	79.9	77.1	66.7

Table 18: Impact of action discrepancy scope in adjacent action discrepancy loss on PKU-MMD v2 (X-sub) dataset. AADL is Adjacent Action Discrepancy Loss, IADL is Instance-scope Action Discrepancy Loss, and BADL is Batch-scope Action Discrepancy Loss.

Spectral Representation	Acc	Edit	F1@{10, 25, 50}
Amplitude Only	75.4	74.5	79.7	76.8	66.6
Complex	75.5	74.4	79.6	76.8	66.6

Table 19: Ablation on spectral representations (Amplitude vs. Complex) for computing the AADL on the PKU-MMD v2 (X-sub) dataset.

Discrepancy Metric	Loss Form	Acc	Edit	F1@{10, 25, 50}
Normalized L1	-log(tanh( $\cdot$ ))	75.4	74.5	79.7	76.8	66.6
Normalized L2	-log(tanh( $\cdot$ ))	75.2	74.0	79.2	76.2	66.3
L-infinity	-log(tanh( $\cdot$ ))	74.8	73.4	78.6	75.4	64.8
Normalized L1	-log(B-sigmoid( $\cdot$ ))	75.2	74.3	79.6	76.4	66.5
Normalized L1	1 - tanh( $\cdot$ )	75.5	74.3	79.4	76.5	66.4
Normalized L1	1 - B-sigmoid( $\cdot$ )	75.3	74.4	79.6	76.7	66.6

Table 20: Effect of different AADL computation strategies on PKU-MMD v2 (X-sub) dataset.

Sampling Strategies	Acc	Edit	F1@{10, 25, 50}
Linear Interpolation	75.4	74.5	79.7	76.8	66.6
Nearest Neighbor	75.6	73.6	79.4	76.3	65.9
AdaptiveAvgPool	75.3	74.3	79.7	76.7	66.3

Table 21: Impact of the sampling strategies for action spectral features on PKU-MMD v2 (X-sub) dataset.

Sampling Length	Acc	Edit	F1@{10, 25, 50}
16	75.3	73.6	78.9	76.1	66.2
32	75.4	74.5	79.7	76.8	66.6
48	75.5	74.6	79.8	76.6	66.5
64	75.6	74.0	79.1	76.5	66.6

Table 22: Effect of fixed sampling length

S_{f}

on action spectral features on PKU-MMD v2 (X-sub) dataset.

Scaling Factor $\alpha$	Acc	Edit	F1@{10, 25, 50}
50	75.0	74.6	79.3	76.1	66.4
100	75.4	74.5	79.7	76.8	66.6
200	75.3	73.9	79.2	76.3	66.0
400	75.0	74.3	79.6	76.7	65.9

Table 23: Impact of discrepancy scaling factor

\alpha

in adjacent action discrepancy loss on PKU-MMD v2 (X-sub) dataset.

Weight $\lambda_{3}$	Acc	Edit	F1@{10, 25, 50}
0.1	75.4	74.4	79.6	76.7	66.3
1.0	75.4	74.5	79.7	76.8	66.6
5.0	75.5	74.5	79.6	76.7	66.7
10.0	75.3	74.6	79.7	76.8	66.6

Table 24: Effects of the weight

\lambda_{3}

for adjacent action discrepancy loss on PKU-MMD v2 (X-sub) dataset.

FFT	PConv	Depth	Acc	Edit	F1@{10, 25, 50}
✗	✗	-	75.1	73.9	79.3	76.3	66.0
✓	✗	-	75.2	74.2	79.7	76.6	66.3
✗	✓	2	75.1	74.1	79.3	76.4	66.1
✓	✓	1	75.3	74.4	79.4	76.2	66.4
✓	✓	2	75.4	74.5	79.7	76.8	66.6
✓	✓	3	75.5	74.5	79.7	76.7	66.7

Table 25: Impact of frequency-aware channel mixer and stacking depth on PKU-MMD v2 (X-sub) dataset.

Conv Strategy	Acc	Edit	F1@{10, 25, 50}
Real-only	74.5	73.5	78.7	75.9	64.5
Imaginary-only	74.7	73.5	77.9	74.7	63.5
Shared Weights	75.4	74.5	79.7	76.8	66.6
Independent Weights	74.8	74.4	78.9	76.1	65.8

Table 26: Ablation on complex feature processing in frequency-aware channel mixer on PKU-MMD v2 (X-sub) dataset.

Post-Processing	Acc	Edit	F1@{10, 25, 50}
Identity	75.4	74.5	79.7	76.8	66.6
ReLU	74.2	73.4	78.6	75.5	64.9
BatchNorm	75.1	74.3	78.9	76.3	65.4
Dropout(0.3)	75.6	74.5	79.6	76.5	66.5

Table 27: Ablation on post-processing operations for frequency-aware mixed features on PKU-MMD v2 (X-sub) dataset.

Kernel Size	Acc	Edit	F1@{10, 25, 50}
1	75.4	74.5	79.7	76.8	66.6
3	74.5	73.9	78.1	75.3	64.6
5	73.8	74.2	78.5	75.7	64.6
7	73.8	74.4	78.9	75.4	64.5

Table 28: Impact of kernel size choices in frequency-aware channel mixer on PKU-MMD v2 (X-sub) dataset.

Normalization Mode	Acc	Edit	F1@{10, 25, 50}
Forward (normalize by $1/S$ )	75.5	74.6	79.8	76.7	66.6
Backward (no normalization)	75.4	74.5	79.7	76.8	66.6
Ortho (normalize by $1/\sqrt{S}$ )	75.3	74.6	79.5	76.4	66.5

Table 29: Comparing FFT normalization strategies in frequency-aware channel mixer on PKU-MMD v2 (X-sub) dataset.

block counts	Acc	Edit	F1@{10, 25, 50}
6	72.1	71.0	76.7	73.4	62.1
8	74.2	73.2	78.5	75.1	64.5
10	75.4	74.5	79.7	76.8	66.6
12	75.7	75.1	80.0	77.0	66.6
14	76.0	75.0	79.6	77.1	66.8

Table 30: Ablation on temporal block depth in temporal modeling on PKU-MMD v2 (X-sub) dataset.

Class stages	Boundary stages	Acc	Edit	F1@{10, 25, 50}
1	1	75.7	74.1	78.2	75.3	66.2
1	2	75.4	74.5	79.7	76.8	66.6
2	2	75.3	74.5	79.6	76.9	66.7
2	3	74.8	74.0	79.6	75.9	66.0
3	3	74.3	74.5	79.5	76.2	65.3

Table 31: Ablation on the stage numbers of classification/boundary refinement branches on PKU-MMD v2 (X-sub) dataset.

Temporal Action Segmentation

Temporal action segmentation labels when actions begin and end in continuous video, enabling fine-grained analysis of human and robotic activity.

See all 49 papers

LaDy: Lagrangian-Dynamic Informed Network for Skeleton-based Action Segmentation via Spatial-Temporal Modulation
Mar 25, 2026Haoyu Ji, Xueting Liu, Yu Gao +5Temporal Action Segmentation Boundary-Aware Segmentation
Harbin Institute of Technology, Shenzhen · Southern University of Science and Technology · Southeast University
Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantization
Apr 16, 2026Umer Ahmed, Syed Ahmed Mahmood, Fawad Javed Fateh +3Temporal Action Segmentation Vector Quantization
Retrocausal, Inc. Redmond, WA
MASQ: Mask-Aware Spatiotemporal Quantization for Unsupervised Skeleton Action Segmentation
Aug 30, 2026Xinyao Qin, Linxiang Peng, Youbao Ye +2Temporal Action Segmentation Unsupervised Representation Learning
University of Science and Technology of China, China

Mar 25, 2026J/K move · Enter open · S save

LaDy: Lagrangian-Dynamic Informed Network for Skeleton-based Action Segmentation via Spatial-Temporal Modulation

Haoyu Ji, Xueting Liu, Yu Gao, Wenze Huang+4

Harbin Institute of Technology, Shenzhen · Southern University of Science and Technology · Southeast University

Temporal Action Segmentation Boundary-Aware Segmentation Physics-Informed ML

Open paper AskPDF arXiv

Skeleton-based Temporal Action Segmentation (STAS) aims to densely parse untrimmed skeletal sequences into frame-level action categories. However, existing methods, while proficient at capturing spatio-temporal kinematics, neglect the underlying physical dynamics that govern human motion. This oversight limits inter-class discriminability between actions with similar kinematics but distinct dynamic intents, and hinders precise boundary localization where dynamic force profiles shift. To address these, we propose the Lagrangian-Dynamic Informed Network (LaDy), a framework integrating principles of Lagrangian dynamics into the segmentation process. Specifically, LaDy first computes generalized coordinates from joint positions and then estimates Lagrangian terms under physical constraints to explicitly synthesize the generalized forces. To further ensure physical coherence, our Energy Consistency Loss enforces the work-energy theorem, aligning kinetic energy change with the work done by the net force. The learned dynamics then drive a Spatio-Temporal Modulation module: Spatially, generalized forces are fused with spatial representations to provide more discriminative semantics. Temporally, salient dynamic signals are constructed for temporal gating, thereby significantly enhancing boundary awareness. Experiments on challenging datasets show that LaDy achieves state-of-the-art performance, validating the integration of physical dynamics for action segmentation. Code is available at https://github.com/HaoyuJi/LaDy.

Abstract

Temporal Action Segmentation

Explore similar work

LaDy: Lagrangian-Dynamic Informed Network for Skeleton-based Action Segmentation via Spatial-Temporal Modulation

Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantization

MASQ: Mask-Aware Spatiotemporal Quantization for Unsupervised Skeleton Action Segmentation