LaDy: Lagrangian-Dynamic Informed Network for Skeleton-based Action Segmentation via Spatial-Temporal Modulation

Haoyu Ji; Xueting Liu; Yu Gao; Wenze Huang; Zhihao Yang; Weihong Ren; Zhiyong Wang; Honghai Liu

cs.CVMar 25, 2026

LaDy: Lagrangian-Dynamic Informed Network for Skeleton-based Action Segmentation via Spatial-Temporal Modulation

Authors: Haoyu Ji, Xueting Liu, Yu Gao, Wenze Huang, Zhihao Yang, Weihong Ren, Zhiyong Wang, Honghai Liu

Organizations: Harbin Institute of Technology, Shenzhen · Southern University of Science and Technology · Southeast University

Abstract

Skeleton-based Temporal Action Segmentation (STAS) aims to densely parse untrimmed skeletal sequences into frame-level action categories. However, existing methods, while proficient at capturing spatio-temporal kinematics, neglect the underlying physical dynamics that govern human motion. This oversight limits inter-class discriminability between actions with similar kinematics but distinct dynamic intents, and hinders precise boundary localization where dynamic force profiles shift. To address these, we propose the Lagrangian-Dynamic Informed Network (LaDy), a framework integrating principles of Lagrangian dynamics into the segmentation process. Specifically, LaDy first computes generalized coordinates from joint positions and then estimates Lagrangian terms under physical constraints to explicitly synthesize the generalized forces. To further ensure physical coherence, our Energy Consistency Loss enforces the work-energy theorem, aligning kinetic energy change with the work done by the net force. The learned dynamics then drive a Spatio-Temporal Modulation module: Spatially, generalized forces are fused with spatial representations to provide more discriminative semantics. Temporally, salient dynamic signals are constructed for temporal gating, thereby significantly enhancing boundary awareness. Experiments on challenging datasets show that LaDy achieves state-of-the-art performance, validating the integration of physical dynamics for action segmentation. Code is available at https://github.com/HaoyuJi/LaDy.

Figures & tables

Figure 1: Conceptual comparison of STAS frameworks. (a) Conventional models capture only spatio-temporal kinematic patterns. (b) Our LaDy framework introduces Lagrangian dynamics to derive generalized forces that modulate the spatio-temporal features, leading to improved discriminability and boundary localization.

Figure 2: Overview of the LaDy framework. The Lagrangian Dynamic Model (lower-left) synthesizes generalized forces from coordinates, constrained by an Energy Consistency Loss for physical coherence. Concurrently, the Spatial Model (upper-left) extracts kinematic features (GCNs) and fuses them with these physics-aware dynamics. The multi-stage Temporal Model (right) processes the fused spatial representation, where each stage is hierarchically gated by force-driven signals to enhance boundary awareness.

Figure 3: Physics-constrained dynamics synthesis and energy-based supervision. Generalized states ( $q,\dot{q},\ddot{q}$ ) are fed into physics-constrained estimators to derive the Lagrangian terms ( $M,C,G,F$ ). These terms are then assembled via the Lagrangian equation to synthesize the generalized forces ( $\tau$ ). Concurrently, the Energy Consistency Loss ( $\mathcal{L}_{EC}$ ) regularizes the forces.

Dataset			PKU-MMD v2 (X-view)					PKU-MMD v2 (X-sub)					LARa
Metric	FLOPs $\downarrow$	Param. $\downarrow$	Acc $\uparrow$	Edit $\uparrow$	F1@{10,25,50} $\uparrow$			Acc $\uparrow$	Edit $\uparrow$	F1@{10,25,50} $\uparrow$			Acc $\uparrow$	Edit $\uparrow$	F1@{10,25,50} $\uparrow$
MS-TCN [ 13 ]	3.13G	0.52M	58.2	56.6	58.6	53.6	39.4	65.5	-	-	-	46.3	65.8	-	-	-	39.6
ASFormer [ 59 ]	6.20G	1.01M	64.8	62.6	65.1	60.0	45.8	68.3	68.1	71.9	68.2	54.5	72.2	62.2	66.1	61.9	49.2
ASRF [ 24 ]	7.00G	1.18M	60.4	59.3	62.5	58.0	46.1	67.7	67.1	72.1	68.3	56.8	71.9	63.0	68.3	65.3	53.2
MS-GCN [ 15 ]	30.97G	0.63M	65.3	58.1	61.3	56.7	44.1	68.5	-	-	-	51.6	65.6	-	-	-	43.6
MTST-GCN [ 4 ]	108.87G	2.58M	66.5	64.0	67.1	62.4	49.9	70.0	65.8	68.5	63.9	50.1	73.7	58.6	63.8	59.4	47.6

Table 1: Comparison with the latest results on PKU-MMD v2 (X-view and X-sub), and LARa. Bold and underline indicate the best and second-best results in each column. FLOPs denote the computational complexity when the input is

X\in\mathbb{R}^{12\times 6000\times 19}

on LARa dataset.

Dataset	MCFS-22					MCFS-130					TCG-15
Metric	Acc $\uparrow$	Edit $\uparrow$	F1@{10, 25, 50} $\uparrow$			Acc $\uparrow$	Edit $\uparrow$	F1@{10, 25, 50} $\uparrow$			Acc $\uparrow$	Edit $\uparrow$	F1@{10, 25, 50} $\uparrow$
MS-TCN [ 13 ]	75.6	74.2	74.3	69.7	59.5	65.7	54.5	56.4	52.2	42.5	85.5	67.7	70.0	67.3	59.1
ASFormer [ 59 ]	78.7	82.3	82.8	77.9	66.9	67.5	69.1	68.3	64.0	55.1	82.3	69.1	71.3	67.7	57.6
ASRF [ 24 ]	75.5	77.3	83.3	80.1	69.2	65.6	65.6	66.7	62.3	51.9	83.9	65.3	71.5	69.9	62.4
MS-GCN [ 15 ]	75.5	72.6	75.7	70.5	57.9	64.9	52.6	52.4	48.8	39.1	86.6	68.7	70.9	69.7	59.4
IDT-GCN [ 32 ]	79.9	84.5	88.0	84.9	74.9	68.6	70.2	70.7	67.3	58.6	-	-	-	-	-

Table 2: Comparison with the latest results on MCFS-22, MCFS-130, and TCG-15 datasets.

Panel of Figure 4 — Figure 4: Qualitative results on PKU-MMD v2 and MCFS-130. The top row is the Ground Truth, followed by the segmentation results (bars) and boundary confidence scores (curves) for LaDy, LaSA, and DeST. Different colors denote distinct action classes. Red boxes and gray dashed vertical lines highlight misclassifications and larger boundary deviations observed in other methods compared to LaDy.

Panel of Figure 5 — Figure 5: Visualization of the representation space on PKU-MMD v2. Each point represents an action segment feature, colored according to its ground-truth class. Quantitative clustering metrics are reported in the top-right corner: Silhouette Coefficient (SC $\uparrow$ ), Calinski-Harabasz Index (CH $\uparrow$ ), and Davies-Bouldin Index (DB $\downarrow$ ). Higher ( $\uparrow$ ) is better for SC/CH; lower ( $\downarrow$ ) is better for DB.

Components	Acc	Edit	F1@{10, 25, 50}
Baseline	73.6	73.0	78.2	74.6	64.3
+LDS+SM	74.8	74.0	78.9	76.0	65.7
+LDS+TM	74.9	73.5	78.1	75.2	65.8
+LDS+STM	75.9	74.6	79.6	76.9	66.3
+LDS+SM+ECLoss	74.7	74.3	79.1	76.3	66.0
+LDS+TM+ECLoss	75.4	74.5	79.6	76.7	66.5

Table 3: Ablation study of Lagrangian Dynamics Synthesis (LDS), Spatio-Temporal Modulation (STM), and Energy Consistency Loss (ECLoss) on PKU-MMD v2 (X-sub) dataset.

Figure 6: Visualization of synthesized generalized force magnitudes for four actions. Limb thickness is proportional to the norm of the estimated torque. Red circles highlight the primary active limbs, illustrating that the learned dynamics are both class-discriminative and physically plausible.

Figure 7: Visualization of layer-averaged dynamic gating signals. The time-series plots for Instantaneous Power, Torque Norm, and Torque Change show signals averaged across all $L$ temporal layers. Note the distinct, sharp troughs (highlighted by ovals) that precisely align with the ground-truth action boundaries (dashed lines), validating their role in boundary localization.

Figure 8: Performance vs. efficiency comparison on the PKU-MMD v2 dataset. The y-axis represents the F1@50 score, and the x-axis indicates computational complexity (FLOPs). Circle area is proportional to the model’s parameter count. LaDy (red star) achieves state-of-the-art performance with a highly competitive computational footprint.

Panel of Figure 9 — Figure 9: F1 scores vs. IoU thresholds on PKU-MMD v2. LaDy (blue) maintains a distinct performance margin over LaSA (orange) and DeST (green), particularly at strict thresholds ( $\text{IoU}>0.7$ ). This demonstrates that integrating Lagrangian dynamics (salient dynamic signals) for temporal modulation yields significantly sharper boundary localization.

Figure 10: Per-class frame-wise F1 improvement of LaDy over the previous method (LaSA) on the PKU-MMD v2 (X-view and X-sub), MCFS-130, and LARa datasets. Blue bars indicate performance gains, while green bars denote declines. The results demonstrate that LaDy yields significant improvements in complex, physically distinct actions, validating that explicitly modeling dynamics effectively resolves kinematic ambiguities.

Panel of Figure 11 — Figure 11: Qualitative results on PKU-MMD v2 (X-view and X-sub), MCFS-22, and LARa. The top row is the Ground Truth, followed by the segmentation results (bars) and boundary confidence scores (curves) for LaDy, LaSA, and DeST. Different colors denote distinct action classes.

Noise	Gaussian Noise					Joint Occlusion					Missing Frames
Metric	Acc	Edit	F1@{10, 25, 50}			Acc	Edit	F1@{10, 25, 50}			Acc	Edit	F1@{10, 25, 50}
DeST	65.5	63.8	69.8	65.4	51.7	58.3	57.4	61.2	56.6	43.7	66.9	65.9	71.7	67.2	52.7
	-6.8%	-7.9%	-6.4%	-7.9%	-12.0%	-17.1%	-17.1%	-17.9%	-20.2%	-25.6%	-4.8%	-5.0%	-3.7%	-5.4%	-10.3%
LaSA	70.4	70.7	75.9	71.5	59.4	60.4	60.0	64.9	59.1	48.0	70.2	69.9	75.5	71.3	58.0
	-4.2%	-3.7%	-3.1%	-4.4%	-6.6%	-17.8%	-18.2%	-17.1%	-21.0%	-24.5%	-4.5%	-4.8%	-3.6%	-4.7%	-8.8%
LaDy	73.6	73.1	78.2	75.4	64.2	70.1	68.6	73.6	70.1	59.5	73.4	71.7	77.1	74.5	62.1

Table 4: Robustness evaluation against various input perturbations (Gaussian noise, joint occlusion, and missing frames) on the PKU-MMD v2 (X-sub) dataset. All models are trained on clean data and evaluated on corrupted test sets. For each method, the first row reports the absolute performance, and the second row indicates the relative performance drop compared to the clean setting.

Loss Formulation	Acc	Edit	F1@{10, 25, 50}
Finite difference (Ours)	76.2	75.1	80.1	77.3	67.0
Gaussian smoothing	75.8	74.6	79.4	76.8	66.5
Savitzky-Golay	75.9	74.9	79.8	76.8	66.4

Table 5: Ablation on the calculation of generalized velocities and accelerations on the PKU-MMD v2 (X-sub) dataset. We compare our simple finite difference approach against explicit pre-smoothing techniques (Gaussian and Savitzky-Golay filtering).

Modeling Strategy	$\mathcal{L}_{EC}$	Acc	Edit	F1@{10, 25, 50}
Baseline (w/o LDS)	-	73.6	73.0	78.2	74.6	64.3
Direct MLP Mapping	-	75.0	74.1	79.0	76.2	65.8
Unconstrained Eq.	-	75.2	74.5	79.7	76.5	66.2
Unconstrained Eq.	✓	75.3	74.7	79.3	76.5	66.7
Physics-Constrained Eq.	-	75.9	74.6	79.6	76.9	66.3
Physics-Constrained Eq.	✓	76.2	75.1	80.1	77.3	67.0

Table 6: Ablation on Dynamics Modeling Strategies on PKU-MMD v2 (X-sub) dataset. We compare direct estimation against equation-based synthesis with and without physical constraints and energy supervision.

Term	Constraint Formulation	Acc	Edit	F1@{10, 25, 50}
$\bm{M}$	Unconstrained	75.4	74.6	79.6	76.6	66.0
	Symmetric Only	75.3	74.1	79.3	76.3	66.0
	Sym. + Semi-Positive Definite	75.9	74.2	78.9	76.5	65.9
	Sym. + Positive Definite (SPD)	76.2	75.1	80.1	77.3	67.0
$\bm{C}$	Unconstrained $C$	75.5	74.2	79.6	76.7	66.2
	Unconstrained $C=0.5(\dot{M}-N)$	75.9	74.8	79.7	76.6	66.6

Table 7: Ablation on Physical Constraints for Lagrangian Terms on PKU-MMD v2 (X-sub) dataset. We evaluate different constraint formulations for Inertia (

M

), Coriolis (

C

), and Gravity (

G

) estimation.

Model Variant	Acc	Edit	F1@{10, 25, 50}
w/o Inertia ( $M$ )	74.5	73.2	78.5	75.9	65.9
w/o Coriolis ( $C$ )	75.5	74.4	78.9	76.2	66.2
w/o Gravity ( $G$ )	75.0	73.9	79.1	76.0	65.9
w/o Non-Conservative ( $F$ )	75.2	73.5	78.8	76.5	65.7
Full Model (LaDy)	76.2	75.1	80.1	77.3	67.0

Table 8: Ablation on the necessity of each Lagrangian term on PKU-MMD v2 (X-sub) dataset. We report results when removing the Inertia (

M

), Coriolis (

C

), Gravity (

G

), or Non-conservative (

F

) terms individually.

Loss Formulation	Acc	Edit	F1@{10, 25, 50}
Naive Formulation	75.7	74.3	79.7	76.9	66.6
w/ Relative Normalization	75.9	74.6	79.6	76.9	66.8
w/ Noise Masking $\mathcal{M}(t)$	76.2	74.4	79.8	77.3	66.8
Full Formulation (Ours)	76.2	75.1	80.1	77.3	67.0

Table 9: Ablation on the internal design of the Energy Consistency Loss (

\mathcal{L}_{EC}

) on PKU-MMD v2 (X-sub) dataset. We compare the naive formulation against versions with Relative Normalization and Noise Masking.

Distance Function	Acc	Edit	F1@{10, 25, 50}
$\ell_{1}$ Loss (MAE Loss)	76.0	74.7	79.9	77.2	66.5
$\ell_{2}$ Loss (MSE Loss)	76.0	74.9	80.0	77.4	67.0
Huber Loss (Smooth $\ell_{1}$ )	76.2	75.1	80.1	77.3	67.0

Table 10: Ablation on the distance metric for

\mathcal{L}_{EC}

on PKU-MMD v2 (X-sub) dataset. We compare standard regression losses against the robust Huber loss.

Coefficient $\lambda_{3}$	Acc	Edit	F1@{10, 25, 50}
1.0	75.9	74.2	79.8	76.9	66.3
0.1	76.2	75.1	80.1	77.3	67.0
0.01	75.7	74.5	79.9	76.9	66.7
0.001	75.7	74.9	79.6	76.9	66.5
0.0001	76.1	75.0	79.8	77.1	66.6

Table 11: Ablation on the regularization weight

\lambda_{3}

for

\mathcal{L}_{EC}

on PKU-MMD v2 (X-sub) dataset.

Injection Strategy	Acc	Edit	F1@{10, 25, 50}
Direct Injection	75.5	74.0	78.8	76.4	66.1
Delayed Start ( $\mathcal{Z}$ ) only	75.9	74.7	79.7	76.9	66.8
Linear Warmup ( $\mathcal{Z}_{w}$ ) only	75.7	74.9	79.2	76.5	66.4
Delayed + Phased Warmup	76.2	75.1	80.1	77.3	67.0

Table 12: Ablation on the Delayed Physics Injection strategy on PKU-MMD v2 (X-sub) dataset. We evaluate the impact of Delayed Start (

\mathcal{Z}

) and Linear Warmup (

\mathcal{Z}_{w}

) on training stability.

Fusion Stage	Acc	Edit	F1@{10, 25, 50}
Early (Input-level)	75.4	74.8	79.8	77.0	66.4
Late (Spatial Merging)	75.7	74.8	79.6	76.7	66.4
Mid (Post-GCN, Ours)	76.2	75.1	80.1	77.3	67.0

Table 13: Ablation on the fusion stage for Spatial Modulation on PKU-MMD v2 (X-sub) dataset. We compare fusing dynamics at the raw input level (Early), after spatial-channel merging (Late), and after GCN encoding (Mid).

Fusion Mechanism	Acc	Edit	F1@{10, 25, 50}
Node Concatenation	75.0	74.0	79.1	76.2	66.4
Element-wise Addition	75.6	74.7	79.7	76.9	66.0
Element-wise Product	75.9	73.9	79.2	76.3	66.2
Cross-Attention (Kin-Query)	75.7	74.7	79.8	76.8	66.7
Channel Concatenation (Ours)	76.2	75.1	80.1	77.3	67.0

Table 14: Ablation on fusion mechanisms for Spatial Modulation on PKU-MMD v2 (X-sub) dataset. We compare our Channel Concatenation strategy against node-level concatenation, cross-attention, and element-wise operations.

Gating Architecture	Acc	Edit	F1@{10, 25, 50}
Single-Stage (Input only)	75.5	74.4	79.4	76.7	66.5
Single-Stage (Output only)	75.3	74.2	79.1	76.5	66.3
Multi-Stage (Static/Repeated)	76.0	74.4	79.9	77.0	66.5
Multi-Stage (Parallel Projection)	75.8	74.2	79.6	76.6	66.6
Multi-Stage (Hierarchical Evolving)	76.2	75.1	80.1	77.3	67.0

Table 15: Ablation on the architecture of Temporal Modulation on PKU-MMD v2 (X-sub) dataset. We compare single-stage gating against multi-stage strategies with static, parallel, or hierarchical topologies.

$g_{\tau}$	$g_{\dot{\tau}}$	$g_{P}$	Acc	Edit	F1@{10, 25, 50}
Dynamic Signals			Acc	Edit	F1@{10, 25, 50}
✓	-	-	75.2	74.2	79.2	75.9	66.2
-	✓	-	74.7	74.1	78.9	76.0	66.2
-	-	✓	75.3	74.5	79.8	76.9	66.7
✓	✓	-	75.2	74.1	79.1	76.0	66.3
✓	-	✓	75.4	74.6	79.8	76.8	66.6

Table 16: Ablation on the combination of Salient Dynamic Signals for temporal gating on PKU-MMD v2 (X-sub) dataset. We evaluate the synergistic effect of Torque (

g_{\tau}

), Torque Change (

g_{\dot{\tau}}

), and Power (

g_{P}

Fusion Strategy	Acc	Edit	F1@{10, 25, 50}
Naive Summation	75.4	74.7	79.6	76.6	66.1
Static Weighted Sum	75.3	74.2	79.3	76.5	66.1
Adaptive Weighted Sum	75.4	74.9	80.0	77.0	66.6
Concat + Conv Fusion (Ours)	76.2	75.1	80.1	77.3	67.0

Table 17: Ablation on the feature fusion strategy within the Temporal Modulation module on PKU-MMD v2 (X-sub) dataset. We compare linear summation methods against learnable concatenation-based fusion.

Interaction Strategy	Acc	Edit	F1@{10, 25, 50}
Early Fusion (Input-level)	75.4	74.2	79.3	76.4	66.5
Step-wise Fusion (Gate-level)	74.8	74.3	79.1	76.0	65.9
Coupled Evolution (Inter-signal)	75.4	74.7	80.0	76.8	66.7
Decoupled Evolution (Ours)	76.2	75.1	80.1	77.3	67.0

Table 18: Ablation on the interaction strategy of dynamic signals within the hierarchical gating mechanism on PKU-MMD v2 (X-sub) dataset. We compare merging signals (Fusion), interacting channels (Coupled), and independent processing (Decoupled).

Signal Dimensionality	Acc	Edit	F1@{10, 25, 50}
Multi-Channel ( $T\times C$ )	76.0	75.0	80.1	77.0	66.8
Channel-Agnostic ( $T\times 1$ )	76.2	75.1	80.1	77.3	67.0

Table 19: Ablation on the channel dimensionality of dynamic gating signals on PKU-MMD v2 (X-sub) dataset. We compare channel-wise modulation (

C

channels) against channel-agnostic scalar modulation (1 channel).

Method	Joint	Bone	J-Motion	B-Motion	Ensemble4
CTR-GCN	90.1	90.2	87.8	87.2	92.4
+LaDy (Ours)	90.3	90.4	88.3	88.9	92.9

Table 20: Preliminary evaluation of LaDy as a plug-and-play feature enhancer for skeleton-based action recognition on the NTU-RGB+D (X-Sub) dataset. The integration consistently improves the CTR-GCN baseline across all individual modalities and the multi-stream ensemble.

Temporal Action Segmentation

Temporal action segmentation labels when actions begin and end in continuous video, enabling fine-grained analysis of human and robotic activity.

See all 49 papers

Spectral Scalpel: Amplifying Adjacent Action Discrepancy via Frequency-Selective Filtering for Skeleton-Based Action Segmentation
Mar 25, 2026Haoyu Ji, Bowen Chen, Zhihao Yang +6Temporal Action Segmentation Boundary-Aware Segmentation
Harbin Institute of Technology, Shenzhen · Shenzhen H&T Intelligent Control Co., Ltd. · Southern University of Science and Technology +1
MASQ: Mask-Aware Spatiotemporal Quantization for Unsupervised Skeleton Action Segmentation
Aug 30, 2026Xinyao Qin, Linxiang Peng, Youbao Ye +2Temporal Action Segmentation Unsupervised Representation Learning
University of Science and Technology of China, China
Improving Temporal Action Segmentation via Constraint-Aware Decoding
May 11, 2026Yeo Keat Ee, Debaditya Roy, Chen Li +2Temporal Action Segmentation Constrained Decoding
Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Indian Institute of Technology Kharagpur, India +1

Mar 25, 2026J/K move · Enter open · S save

Spectral Scalpel: Amplifying Adjacent Action Discrepancy via Frequency-Selective Filtering for Skeleton-Based Action Segmentation

Haoyu Ji, Bowen Chen, Zhihao Yang, Wenze Huang+5

Harbin Institute of Technology, Shenzhen · Shenzhen H&T Intelligent Control Co., Ltd. · Southern University of Science and Technology+1

Temporal Action Segmentation Boundary-Aware Segmentation Frequency-Domain Feature Learning

Open paper AskPDF arXiv

Skeleton-based Temporal Action Segmentation (STAS) seeks to densely segment and classify diverse actions within long, untrimmed skeletal motion sequences. However, existing STAS methodologies face challenges of limited inter-class discriminability and blurred segmentation boundaries, primarily due to insufficient distinction of spatio-temporal patterns between adjacent actions. To address these limitations, we propose Spectral Scalpel, a frequency-selective filtering framework aimed at suppressing shared frequency components between adjacent distinct actions while amplifying their action-specific frequencies, thereby enhancing inter-action discrepancies and sharpening transition boundaries. Specifically, Spectral Scalpel employs adaptive multi-scale spectral filters as scalpels to edit frequency spectra, coupled with a discrepancy loss between adjacent actions serving as the surgical objective. This design amplifies representational disparities between neighboring actions, effectively mitigating boundary localization ambiguities and inter-class confusion. Furthermore, complementing long-term temporal modeling, we introduce a frequency-aware channel mixer to strengthen channel evolution by aggregating spectra across channels. This work presents a novel paradigm for STAS that extends conventional spatio-temporal modeling by incorporating frequency-domain analysis. Extensive experiments on five public datasets demonstrate that Spectral Scalpel achieves state-of-the-art performance. Code is available at https://github.com/HaoyuJi/SpecScalpel.

Abstract

Temporal Action Segmentation

Explore similar work

Spectral Scalpel: Amplifying Adjacent Action Discrepancy via Frequency-Selective Filtering for Skeleton-Based Action Segmentation

MASQ: Mask-Aware Spatiotemporal Quantization for Unsupervised Skeleton Action Segmentation

Improving Temporal Action Segmentation via Constraint-Aware Decoding