E2E-GNet: An End-to-End Skeleton-based Geometric Deep Neural Network for Human Motion Recognition

Mubarak Olaoluwa; Hassen Drira

cs.CVMar 3, 2026

E2E-GNet: An End-to-End Skeleton-based Geometric Deep Neural Network for Human Motion Recognition

Authors: Mubarak Olaoluwa, Hassen Drira

Organizations: University of Strasbourg, ICube Laboratory, UMR 7357, F-67412 Strasbourg, France

Abstract

Geometric deep learning has recently gained significant attention in the computer vision community for its ability to capture meaningful representations of data lying in a non-Euclidean space. To this end, we propose E2E-GNet, an end-to-end geometric deep neural network for skeleton-based human motion recognition. To enhance the discriminative power between different motions in the non-Euclidean space, E2E-GNet introduces a geometric transformation layer that jointly optimizes skeleton motion sequences on this space and applies a differentiable logarithm map activation to project them onto a linear space. Building on this, we further design a distortion-aware optimization layer that limits skeleton shape distortions caused by this projection, enabling the network to retain discriminative geometric cues and achieve a higher motion recognition rate. We demonstrate the impact of each layer through ablation studies and extensive experiments across five datasets spanning three domains show that E2E-GNet outperforms other methods with lower cost.

Figures & tables

Figure 1: Illustration of proposed E2E-GNet . The input motion sequences are modeled onto the pre-shape space of a unit sphere. Then the geometric transformation layer (GTL) learns and applies optimal transforms to the sequences for improved feature extraction, with projection onto the tangent space via a log-map activation function. The distortion minimization layer (DML) then acts on these tangent representative skeletons to reduce projection-induced distortions. Finally, convolution and LSTM modules extract discriminative spatio-temporal features for classification, and the optimization is done in an end-to-end manner spanning both manifold and tangent spaces.

Parameters	NTU-60/120	EHE	KIMORE	UI-PRMD
Batch size	64	12	16	16
Epochs	50	35	50	40
Learning rate	1e-4	1e-3	1e-3	1e-3
Sequence Length	100	221	150	150

Table 1: Datasets other experimental settings

Category	Methods	Venue	Params (M)	FLOPs (G)	NTU-60 (%)		NTU-120 (%)
Category	Methods	Venue	Params (M)	FLOPs (G)	X-Sub	X-View	X-Sub	X-Setup
Graph Convolution	ST-GCN [ 75 ]	AAAI 2018	3.08	5.34	81.5	88.3	70.7	73.2
	2s-AGCN [ 60 ]	CVPR 2019	–	–	88.5	95.1	82.5	84.2
	MS-G3D [ 47 ]	CVPR 2020	3.17	10.27	91.5	96.2	86.9	88.4
	MST-GCN [ 16 ]	AAAI 2021	3.10	–	91.5	96.6	87.5	88.8
	CTR-GCN [ 15 ]	ICCV 2021	1.46	1.97	92.4	96.8	88.9	90.6

Table 2: Performance comparison (% classification accuracy and cost) against SOTA methods on NTU-60 and NTU-120 datasets.

Methods	Venue	Params (M)	FLOPs (G)	EHE	KIMORE	UI-PRMD
SPDNet [ 35 ]	AAAI 2017	–	–	78.76	82.57	88.87
GCN [ 10 ]	TransAI 2020	3.10	3.80	77.63	76.27	75.44
KShapeNet [ 28 ]	ICCV 2021	0.07	0.003	83.26	76.60	88.22
EGCN [ 76 ]	IJCAI 2022	6.20	7.60	78.87	80.14	86.90
EGCN++ [ 8 ]	TPAMI 2024	6.20	7.60	86.38	83.56	89.95
ManSPD [ 4 ]	SciReports 2025	267.20	0.60	84.74	85.05	92.40

Table 3: Performance comparison (% classification accuracy and cost) against SOTA methods on EHE, KIMORE and UI-PRMD datasets.

Protocol	Action				Disease		RHB
	NTU-60		NTU-120		EHE	KIMORE	UI-PRMD
	X-Sub	X-View	X-Sub	X-Set
Baseline (BL)	93.2	95.8	89.1	81.7	81.69	74.65	81.40
BL+GTL	95.2	96.8	92.3	88.4	83.15	80.09	87.06
BL+GTL+DML	97.1	98.6	95.2	93.3	87.14	85.93	95.19

Table 4: Ablation results (% accuracy) on the contribution of each proposed layer in E2E-GNet. E2E-GNet is BL+GTL+DML .

Variants	Action				Disease		RHB
	NTU-60		NTU-120		EHE	KIMORE	UI-PRMD
	X-Sub	X-View	X-Sub	X-Set
R-Ctr	93.8	95.7	91.3	86.8	83.15	80.09	87.06
R-Utr	94.7	95.7	91.9	86.8	81.63	77.09	87.00
NR-Ctr	94.6	96.6	91.1	88.3	80.03	79.05	85.02
NR-Utr	95.2	96.8	92.3	88.4	79.25	77.77	84.80

Table 5: Performance (% accuracy) of GTL variants on all datasets. Best are bolded. RHB is Rehabilitation. R, NR, Ctr and Utr are rigid, non-rigid, constrained and unconstrained.

Variants	Action				Disease		RHB
	NTU-60		NTU-120		EHE	KIMORE	UI-PRMD
	X-Sub	X-View	X-Sub	X-Set
GH	97.0	98.5	95.0	92.7	87.14	85.93	95.19
GIN	97.1	98.6	95.2	93.3	86.51	84.78	93.95
LH	96.8	98.6	95.2	93.1	85.85	85.23	93.89
LIN	96.6	98.6	95.2	92.5	86.18	85.96	94.85

Table 6: Performance (% accuracy) of DML variants on best transformation for each domain from Tab. 5 . Best are bolded. G, L, H and IN are global, local, homogeneous and inhomogeneous.

	1st	10th	20th	30th	40th	50th	60th	70th	80th	90th
without DML	92.3	90.6	91.1	90.6	89.1	89.1	89.0	89.2	89.0	88.8
with DML	95.2	95.0	95.3	95.0	95.0	95.2	95.1	95.3	95.0	95.0

Table 7: Performance (% accuracy) of proposed DML on different choice of reference frame on NTU-120 X-Sub dataset.

Protocol	Action				Disease		RHB
	NTU-60		NTU-120		EHE	KIMORE	UI-PRMD
	X-Sub	X-View	X-Sub	X-Set
FS	95.2	96.8	92.3	88.4	83.15	80.09	87.06
PT	95.8 $(\uparrow$ 0.9)	98.4 $(\uparrow$ 1.6)	94.3 $(\uparrow$ 2.0)	91.2 $(\uparrow$ 2.8)	78.81 $(\downarrow$ 4.34)	73.41 $(\downarrow$ 6.68)	89.23 $(\uparrow$ 2.17)
DML	97.1 * $(\uparrow$ 1.9)	98.6 * $(\uparrow$ 1.8)	95.2 * $(\uparrow$ 2.9)	93.3 * $(\uparrow$ 4.9)	87.14 * $(\uparrow$ 3.99)	85.93 * $(\uparrow$ 5.84)	95.19 * $(\uparrow$ 8.13)

Table 8: Comparison (% accuracy) of PT and DML under projection at the first shape (FS). Values in blue indicate improvements over FS. * denotes

p\leq 0.05

between PT and DML.

Figure 2: Five consecutive shapes of normal (left) and Alzheimer’s subjects (right) for Bend Waist exercise in EHE dataset. Black values are geodesic distance between the current shape and previous.

Protocol	X-Sub (% acc.)	X-Set (% acc.)
TL by [ 28 ] on pre-shape space	57.2	63.8
TL by [ 28 ] on the tangent space	90.6	86.7
GTL (ours)	92.3	88.4

Table 9: Performance comparison of our proposed GTL vs. transformation layer (TL) proposed by [ 28 ] on NTU-120 dataset.

Figure 3: Rotation difference between consecutive frames on NTU-60 X-Sub.

Figure 4: Visualization of DML effect on some representative skeleton shapes for ‘pick up’ action of NTU dataset. Red and blue values are geodesic distances before and after DML respectively.

	NTU-60/120	EHE	KIMORE	UI-PRMD
IT (sec.)	0.004	0.003	0.0025	0.0025

Table 10: Complete inference time (IT) per sample in each dataset

DML Variants	NTU-60 X-Sub				NTU-60 X-View				NTU-120 X-Sub				NTU-120 X-Set
DML Variants	R-Ctr	R-Utr	NR-Ctr	NR-Utr	R-Ctr	R-Utr	NR-Ctr	NR-Utr	R-Ctr	R-Utr	NR-Ctr	NR-Utr	R-Ctr	R-Utr	NR-Ctr	NR-Utr
GH	96.2	96.5	96.4	97.0	98.4	98.7	98.3	98.5	93.9	94.6	95.0	95.0	91.1	91.9	92.3	92.7
GIN	97.0	97.0	97.0	97.1	98.5	98.8	98.4	98.6	95.0	95.4	95.1	95.2	92.0	92.9	93.0	93.3
LH	96.5	96.6	96.6	96.8	98.5	98.6	98.3	98.6	95.1	94.9	95.0	95.2	91.3	92.0	93.0	93.1
LIN	96.1	96.5	96.1	96.6	98.2	98.5	98.7	98.6	94.5	95.1	94.9	95.2	92.2	92.1	92.3	92.5

Table 11: Performance (% accuracy) of each DML variant under each geometric transformation setting for action recognition datasets. G, L, H and IN are global, local, homogeneous and inhomogeneous respectively. R-Ctr, R-Utr, NR-Ctr and NR-Utr denote rigid-constrained, rigid-unconstrained, non-rigid-constrained and non-rigid-unconstrained respectively.

DML Variants	EHE				KIMORE				UI-PRMD
DML Variants	R-Ctr	R-Utr	NR-Ctr	NR-Utr	R-Ctr	R-Utr	NR-Ctr	NR-Utr	R-Ctr	R-Utr	NR-Ctr	NR-Utr
GH	87.14	86.74	86.62	87.00	85.93	85.19	84.81	85.31	95.19	92.60	92.93	93.95
GIN	86.51	86.27	85.24	85.77	84.78	84.49	83.78	84.81	93.95	91.47	91.38	95.47
LH	85.85	86.62	85.55	86.63	85.23	85.09	84.88	85.84	93.89	92.64	92.63	95.61
LIN	86.18	85.46	85.90	85.86	85.96	83.75	83.67	83.16	94.85	92.54	93.39	95.76

Table 12: Performance (% accuracy) of each DML variant under each geometric transformation setting for disease and rehabilitation datasets. G, L, H and IN are global, local, homogeneous and inhomogeneous respectively. R-Ctr, R-Utr, NR-Ctr and NR-Utr denote rigid-constrained, rigid-unconstrained, non-rigid-constrained and non-rigid-unconstrained respectively.

Folds	EHE						KIMORE								UI-PRMD
	E1	E2	E3	E4	E5	E6	Es1	Es2 (L)	Es2 (R)	Es3 (L)	Es3 (R)	Es4 (L)	Es4 (R)	Es5	E1	E2	E3	E4	E5	E6	E7	E8	E9	E10
F1	33	27	27	30	14	14	58	53	54	54	51	47	56	58	38	30	30	18	36	20	18	16	20	36
F2	34	27	35	26	12	12	55	47	44	48	48	38	61	51	38	34	18	36	34	38	18	28	18	18
F3	41	30	42	35	16	16	49	34	34	38	39	49	34	53	34	14	18	18	36	18	18	16	18	18
F4	45	30	52	52	16	15	43	31	34	37	38	33	43	48	36	14	18	36	34	32	36	30	36	18
F5	45	30	40	43	16	14	50	31	35	33	33	56	26	45	34	18	18	32	28	38	36	36	28	18

Table 13: Exercise Repetitions in the test set of each fold for EHE, KIMORE and UI-PRMD datasets. E and Es represent Exercise while R and L are right and left respectively.

Methods	EHE				UI-PRMD
	Sigmoid		Softmax		Sigmoid		Softmax
	$\mathbf{E_{D}}$ $\Downarrow$	$\mathbf{C_{R}}$ $\Uparrow$	$\mathbf{E_{D}}$ $\Downarrow$	$\mathbf{C_{R}}$ $\Uparrow$	$\mathbf{S_{D}}$ $\Uparrow$	$\mathbf{D_{M}}$ $\Uparrow$	$\mathbf{S_{D}}$ $\Uparrow$	$\mathbf{D_{M}}$ $\Uparrow$
GCN [ 10 ] , [ 9 ]	1.7357	0.3458	1.5382	0.4878	0.3035	0.7736	0.4336	0.7830
KShapeNet [ 28 ]	1.7697	0.6732	1.7856	0.6625	0.5772	0.9284	0.4626	0.8895
EGCN [ 76 ]	1.5893	0.5111	1.5077	0.5531	0.4142	0.8280	0.5307	0.8494
EGCN++ [ 8 ]	1.5456	0.6254	1.4444	0.6713	0.5419	0.8702	0.6485	0.8916

Table 14:

\mathbf{E_{D}}

\mathbf{C_{R}}

\mathbf{S_{D}}

, and

\mathbf{D_{M}}

results for EHE and UI-PRMD datasets, with comparison to other state-of-the-art methods. Best results are in bold, second best underlined.

Dataset	Exercise ID						Average
	E1	E2	E3	E4	E5	E6
EHE	89.51	88.52	88.63	83.60	90.36	82.24	87.14

Table 15: Performance results (% accuracy per exercise) of the proposed E2E-GNet for the EHE Dataset.

Dataset	Exercise ID								Average
	Es1	Es2(L)	Es2(R)	Es3(L)	Es3(R)	Es4(L)	Es4(R)	Es5
KIMORE	86.97	81.61	88.06	87.38	90.33	84.79	84.11	84.21	85.93

Table 16: Performance results (% accuracy per exercise) of the proposed E2E-GNet for the KIMORE Dataset.

Dataset	Exercise ID										Average
	E1	E2	E3	E4	E5	E6	E7	E8	E9	E10
UI-PRMD	93.94	95.88	94.56	100	100	100	93.33	93.18	90.56	90.40	95.19

Table 17: Performance results (% accuracy per exercise) of the proposed E2E-GNet for the UI-PRMD Dataset.

Explore similar work

An Elastic Shape Variational Autoencoder for Skeleton Pose Trajectories
May 10, 2026Arafat Rahman, Shashwat Kumar, Laura E. Barnes +1Variational Autoencoders Skeleton-Based Action Recognition
Systems and Information Engineering, University of Virginia, Charlottesville, VA, USA · Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA · Dept. of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD, USA
MHRGait: Gait Recognition from Momentum Human Rig Pose
Jul 31, 2026Huiran Duan, Qian Zhou, Xianda Guo +4Gait Recognition
City University of New York · School of Computer Science, Wuhan University · ELLIS Institute Finland +1
Retargeting Motions to Diverse Skeletons via Learnable Flattening
Sep 29, 2026Kia-Jüng Yang, Fabian H. Sinz, Paweł A. PierzchlewiczZero-Shot Learning Graph Representation Learning
Institute of Computer Science, University of Göttingen · Campus Institute Data Science, University Göttingen · Pantomim P.S.A

May 10, 2026J/K move · Enter open · S save

An Elastic Shape Variational Autoencoder for Skeleton Pose Trajectories

Arafat Rahman, Shashwat Kumar, Laura E. Barnes, Anuj Srivastava

Systems and Information Engineering, University of Virginia, Charlottesville, VA, USA · Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA · Dept. of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD, USA

Variational Autoencoders Skeleton-Based Action Recognition Gait Analysis Human Motion Generation Geometric Representation Learning

Open paper AskPDF arXiv

Deep generative models provide flexible frameworks for modeling complex, structured data such as images, videos, 3D objects, and texts. However, when applied to sequences of human skeletons, standard variational autoencoders (VAEs) often allocate substantial capacity to nuisance factors-such as camera orientation, subject scale, viewpoint, and execution speed-rather than the intrinsic geometry of shapes and their motion. We propose the Elastic Shape - Variational Autoencoder (ES-VAE), a geometry-aware generative model for skeletal trajectories that leverages the transported square-root velocity field (TSRVF) representation on Kendall's shape manifold. This representation inherently removes rigid translations, rotations, and global scaling of shapes, and temporal rate variability of sequences, isolating the underlying shape dynamics. The ES-VAE encoder maps skeletal sequences to a low-dimensional latent space incorporating the Riemannian logarithm map, while the decoder reconstructs sequences using the corresponding exponential map. We demonstrate the effectiveness of ES-VAE on two datasets. First, we analyze skeletal gait cycles to predict clinical mobility scores and classify subjects into healthy and post-stroke groups. Second, we evaluate action recognition on the NTU RGB+D dataset. Across both settings, ES-VAE consistently outperforms standard VAEs and a range of sequence modeling baselines, including temporal convolutional networks, transformers, and graph convolutional networks. More broadly, ES-VAE provides a principled framework for learning generative models of longitudinal data on pose shape manifolds, offering improved latent representation and downstream performance compared to existing deep learning approaches.

Abstract

Explore similar work

An Elastic Shape Variational Autoencoder for Skeleton Pose Trajectories

MHRGait: Gait Recognition from Momentum Human Rig Pose

Retargeting Motions to Diverse Skeletons via Learnable Flattening