Sparkle: A Robust and Versatile Representation for Point Cloud based Human Motion Capture

Yiming Ren; Yujing Sun; Aoru Xue; Kwok-Yan Lam; Yuexin Ma

cs.CVApr 1, 2026

Sparkle: A Robust and Versatile Representation for Point Cloud based Human Motion Capture

Authors: Yiming Ren, Yujing Sun, Aoru Xue, Kwok-Yan Lam, Yuexin Ma

Organizations: ShanghaiTech University, China · Digital Trust Centre, Nanyang Technological University, Singapore

Abstract

Point cloud-based motion capture leverages rich spatial geometry and privacy-preserving sensing, but learning robust representations from noisy, unstructured point clouds remains challenging. Existing approaches face a struggle trade-off between point-based methods (geometrically detailed but noisy) and skeleton-based ones (robust but oversimplified). We address the fundamental challenge: how to construct an effective representation for human motion capture that can balance expressiveness and robustness. In this paper, we propose Sparkle, a structured representation unifying skeletal joints and surface anchors with explicit kinematic-geometric factorization. Our framework, SparkleMotion, learns this representation through hierarchical modules embedding geometric continuity and kinematic constraints. By explicitly disentangling internal kinematic structure from external surface geometry, SparkleMotion achieves state-of-the-art performance not only in accuracy but crucially in robustness and generalization under severe domain shifts, noise, and occlusion. Extensive experiments demonstrate our superiority across diverse sensor types and challenging real-world scenarios.

Figures & tables

Figure 2: The pipeline of SparkleMotion. It can take point clouds of diverse patterns as input in different challenge scenarios, as shown on the left. SparkleMotion consists of three primary modules, the Point-aligned Skeleton Tracker, and Skeleton-guided Anchor Estimator construct the Sparkle Representation, and the Sparkle-based SMPL Solver for motion reconstruction.

Figure 3: Qualitative comparisons. Due to the natural spatial information of point clouds, we simultaneously displayed both point clouds and global human mesh to reflect the accuracy. For multi-view MoCap, we present point clouds from three perspectives and unique fusion results from multiple perspectives. The red rounded rectangle indicates where other methods did not work correctly.

Method	FreeMotion ( Ren et al., 2024b )			Sloper4D ( Dai et al., 2023 )			FreeMotion-OBJ ( Ren et al., 2024a )			NoiseMotion ( Ren et al., 2024a )
Method	J/V Err(L)	J/V Err(G)	Ang Err	J/V Err(L)	J/V Err(G)	Ang Err	J/V Err(L)	J/V Err(G)	Ang Err	J/V Err(L)	J/V Err(G)	Ang Err
LiDARCap	86.4/104.3	180.4/188.6	15.51	71.6/84.2	138.7/147.8	13.72	84.1/100.6	181.8/189.3	16.61	52.6/64.7	400.7/402.6	10.87
LiveHPS	74.7/90.8	130.4/141.1	16.96	53.4/63.2	88.4/95.9	13.08	70.7/88.4	146.8/158.0	17.81	48.4/60.4	74.7/83.8	12.19
LiveHPS++	61.9/75.3	112.1/120.4	15.40	42.7/50.6	77.0/81.7	11.92	58.1/72.5	128.60/136.94	15.85	34.0/42.8	58.5/64.5	10.63
Ours	59.0/73.2	105.1/113.9	9.66	41.4/50.5	70.9/77.1	10.48	50.8/62.7	104.1/110.5	8.49	27.8/36.3	38.8/45.8	7.57

Table 1: Evaluations on general scenarios with noisy and occlusion in 4 datasets. Notice our substantial improvement on noisy datasets, FreeMotion-OBJ and NoiseMotion.

Dataset	LiveHPS++ ( Ren et al., 2024b )			Ours
Dataset	J/V Err(L)	J/V Err(G)	Ang Err	J/V Err(L)	J/V Err(G)	Ang Err
Interhuman	41.7/62.8	55.0/73.8	18.47	30.4/39.9	40.4/48.4	6.75
Chi3D	39.8/62.8	55.0/74.4	23.28	33.7/46.4	47.3/57.8	12.09
Hi4D	47.5/71.9	67.9/88.8	25.29	38.6/52.4	54.5/66.2	13.11

Table 2: Evaluation on close-interaction datasets.

Method	GTA-Human-Point ( Cai et al., 2024 )			HuMMan-Point ( Cai et al., 2022 )
Method	J/V Err(L)	J/V Err(G)	Ang Err	J/V Err(L)	J/V Err(G)	Ang Err
VoteHMR	95.7/114.4	-/-	-	81.1/97.6	-/-	-
PointHPS	92.9/107.0	-/-	-	70.1/81.9	-/-	-
LiveHPS++	77.5/90.4	127.7/137.3	13.17	71.4/92.4	95.6/113.1	21.47
Ours	69.1/82.2	122.7/131.9	11.52	63.5/76.4	87.5/97.6	12.60

Table 3: Evaluation on cross-sensor generalization.

Datasets	FreeCap ( Xue et al., 2025 )			Ours
Datasets	J/V Err(L)	J/V Err(G)	Ang Err	J/V Err(L)	J/V Err(G)	Ang Err
FreeMotion	68.78/85.74	91.04/102.38	12.24	55.1/68.2	90.2/98.7	9.61
HuMMan-Point	72.77/87.60	88.07/98.06	13.12	61.8/74.3	83.3/93.1	12.14

Table 4: Evaluation on multi-view datasets.

Datasets	Network Structure			Point-alighed Skeleton Tracker (PST)		Skeleton-guided Anchor Estimator (SAE)		Sparkle-based SMPL Solver (SSS)		Ours
Datasets	w/o PST	w/o SAE	w/o SSS	w/o Offset	w/o OP	w/o Initialization	w/o OP	w/o Initialization	w/o OP	Ours
FreeMotion	149.2/159.6	116.0/125.3	112.6/122.4	107.2/116.7	106.3/115.4	108.3/118.4	107.8/117.0	108.8/117.9	106.3/119.6	105.1/113.9
HuMMan-Point	112.3/123.9	95.3/108.6	91.9/101.6	89.7/101.3	91.2/102.3	89.4/99.1	90.4/100.2	91.9/102.0	88.0/101.1	87.5/97.6
Interhuman	58.4/66.1	55.3/64.9	56.7/67.2	43.2/52.0	42.2/50.3	44.1/53.3	42.6/49.8	43.9/51.6	46.4/62.9	40.4/48.4
FreeMotion-MV	135.4/142.8	99.7/107.5	95.4/104.3	92.4/100.5	92.0/99.8	93.6/102.0	92.6/101.2	93.9/102.7	92.1/100.7	90.2/98.7

Table 5: Ablation studies for our network modules across multiple datasets.

Datasets	PCA Selection			Random Selection				Manual Selection			Ours
Datasets	16 Anchors	64 Anchors	96 Anchors	16 Anchors	32 Anchors	64 Anchors	96 Anchors	41 Anchors	50 Anchors	60 Anchors	Ours
FreeMotion	112.7/121.0	107.4/115.3	110.1/118.9	110.8/118.7	117.2/126.3	116.4/125.0	111.2/118.7	104.8 /114.1	105.8/114.7	107.3/114.9	105.1/ 113.9
HuMMan-Point	93.1/104.2	94.2/105.1	95.7/106.3	91.6/101.6	94.6/103.8	90.2/99.7	91.1/102.4	87.1/97.0	86.5/97.2	85.3 / 96.4	87.5/97.6
Interhuman	50.2/58.4	47.7/55.1	49.6/57.1	51.3/58.6	52.2/60.0	49.3/56.4	51.7/58.6	40.8/48.6	41.2/49.1	48.0/55.9	40.4/48.4
FreeMotion-MV	97.4/106.3	93.2/102.7	95.3/104.1	95.2/104.1	102.6/113.1	102.4/111.7	96.9/105.2	89.3/97.2	90.1/98.0	92.4/100.1	90.2/98.7

Table 6: Ablation studies for our network in different selections and numbers of surface anchors.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: The details of Point-aligned Skeleton Tracker.

Figure 5: The details of Skeleton-guided Anchor Estimator.

Figure 6: The pipeline of the multi-view SparkleMotion. Our method can take point clouds from any perspective as input to obtain optimized results, demonstrating the powerful scalability of Sparkle .

Distance Range	J/V Err(L)	J/V Err(G)	Ang Err
5m - 10m	52.6/65.4	89.9/98.3	9.01
10m - 20m	61.6/76.1	115.4/123.6	9.99
20m - 30m	63.7/79.0	114.2/124.0	10.04

Appendix

Table 7: Performance analysis under different capture distances on the FreeMotion dataset.

Occlusion Ratio	J/V Err(L)	J/V Err(G)	Ang Err
0%	59.0/73.2	105.1/113.9	9.66
30%	60.0/74.4	107.8/116.7	9.81
50%	61.7/76.5	110.8/119.8	10.07
70%	66.7/82.6	118.7/128.3	10.92
90%	97.4/121.2	163.3/179.0	14.18

Appendix

Table 8: Performance analysis under different occlusion ratios. Metrics are reported on a held-out test set with simulated occlusion.

Refinement Strategy	J/V Err(L)	J/V Err(G)	Ang Err
Direct Prediction (w/o offset)	68.1/83.5	107.2/116.7	10.23
Iterative Refinement (2 stages)	58.5/72.8	104.8/113.5	9.61
Attention-based Refinement	59.8/74.1	106.2/114.9	9.80
Residual Offset (Ours)	59.0/73.2	105.1/113.9	9.66

Appendix

Table 9: Detailed ablation of different refinement strategies for the PST module on the FreeMotion dataset.

A. Initialization Strategy Analysis
Method	J/V Err(L)	J/V Err(G)
Random Initialization	95.4/115.2	155.7/168.3
MLP-based Initialization	65.3/80.1	112.4/121.0
Linear Mapping (Ours)	59.0/73.2	105.1/113.9
B. Refinement Strategy Analysis (using Linear Mapping Init)
No Refinement (w/o op)	66.2/81.0	107.8/117.0

Appendix

Table 10: Detailed ablation of initialization and refinement strategies for the SAE module on the FreeMotion dataset.

T-Data	LiveHPS++	Ours
0%	84.0	65.2
25%	78.4	63.1
50%	68.9	61.5
75%	63.2	59.8
100%	61.9	59.0

Appendix

Table 11: Cross-dataset generalization and data efficiency.

Figure 7: More sports demo shows and comparison with optical-based MoCap System.

Method	MOTA $\uparrow$	IDF1 $\uparrow$	IDs $\downarrow$	HOTA $\uparrow$
ByteTrack	0.9485	0.7652	33	0.8451
Ours	0.9883	0.8172	18	0.9975

Appendix

Table 12: Multi-person tracking performance comparison on the FIFA dataset.

Motion Representation Learning

Motion representation learning develops structured encodings of movement for recognition, prediction, generation, and control.

See all 166 papers

DirtyMoCap: Robust Motion Capture from Unconstrained Markers
Sep 17, 2026Long Wang, Shuting Zhao, Shen Yan +5Gauss-Newton Optimization 3D Human Pose Estimation
Zhejiang University, Hangzhou, China · Westlake University, Hangzhou, China · Fudan University, Shanghai, China +3
Sen-Cap: Sensor-Flexible and Noise-Resilient Human Motion Capture via LiDAR-Camera Integration
Aug 3, 2026Aoru Xue, Yujing Sun, Yiming Ren +3Markerless Motion Capture Multimodal Robustness
ShanghaiTech University, Shanghai, China · ShanghaiTech University, China · Digital Trust Centre, Nanyang Technological University, Singapore +1
Bézier Degradation Modeling for LiDAR-based Human Motion Capture
May 19, 2026Xiaoqi An, Lin Zhao, Jun Li +2Point Cloud Learning 3D Human Pose Estimation
PCA Lab, School of Computer Science and Engineering, Nanjing University of Science and Technology · PCA Lab, School of Intelligence Science and Technology, Nanjing University

Sep 17, 2026J/K move · Enter open · S save

DirtyMoCap: Robust Motion Capture from Unconstrained Markers

Long Wang, Shuting Zhao, Shen Yan, Siyuan Yu+4

Zhejiang University, Hangzhou, China · Westlake University, Hangzhou, China · Fudan University, Shanghai, China+3

Gauss-Newton Optimization 3D Human Pose Estimation 3D Human Reconstruction

Open paper AskPDF arXiv

Optical motion capture delivers high-fidelity human motion, but its reliance on strict marker layouts and clean trajectories severely limits its real-world applicability. In practice, tracking systems frequently output unconstrained markers: sparse, noisy, and unordered point clouds with unknown or varying configurations. To bridge the gap between corrupted raw markers and parametric human models, we introduce DirtyMoCap, a robust, marker-layout-free framework. Our core insight is to map unordered marker observations to a fixed set of "proxy anchors" comprising skeletal joints and body surface points, which serve as a stable intermediate representation. We first initialize and track these anchors over long sequences using a recurrent sliding-window architecture. Then, a custom differentiable Gauss-Newton solver fits the SMPL-H model to the tracked anchors to recover full-body pose, translation, and shape. By explicitly deriving geometric residuals, our solver learns adaptive observation confidence, smoothness, and prior weights end-to-end, adapting dynamically to the reliability of the input data. Extensive experiments on diverse, noisy marker configurations demonstrate that DirtyMoCap successfully generalizes across arbitrary layouts using only a single trained model. It consistently outperforms state-of-the-art configuration-specific baselines in both joint and vertex reconstruction accuracy, while our custom CUDA solver achieves up to a 100x speedup over standard PyTorch implementations. We further apply DirtyMoCap to heterogeneous raw optical MoCap recordings of traditional Chinese martial arts, yielding a Kung Fu motion dataset of temporally coherent SMPL-H reconstructions. Code and data are available at https://wanglongzju.github.io/DirtyMoCap-Project-Page.

Abstract

Motion Representation Learning

Explore similar work

DirtyMoCap: Robust Motion Capture from Unconstrained Markers

Sen-Cap: Sensor-Flexible and Noise-Resilient Human Motion Capture via LiDAR-Camera Integration

Bézier Degradation Modeling for LiDAR-based Human Motion Capture