MySign: A High-Fidelity Motion-Capture Dataset for 3D Sign Generation in Bahasa Isyarat Malaysia
Authors: Jiayu Shen, Kalin Stefanov, Lay-Ki Soon, Vee Yee Chong, Alpha Agape Gopalai, KokSheik Wong
Organizations: School of Information Technology, Monash University Malaysia · Action Lab Malaysia, Monash University Malaysia · Japan Advanced Institute of Science and Technology · Jeffrey Cheah School of Medicine & Health Sciences, Monash University Malaysia · School of Engineering, Monash University Malaysia
Progress in 3D sign generation is constrained by the scarcity of high-fidelity 3D sign language datasets, a limitation that is particularly severe for under-resourced languages. We introduce MySign, the first motion-capture dataset for isolated signs in Bahasa Isyarat Malaysia (BIM). Its 5,000 samples cover 1,000 lexical items (glosses) and were produced by five Deaf native signers. Prior datasets that rely on SMPL-X parameters regressed from video provide pseudo-labels, which are often affected by depth ambiguity, inaccurate finger poses, and temporal jitter. MySign instead provides ground-truth SMPL-X sequences retargeted from recordings made with a motion-capture setup tailored for sign language; these sequences offer higher-fidelity motion, particularly in fine-grained hand and finger articulation. A controlled comparison on the same glosses and signers indicates that training on these sequences improves generation over training on such pseudo-labels. We establish MySign as a challenging benchmark for gloss-conditioned 3D sign generation: state-of-the-art models produce visually plausible outputs but fall short of the ground truth, most notably in fine-grained articulation, revealing persistent gaps in modeling articulated sign motion. In contrast, recognition baselines consistently achieve high lexical accuracy, indicating that the dataset reliably supports supervised learning. The dataset is available at https://huggingface.co/datasets/mysigner/MySign
Figures & tables
Dataset
Year
Lang.
Scale
Capture
Hand
Body
Task
Gloss
Signer
Sample
Modality
Source
Model
Video-based
CSL 500 [ 20 ]
2018
CSL
500
50
125K
RGB-D
D-skel
–
SLR/SLT
Phoenix-2014T [ 6 ]
2018
DGS
1,066
9
8,257
RGB
–
–
SLR/SLT
BosphorusSign22k [ 31 ]
2020
TİD
744
6
22,542
RGB-D
D-skel
–
SLR/SLT
AUTSL [ 42 ]
2020
TİD
226
43
38,336
RGB-D
D-skel
–
SLR/SLT
Table 1: Comparison of sign language datasets, grouped by 3D signal acquisition. Hand Source gives the origin of hand-articulation labels: Glove or Markers (direct capture), Fit (parametric fitting to RGB), RGB-kp (RGB-derived 2D/3D keypoints), or D-skel (depth-sensor skeleton). TİD: Turkish Sign Language; ČZJ: Czech Sign Language; –: not available or not applicable.
Category
Joint group
# joints
Clip rate (%)
Mean corr. (°)
Arm
Shoulders (L/R)
2
3.11
2.87
Elbows (L/R)
2
0.70
0.63
Wrists (L/R)
2
5.89
2.43
Hand
Finger MCP (proximal)
8
7.36
3.21
Finger PIP (middle)
8
13.12
3.75
Finger DIP (distal)
8
21.04
6.08
Table 2: Per-joint-group ROM clamping statistics. Clip rate : percentage of frames in which at least one Euler component of any joint in the group is actively clipped. Mean corr. : mean magnitude of the clipping correction over the affected components.
Figure 2: Overview of the lexical composition of MySign.
Method
Top-1
Top-5
ST-GCN [ 49 ]
85.70
92.76
SignBERT [ 19 ]
87.33
94.35
BEST [ 57 ]
90.25
97.33
SignBERT+ [ 18 ]
92.90
98.48
Table 3: Isolated sign language recognition accuracy (%) of representative baselines on MySign, averaged over 20 evaluation repetitions.
Method
FID ↓
Acc ↑
Div →
Multimod →
Real
0.023 ±.020
0.857 ±.002
46.881 ±.300
18.719 ±.250
ACTOR [ 34 ]
143.293 ±5.073
0.610 ±.005
36.472 ±.590
32.991 ±.320
MDM [ 45 ]
119.367 ±5.802
0.632 ±.007
35.621 ±.600
32.862 ±.380
MLD [ 9 ]
92.011 ±4.241
0.626 ±.009
36.483 ±.600
36.824 ±.420
MotionDiffuse [ 56 ]
83.564 ±4.310
0.639 ±.007
36.538 ±.480
32.076 ±.340
LS-GAN [ 2 ]
131.331 ±5.106
0.605 ±.011
36.531 ±.420
33.718 ±.340
Table 4: Gloss-conditioned isolated sign generation on MySign. ↑ / ↓ denote that higher / lower is better; → denotes that values closer to the Real reference are preferred, and ± indicates the 95% confidence interval.
Held-out signer
ST-GCN
FID ↓
Acc ↑
Top-1 (%)
wSignGen
MotionDiffuse
wSignGen
MotionDiffuse
S1
87.42
44.018 ±4.370
87.914 ±6.075
0.771 ±.027
0.649 ±.044
S2
84.18
42.698 ±2.072
85.453 ±1.540
0.747 ±.011
0.622 ±.005
S3
88.39
36.821 ±1.480
76.530 ±3.061
0.784 ±.023
0.669 ±.008
S4
83.64
47.556 ±1.991
98.611 ±4.525
0.736 ±.019
0.611 ±.011
S5
85.70
41.465 ±2.069
83.564 ±4.310
0.758 ±.013
0.639 ±.007
Table 5: Five-fold leave-one-signer-out evaluation. ± denotes the 95% CI. S5 corresponds to the predefined signer-independent split in Table 3 .
Supervision
FID to MoCap-real ↓
FID to RGB-real ↓
Acc ↑
MoCap-grounded
42.800 ±2.200
45.600 ±2.400
0.747 ±.012
RGB-fitted
51.400 ±2.700
49.300 ±2.600
0.704 ±.014
Table 6: MoCap-grounded vs. RGB-fitted supervision for wSignGen [ 13 ] . MoCap-real/RGB-real: real MoCap-grounded/RGB-fitted SMPL-X motions used as FID references.
Figure 3: Qualitative comparison on the ASL3DWord (left) and MySign (right) test sets. A→B : pretrained on A , fine-tuned on B . The panels contain different glosses and are compared only within each panel. The MySign → ASL3DWord (left) and MySign-only (right) models produce hand configurations closer to the ground truth and fewer visible finger artifacts than their counterparts, as highlighted in the boxes.
Source
ASL3DWord [ 12 ] (Test)
MySign (Test)
FID ↓
Acc ↑
Div →
Multimod →
FID ↓
Acc ↑
Div →
Multimod →
Real
0.100 ±.010
0.818 ±.005
30.599 ±.450
12.289 ±.200
0.023 ±.020
0.857 ±.002
46.881 ±.300
18.719 ±.250
ASL3DWord [ 12 ]
71.809 ±2.236
0.733 ±.019
27.811 ±.430
10.483 ±.170
65.037 ±5.015
0.581 ±.029
43.715 ±1.490
13.726 ±.315
MySign
63.831 ±3.511
0.751 ±.030
29.131 ±.605
10.726 ±.203
54.166 ±2.010
0.727 ±.021
48.299 ±.730
22.913 ±.200
Table 7: Comparison between MySign and ASL3DWord using SignAvatar [ 12 ] . Each column group is a target dataset, evaluated with its own recognizer; metrics are comparable only within a group.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Dataset recording setup.
Joint group
Median (°)
95th percentile (°)
Shoulders
3.0
8.5
Elbows
3.5
9.5
Wrists
5.0
13.0
Finger MCP
4.0
11.0
Finger PIP
5.0
13.0
Finger DIP
6.5
17.0
Appendix
Table 8: Retargeting validation. Bone-direction angle between the fused skeleton and the pre-cleaning SMPL-X output, reported per joint group over the evaluated sequences.
Scope
DIP clip rate (%)
∣Δe∣ (°)
Fingertip displacement (mm)
Median
P95
Max
Median
P95
Max
Full dataset
21.04
3.4
16.5
82
0.15
3.1
28.1
HIGH
46
5.2
25.3
62
1.9
8.4
26
Appendix
Table 9: Paired clamping ablation: geometric correction. Clip rate: as in Table 2 . ∣Δe∣ : magnitude of the Euler-angle correction per clipped DIP component. Fingertip displacement: computed with wrist-anchored hand-local forward kinematics. HIGH: top 10% of test sequences by per-sequence P95 fingertip displacement ( N=100 ), fixed before any recognition output was inspected.
Scope
N
U Top-1 (%)
H Top-1 (%)
Δ (pp)
Pred. agreement (%)
All test
1,000
85.0±0.3
85.7±0.2
+0.5
96.8
HIGH
100
82.0±0.7
83.0±0.5
+1.0
92.0
Appendix
Table 10: Paired clamping ablation: recognition with the frozen ST-GCN evaluator (Sec. 4.1 ). U: without hand clamping; H: hand-clamped; Δ = H − U. Top-1 is mean ± 95% CI over 20 evaluation repetitions. Agreement: share of sequences for which U and H receive the same predicted gloss. HIGH as in Table 9 .
Figure 5: Web interface used by domain experts for the perceptual validation study.
Figure 6: Finger joint articulation trajectories for a representative sign sequence. Gray bands mark major handshape transition intervals.
Figure 7: PCA projection of bimanual hand poses. Color denotes the percentile rank of per-frame kernel density in the 2D projection.
n
Train seq.
# subsets
FID ↓
Acc ↑
Div →
Multimod →
Real
–
–
0.023 ±.020
0.857 ±.002
46.881 ±.300
18.719 ±.250
1
1,000
4
123.503 ±6.544
0.500 ±.021
55.508 ±1.300
30.214 ±.463
2
2,000
6
82.549 ±4.416
0.629 ±.017
52.442 ±1.054
26.520 ±.382
3
3,000
4
58.536 ±3.035
0.705 ±.014
50.739 ±.904
24.426 ±.314
4
4,000
1
41.465 ±2.069
0.758 ±.013
49.407 ±.810
22.825 ±.230
Appendix
Table 11: Samples-per-gloss scaling ablation with wSignGen. n denotes the number of signers per gloss used for training; for n<4 , results are averaged over all subsets of the four training signers (4, 6, and 4 subsets). All values, including the Real row, are mean ± 95% CI over 20 repetitions.
Sign language datasets provide annotated video, motion, and linguistic resources for training and evaluating recognition, translation, and generation systems.
In this work, we focus on photorealistic sign avatar modeling, which is crucial for effective communication with the Deaf community and is characterized by complex hand gestures and nuanced facial expressions. To this end, we introduce MVSign, the first multi-view Chinese sign language dataset co-designed with Deaf experts, featuring diverse gestures and rich annotations. For precise SMPL-X annotation, we develop a hybrid fitting pipeline that produces accurate body, hand, and facial parameters and can also be applied to the monocular setting. Building on MVSign, we propose a decoupled sign avatar representation that isolates body, head, and hand components to capture complex articulations, together with a motion-aware sampling strategy to handle motion blur and balance gesture diversity. Extensive experiments demonstrate that our method achieves high-fidelity visual results on MVSign, particularly in detailed hand and facial regions, and generalizes well to in-the-wild monocular sign language videos. Project page: https://naaapi.github.io/PHOSA.
Haodong Wang, Hezhen Hu, Wengang Zhou +1
University of Science and Technology of China · University of Texas at Austin
Large-scale sign language resources usually pair RGB videos with text and support recognition, translation, and generation. However, RGB frames contain backgrounds, clothing, and other appearance information unrelated to signing motion. Representing the videos as whole-body keypoints such as DWPose removes much of this variation, produces a compact description of body, hand, and facial movement, and matches the input used by recent pose-guided video models. Existing datasets do not provide this representation at scale for in-the-wild sign language videos. We present SignVerse-2M, a multilingual dataset containing approximately two million pose clips from more than 55 sign languages. We process publicly available sign language videos with a shared DWPose pipeline, representing body, hand, and facial motion in the same keypoint format across all clips. The source videos include diverse signers, viewpoints, and recording environments. We also release the processing pipeline, task definitions, and SignDW Transformer baselines. Experiments show that the data supports multilingual text-to-pose modeling and that the generated poses can directly drive several recent pose-conditioned rendering systems.
Sen Fang, Hongbin Zhong, Yanxin Zhang +2
Rutgers University · Georgia Institute of Technology · Wisconsin-Madison
The goal of Sign Language Production (SLP) is to translate natural language text into corresponding sign language movements performed by a virtual avatar. The quality of the generated motion is typically evaluated by a motion-space Frechet distance (FID) and back-translation BLEU score on benchmarks such as How2Sign. The existing approaches can improve both metrics substantially while the underlying generator fails to represent the input sentence, a failure we call conditional collapse. In this work we propose to evaluate the generated motion at three independent levels: (tau1) initial-pose conditioning, (tau2) output diversity, and (tau3) target faithfulness. We compute these as pairwise-distance ratios using latent representations of a frozen motion autoencoder (MoAE), judged against each dataset's own GT-vs-GT reference. We evaluate diffusion and regression SLP models on How2Sign, including a re-implemented Neural Sign Actors, and show that tau3 faithfulness is never attained, while FID varies by nearly two orders of magnitude and is uncorrelated with faithfulness. We then apply the same models and diagnostic to Phoenix-2014T, CSL-Daily, and gloss-level ASL3DWord and show that faithfulness varies proportionally to how densely the training sentences cover the test conditions (conditioning coverage) and not corpus size. The smaller but densely covered Phoenix-2014T and ASL3DWord datasets generate faithful sign motion, while CSL-Daily follows the input text yet falls short of sign language faithfulness. Using the largest, open-domain How2Sign dataset, the models collapse at every scale. We introduce two measurements of the text data that rank the corpora in the order of how densely the training sentences fill the embedding space, and how close each test sentence lies to its nearest training sentence.