MySign: A High-Fidelity Motion-Capture Dataset for 3D Sign Generation in Bahasa Isyarat Malaysia
Authors: Jiayu Shen, Kalin Stefanov, Lay-Ki Soon, Vee Yee Chong, Alpha Agape Gopalai, KokSheik Wong
Organizations: School of Information Technology, Monash University Malaysia · Action Lab Malaysia, Monash University Malaysia · Japan Advanced Institute of Science and Technology · Jeffrey Cheah School of Medicine & Health Sciences, Monash University Malaysia · School of Engineering, Monash University Malaysia
Progress in 3D sign generation is constrained by the scarcity of high-fidelity 3D sign language datasets, a limitation that is particularly severe for under-resourced languages. We introduce MySign, the first motion-capture dataset for isolated signs in Bahasa Isyarat Malaysia (BIM). Its 5,000 samples cover 1,000 lexical items (glosses) and were produced by five Deaf native signers. Prior datasets that rely on SMPL-X parameters regressed from video provide pseudo-labels, which are often affected by depth ambiguity, inaccurate finger poses, and temporal jitter. MySign instead provides ground-truth SMPL-X sequences retargeted from recordings made with a motion-capture setup tailored for sign language; these sequences offer higher-fidelity motion, particularly in fine-grained hand and finger articulation. A controlled comparison on the same glosses and signers indicates that training on these sequences improves generation over training on such pseudo-labels. We establish MySign as a challenging benchmark for gloss-conditioned 3D sign generation: state-of-the-art models produce visually plausible outputs but fall short of the ground truth, most notably in fine-grained articulation, revealing persistent gaps in modeling articulated sign motion. In contrast, recognition baselines consistently achieve high lexical accuracy, indicating that the dataset reliably supports supervised learning. The dataset is available at https://huggingface.co/datasets/mysigner/MySign
Figures & tables
Dataset
Year
Lang.
Scale
Capture
Hand
Body
Task
Gloss
Signer
Sample
Modality
Source
Model
Video-based
CSL 500 [ 20 ]
2018
CSL
500
50
125K
RGB-D
D-skel
–
SLR/SLT
Phoenix-2014T [ 6 ]
2018
DGS
1,066
9
8,257
RGB
–
–
SLR/SLT
BosphorusSign22k [ 31 ]
2020
TİD
744
6
22,542
RGB-D
D-skel
–
SLR/SLT
AUTSL [ 42 ]
2020
TİD
226
43
38,336
RGB-D
D-skel
–
SLR/SLT
Table 1: Comparison of sign language datasets, grouped by 3D signal acquisition. Hand Source gives the origin of hand-articulation labels: Glove or Markers (direct capture), Fit (parametric fitting to RGB), RGB-kp (RGB-derived 2D/3D keypoints), or D-skel (depth-sensor skeleton). TİD: Turkish Sign Language; ČZJ: Czech Sign Language; –: not available or not applicable.
Category
Joint group
# joints
Clip rate (%)
Mean corr. (°)
Arm
Shoulders (L/R)
2
3.11
2.87
Elbows (L/R)
2
0.70
0.63
Wrists (L/R)
2
5.89
2.43
Hand
Finger MCP (proximal)
8
7.36
3.21
Finger PIP (middle)
8
13.12
3.75
Finger DIP (distal)
8
21.04
6.08
Table 2: Per-joint-group ROM clamping statistics. Clip rate : percentage of frames in which at least one Euler component of any joint in the group is actively clipped. Mean corr. : mean magnitude of the clipping correction over the affected components.
Figure 2: Overview of the lexical composition of MySign.
Method
Top-1
Top-5
ST-GCN [ 49 ]
85.70
92.76
SignBERT [ 19 ]
87.33
94.35
BEST [ 57 ]
90.25
97.33
SignBERT+ [ 18 ]
92.90
98.48
Table 3: Isolated sign language recognition accuracy (%) of representative baselines on MySign, averaged over 20 evaluation repetitions.
Method
FID ↓
Acc ↑
Div →
Multimod →
Real
0.023 ±.020
0.857 ±.002
46.881 ±.300
18.719 ±.250
ACTOR [ 34 ]
143.293 ±5.073
0.610 ±.005
36.472 ±.590
32.991 ±.320
MDM [ 45 ]
119.367 ±5.802
0.632 ±.007
35.621 ±.600
32.862 ±.380
MLD [ 9 ]
92.011 ±4.241
0.626 ±.009
36.483 ±.600
36.824 ±.420
MotionDiffuse [ 56 ]
83.564 ±4.310
0.639 ±.007
36.538 ±.480
32.076 ±.340
LS-GAN [ 2 ]
131.331 ±5.106
0.605 ±.011
36.531 ±.420
33.718 ±.340
Table 4: Gloss-conditioned isolated sign generation on MySign. ↑ / ↓ denote that higher / lower is better; → denotes that values closer to the Real reference are preferred, and ± indicates the 95% confidence interval.
Held-out signer
ST-GCN
FID ↓
Acc ↑
Top-1 (%)
wSignGen
MotionDiffuse
wSignGen
MotionDiffuse
S1
87.42
44.018 ±4.370
87.914 ±6.075
0.771 ±.027
0.649 ±.044
S2
84.18
42.698 ±2.072
85.453 ±1.540
0.747 ±.011
0.622 ±.005
S3
88.39
36.821 ±1.480
76.530 ±3.061
0.784 ±.023
0.669 ±.008
S4
83.64
47.556 ±1.991
98.611 ±4.525
0.736 ±.019
0.611 ±.011
S5
85.70
41.465 ±2.069
83.564 ±4.310
0.758 ±.013
0.639 ±.007
Table 5: Five-fold leave-one-signer-out evaluation. ± denotes the 95% CI. S5 corresponds to the predefined signer-independent split in Table 3 .
Supervision
FID to MoCap-real ↓
FID to RGB-real ↓
Acc ↑
MoCap-grounded
42.800 ±2.200
45.600 ±2.400
0.747 ±.012
RGB-fitted
51.400 ±2.700
49.300 ±2.600
0.704 ±.014
Table 6: MoCap-grounded vs. RGB-fitted supervision for wSignGen [ 13 ] . MoCap-real/RGB-real: real MoCap-grounded/RGB-fitted SMPL-X motions used as FID references.
Figure 3: Qualitative comparison on the ASL3DWord (left) and MySign (right) test sets. A→B : pretrained on A , fine-tuned on B . The panels contain different glosses and are compared only within each panel. The MySign → ASL3DWord (left) and MySign-only (right) models produce hand configurations closer to the ground truth and fewer visible finger artifacts than their counterparts, as highlighted in the boxes.
Source
ASL3DWord [ 12 ] (Test)
MySign (Test)
FID ↓
Acc ↑
Div →
Multimod →
FID ↓
Acc ↑
Div →
Multimod →
Real
0.100 ±.010
0.818 ±.005
30.599 ±.450
12.289 ±.200
0.023 ±.020
0.857 ±.002
46.881 ±.300
18.719 ±.250
ASL3DWord [ 12 ]
71.809 ±2.236
0.733 ±.019
27.811 ±.430
10.483 ±.170
65.037 ±5.015
0.581 ±.029
43.715 ±1.490
13.726 ±.315
MySign
63.831 ±3.511
0.751 ±.030
29.131 ±.605
10.726 ±.203
54.166 ±2.010
0.727 ±.021
48.299 ±.730
22.913 ±.200
Table 7: Comparison between MySign and ASL3DWord using SignAvatar [ 12 ] . Each column group is a target dataset, evaluated with its own recognizer; metrics are comparable only within a group.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Dataset recording setup.
Joint group
Median (°)
95th percentile (°)
Shoulders
3.0
8.5
Elbows
3.5
9.5
Wrists
5.0
13.0
Finger MCP
4.0
11.0
Finger PIP
5.0
13.0
Finger DIP
6.5
17.0
Appendix
Table 8: Retargeting validation. Bone-direction angle between the fused skeleton and the pre-cleaning SMPL-X output, reported per joint group over the evaluated sequences.
Scope
DIP clip rate (%)
∣Δe∣ (°)
Fingertip displacement (mm)
Median
P95
Max
Median
P95
Max
Full dataset
21.04
3.4
16.5
82
0.15
3.1
28.1
HIGH
46
5.2
25.3
62
1.9
8.4
26
Appendix
Table 9: Paired clamping ablation: geometric correction. Clip rate: as in Table 2 . ∣Δe∣ : magnitude of the Euler-angle correction per clipped DIP component. Fingertip displacement: computed with wrist-anchored hand-local forward kinematics. HIGH: top 10% of test sequences by per-sequence P95 fingertip displacement ( N=100 ), fixed before any recognition output was inspected.
Scope
N
U Top-1 (%)
H Top-1 (%)
Δ (pp)
Pred. agreement (%)
All test
1,000
85.0±0.3
85.7±0.2
+0.5
96.8
HIGH
100
82.0±0.7
83.0±0.5
+1.0
92.0
Appendix
Table 10: Paired clamping ablation: recognition with the frozen ST-GCN evaluator (Sec. 4.1 ). U: without hand clamping; H: hand-clamped; Δ = H − U. Top-1 is mean ± 95% CI over 20 evaluation repetitions. Agreement: share of sequences for which U and H receive the same predicted gloss. HIGH as in Table 9 .
Figure 5: Web interface used by domain experts for the perceptual validation study.
Figure 6: Finger joint articulation trajectories for a representative sign sequence. Gray bands mark major handshape transition intervals.
Figure 7: PCA projection of bimanual hand poses. Color denotes the percentile rank of per-frame kernel density in the 2D projection.
n
Train seq.
# subsets
FID ↓
Acc ↑
Div →
Multimod →
Real
–
–
0.023 ±.020
0.857 ±.002
46.881 ±.300
18.719 ±.250
1
1,000
4
123.503 ±6.544
0.500 ±.021
55.508 ±1.300
30.214 ±.463
2
2,000
6
82.549 ±4.416
0.629 ±.017
52.442 ±1.054
26.520 ±.382
3
3,000
4
58.536 ±3.035
0.705 ±.014
50.739 ±.904
24.426 ±.314
4
4,000
1
41.465 ±2.069
0.758 ±.013
49.407 ±.810
22.825 ±.230
Appendix
Table 11: Samples-per-gloss scaling ablation with wSignGen. n denotes the number of signers per gloss used for training; for n<4 , results are averaged over all subsets of the four training signers (4, 6, and 4 subsets). All values, including the Real row, are mean ± 95% CI over 20 repetitions.
Sign language datasets provide annotated video, motion, and linguistic resources for training and evaluating recognition, translation, and generation systems.