cs.CVNov 8, 2026

MySign: A High-Fidelity Motion-Capture Dataset for 3D Sign Generation in Bahasa Isyarat Malaysia

Authors: Jiayu Shen, Kalin Stefanov, Lay-Ki Soon, Vee Yee Chong, Alpha Agape Gopalai, KokSheik Wong

Organizations: School of Information Technology, Monash University Malaysia · Action Lab Malaysia, Monash University Malaysia · Japan Advanced Institute of Science and Technology · Jeffrey Cheah School of Medicine & Health Sciences, Monash University Malaysia · 

Abstract

Progress in 3D sign generation is constrained by the scarcity of high-fidelity 3D sign language datasets, a limitation that is particularly severe for under-resourced languages. We introduce MySign, the first motion-capture dataset for isolated signs in Bahasa Isyarat Malaysia (BIM). Its 5,000 samples cover 1,000 lexical items (glosses) and were produced by five Deaf native signers. Prior datasets that rely on SMPL-X parameters regressed from video provide pseudo-labels, which are often affected by depth ambiguity, inaccurate finger poses, and temporal jitter. MySign instead provides ground-truth SMPL-X sequences retargeted from recordings made with a motion-capture setup tailored for sign language; these sequences offer higher-fidelity motion, particularly in fine-grained hand and finger articulation. A controlled comparison on the same glosses and signers indicates that training on these sequences improves generation over training on such pseudo-labels. We establish MySign as a challenging benchmark for gloss-conditioned 3D sign generation: state-of-the-art models produce visually plausible outputs but fall short of the ground truth, most notably in fine-grained articulation, revealing persistent gaps in modeling articulated sign motion. In contrast, recognition baselines consistently achieve high lexical accuracy, indicating that the dataset reliably supports supervised learning. The dataset is available at https://huggingface.co/datasets/mysigner/MySign

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. PHOSA: Photorealistic 3D Sign Avatar Modeling and Benchmark

    Sep 24, 2026Haodong Wang, Hezhen Hu, Wengang Zhou +13D Avatar Generation3D Human Reconstruction

  2. SignVerse-2M: A Two-Million-Clip Pose-Native Universe of 55+ Sign Languages

    May 3, 2026Sen Fang, Hongbin Zhong, Yanxin Zhang +2Sign Language RecognitionSign Language Datasets