JSL-DC: A Word-Level Japanese Sign Language Dataset with Linguist-Derived Descriptions for Distinguishing Confusable Signs
Authors: Ken Takaki, Asuka Ando, Misa Suzuki, Uiko Yano, Masaya Tsujimoto, Bill Neubauer, Ananay Vikram Gupta, Rose Shao, +7 more
Organizations: The University of Tokyo · Independent · Kwansei Gakuin University · Georgia Institute of Technology · Keio University · Technical University of Clausthal
Effective sign language (SL) acquisition is crucial for deaf children, yet 95% are born to hearing parents who often lack proficiency in SL. SL recognition can power learning tools to help parents communicate with their children. However, Japanese Sign Language (JSL) lacks large-scale, multi-signer datasets, hindering the development of models that can generalize to new users. To address this gap, we introduce JSL-DC, the largest JSL dataset by video count, comprising 36.7K videos from 19 signers. The entire process was Deaf-centric: the lexicon comprising 270 JSL words was selected by Deaf and Coda linguists to facilitate parent-child communication, all participants were Deaf individuals who use JSL daily, and the data underwent a two-stage review process involving Deaf linguists. Moreover, we provide linguist-derived descriptions for distinguishing confusable signs. We demonstrate that the proposed model inspired by the descriptions outperforms state-of-the-art recognition methods by 9.8% on the confusable subset. The dataset, along with its linguistic description that inspires new models, will be released under a CC-BY 4.0 license to accelerate research in SL recognition.
Figures & tables
Dataset
Country
Signs
Signers
Videos
Ave. Videos/Sign
Source
Distinguish Note
Purdue RVL-SLLL [ 32 ]
USA
39
14
2.6K
67
Studio
RWTH-BOSTON 50 [ 56 ]
USA
50
3
0.5K
10
Studio
ASLLVD [ 4 ]
USA
3,000
6
9.8K
3
Studio
WLASL [ 31 ]
USA
2,000
119
21.1K
11
Web
MS-ASL [ 27 ]
USA
1,000
222
25.5K
26
Web
ASL Citizen [ 12 ]
USA
2,731
52
83.9K
31
Webcam
Table 1: Comparison of existing ISLR datasets. Distinguish Note is a description by a Deaf linguist for distinguishing confusable signs.
Figure 1: Overview of our Deaf-centric process of data collection involving deaf participants. Deaf authors D1, D2, and the Coda author C led the process.
Figure 2: Sample reference videos used in the data collection. (a) is “scary” (怖い, kowai ), and (b) is “cold” (寒い, samui ), with the same body movement but different mouth movements.)
Figure 3: Screenshot of the JSL recorder tablet application. The participant swipes to the left after finish signing each example.)
Figure 4: Frequently confused examples. (a) is “bitter (taste)” (苦い, nigai ), and was misclassified as “spicy” (辛い, karai ), which is (b). They could be classified by the mouth movements.
All (270 glosses)
Confusable (22 glosses)
Others (248 glosses)
Method
input
MACs (G)
Top-1
F1
Top-1
F1
Top-1
F1
SPOTER [ 6 ]
keypoint
0.38
78.7
78.5
51.7
50.9
81.1
80.9
PopSign ASL v2.0 [ 38 ]
keypoint
1.6
81.9
81.4
55.3
50.1
84.3
84.2
MASA [ 57 ]
keypoint
3.9
82.8
83.0
54.8
51.1
85.3
85.8
NLA-SLR [ 59 ]
keypoint
55.7
91.6
91.9
61.2
58.1
94.3
94.9
NLA-SLR + mouthing (ours)
keypoint
58.0
92.2
92.5
68.3
65.4
94.3
94.9
Table 2: Results across confusable vs. other glosses.
Figure 5: The proposed mouthing-based reranker to classify the confusable sets.
Gloss
Δ Top-1
Δ F1
Thank you for the meal. (s0006_itatakimasu)
−19.0
−3.5
Thank you for the meal. (s0033_kochisousamateshita)
+31.2
+38.5
scared (s0034_kowai_Bu_i_)
+37.5
+33.8
cold (s0036_samui_Han_i_)
+10.0
+15.5
cold (s0057_tsumetai_Leng_tai_)
+44.4
+43.9
spicy (s0024_karai_Xin_i_)
0.0
+0.8
Table 3: Change in Top-1 accuracy and F1 (percentage points) after the mouthing rerank, for confusable pairs distinguished mainly by mouthing/mouth cues.
Figure 6: A screenshot of the playable version of PopSign JSL, where (a) the user is signing “warm” (暖かい, atatakai ) and hand landmarks are recognized by MediaPipe (only the left hand is visualized). The landmarks are fed into a Bi-LSTM model trained with our dataset. (b) The bubble labeled with the recognition result is released.