This paper investigates the recognition of the Russian fingerspelling alphabet, also known as the Russian Sign Language (RSL) dactyl. Dactyl is a component of sign languages where distinct hand movements represent individual letters of a written language. This method is used to spell words without specific signs, such as proper nouns or technical terms. The alphabet learning simulator is an essential isolated dactyl recognition application. There is a notable issue of data shortage in isolated dactyl recognition: existing Russian dactyl datasets lack subject heterogeneity, contain insufficient samples, or cover only static signs. We provide Bukva, the first full-fledged open-source video dataset for RSL dactyl recognition. It contains 3,757 videos with more than 101 samples for each RSL alphabet sign, including dynamic ones. We utilized crowdsourcing platforms to increase the subject's heterogeneity, resulting in the participation of 155 deaf and hard-of-hearing experts in the dataset creation. We use a TSM (Temporal Shift Module) block to handle static and dynamic signs effectively, achieving 83.6% top-1 accuracy with a real-time inference with CPU only. The dataset, demo code, and pre-trained models are publicly available.
Figures & tables
Figure 1: Each of the RSL alphabet’s 33 signs corresponds to a letter of the Russian language alphabet. Arrows indicate the direction of movement for 8 dynamic signs.
Dataset
Letters
Samples
Signers
Resolution
Language
Images
MU HandImages ASL, 2011 ( Barczak et al., 2011 )
26
2,425
5
crop
American
ASL Fingerspelling, 2015 ( Kang et al., 2015 )
24
24,000
5
320 × 240
American
RSL dactyl, 2019 ( Makarov et al., 2019 )
26
520
20
crop
Russian
BdSL alphabet, 2021 ( Podder et al., 2020 )
37
35,149
350
crop
Bangla
PSL Alphabet 100, 2021 ( Szyszka, 2021 )
36
3,707
–
crop
Polish
Table 1: The main parameters of widespread SL alphabets in different languages. The datasets are divided into two groups, containing predominantly images or videos. Relevant Russian SL datasets are highlighted in blue. In the Resolution column, “crop” denotes images cropped to the hand region, and “signals” denotes data glove sensor recordings rather than RGB samples. The WLASL row refers only to the alphabet-letter subset of this word-level dataset.
Figure 2: Video resolution, brightness, signers, classes, and splitting analysis. a) class-based video distribution: number of videos for each RSL alphabet letter in the train and test sets; b) brightness distribution: the central image of each video sign part is extracted, converted to grayscale, and then the average pixel brightness is calculated; c) distance distribution: the distance is approximately estimated in meters by computing the length between the signers’ left and right shoulders using MediaPipe ( Zhang et al., 2020 ) ; d) signers’ devices; e) video resolution; f) signers’ gender; g) signers’ age distribution: the age is determined by the MiVOLO model ( Kuprashevich and Tolstykh, 2023 ) .
Model
Model Size (MB)
Parameters (M)
Inference Time (ms)
top-1 acc.
ResNet-18
44.8
11.2
191
72.7
ResNet-50
94.6
23.6
333
79.2
MobileNetV2
10.3
2.3
145
83.6
MobileOne S0
4.4
1.1
94
71.3
MobileOne S1
14.1
3.5
165
75.7
MobileOne S2
23.4
5.8
232
77.2
Table 2: The models’ training results on the Bukva. We evaluate models with top-1 accuracy as the standard metric in the action recognition task. All models are tested on a MacBook M1 Pro using a single core. We measured the model’s inference 100 times with further averaging.
Figure 3: The confusion matrix for test samples using the MobileNetV2 model. Off-diagonal cells indicate misclassifications. The model confuses the classes ‘‘м’’ (“m”) and ‘‘л’’ (“l”), as their signs are nearly identical.
Figure 4: Russian sign language demo stand for education.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Example of a web page with an exam on crowdsourcing platforms translated into English.
Figure 8: All crowdworkers are required to pass an exam before getting access to all crowd stages. During data collection, signers recorded the sign from the video template with further filtering to remove low-quality videos and duplicates. Only the videos remaining after the filtering stage were verified by at least three crowdworkers to ensure their correctness compared to the template. The annotation stage was also applied only to the samples remaining after the previous stage. Three annotators are asked to mark the video’s sign start and end marks. Received marks are aggregated by the algorithm proposed in Kapitanov et al. (2023) .
Letters
Sign length
Brightness
Gender
Devices
Age
Total
Min
Max
Median
Min
Max
Median
Male
Female
Smartphone
PC
Min
Max
Median
А (A)
23
94
45.0
59
185
131
26
92
101
17
11.1
71.3
30.7
118
Б (B)
21
112
50.0
62
191
132
17
92
94
15
13.5
76.6
31.7
109
В (V)
22
105
44.0
57
192
129
21
95
95
21
10.4
71.7
32.4
116
Г (G)
25
100
43.0
69
190
125
19
92
94
18
12.7
63.9
31.4
112
Д (D)
32
105
50.0
65
190
130
20
82
84
18
11.4
66.1
31.1
102
Appendix
Table 3: Dataset statistics. “Sign length” in frames, “Brightness” in grayscale levels, “Gender”, “Devices” by records, “Age” of crowdworkers and “Total” number of videos. The Latin transliteration of each Cyrillic letter is given in brackets.
Fingerspelling - the sequential signing of letters that compose proper names, technical terms, and out-of-vocabulary words - is an integral component of any sign language. Despite Russian Sign Language (RSL) being used by more than 150,000 native signers, no public fingerspelling dataset for RSL exists, in contrast to American, British, German, and several other sign languages. We introduce Znaki the first open RSL fingerspelling dataset: 37,252 video recordings (35.4 hours) of 1,593 unique sequences - proper names sampled from 16 domain categories - recorded by 68 native RSL signers (37 Deaf, 29 hard-of-hearing, and 2 certified interpreters) through a crowdsourcing pipeline with a two-stage signer qualification procedure and expert validation. The dataset covers all 33 Cyrillic handshapes, and its letter-frequency distribution matches the Russian National Corpus. As reference points for future research, we establish two baselines adapted from American Sign Language fingerspelling recognition: a Squeezeformer-based encoder-decoder, reaching a character-level accuracy (1−CER) of 0.932 and a sequence accuracy of 0.596, and Fingerspelling-PoseNet, reaching 0.886 and 0.545, respectively.
Petr Surovtsev, Pavel Korotaev, Alexander Nagaev +1
Department of Artificial Intelligence and Machine Learning Development, DAIMLD
Effective sign language (SL) acquisition is crucial for deaf children, yet 95% are born to hearing parents who often lack proficiency in SL. SL recognition can power learning tools to help parents communicate with their children. However, Japanese Sign Language (JSL) lacks large-scale, multi-signer datasets, hindering the development of models that can generalize to new users. To address this gap, we introduce JSL-DC, the largest JSL dataset by video count, comprising 36.7K videos from 19 signers. The entire process was Deaf-centric: the lexicon comprising 270 JSL words was selected by Deaf and Coda linguists to facilitate parent-child communication, all participants were Deaf individuals who use JSL daily, and the data underwent a two-stage review process involving Deaf linguists. Moreover, we provide linguist-derived descriptions for distinguishing confusable signs. We demonstrate that the proposed model inspired by the descriptions outperforms state-of-the-art recognition methods by 9.8% on the confusable subset. The dataset, along with its linguistic description that inspires new models, will be released under a CC-BY 4.0 license to accelerate research in SL recognition.
Ken Takaki, Asuka Ando, Misa Suzuki +12
The University of Tokyo · Independent · Kwansei Gakuin University +3
We present the first experiments on isolated sign language recognition (ISLR) for Icelandic Sign Language (ÍTM). We use ÍTM SignWiki, a dataset derived from a bilingual Icelandic--ÍTM online dictionary. It is genuinely low-resource: 1,845 videos cover 849 classes, 86% of which have only two examples, making the full task effectively one-shot recognition across signers. We compare two open-source ISLR frameworks, OpenHands and SPOTER, on three tasks of increasing vocabulary size (22, 117 and 849 classes), and evaluate three pose estimators and two forms of cross-lingual transfer. With ÍTM data alone, SPOTER outperforms OpenHands on all three tasks, and MediaPipe poses give better results than AlphaPose or SDPose. Cross-lingual transfer brings the largest gains: pretraining SPOTER on American Sign Language data before finetuning on ÍTM raises accuracy by 14--24 percentage points, to 72.7%, 47.9% and 22.6% on the three tasks, and multilingual training with data from six other sign languages lifts OpenHands from 1.41% to 28.86% on the full task. Although far from practical use, the results suggest that transfer from better-resourced sign languages is promising for very low-resource ones. We release our adapted versions of both frameworks.
Finnur Ágúst Ingimundarson, Guðný Björk Þorvaldsdóttir, Mathias Müller +1
Department of Computational Linguistics, University of Zurich · Communication Centre for the Deaf and Hard of Hearing in Iceland