Fingerspelling is a significant component of Sign Language (SL), allowing the interpretation of proper names, characterized by fast hand movements during signing. Although previous works on fingerspelling recognition have focused on processing the temporal dimension of videos, there remains room for improving the accuracy of these approaches. This paper introduces HandReader, a group of three architectures designed to address the fingerspelling recognition task. HandReaderRGB employs the novel Temporal Shift-Adaptive Module (TSAM) to process RGB features from videos of varying lengths while preserving important sequential information. HandReaderKP is built on the proposed Temporal Pose Encoder (TPE) operated on keypoints as tensors. Such keypoints composition in a batch allows the encoder to pass them through 2D and 3D convolution layers, utilizing temporal and spatial information and accumulating keypoints coordinates. We also introduce HandReader_RGB+KP - architecture with a joint encoder to benefit from RGB and keypoint modalities. Each HandReader model possesses distinct advantages and achieves state-of-the-art results on the ChicagoFSWild and ChicagoFSWild+ datasets. Moreover, the models demonstrate high performance on the first open dataset for Russian fingerspelling, Znaki, presented in this paper. The Znaki dataset and HandReader pre-trained models are publicly available.
Figures & tables
Figure 1: Three HandReader architectures. a) The videos with different lengths b) are processed to get crops and keypoints for HandReader RGB and HandReader KP , respectively, c) form the batches, and d) pass through the encoders. Further, e) the GRU decoder is applied to the encoder outputs. The final output of the HandReader RGB+KP is obtained by summing the outputs of TSAM and TPE. Red-colored encoders are the paper’s contribution.
Figure 2: Comparison of the TSAM architecture (outlined in red at the top) with the original TSM (outlined in gray at the bottom). The input batch includes two videos of different lengths, with specific frames highlighted in blue and yellow. While the TSM model requires fixed-length inputs utilizing padding (shown in white) or trimming frames (shown in red) when necessary, the TSAM model employs individual shift counters to manage variable-length sequences without additional padding or trimming. The shift counter displayed at the top of the TSAM block shows how the shifts are activated individually for each video, incrementing the shift counter until it reaches the length of that specific video.
Figure 3: Demonstration of the Temporal Pose Encoder operating on a single batch. a) The encoder receives an input tensor containing N frames and K keypoints. All videos are padded to match the length of the most extended video clip in the batch, allowing them to be stacked. b) 2D convolutions process multiple keypoints across several frames, extracting temporal features. c) Subsequently, a 3D convolution in the form of a “tube” aggregates the extracted motion information for each body part into final feature representations F . d) After applying TPE, features are processed with linear layer, e) followed by MLP with convolution modules from Gulati et al. (2020) .
Dataset
Lang
Sequences
Chars
Hours
Signers
Source
ChicagoFSWild
ASL
7K
38K
2
160
Web
ChicagoFSWild+
ASL
55K
0.3M
14
260
Web
Table 1: Open fingerspelling recognition datasets used in this work.
SiameseNet+Iterative model ( Kumwilaisak et al., 2022 )
49.6
–
RGB
Iter train + SupCon ( Pannattee et al., 2024 )
64.9
71.7*
RGB
Pose-based transformer ( Fayyazsanavi et al., 2024 )
66.3
71.1
KP
Table 3: Results comparison on ChicagoFSWild and ChicagoFSWild+ test sets. * indicates the model was trained on both ChicagoFSWild and ChicagoFSWild+ training sets. ** means two convolution modules were used for HandReader KP encoder, while only one was employed for other metrics.
PoseBased ( Fayyazsanavi et al., 2024 ) *
KP
RGB+KP
RGB
Reference
macah
mach
mach
mach
mach
meach
meah
beack
fealc
beach
hospital
hospital
hospital
hospital
hospital
-
outube
iutube
iutube
youtube
-
organic
organic
orgadic
organic
Table 4: Results comparison of predictions between previous SotA solution and our models, predictions for PoseBased model were taken from the original paper.
Gt
KP
RGB
RGB+KP
youtube
outube
iutube
iutube
issue
isue
isue
isue
organic
or ganic
orgadic
organic
pharmacology
opharmacoint
opharoacolohy
opharmacolgi
pharmacology
harmcogiy
haroscothy
harmacotgy
physical
phisical
phylsical
phistical
Table 5: Result of comparison of predictions between different models on ChicagoFSWild dataset.
Model
D
S
I
RGB
677 / 628
462 / 478
122 / 130
KP
857 / 789
382 / 406
148 / 161
RGB+KP
683 / 621
386 / 409
160 / 164
Table 6: SDI breakdown on ChicagoFSWild. Each cell shows Greedy / Beam decoding.
Experiment
Packed sequence
RNN
Count shift
Shift type
Image Augs.
BS
Letter accuracy
best
✓
GRU
✓
TSAM
✓
4
72.0
Packed sequence
✗
GRU
✗
TSM
✓
4
70.6_{\color[rgb]{1,0,0}-1.4}
RNN
✓
LSTM
✓
TSAM
✓
4
70.31_{\color[rgb]{1,0,0}-1.69}
✓
✗
✓
TSAM
✓
4
61.0_{\color[rgb]{1,0,0}-11.0}
Count shift
✓
GRU
✗
TSAM
✓
4
70.51_{\color[rgb]{1,0,0}-1.49}
Image Augs.
✓
GRU
✓
TSAM
✗
4
70.49_{\color[rgb]{1,0,0}-1.51}
Table 7: Ablation study of HandReader RGB components. * denotes the baseline setup with a ResNet34 configuration (without TSM or TSAM shift operations), which already outperforms previous state-of-the-art solutions. Here, we define BS as batch size and RNN as the type of recurrent neural network used, which may include LSTM, GRU, or a linear layer without recurrence. “Packed sequence” refers to a modification of the TSM mechanism that enables processing each video independently during the shift operation, as described in Section 3.1 .
Experiment
Keypoints
RNN
Conv. modules
Image Augs.
BS
Letter accuracy
hand
pose
face
best
✓
✓
✗
GRU
1
✓
4
69.31
Keypoints
✓
✗
✗
GRU
1
✓
4
68.54_{\color[rgb]{1,0,0}-0.77}
✓
✓
✓
✓
69.08_{\color[rgb]{1,0,0}-0.23}
✓
✗
✓
✓
67.59_{\color[rgb]{1,0,0}-1.72}
✗
✓
✓
✓
10.02_{\color[rgb]{1,0,0}-59.29}
Table 8: Ablation study of HandReader KP components. Here, “Conv Modules” denotes the number of convolutional modules from the Conformer architecture that are used inside the encoder. Other terms are defined in Table 7 .
Fingerspelling is a component of sign languages in which words are spelled out letter by letter using specific hand poses. Automatic fingerspelling recognition plays a crucial role in bridging the communication gap between Deaf and hearing communities, yet it remains challenging due to the signing-hand ambiguity issue, the lack of appropriate training losses, and the out-of-vocabulary (OOV) problem. Prior fingerspelling recognition methods rely on explicit signing-hand detection, which often leads to recognition failures, and on a connectionist temporal classification (CTC) loss, which exhibits the peaky behavior problem. To address these issues, we develop OpenFS, an open-source approach for fingerspelling recognition and synthesis. We propose a multi-hand-capable fingerspelling recognizer that supports both single- and multi-hand inputs and performs implicit signing-hand detection by incorporating a dual-level positional encoding and a signing-hand focus (SF) loss. The SF loss encourages cross-attention to focus on the signing hand, enabling implicit signing-hand detection during recognition. Furthermore, without relying on the CTC loss, we introduce a monotonic alignment (MA) loss that enforces the output letter sequence to follow the temporal order of the input pose sequence through cross-attention regularization. In addition, we propose a frame-wise letter-conditioned generator that synthesizes realistic fingerspelling pose sequences for OOV words. This generator enables the construction of a new synthetic benchmark, called FSNeo. Through comprehensive experiments, we demonstrate that our approach achieves state-of-the-art performance in recognition and validate the effectiveness of the proposed recognizer and generator. Codes and data are available in: https://github.com/AIRC-KETI/OpenFS.
Fingerspelling - the sequential signing of letters that compose proper names, technical terms, and out-of-vocabulary words - is an integral component of any sign language. Despite Russian Sign Language (RSL) being used by more than 150,000 native signers, no public fingerspelling dataset for RSL exists, in contrast to American, British, German, and several other sign languages. We introduce Znaki the first open RSL fingerspelling dataset: 37,252 video recordings (35.4 hours) of 1,593 unique sequences - proper names sampled from 16 domain categories - recorded by 68 native RSL signers (37 Deaf, 29 hard-of-hearing, and 2 certified interpreters) through a crowdsourcing pipeline with a two-stage signer qualification procedure and expert validation. The dataset covers all 33 Cyrillic handshapes, and its letter-frequency distribution matches the Russian National Corpus. As reference points for future research, we establish two baselines adapted from American Sign Language fingerspelling recognition: a Squeezeformer-based encoder-decoder, reaching a character-level accuracy (1−CER) of 0.932 and a sequence accuracy of 0.596, and Fingerspelling-PoseNet, reaching 0.886 and 0.545, respectively.
Petr Surovtsev, Pavel Korotaev, Alexander Nagaev +1
Department of Artificial Intelligence and Machine Learning Development, DAIMLD
Fingerspelling is a critical component of British Sign Language (BSL), used to spell proper names, technical terms, and words that lack established lexical signs. Fingerspelling recognition is challenging due to the rapid pace of signing and common letter omissions by native signers, while existing BSL fingerspelling datasets are either small in scale or temporally and letter-wise inaccurate. In this work, we introduce a new large-scale BSL fingerspelling dataset, FS23K, constructed using an iterative annotation framework. In addition, we propose a fingerspelling recognition model that explicitly accounts for bi-manual interactions and mouthing cues. As a result, with refined annotations, our approach halves the character error rate (CER) compared to the prior state of the art on fingerspelling recognition. These findings demonstrate the effectiveness of our method and highlight its potential to support future research in sign language understanding and scalable, automated annotation pipelines. The project page can be found at https://taeinkwon.com/projects/fs23k/.
Alyssa Chan, Taein Kwon, Andrew Zisserman
Visual Geometry Group, Department of Engineering Science, University of Oxford, UK