HandReader: Advanced Techniques for Efficient Fingerspelling Recognition

Pavel Korotaev; Petr Surovtsev; Alexander Kapitanov; Karina Kvanchiani; Alexander Nagaev

cs.CVMay 15, 2025

HandReader: Advanced Techniques for Efficient Fingerspelling Recognition

Authors: Pavel Korotaev, Petr Surovtsev, Alexander Kapitanov, Karina Kvanchiani, Alexander Nagaev

Organizations: SberDevices, Russia

Abstract

Fingerspelling is a significant component of Sign Language (SL), allowing the interpretation of proper names, characterized by fast hand movements during signing. Although previous works on fingerspelling recognition have focused on processing the temporal dimension of videos, there remains room for improving the accuracy of these approaches. This paper introduces HandReader, a group of three architectures designed to address the fingerspelling recognition task. HandReader $_{RGB}$ employs the novel Temporal Shift-Adaptive Module (TSAM) to process RGB features from videos of varying lengths while preserving important sequential information. HandReader $_{KP}$ is built on the proposed Temporal Pose Encoder (TPE) operated on keypoints as tensors. Such keypoints composition in a batch allows the encoder to pass them through 2D and 3D convolution layers, utilizing temporal and spatial information and accumulating keypoints coordinates. We also introduce HandReader_RGB+KP - architecture with a joint encoder to benefit from RGB and keypoint modalities. Each HandReader model possesses distinct advantages and achieves state-of-the-art results on the ChicagoFSWild and ChicagoFSWild+ datasets. Moreover, the models demonstrate high performance on the first open dataset for Russian fingerspelling, Znaki, presented in this paper. The Znaki dataset and HandReader pre-trained models are publicly available.

Figures & tables

Figure 1: Three HandReader architectures. a) The videos with different lengths b) are processed to get crops and keypoints for HandReader RGB and HandReader KP , respectively, c) form the batches, and d) pass through the encoders. Further, e) the GRU decoder is applied to the encoder outputs. The final output of the HandReader RGB+KP is obtained by summing the outputs of TSAM and TPE. Red-colored encoders are the paper’s contribution.

Figure 2: Comparison of the TSAM architecture (outlined in red at the top) with the original TSM (outlined in gray at the bottom). The input batch includes two videos of different lengths, with specific frames highlighted in blue and yellow. While the TSM model requires fixed-length inputs utilizing padding (shown in white) or trimming frames (shown in red) when necessary, the TSAM model employs individual shift counters to manage variable-length sequences without additional padding or trimming. The shift counter displayed at the top of the TSAM block shows how the shifts are activated individually for each video, incrementing the shift counter until it reaches the length of that specific video.

Figure 3: Demonstration of the Temporal Pose Encoder operating on a single batch. a) The encoder receives an input tensor containing $N$ frames and $K$ keypoints. All videos are padded to match the length of the most extended video clip in the batch, allowing them to be stacked. b) 2D convolutions process multiple keypoints across several frames, extracting temporal features. c) Subsequently, a 3D convolution in the form of a “tube” aggregates the extracted motion information for each body part into final feature representations $F$ . d) After applying TPE, features are processed with linear layer, e) followed by MLP with convolution modules from Gulati et al. (2020) .

Dataset	Lang	Sequences	Chars	Hours	Signers	Source
ChicagoFSWild	ASL	7K	38K	2	160	Web
ChicagoFSWild+	ASL	55K	0.3M	14	260	Web

Table 1: Open fingerspelling recognition datasets used in this work.

Augmentation	Parameters	Probability	Modality
Resample	rate=(0.5, 1.5)	0.8	KP, RGB+KP
SpatialAffine	scale=(0.8, 1.2), shear=(-0.15, 0.15), shift=(-0.1, 0.1), degree=(-30, 30)	0.75	KP
TemporalMask	size=(0.2, 0.4), value=0.0	0.5	KP
SpatialMask	size=(0.05, 0.1), value=0.0, mode="relative"	0.5	KP
HorizontalFlip	—	0.5	RGB, RGB+KP, KP
Rotation	angle=(-10, 10)	0.5	RGB, RGB+KP

Table 2: Augmentations used during training.

Model	ChicagoFSWild	ChicagoFSWild+	Modality
Hand Det + CNN + RNN ( Shi et al., 2018 )	41.9	41.2	RGB
Iterative Attention ( Shi et al., 2019 )	45.1	46.7	RGB
3d hand pose + RGB ( Parelli et al., 2020 )	47.9	–	RGB + KP
SiameseNet+Iterative model ( Kumwilaisak et al., 2022 )	49.6	–	RGB
Iter train + SupCon ( Pannattee et al., 2024 )	64.9	71.7*	RGB
Pose-based transformer ( Fayyazsanavi et al., 2024 )	66.3	71.1	KP

Table 3: Results comparison on ChicagoFSWild and ChicagoFSWild+ test sets. * indicates the model was trained on both ChicagoFSWild and ChicagoFSWild+ training sets. ** means two convolution modules were used for HandReader KP encoder, while only one was employed for other metrics.

PoseBased ( Fayyazsanavi et al., 2024 ) *	KP	RGB+KP	RGB	Reference
macah	mach	mach	mach	mach
meach	meah	beack	fealc	beach
hospital	hospital	hospital	hospital	hospital
-	outube	iutube	iutube	youtube
-	organic	organic	orgadic	organic

Table 4: Results comparison of predictions between previous SotA solution and our models, predictions for PoseBased model were taken from the original paper.

Gt	KP	RGB	RGB+KP
youtube	outube	iutube	iutube
issue	isue	isue	isue
organic	or ganic	orgadic	organic
pharmacology	opharmacoint	opharoacolohy	opharmacolgi
pharmacology	harmcogiy	haroscothy	harmacotgy
physical	phisical	phylsical	phistical

Table 5: Result of comparison of predictions between different models on ChicagoFSWild dataset.

Model	D	S	I
RGB	677 / 628	462 / 478	122 / 130
KP	857 / 789	382 / 406	148 / 161
RGB+KP	683 / 621	386 / 409	160 / 164

Table 6: SDI breakdown on ChicagoFSWild. Each cell shows Greedy / Beam decoding.

Experiment	Packed sequence	RNN	Count shift	Shift type	Image Augs.	BS	Letter accuracy
best	✓	GRU	✓	TSAM	✓	4	72.0
Packed sequence	✗	GRU	✗	TSM	✓	4	$70.6_{\color[rgb]{1,0,0}-1.4}$
RNN	✓	LSTM	✓	TSAM	✓	4	$70.31_{\color[rgb]{1,0,0}-1.69}$
RNN	✓	✗	✓	TSAM	✓	4	$61.0_{\color[rgb]{1,0,0}-11.0}$
Count shift	✓	GRU	✗	TSAM	✓	4	$70.51_{\color[rgb]{1,0,0}-1.49}$
Image Augs.	✓	GRU	✓	TSAM	✗	4	$70.49_{\color[rgb]{1,0,0}-1.51}$

Table 7: Ablation study of HandReader RGB components. * denotes the baseline setup with a ResNet34 configuration (without TSM or TSAM shift operations), which already outperforms previous state-of-the-art solutions. Here, we define BS as batch size and RNN as the type of recurrent neural network used, which may include LSTM, GRU, or a linear layer without recurrence. “Packed sequence” refers to a modification of the TSM mechanism that enables processing each video independently during the shift operation, as described in Section 3.1 .

Experiment	Keypoints			RNN	Conv. modules	Image Augs.	BS	Letter accuracy
Experiment	hand	pose	face	RNN	Conv. modules	Image Augs.	BS	Letter accuracy
best	✓	✓	✗	GRU	1	✓	4	69.31
Keypoints	✓	✗	✗	GRU	1	✓	4	$68.54_{\color[rgb]{1,0,0}-0.77}$
	✓	✓	✓			✓		$69.08_{\color[rgb]{1,0,0}-0.23}$
	✓	✗	✓			✓		$67.59_{\color[rgb]{1,0,0}-1.72}$
	✗	✓	✓			✓		$10.02_{\color[rgb]{1,0,0}-59.29}$

Table 8: Ablation study of HandReader KP components. Here, “Conv Modules” denotes the number of convolutional modules from the Conformer architecture that are used inside the encoder. Other terms are defined in Table 7 .

Fingerspelling Recognition

See all 8 papers

OpenFS: Multi-Hand-Capable Fingerspelling Recognition with Implicit Signing-Hand Detection and Frame-Wise Letter-Conditioned Synthesis
Feb 26, 2026Junuk Cha, Jihyeon Kim, Han-Mu ParkFingerspelling Recognition Sign Language Recognition
KAIST · KT · KETI
Znaki: A Russian Fingerspelling Recognition Dataset
Nov 2, 2026Petr Surovtsev, Pavel Korotaev, Alexander Nagaev +1Fingerspelling Recognition Sign Language Datasets
Department of Artificial Intelligence and Machine Learning Development, DAIMLD
Recognising BSL Fingerspelling in Continuous Signing Sequences
Mar 19, 2026Alyssa Chan, Taein Kwon, Andrew ZissermanFingerspelling Recognition Sign Language Recognition
Visual Geometry Group, Department of Engineering Science, University of Oxford, UK

Feb 26, 2026J/K move · Enter open · S save

OpenFS: Multi-Hand-Capable Fingerspelling Recognition with Implicit Signing-Hand Detection and Frame-Wise Letter-Conditioned Synthesis

Junuk Cha, Jihyeon Kim, Han-Mu Park

KAIST · KT · KETI

Fingerspelling Recognition Sign Language Recognition Sign Language Generation Sign Language Datasets

Open paper AskPDF arXiv

Fingerspelling is a component of sign languages in which words are spelled out letter by letter using specific hand poses. Automatic fingerspelling recognition plays a crucial role in bridging the communication gap between Deaf and hearing communities, yet it remains challenging due to the signing-hand ambiguity issue, the lack of appropriate training losses, and the out-of-vocabulary (OOV) problem. Prior fingerspelling recognition methods rely on explicit signing-hand detection, which often leads to recognition failures, and on a connectionist temporal classification (CTC) loss, which exhibits the peaky behavior problem. To address these issues, we develop OpenFS, an open-source approach for fingerspelling recognition and synthesis. We propose a multi-hand-capable fingerspelling recognizer that supports both single- and multi-hand inputs and performs implicit signing-hand detection by incorporating a dual-level positional encoding and a signing-hand focus (SF) loss. The SF loss encourages cross-attention to focus on the signing hand, enabling implicit signing-hand detection during recognition. Furthermore, without relying on the CTC loss, we introduce a monotonic alignment (MA) loss that enforces the output letter sequence to follow the temporal order of the input pose sequence through cross-attention regularization. In addition, we propose a frame-wise letter-conditioned generator that synthesizes realistic fingerspelling pose sequences for OOV words. This generator enables the construction of a new synthetic benchmark, called FSNeo. Through comprehensive experiments, we demonstrate that our approach achieves state-of-the-art performance in recognition and validate the effectiveness of the proposed recognizer and generator. Codes and data are available in: https://github.com/AIRC-KETI/OpenFS.

Abstract

Fingerspelling Recognition

Explore similar work

OpenFS: Multi-Hand-Capable Fingerspelling Recognition with Implicit Signing-Hand Detection and Frame-Wise Letter-Conditioned Synthesis

Znaki: A Russian Fingerspelling Recognition Dataset

Recognising BSL Fingerspelling in Continuous Signing Sequences