Fingerspelling is a critical component of British Sign Language (BSL), used to spell proper names, technical terms, and words that lack established lexical signs. Fingerspelling recognition is challenging due to the rapid pace of signing and common letter omissions by native signers, while existing BSL fingerspelling datasets are either small in scale or temporally and letter-wise inaccurate. In this work, we introduce a new large-scale BSL fingerspelling dataset, FS23K, constructed using an iterative annotation framework. In addition, we propose a fingerspelling recognition model that explicitly accounts for bi-manual interactions and mouthing cues. As a result, with refined annotations, our approach halves the character error rate (CER) compared to the prior state of the art on fingerspelling recognition. These findings demonstrate the effectiveness of our method and highlight its potential to support future research in sign language understanding and scalable, automated annotation pipelines. The project page can be found at https://taeinkwon.com/projects/fs23k/.
Figures & tables
Figure 1: BSL fingerspelling recognition. Video frames from continuous signing where a fingerspelling temporal interval is detected, and hand and lip features are used to correctly recognize the signed letters.
Figure 2: The British Sign Language (BSL) fingerspelling alphabet. Unlike many other sign languages, BSL employs bi-manual fingerspelling, which poses additional challenges for recognition due to frequent occlusions between the two hands. Note, these examples are for a left-handed signer.
Dataset
Language
Anno. Method
Tem. Boundary
GT Letters
Scale
Auslan-Daily ?
Auslan
Subtitle-aligned
Tight word-level
✗
2K clips
Bukva ?
RSL
Recording
Tight letter-level
✓
3.7K letters
One-Stage-TFS ?
TSL
Recording
Img-level
✓
7.2k letters
ChicagoFSWild ?
ASL
Manual
Tight word-level
✓
38K letters
ChicagoFSWild+ ?
ASL
Manual
Tight word-level
✓
0.3M letters
FSboard ?
ASL
Recording
Tight phrase-level
✓
3.2M letters
Table 1: Fingerspelling datasets. ‘Tem. boundary’ refers to how tightly the fingerspelling segments are localised in time. ‘GT Letters’ indicates whether the provided annotations correspond to the exact fingerspelt letters or to the associated word being fingerspelt.
Figure 3: Fingerspelling recognition network architecture. The model leverages two complementary feature modalities: lip features extracted using Auto-AVSR ? and hand features obtained from HaMeR ? . Each modality is first passed through an individual linear projection to align feature dimensions, followed by separate Transformer encoders. The encoded features are then concatenated and further processed by a Transformer encoder. Finally, a two-layer MLP predicts per-frame letter labels, which are also used as inputs to the CTC decoder. The dimensions are shown in the numbers beside the arrows. The 384 dimensional hand feature vector represents both hands.
#tem.-boundary
#words
#letters
#frames
hours
#signers
133,909
23,073
139,031
905,744
10.06
27
Table 2: Dataset statistics. FS23K provides large-scale annotations, covering a wide range of samples and diverse signers.
Figure 6Figure 7
Methods
CER on CSLR test set ↓
CER on Transpeller test set ↓
Transpeller ?
0.581
0.533
Transpeller ? fine-tuned on FS23K
0.549
0.496
Ours
0.250
0.389
Table 3: Comparison between Transpeller and our method. Our approach significantly improves performance on the CSLR and Transpeller test sets, achieving a reduction of 0.331 and 0.144 respectively in character error rate (CER).
Figure 8: Example correct fingerspelling predictions. Top: a subset of the video frames with corresponding letters. Bottom: full-word predictions from Transpeller and our model with character error rates (CER). Colours indicate correct letters ( green ), substitutions ( red ), and insertions/deletions ( blue ).
Figure 9: Examples of ground truth and model predictions highlighting where our model struggles. The figures on the right are examples where letters can be easily confused with each other due to occlusion or similar hand positions. As previously noted, ‘n’ and ‘v’ are visually similar. Many letters involve the index finger outstretched and touching the other hand – ‘l, t, a, e, i, o, u’, and therefore these tend to have lower classification accuracies. This is shown in examples two and three.
Figure 10: Confusion Matrix: The darker blue areas indicate higher accuracy for each letter class, with the diagonals representing the percentage of true positives. All the off diagonal entries are false positives. The frames on the right show the letters that are often confused. The letters ’n’ and ’v’, differ in the configuration of two fingers on the palm – the fingers are together for ’n’ and apart for ’v’. This distinction is subtle and often obscured by occlusions. The letters ’m’ and ’z’ share an outstretched palm.
Large-scale Anno.
Per-frame Loss
Weighted Sampling
Data Aug.
Average Class Acc.
CER ↓
✗
✗
✗
✗
46.92%
0.4472
✗
✗
✓
✗
50.48%
0.4490
✗
✗
✓
✓
52.61%
0.4447
✗
✓
✓
✓
53.03%
0.4396
✓
✓
✓
✓
70.65%
0.2497
Table 4: Ablation study. ‘Large-scale Anno.’ indicates whether the model is trained on FS23K, whereas the absence of ’Large-scale Anno.’ means that the model is trained only on the CSLR training dataset. This ablation study shows that our model consistently improves both CER and average class accuracy on the CSLR test set, validating the effectiveness of our design choices.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : Data Augmentation Examples. 2D keypoints with no augmentation (left) and 12 pixels (right) in horizontal and vertical directions.
Figure A2 : Visual representation of each method used to annotate letters in CSLR dataset. The word ‘mule’ is being spelt. The ‘correct’ annotation shows the actual letters being spelt in the frame with transition periods shown with a dashed line.
Figure A3 : Example visualisation on the CSLR test set. The dashed lines correspond to the frames displayed at the top of the figure. The blue lines show the ground truth. There are four true positive events with IoU≥0.5 . The bottom panel depicts the frame-wise predictions and the top panel shows the results after applying the sliding window, both in red.
Figure A4: Visualisation of an example with both CSLR and Transpeller annotations. The dashed lines correspond to the frames displayed at the top of the figure. The top panel shows the ground truth from CSLR in blue and our smoothed predictions in a red dashed line. The bottom panel shows the Transpeller automatic annotations in solid red and the ground truth from CSLR in blue. We observe that the Transpeller annotations span much wider intervals that include non-fingerspelling frames, whereas our predictions closely match the ground truth and precisely localise the frames where fingerspelling occurs.
Subset
# Entries
(1) Full noisy labels
157,182
(2) Removing non-existing videos dataset
154,781
(3) Removing entries that have no fingerspelling or multiple fingerspelling events
133,909
(3) Keeping entries from the training videos
113,100
(4) Removing missing word annotations
58,335
(5) Restricting entries with less than 0.3 CER between the associated and predicted word
23,073
Appendix
Table A1 : Statistics of the automatically annotated Transpeller dataset after successive filtering steps.
Only Lip features
Only Hand features
Combined features
CER ↓
✓
✗
✗
0.8756
✗
✓
✗
0.2945
✗
✗
✓
0.2497
Appendix
Table A2: Feature ablation study showing the improvement in CER when using combined features. Used our FS23K, per-frame loss and data augmentation.
Figure A5 : Fingerspelling detector ROC curve. The detector achieves an area under the curve (AUC) of 0.92.
Fingerspelling is a component of sign languages in which words are spelled out letter by letter using specific hand poses. Automatic fingerspelling recognition plays a crucial role in bridging the communication gap between Deaf and hearing communities, yet it remains challenging due to the signing-hand ambiguity issue, the lack of appropriate training losses, and the out-of-vocabulary (OOV) problem. Prior fingerspelling recognition methods rely on explicit signing-hand detection, which often leads to recognition failures, and on a connectionist temporal classification (CTC) loss, which exhibits the peaky behavior problem. To address these issues, we develop OpenFS, an open-source approach for fingerspelling recognition and synthesis. We propose a multi-hand-capable fingerspelling recognizer that supports both single- and multi-hand inputs and performs implicit signing-hand detection by incorporating a dual-level positional encoding and a signing-hand focus (SF) loss. The SF loss encourages cross-attention to focus on the signing hand, enabling implicit signing-hand detection during recognition. Furthermore, without relying on the CTC loss, we introduce a monotonic alignment (MA) loss that enforces the output letter sequence to follow the temporal order of the input pose sequence through cross-attention regularization. In addition, we propose a frame-wise letter-conditioned generator that synthesizes realistic fingerspelling pose sequences for OOV words. This generator enables the construction of a new synthetic benchmark, called FSNeo. Through comprehensive experiments, we demonstrate that our approach achieves state-of-the-art performance in recognition and validate the effectiveness of the proposed recognizer and generator. Codes and data are available in: https://github.com/AIRC-KETI/OpenFS.
AI-driven sign language interpretation is limited by a lack of high-quality annotated data. New datasets including ASL STEM Wiki and FLEURS-ASL contain professional interpreters and 100s of hours of data but remain only partially annotated and thus underutilized, in part due to the prohibitive costs of annotating at this scale. In this work, we develop a pseudo-annotation pipeline that takes signed video and English as input and outputs a ranked set of likely annotations, including time intervals, for glosses, fingerspelled words, and sign classifiers. Our pipeline uses sparse predictions from our fingerspelling recognizer and isolated sign recognizer (ISR), along with a K-Shot LLM approach, to estimate these annotations. In service of this pipeline, we establish simple yet effective baseline fingerspelling and ISR models, achieving state-of-the-art on FSBoard (6.7% CER) and on ASL Citizen datasets (74% top-1 accuracy). To validate and provide a gold-standard benchmark, a professional interpreter annotated nearly 500 videos from ASL STEM Wiki with sequence-level gloss labels containing glosses, classifiers, and fingerspelling signs. These human annotations and over 300 hours of pseudo-annotations are being released in supplemental material.
Sign language processing advances rapidly for high-resource languages such as American Sign Language (ASL), yet most of the world's sign languages lack the phonological annotations new methods require. We present the first zero-shot cross-lingual framework for handshape recognition, transferring from ASL to Catalan Sign Language (LSC). Our approach leverages the decomposition of handshapes into five phonological features -- selected fingers, flexion, spread, thumb position, and thumb contact -- shared across both languages, to decode LSC handshapes from predicted features via a composite phonological distance metric. We evaluate three architectures (MLP, SL-GCN, SHuBERT) trained on two ASL corpora (PopSign, Sem-Lex) against a 37-handshape, single-signer LSC benchmark. Zero-shot transfer proves viable once recording-format disparities are harmonized, reaching 80.0% phonological feature accuracy and 54.5% expected handshape accuracy. Phonological decomposition thus offers a bridge for extending sign language technologies to low-resource languages without any target-language video training labels.
Marcel Granero-Moya, Carolina del Corral Farrarós, Gloria Haro +2