cs.CVMar 19, 2026
SaveRecognising BSL Fingerspelling in Continuous Signing Sequences
Organizations: Visual Geometry Group, Department of Engineering Science, University of Oxford, UK
Abstract
Fingerspelling is a critical component of British Sign Language (BSL), used to spell proper names, technical terms, and words that lack established lexical signs. Fingerspelling recognition is challenging due to the rapid pace of signing and common letter omissions by native signers, while existing BSL fingerspelling datasets are either small in scale or temporally and letter-wise inaccurate. In this work, we introduce a new large-scale BSL fingerspelling dataset, FS23K, constructed using an iterative annotation framework. In addition, we propose a fingerspelling recognition model that explicitly accounts for bi-manual interactions and mouthing cues. As a result, with refined annotations, our approach halves the character error rate (CER) compared to the prior state of the art on fingerspelling recognition. These findings demonstrate the effectiveness of our method and highlight its potential to support future research in sign language understanding and scalable, automated annotation pipelines. The project page can be found at https://taeinkwon.com/projects/fs23k/.
Figures & tables
Figure 1: BSL fingerspelling recognition. Video frames from continuous signing where a fingerspelling temporal interval is detected, and hand and lip features are used to correctly recognize the signed letters.
Figure 2: The British Sign Language (BSL) fingerspelling alphabet. Unlike many other sign languages, BSL employs bi-manual fingerspelling, which poses additional challenges for recognition due to frequent occlusions between the two hands. Note, these examples are for a left-handed signer.
| Dataset | Language | Anno. Method | Tem. Boundary | GT Letters | Scale |
|---|---|---|---|---|---|
| Auslan-Daily ? | Auslan | Subtitle-aligned | Tight word-level | ✗ | 2K clips |
| Bukva ? | RSL | Recording | Tight letter-level | ✓ | 3.7K letters |
| One-Stage-TFS ? | TSL | Recording | Img-level | ✓ | 7.2k letters |
| ChicagoFSWild ? | ASL | Manual | Tight word-level | ✓ | 38K letters |
| ChicagoFSWild+ ? | ASL | Manual | Tight word-level | ✓ | 0.3M letters |
| FSboard ? | ASL | Recording | Tight phrase-level | ✓ | 3.2M letters |
Table 1: Fingerspelling datasets. ‘Tem. boundary’ refers to how tightly the fingerspelling segments are localised in time. ‘GT Letters’ indicates whether the provided annotations correspond to the exact fingerspelt letters or to the associated word being fingerspelt.
Figure 3: Fingerspelling recognition network architecture. The model leverages two complementary feature modalities: lip features extracted using Auto-AVSR ? and hand features obtained from HaMeR ? . Each modality is first passed through an individual linear projection to align feature dimensions, followed by separate Transformer encoders. The encoded features are then concatenated and further processed by a Transformer encoder. Finally, a two-layer MLP predicts per-frame letter labels, which are also used as inputs to the CTC decoder. The dimensions are shown in the numbers beside the arrows. The 384 dimensional hand feature vector represents both hands.
| #tem.-boundary | #words | #letters | #frames | hours | #signers |
|---|---|---|---|---|---|
| 133,909 | 23,073 | 139,031 | 905,744 | 10.06 | 27 |
Table 2: Dataset statistics. FS23K provides large-scale annotations, covering a wide range of samples and diverse signers.
| Methods | CER on CSLR test set | CER on Transpeller test set |
|---|---|---|
| Transpeller ? | 0.581 | 0.533 |
| Transpeller ? fine-tuned on FS23K | 0.549 | 0.496 |
| Ours | 0.250 | 0.389 |
Table 3: Comparison between Transpeller and our method. Our approach significantly improves performance on the CSLR and Transpeller test sets, achieving a reduction of 0.331 and 0.144 respectively in character error rate (CER).
Figure 8: Example correct fingerspelling predictions. Top: a subset of the video frames with corresponding letters. Bottom: full-word predictions from Transpeller and our model with character error rates (CER). Colours indicate correct letters ( green ), substitutions ( red ), and insertions/deletions ( blue ).
Figure 9: Examples of ground truth and model predictions highlighting where our model struggles. The figures on the right are examples where letters can be easily confused with each other due to occlusion or similar hand positions. As previously noted, ‘n’ and ‘v’ are visually similar. Many letters involve the index finger outstretched and touching the other hand – ‘l, t, a, e, i, o, u’, and therefore these tend to have lower classification accuracies. This is shown in examples two and three.
Figure 10: Confusion Matrix: The darker blue areas indicate higher accuracy for each letter class, with the diagonals representing the percentage of true positives. All the off diagonal entries are false positives. The frames on the right show the letters that are often confused. The letters ’n’ and ’v’, differ in the configuration of two fingers on the palm – the fingers are together for ’n’ and apart for ’v’. This distinction is subtle and often obscured by occlusions. The letters ’m’ and ’z’ share an outstretched palm.
| Large-scale Anno. | Per-frame Loss | Weighted Sampling | Data Aug. | Average Class Acc. | CER |
| ✗ | ✗ | ✗ | ✗ | 46.92% | 0.4472 |
| ✗ | ✗ | ✓ | ✗ | 50.48% | 0.4490 |
| ✗ | ✗ | ✓ | ✓ | 52.61% | 0.4447 |
| ✗ | ✓ | ✓ | ✓ | 53.03% | 0.4396 |
| ✓ | ✓ | ✓ | ✓ | 70.65% | 0.2497 |
Table 4: Ablation study. ‘Large-scale Anno.’ indicates whether the model is trained on FS23K, whereas the absence of ’Large-scale Anno.’ means that the model is trained only on the CSLR training dataset. This ablation study shows that our model consistently improves both CER and average class accuracy on the CSLR test set, validating the effectiveness of our design choices.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : Data Augmentation Examples. 2D keypoints with no augmentation (left) and 12 pixels (right) in horizontal and vertical directions.
Figure A2 : Visual representation of each method used to annotate letters in CSLR dataset. The word ‘mule’ is being spelt. The ‘correct’ annotation shows the actual letters being spelt in the frame with transition periods shown with a dashed line.
Figure A3 : Example visualisation on the CSLR test set. The dashed lines correspond to the frames displayed at the top of the figure. The blue lines show the ground truth. There are four true positive events with . The bottom panel depicts the frame-wise predictions and the top panel shows the results after applying the sliding window, both in red.
Figure A4: Visualisation of an example with both CSLR and Transpeller annotations. The dashed lines correspond to the frames displayed at the top of the figure. The top panel shows the ground truth from CSLR in blue and our smoothed predictions in a red dashed line. The bottom panel shows the Transpeller automatic annotations in solid red and the ground truth from CSLR in blue. We observe that the Transpeller annotations span much wider intervals that include non-fingerspelling frames, whereas our predictions closely match the ground truth and precisely localise the frames where fingerspelling occurs.
| Subset | # Entries |
|---|---|
| (1) Full noisy labels | 157,182 |
| (2) Removing non-existing videos dataset | 154,781 |
| (3) Removing entries that have no fingerspelling or multiple fingerspelling events | 133,909 |
| (3) Keeping entries from the training videos | 113,100 |
| (4) Removing missing word annotations | 58,335 |
| (5) Restricting entries with less than 0.3 CER between the associated and predicted word | 23,073 |
Table A1 : Statistics of the automatically annotated Transpeller dataset after successive filtering steps.
| Only Lip features | Only Hand features | Combined features | CER |
| ✓ | ✗ | ✗ | 0.8756 |
| ✗ | ✓ | ✗ | 0.2945 |
| ✗ | ✗ | ✓ | 0.2497 |
Table A2: Feature ablation study showing the improvement in CER when using combined features. Used our FS23K, per-frame loss and data augmentation.
Figure A5 : Fingerspelling detector ROC curve. The detector achieves an area under the curve (AUC) of 0.92.
More in this topic
Sign Language Recognition
Sign language recognition uses video, images, or sensor data to identify signs, fingerspelling, and continuous signing across people and settings.
See all 56 papers