OpenFS: Multi-Hand-Capable Fingerspelling Recognition with Implicit Signing-Hand Detection and Frame-Wise Letter-Conditioned Synthesis

Junuk Cha; Jihyeon Kim; Han-Mu Park

cs.CVFeb 26, 2026

OpenFS: Multi-Hand-Capable Fingerspelling Recognition with Implicit Signing-Hand Detection and Frame-Wise Letter-Conditioned Synthesis

Authors: Junuk Cha, Jihyeon Kim, Han-Mu Park

Organizations: KAIST · KT · KETI

Abstract

Fingerspelling is a component of sign languages in which words are spelled out letter by letter using specific hand poses. Automatic fingerspelling recognition plays a crucial role in bridging the communication gap between Deaf and hearing communities, yet it remains challenging due to the signing-hand ambiguity issue, the lack of appropriate training losses, and the out-of-vocabulary (OOV) problem. Prior fingerspelling recognition methods rely on explicit signing-hand detection, which often leads to recognition failures, and on a connectionist temporal classification (CTC) loss, which exhibits the peaky behavior problem. To address these issues, we develop OpenFS, an open-source approach for fingerspelling recognition and synthesis. We propose a multi-hand-capable fingerspelling recognizer that supports both single- and multi-hand inputs and performs implicit signing-hand detection by incorporating a dual-level positional encoding and a signing-hand focus (SF) loss. The SF loss encourages cross-attention to focus on the signing hand, enabling implicit signing-hand detection during recognition. Furthermore, without relying on the CTC loss, we introduce a monotonic alignment (MA) loss that enforces the output letter sequence to follow the temporal order of the input pose sequence through cross-attention regularization. In addition, we propose a frame-wise letter-conditioned generator that synthesizes realistic fingerspelling pose sequences for OOV words. This generator enables the construction of a new synthetic benchmark, called FSNeo. Through comprehensive experiments, we demonstrate that our approach achieves state-of-the-art performance in recognition and validate the effectiveness of the proposed recognizer and generator. Codes and data are available in: https://github.com/AIRC-KETI/OpenFS.

Figures & tables

Figure 1: Motivation for multi-hand-capable fingerspelling recognition. The input consists of the hand pose sequence extracted from the video in which the word “nad” is fingerspelled using the right hand . Existing methods [ 55 , 57 , 13 ] rely on explicit signing-hand detection. However, they misidentify the signing hand when the non-signing hand exhibits large motion changes, which subsequently causes recognition failures. In contrast, our multi-hand-capable fingerspelling recognizer implicitly detects the signing hand from the multi-hand pose sequence to infer the target word. As evidence, the cross-attention map presents high attention values on the frames of the right hand when predicting the word “nad” .

Figure 2: Overview of the multi-hand-capable fingerspelling recognizer. The hand pose sequence is embedded into a feature space and encoded using our proposed dual-level positional encoding, which consists of hand-identity encoding (

\tau

) and temporal positional encoding (

\eta

). The recognizer’s decoder then predicts the next letter token based on the pose-aware, semantically rich encoder features.

\psi

denotes the standard positional encoding [ 65 ] , and

W_{i}

represents the

i

-th letter of the word. <start> and <end> are special tokens indicating the start and end of the letter token sequence, respectively.

Figure 3: Overview of the signing-hand (SF) and monotonic alignment (MA) losses. (a) The signing-hand focus (SF) loss $\cal{L}_{\text{SF}}$ measures the entropy of the hand-wise attention distribution derived from the cross-attention map between input hand pose tokens and output letter tokens. Minimizing this entropy encourages the recognizer to focus on the single signing hand. (b) The monotonic alignment (MA) loss $\cal{L}_{\text{MA}}$ penalizes misalignments that violate the natural temporal order between input hand pose tokens and output letter tokens in fingerspelling. Reducing these violations encourages the model to interpret the hand pose tokens in a temporally coherent manner to predict the letter token.

Figure 4: Overview of the coarse-to-fine frame-wise letter annotation method. (a) We utilize cross-attention map between input hand pose tokens and output letter tokens to generate coarse frame-wise letter annotations, where $\phi$ denotes a non-letter annotation. (b) To refine the coarse frame-wise letter annotations, we freeze the pre-trained recognizer and train a frame-wise annotation refiner supervised by the coarse frame-wise letter annotations. (c) The trained frame-wise annotation refiner produces refined frame-wise letter annotations. The coarse and refined annotations are compared with the corresponding image frames, where each label–frame pair is linked with arrows, and mismatched cases are highlighted in red.

Figure 5: Overview of the frame-wise letter-conditioned generator.

W_{i}

is the

i

-th letter of the word,

|W|

is the word length,

\otimes

denotes concatenation, and

\psi

denotes the standard positional encoding [ 65 ] . The generator embeds each letter token and each noised pose vector through their respective embedding layers. The resulting letter and pose embeddings are concatenated frame-wise and, given a diffusion timestep, are denoised by the generator encoder to produce a clean hand-pose sequence.

Method	CFSW [ 55 ]	CFSWP [ 57 ]	FSNeo
Shi et al . [ 55 ]	57.5	58.3	-
Shi et al . [ 57 ]	61.2	62.3	-
FSS-Net [ 54 ]	52.5	64.4	-
PoseNet [ 13 ]	61.6	61.0	61.2
Ours	75.4	70.5	80.5
PoseNet † [ 13 ]	69.2( +7.6 )	69.4( +8.4 )	94.9( +33.7 )

Table 1: We evaluate our model on CFSW [ 55 ] , CFSWP [ 57 ] and FSNeo, in terms of letter accuracy, comparing it with both RGB-based approaches [ 55 , 57 , 54 ] and the pose-based approach, PoseNet [ 13 ] . Parenthesized numbers indicate the accuracy improvements achieved by using the additional synthetic data † .

Method	Fair Acc.	IV Acc.	OOV Acc.
PoseNet [ 13 ]	66.2	65.0	58.8
Ours	76.0	80.1	71.8
PoseNet † [ 13 ]	74.5( +8.3 )	70.9( +5.9 )	67.9( +9.1 )
Ours †	77.9 (+1.9)	84.5 (+4.4)	72.5 (+0.7)

Table 2: On the ChicagoFSWild [ 55 ] dataset, Fair Acc. denotes letter accuracy on samples where PoseNet [ 13 ] successfully detects the signing hand. IV Acc. denotes letter accuracy on in-word samples, while OOV Acc. denotes letter accuracy on out-of-vocabulary samples. Parenthesized numbers indicate the accuracy improvements achieved by using the additional synthetic data † .

Method	Accuracy	Method	Accuracy
PoseNet [ 13 ]	90.4	Ours/Ours †	99.9

Table 3: Comparison of signing-hand detection accuracy on the ChicagoFSWild [ 55 ] . The symbol

\dagger

denotes methods trained with additional synthetic data.

Figure 6: Top: image frames corresponding to the letters “A”, “S”, and “L”. Bottom: frame-wise similarity maps of encoder features for PoseNet [ 13 ] and our recognizer on the “ASL” sample, presenting how similarly each encoder represents hand poses across frames. Blue, red, and green borders indicate frames with similar features to the letters “A”, “S”, and “L”, respectively.

Method	CFSW [ 55 ]
w standard pos. enc. & w/o $\cal{L}_{\text{aux}}$	73.2
w standard pos. enc. & w $\cal{L}_{\text{aux}}$	73.1
w dual-level pos. enc. & w/o $\cal{L}_{\text{aux}}$	74.8
w dual-level pos. enc. & w $\cal{L}_{\text{aux}}$	75.4

Table 4: Ablation study on the effects of positional encoding (pos. enc.) and auxiliary loss (

\mathcal{L}_{\text{aux}}

) on letter accuracy. “w” and “w/o” denote “with” and “without,” respectively. The standard pos. enc. refers to the standard sinusoidal encoding [ 65 ] , while the dual-level pos. enc. represents our proposed encoding that captures both temporal and hand identity information. The last row corresponds to our full model (Ours).

Generator condition	PoseNet [ 13 ]	Our recognizer
WC [ 50 ]	19.9	23.3
LC	26.4	40.2
FWLC (Ours)	63.5	82.3

Table 5: Evaluation of different conditioning strategies in the generator. The results show the letter accuracy of each recognizer on the generated fingerspelling pose sequences based on the unique test-set words in CFSW [ 55 ] and CFSWP [ 57 ] . WC, LC, and FWLC denote word-conditioned, letter-conditioned, and frame-wise letter-conditioned, respectively.

Method	BS	$t_{lat}$ ( $\downarrow$ )	$R_{tp}$ ( $\uparrow$ )	$R_{lps}$ ( $\uparrow$ )	FPS ( $\uparrow$ )
PoseNet [ 13 ]	1	4,282	0.2	1	6
Ours	1	39	22.0	106	962
Ours	32	6	149.8	725	6,356

Table 6: Latency, throughput and letters per second comparison measured on 868 samples from CFSW [ 55 ] . BS denotes batch size.

Figure 7: Generated fingerspelling pose sequences for the prompt “denver” under FWLC (frame-wise letter-conditioned), LC (letter-conditioned), and WC (word-conditioned) [ 50 ] . The prediction results from our recognizer are shown below each generated sequence. Incorrect poses are highlighted by red dashed rectangles, while incorrect predictions are shown in red.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Recognizer
Architecture	Transformer encoder–decoder
Layers	$L_{\text{enc}}/L_{\text{dec}}=3/3$
Hidden dimension	256
Feed-forward dimension	2,048
Attention heads	8
Activation	GELU

Appendix

Table S7 : Implementation details of the Transformer and training configurations for the recognizer.

Encoder: Pose embedding
Layer	Input Dimension	Output Dimension
Linear	pose_dim	$256$
LayerNorm	$256$	$256$
ReLU	$256$	$256$
Linear	$256$	$256$
LayerNorm	$256$	$256$

Appendix

Table S8 : Architectures of the encoder-side pose embedding, decoder-side character embedding, and decoder head used in the recognizer . pose_dim corresponds to

21~(\text{joints})\times 2~(x,y)=42

. We set char_size = 33, and the additional index is reserved for padding.

Component	Generator (DDIM)
Architecture	Transformer encoder
Layers	$L=8$
Hidden dimension	256
Feed-forward dimension	1,024
Attention heads	4
Activation	GELU

Appendix

Table S9 : Implementation details of the Transformer architecture and training configurations for the generator, and the diffusion process.

Time embedding
Layer	Input Dimension	Output Dimension
Sinusoidal	1	256
Linear	256	256
SiLU	256	256
Linear	256	256
Pose embedding

Appendix

Table S10 : Architectures of the embedding layers and output head used in the generator . We use char_size = 33, and the additional index is reserved for padding. pose_dim corresponds to

21~(\text{joints})\times 3~(x,y,z)=63

Layer	Input Dimension	Output Dimension
Linear	256	256
LayerNorm	256	256
ReLU	256	256
Dropout ( $p=0.1$ )	256	256
Linear	256	char_size

Appendix

Table S11 : Architecture of the frame-wise annotation refiner. We set char_size = 33.

Coordinate Dim.	Concatenation	Letter Accuracy
3D	frame-wise	73.4
2D	joint-wise	71.6
2D	frame-wise	75.4

Appendix

Table S12 : Ablation on input pose representations on CFSW [ 55 ] , comparing coordinate dimension (2D vs 3D) and concatenation strategy (frame-wise vs joint-wise).

Method	FID ( $\downarrow$ )	Diversity ( $\rightarrow$ )
Real	-	1.1107 ± 0.0115
WC	0.4036 ± 0.0408	0.7803 ± 0.0015
LC	0.4159 ± 0.0516	0.8987 ± 0.0009
FWLC	0.3695 ± 0.0522	0.8996 ± 0.0006

Appendix

Table S13 : FID and Diversity results across different conditioning methods. WC denotes word-conditioned generation, LC denotes letter-conditioned generation, and FWLC denotes frame-wise letter-conditioned generation. Lower FID indicates better fidelity, and Diversity results are better when the values are closer to the real data. All evaluations are repeated 20 times, and

\pm

indicates the 95% confidence interval.

Figure S8 : Comparison of the three conditioning strategies. We omit the details of standard positional encoding [ 65 ] and the embedding layers. For illustration, we consider a word of length 4. The symbol

\mathbf{w}

denotes the word-level textual embedding,

\mathbf{w}_{i}

denotes the embedding of the

i

-th letter in the word, and

\mathbf{p}_{t}^{k}

denotes a pose embedding token, where

t

is the frame index and

k

is the diffusion timestep (

k=0

indicates a clean pose). The diffusion timestep embedding vector is represented as

\mathbf{k}

. (a) WC uses a single word-level textual embedding token as input. (b) LC takes a sequence of letter tokens, with one token corresponding to each character. (c) FWLC aligns each pose embedding token with its corresponding letter embedding token and concatenates them in a one-to-one, frame-wise manner. The different conditioning inputs are highlighted with red rectangles.

Method	Model Size (MB)	Params. (M)	FPS
PoseNet [ 13 ]	33.5	8.78	6
MHC recognizer	33.7	8.81	962
FWLC generator	24.8	6.48	96

Appendix

Table S14 : Model size, number of parameters, and inference speed (FPS) of PoseNet [ 13 ] and our models, grouped into the multi-hand-capable (MHC) recognizer and the Frame-Wise Letter-Conditioned (FWLC) generator.

Method	Deletions	Substitutions	Insertions
PoseNet [ 13 ]	962(21.8%)	502(11.4%)	235(5.3%)
Ours	458(10.4%)	390(8.8%)	239(5.4%)
PoseNet † [ 13 ]	850(19.2%)	339(7.7%)	173(3.9%)
Ours †	336(7.6%)	344(7.8%)	304(6.9%)

Appendix

Table S15 : Comparison of deletion, substitution, and insertion error counts and their corresponding rates across different methods on CFSW [ 55 ] . The symbol

{\dagger}

denotes methods trained with additional synthetic data.

Method	1-4	5+	Diff.
PoseNet [ 13 ]	65.6	59.4	6.2
Ours	79.2	73.4	5.8
PoseNet † [ 13 ]	69.2	68.8	0.4
Ours †	80.0	76.6	3.4

Appendix

Table S16 : Letter accuracy across word-length bins (1–4 and 5+) on CFSW [ 55 ] . Diff. denotes the accuracy drop between short (1–4) and long (5+) words. Both PoseNet [ 13 ] and our model show accuracy drops on longer words, while their synthetic-data variants † markedly reduce this gap.

Method	CFSW	CFSWP	FSNeo
PoseNet [ 13 ]	28.8	25.7	4.7
Ours	44.6	40.4	24.1
PoseNet † [ 13 ]	36.9( +8.1 )	36.0( +10.3 )	64.2( +59.5 )
Ours †	50.5 (+5.9)	48.5 (+8.1)	82.0 (+57.9)

Appendix

Table S17 : We compare PoseNet [ 13 ] , Ours, and their variants † trained with additional synthetic data on CFSW [ 55 ] , CFSWP [ 57 ] and FSNeo, using top-1 accuracy as the evaluation metric. Parenthesized numbers indicate the accuracy improvements achieved by using the additional synthetic data † .

Method	Letter Acc.	Top-1 Acc.	Params. (M)
ByT5-s [ 15 , 66 ]	66.2	17.9	300M
ByT5-p [ 15 , 66 ]	88.9	52.9	300M
Ours	93.7	59.4	8.81M

Appendix

Table S18 : On the FSboard dataset [ 15 ] , our multi-hand-capable recognizer substantially outperforms ByT5-based methods [ 15 , 66 ] while using 34

\times

fewer parameters. ByT5-s and ByT5-p denote the ByT5 model trained from scratch and the pretrained variant, respectively.

Figure S9 : The only failure case of implicit signing-hand detection. Although the signer uses the right hand for fingerspelling, the short video and motion blur in early frames cause the model to incorrectly identify the left hand as the signing hand. The ground-truth is “so” with the right hand, whereas the prediction is “of” with the left hand.

Figure S10 : Robustness to pose noise. We increment the noise factor by 0.01 at each step. Ours ${}^{\text{SH}}$ refers to the variant that first identifies the signing hand via cross-attention and then performs recognition using only the selected hand. This single-hand variant maintains substantially higher robustness than both our full model (Ours) and PoseNet across all noise factors.

Figure S11 : Qualitative recognition results on ChicagoFSWild [ 55 ] . For each example, we show the input frames, the ground-truth letters, and the predictions from PoseNet, PoseNet † , Ours, and Ours † . The symbol $\dagger$ denotes models trained with additional synthetic data generated by our frame-wise letter-conditioned generator. We also report the letter accuracy for each prediction. Colored blocks indicate different types of prediction outcomes: blue for ground-truth letters, green for correct predictions, purple for substitution errors, red for deletion errors, and yellow for insertion errors.

Figure S12 : Qualitative recognition results on ChicagoFSWild [ 55 ] . For each example, we show the input frames, the ground-truth letters, and the predictions from PoseNet, PoseNet † , Ours, and Ours † . The symbol $\dagger$ denotes models trained with additional synthetic data generated by our frame-wise letter-conditioned generator. We also report the letter accuracy for each prediction. Colored blocks indicate different types of prediction outcomes: blue for ground-truth letters, green for correct predictions, purple for substitution errors, red for deletion errors, and yellow for insertion errors. The symbol ^ denotes a space character.

Figure S13 : Qualitative recognition results on ChicagoFSWild [ 55 ] . For each example, we show the input frames, the ground-truth letters, and the predictions from PoseNet, PoseNet † , Ours, and Ours † . The symbol $\dagger$ denotes models trained with additional synthetic data generated by our frame-wise letter-conditioned generator. We also report the letter accuracy for each prediction. Colored blocks indicate different types of prediction outcomes: blue for ground-truth letters, green for correct predictions, purple for substitution errors, red for deletion errors, and yellow for insertion errors. The symbol ^ denotes a space character. For the first example, earlier input frames are omitted for space.

Figure S14 : Qualitative examples of coarse-to-fine frame-wise letter annotation. For each word-level label (top: garden, middle: asl, bottom: sk), selected frames are shown with extracted hand keypoints. We compare coarse pseudo labels, fine-grained refined labels, and human annotations. The symbol $\phi$ denotes the absence of a letter label for the corresponding frame. The results show that the refined fine labels align more closely with human annotations, especially in frames where the coarse labels fail to assign a valid letter.

Sign Language Recognition

Sign language recognition uses video, images, or sensor data to identify signs, fingerspelling, and continuous signing across people and settings.

See all 56 papers

Recognising BSL Fingerspelling in Continuous Signing Sequences
Mar 19, 2026Alyssa Chan, Taein Kwon, Andrew ZissermanSign Language Recognition Sign Language Datasets
Visual Geometry Group, Department of Engineering Science, University of Oxford, UK
Zero-Shot Cross-Lingual Recognition of Sign Language Handshapes
Sep 16, 2026Marcel Granero-Moya, Carolina del Corral Farrarós, Gloria Haro +2Zero-Shot Learning Sign Language Recognition
Universitat Pompeu Fabra
A HamNoSys-Guided Dataset and Baselines for Fine-Grained Isolated Handshape Recognition in Sign Language
Aug 11, 2026Ushnish Sarkar, Suvajit Patra, Bhaswar Chattopadhyay +2Sign Language Recognition Sign Language Datasets
Computer and Informatics Group, Variable Energy Cyclotron Centre, Kolkata, 700064, India. · Homi Bhabha National Institute, Mumbai, 400094, India. · Ramakrishna Mission Vivekananda Educational and Research Institute, Belur, 711202, India.

Mar 19, 2026J/K move · Enter open · S save

Recognising BSL Fingerspelling in Continuous Signing Sequences

Alyssa Chan, Taein Kwon, Andrew Zisserman

Visual Geometry Group, Department of Engineering Science, University of Oxford, UK

Sign Language Recognition Sign Language Datasets

Open paper AskPDF arXiv

Fingerspelling is a critical component of British Sign Language (BSL), used to spell proper names, technical terms, and words that lack established lexical signs. Fingerspelling recognition is challenging due to the rapid pace of signing and common letter omissions by native signers, while existing BSL fingerspelling datasets are either small in scale or temporally and letter-wise inaccurate. In this work, we introduce a new large-scale BSL fingerspelling dataset, FS23K, constructed using an iterative annotation framework. In addition, we propose a fingerspelling recognition model that explicitly accounts for bi-manual interactions and mouthing cues. As a result, with refined annotations, our approach halves the character error rate (CER) compared to the prior state of the art on fingerspelling recognition. These findings demonstrate the effectiveness of our method and highlight its potential to support future research in sign language understanding and scalable, automated annotation pipelines. The project page can be found at https://taeinkwon.com/projects/fs23k/.

Abstract

Sign Language Recognition

Explore similar work

Recognising BSL Fingerspelling in Continuous Signing Sequences

Zero-Shot Cross-Lingual Recognition of Sign Language Handshapes

A HamNoSys-Guided Dataset and Baselines for Fine-Grained Isolated Handshape Recognition in Sign Language