Bootstrapping Sign Language Annotations with Sign Language Models

Colin Lea; Vasileios Baltatzis; Connor Gillis; Raja Kushalnagar; Lorna Quandt; Leah Findlater

cs.CVApr 8, 2026

Bootstrapping Sign Language Annotations with Sign Language Models

Authors: Colin Lea, Vasileios Baltatzis, Connor Gillis, Raja Kushalnagar, Lorna Quandt, Leah Findlater

Organizations: Apple · Gallaudet University

Abstract

AI-driven sign language interpretation is limited by a lack of high-quality annotated data. New datasets including ASL STEM Wiki and FLEURS-ASL contain professional interpreters and 100s of hours of data but remain only partially annotated and thus underutilized, in part due to the prohibitive costs of annotating at this scale. In this work, we develop a pseudo-annotation pipeline that takes signed video and English as input and outputs a ranked set of likely annotations, including time intervals, for glosses, fingerspelled words, and sign classifiers. Our pipeline uses sparse predictions from our fingerspelling recognizer and isolated sign recognizer (ISR), along with a K-Shot LLM approach, to estimate these annotations. In service of this pipeline, we establish simple yet effective baseline fingerspelling and ISR models, achieving state-of-the-art on FSBoard (6.7% CER) and on ASL Citizen datasets (74% top-1 accuracy). To validate and provide a gold-standard benchmark, a professional interpreter annotated nearly 500 videos from ASL STEM Wiki with sequence-level gloss labels containing glosses, classifiers, and fingerspelling signs. These human annotations and over 300 hours of pseudo-annotations are being released in supplemental material.

Figures & tables

Figure 1: Our pseudo-annotation pipeline takes in English text and ASL video and outputs likely gloss annotations. There are three steps: K-Shot Candidate Translation, Fingerspelling Alignment, and Sign Alignment.

Figure 2: (left) Architecture and design of the (left) Fingerspelling and (right) Isolated Sign Recognition models.

Model	Character Error Rate $\downarrow$
Kaggle Competition Winner [ 8 ]	16.4
ByT5 [ 15 ]	11.1
ByT5 with YouTube-ASL [ 42 ]	8.9*
Ours (no LLM)	7.34
Ours (with LLM)	6.75

Table 1: Character Error Rate (CER) on the FSBoard dataset. Lower is better. *=Uses 2,800 hours of pretraining

Model	R@1 $\uparrow$	R@5 $\uparrow$	R@10 $\uparrow$
Baseline [ 10 ]	0.63	0.86	0.91
SignClip [ 50 ]	0.60	0.84	0.89
SHuBERT [ 19 ]	0.65	0.87	0.91
Ours	0.69	0.89	0.92
Ours (handed)	0.74	0.91	0.94

Table 3: Isolated SL Recognition on ASL Citizen. R@k means recall for the top 1, 5, or 10 probability classes for a given video.

Panel of Figure 3 — Figure 3: Candidate forced alignment plots using our pseudo-annotation pipeline. Gray areas are fingerspelled regions as defined by detections of the first and last letter in a word. Gloss detections with a solid background are in-vocabulary and glosses that are outlined are out-of-vocabulary detections. Each colored line corresponds to the score of each gloss class. (fs) refers to a fingerspelled word.

Source	Output
English	At 23, the Kiev grid controller allowed the reactor shutdown to resume.
Annotation	FS-23 FS-KIEV SPREADSHEET CONTROL+PERSON ALLOW REACT SHUT-DOWN
Pipeline	AGE 23 FS-KIEV CONTROLLER GRID ALLOW REACTOR SHUTDOWN START+AGAIN
English	The commercial banana is an example of a sterile, seedless triploid hybrid.
Annotation	BANANA THOSE BUSINESS MAKE EXAMPLE FS-STERILE FS-TRIPLOID FS-HYBRID
Pipeline	BANANA COMMERCIAL SHOW EXAMPLE FS-STERILE SEEDLESS FS-TRIPLOID FS-HYBRID

Table 4: Examples of English input, annotated input, and the best scoring translation.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Phrase
English Reference	I will go to the store tomorrow.	BLEU	ChrF	BLEURT	Comet 22
ASL-like glosses	TOMORROW STORE TRAVEL I
ASL-like back translation	Tomorrow I will travel to the store.	27.05	59.72	0.767	0.84
SEE-like glosses	I TRAVEL STORE TOMORROW
SEE-like back translation	I will travel to the store tomorrow.	59.46	77.58	0.795	0.89
English Reference	The window was broken by the baseball.	BLEU	ChrF	BLEURT	Comet 22

Appendix

Table 5: Results across NLP metrics when back-translating an English reference with ASL-like glosses and Signed English (SEE)-like glosses. All back-translations accurately represent the English reference and thus are high quality, but scores for BLEU are very different, ChrF metrics are somewhat different, and both Comet and BLEURT are similar.

Dataset	Conformer		TCN
	All	Long	All	Long
FSBoard (CER $\downarrow$ )	4.9%	–	7.3%	–
ASL STEM Wiki (AUC $\uparrow$ )	0.70	0.79	0.80	0.89
FLEURS-ASL (AUC $\uparrow$ )	0.69	0.73	0.72	0.76

Appendix

Table 6: Fingerspelling results comparing Conformer and TCN architectures. FSBoard is in-domain (only fingerspelling); ASL STEM Wiki and FLEURS-ASL are out-of-domain (signing & fingerspelling). “Long” refers to words with

\geq

3 characters.

Model	BLEURT $\uparrow$
Gemma-3:12b	0.52
Gemma-3:27b	0.55
Claude Sonnet 4.0	0.59
Gemini-2.5-flash	0.60

Appendix

Table 7: BLEURT scores for different LLMs on gloss-to-English back-translation using ASL STEM Wiki manual annotations.

$K$	Gloss-level ChrF $\uparrow$
1	47.7
4	49.8
10	50.8

Appendix

Table 8: Effect of K-shot prompting on gloss-level ChrF, comparing manual annotations to the best-case LLM translation.

Sign Language Recognition

Sign language recognition uses video, images, or sensor data to identify signs, fingerspelling, and continuous signing across people and settings.

See all 56 papers

SignAgent: Agentic LLMs for Linguistically-Grounded Sign Language Annotation and Dataset Curation
Mar 19, 2026Oliver Cory, Ozge Mercanoglu Sincan, Richard BowdenLLM-Assisted Annotation Sign Language Datasets
Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, UK
Isolated Sign Language Recognition for Icelandic Sign Language: Experiments in a Low-resource Setting
Sep 22, 2026Finnur Ágúst Ingimundarson, Guðný Björk Þorvaldsdóttir, Mathias Müller +1Few-Shot Learning Sign Language Recognition
Department of Computational Linguistics, University of Zurich · Communication Centre for the Deaf and Hard of Hearing in Iceland
Sign Language Recognition in the Age of LLMs
Apr 13, 2026Vaclav Javorek, Jakub Honzik, Ivan Gruber +2Sign Language Recognition VLM Evaluation
University of West Bohemia, Czech Republic · Eindhoven University of Technology, The Netherlands

Mar 19, 2026J/K move · Enter open · S save

SignAgent: Agentic LLMs for Linguistically-Grounded Sign Language Annotation and Dataset Curation

Oliver Cory, Ozge Mercanoglu Sincan, Richard Bowden

Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, UK

LLM-Assisted Annotation Sign Language Datasets LLM Agents

Open paper AskPDF arXiv

This paper introduces SignAgent, a novel agentic framework that utilises Large Language Models (LLMs) for scalable, linguistically-grounded Sign Language (SL) annotation and dataset curation. Traditional computational methods for SLs often operate at the gloss level, overlooking crucial linguistic nuances, while manual linguistic annotation remains a significant bottleneck, proving too slow and expensive for the creation of large-scale, phonologically-aware datasets. SignAgent addresses these challenges through SignAgent Orchestrator, a reasoning LLM that coordinates a suite of linguistic tools, and SignGraph, a knowledge-grounded LLM that provides lexical and linguistic grounding. We evaluate our framework on two downstream annotation tasks. First, on Pseudo-gloss Annotation, where the agent performs constrained assignment, using multi-modal evidence to extract and order suitable gloss labels for signed sequences. Second, on ID Glossing, where the agent detects and refines visual clusters by reasoning over both visual similarity and phonological overlap to correctly identify and group lexical sign variants. Our results demonstrate that our agentic approach achieves strong performance for large-scale, linguistically-aware data annotation and curation.

Abstract

Sign Language Recognition

Explore similar work

SignAgent: Agentic LLMs for Linguistically-Grounded Sign Language Annotation and Dataset Curation

Isolated Sign Language Recognition for Icelandic Sign Language: Experiments in a Low-resource Setting

Sign Language Recognition in the Age of LLMs