Maswao: An automatically collected, prepared, and benchmarked dataset for the South African Sign language

Thapelo Sindane; Unarine Netshifhefhe; Vukosi Marivate; Abiodun Modupe; Herkulaas Combrink; Ketro Sithole

cs.CLNov 2, 2026

Maswao: An automatically collected, prepared, and benchmarked dataset for the South African Sign language

Authors: Thapelo Sindane, Unarine Netshifhefhe, Vukosi Marivate, Abiodun Modupe, Herkulaas Combrink, Ketro Sithole

Organizations: Data Science For Social Impact Lab Department of Computer Science University of Pretoria · Lelapa AI · interdisciplinary center for digital futures Faculty of Economic and Management Sciences University of the Free State

Abstract

Recent advancements in sign language processing can be attributed to developments in both large-scale dataset collection and algorithmic complexity in areas such as pose estimation techniques, signal processing through Transformer models, and model optimization. However, only a handful of sign languages, such as the American, British, and German, from over 200 globally recognized sign languages, reap the benefits of such advancements, while the rest lag behind. In this study, we aim to address this technological disparity by proposing a framework for the automatic collection and preparation of sign language datasets, with very minimal human-labour intervention, using the South African Sign Language (SASL) as a testbed. Specifically, we automatically collect SASL video content using public resources such as news broadcasts. Additionally, we convert their accompanying audio streams into text using available Automatic Speech Recognition (ASR) tools. Furthermore, we crop the sign language area in the videos and use gradient descent to locate valleys of non-motion to determine end-of-sentence points within long videos to create our weakly sentence-aligned parallel corpora -- the Maswao (Signs) dataset. Finally, we benchmark the weakly aligned dataset on the task of Sign-to-text Translation using Recurrent Neural Networks and Transformer models to compare their individual intricacies. Our results show that Transformer models proposed in the literature outperform recurrent baselines, with the best-performing Transformer achieving BLEU-1 to BLEU-4 scores of 13.78, 5.21, 2.52, and 1.54, respectively, as well as ChrF = 18.09 and ROUGE-L = 10.51. These results indicate that the task remains promising and warrants further investigations.

Figures & tables

Panel of Figure 1 — Figure 1: News video example with sign language interpretation example (left) with topic model for all collected videos (right).

Figure 2: Overview of the framework used to create the Maswao dataset.

Method	Total	Pred.	GT	TP	FP	FN	TN	Prec.	Rec.	F1	FPR
Maswao-Interpreter-Extractor	377	82	67	55	27	12	283	0.671	0.821	0.738	0.087

Table 2: Performance of the MediaPipe-based interpreter detection on the English subset, evaluated against the manually labelled ground truth. Total : number of English videos processed. Pred. : videos the method flagged as containing an interpreter. GT : videos that contain an interpreter according to the manually corrected ground truth. TP (true positives): flagged by the method and present in the ground truth. FP (false positives): flagged by the method but absent from the ground truth. FN (false negatives): not flagged by the method but present in the ground truth. TN (true negatives): not flagged and absent from the ground truth. Prec. :

TP/(TP+FP)

. Rec. :

TP/(TP+FN)

. F1 : harmonic mean of precision and recall. FPR : false positive rate,

FP/(FP+TN)

Panel of Figure 3 — Figure 3: Representative video frames sampled uniformly from downloaded sign language videos.

Panel of Figure 4 — Figure 4: Left: A motion-halt detection graph with gradient-informed segmentation points with sentence-size ratios enforced (S(N)), and gradient detection without sentence ratios (E(N)) breaking points. N is the number of sentences determined during the audio transcription phase. Right: dataset splitting with stratified sampling to maintain similar sentence distributions in training, development, and testing.

Panel of Figure 5 — Figure 5: BLEU Score Heatmaps: Impact of Batch Size and Learning Rate. Higher scores indicate better translation quality.

Model	DEV						Test
	B1	B2	B3	B4	CHRF	ROUGE	B1	B2	B3	B4	CHRF	ROUGE
NSLT-gru-luo	3.73	0.66	0.20	0.0	10.16	3.58	3.81	0.71	0.0	0.0	10.11	3.3
Joint SLT-1024	13.78	5.26	2.71	1.74	17.86	10.81	13.78	5.21	2.52	1.54	18.09	10.51
Joint SLT-4096	12.28	5.05	3	2.17	18.36	9.93	11.63	4.18	2.19	1.45	17.95	8.80
GloFE	12.66	3.68	1.79	1.18	–	–	12.37	3.39	1.35	0.79	–	–
Our SLT	12.93	4.92	2.68	1.79	18.33	10.03	12.86	4.67	2.28	1.46	18.25	9.64

Table 3: Translation performance on the Maswao DEV and Test sets. We report BLEU-1 to BLEU-4 (B1–B4), CHRF, and ROUGE scores for NSLT, GloFE, and Joint SLT models, and our proposed STT model. We ran each model five times with a different random seed and reported the average results.

Model	BERTScore			MetricX
	P	R	F1	Score
Our Model	12.20	14.60	13.23	21.85

Table 4: Complementary evaluation of the best-performing sign-to-text model using BERTScore and MetricX.

GT	And it was incumbent upon us to set up policies.
Our STT	This is but we do that we had the policies which we had to set up regarding the language.
GT	Do you see police visibility?
Our STT	They are still shooting on the police around here?
GT	And it is spoken by blacks in not only so - called colored townships.
Our STT	That’s the Afrikaans language has been spoken as you just heard.
GT	The 6 am Central African Time you’re watching SABC News live from Johannesburg.

Table 5: Translations from our neural network model using Maswao datasets. GT = Ground Truth, STT = System Output. Highlighted words indicate correct matches.

Sign Language Datasets

Sign language datasets provide annotated video, motion, and linguistic resources for training and evaluating recognition, translation, and generation systems.

See all 36 papers

Aug 3, 2026

TransSLR: A Lightweight Transformer for Sign Language Recognition

Automated sign language recognition for underrepresented languages remains a largely unsolved problem. Central African Sign Language (CASL) exemplifies this challenge: the CASL-W60 benchmark contains 60 isolated signs, and the best published result is 69.93%. We present TransSLR, a lightweight Temporal Transformer Encoder trained from scratch on normalized 64-frame pose sequences. By operating on geometric keypoint representations rather than raw RGB, TransSLR achieves 80.39% top-1 accuracy and 91.07% top-5 accuracy on a signer-independent evaluation split. This result is 10.46 percentage points higher than the published 69.93% result. TransSLR has 8.67 million trainable parameters. We also evaluate zero-shot transfer from a high-resource sign-language model, which achieves 0.00% exact-match accuracy under our manual gloss-matching protocol. Our experiments compare pose-based, RGB-based, and multimodal approaches to sign-language recognition for a low-resource language.

Lucia Yen Wanchi, Samuel Johnny, Victor Tolulope Olufemi +2

Carnegie Mellon University Africa

Sign Language Recognition

Ask PDF

Apr 22, 2026

SignDATA: Data Pipeline for Sign Language Translation

Sign-language datasets are difficult to preprocess consistently because they vary in annotation schema, clip timing, signer framing, and privacy constraints. Existing work usually reports downstream models, while the preprocessing pipeline that converts raw video into training-ready pose or video artifacts remains fragmented, backend-specific, and weakly documented. We present SignDATA, a config-driven preprocessing toolkit that standardizes heterogeneous sign-language corpora into comparable outputs for learning. The system supports two end-to-end recipes: a pose recipe that performs acquisition, manifesting, person localization, clipping, cropping, landmark extraction, normalization, and WebDataset export, and a video recipe that replaces pose extraction with signer-cropped video packaging. SignDATA exposes interchangeable MediaPipe and MMPose backends behind a common interface, typed job schemas, experiment-level overrides, and per-stage checkpointing with config- and manifest-aware hashes. We validate the toolkit through a research-oriented evaluation design centered on backend comparison, preprocessing ablations, and privacy-aware video generation on datasets. Our contribution is a reproducible preprocessing layer for sign-language research that makes extractor choice, normalization policy, and privacy tradeoffs explicit, configurable, and empirically comparable.Code is available at https://github.com/balaboom123/signdata-slt.

Kuanwei Chen, Tingyi Lin

Computer Science and Information Engineering National Central University, Zhongli, Taiwan · Electrical Engineering National Changhua University Of Education, Changhua, Taiwan

Sign Language Translation Training Data Curation Sign Language Datasets

Ask PDF

Mar 31, 2026

SyriSign: A Parallel Corpus for Arabic Text to Syrian Arabic Sign Language Translation

Sign language is the primary approach of communication for the Deaf and Hard-of-Hearing (DHH) community. While there are numerous benchmarks for high-resource sign languages, low-resource languages like Arabic remain underrepresented. Currently, there is no publicly available dataset for Syrian Arabic Sign Language (SyArSL). To overcome this gap, we introduce SyriSign, a dataset comprising 1500 video samples across 150 unique lexical signs, designed for text-to-SyArSL translation tasks. This work aims to reduce communication barriers in Syria, as most news are delivered in spoken or written Arabic, which is often inaccessible to the deaf community. We evaluated SyriSign using three deep learning architectures: MotionCLIP for semantic motion generation, T2M-GPT for text-conditioned motion synthesis, and SignCLIP for bilingual embedding alignment. Experimental results indicate that while generative approaches show strong potential for sign representation, the limited dataset size constrains generalization performance. We will release SyriSign publicly, hoping it serves as an initial benchmark.

Mohammad Amer Khalil, Raghad Nahas, Ahmad Nassar +1

Arab International University, Daraa, Syria

Sign Language Datasets Sign Language Translation Sign Language Generation Low-Resource Language Processing

Ask PDF

Abstract

Sign Language Datasets

Explore similar work

TransSLR: A Lightweight Transformer for Sign Language Recognition

SignDATA: Data Pipeline for Sign Language Translation

SyriSign: A Parallel Corpus for Arabic Text to Syrian Arabic Sign Language Translation