SignMAE: Segmentation-Driven Self-Supervised Learning for Sign Language Recognition

Kunyuan Xie; Zhixi Cai; Kalin Stefanov

cs.CVMay 3, 2026

SignMAE: Segmentation-Driven Self-Supervised Learning for Sign Language Recognition

Authors: Kunyuan Xie, Zhixi Cai, Kalin Stefanov

Organizations: Monash University, Australia

Abstract

Subtle hand differences make sign language recognition challenging, yet many existing methods rely on encoders pretrained on generic action datasets that poorly capture such fine-grained cues. We propose a self-supervised pretraining method for sign language recognition that uses segmentation-based masking to adapt to the presence and motion of key body parts, rather than treating hand poses as static visual tokens. The resulting mask-and-reconstruct objective improves fine-grained sign representation learning. On WLASL, NMFs-CSL, and Slovo, our encoder achieves state-of-the-art performance, improving per-instance and per-class Top-1 accuracy while using fewer input frames and modalities than comparable encoders.

Explore similar work

TransSLR: A Lightweight Transformer for Sign Language Recognition
Aug 3, 2026Lucia Yen Wanchi, Samuel Johnny, Victor Tolulope Olufemi +2Sign Language Recognition 3D Encoder
SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval
Sep 3, 2026Sobhan Asasi, Ozge Mercanoglu Sincan, Richard BowdenSign Language Recognition Sign Language
A Reverse Sign Language Dictionary: Open-Vocabulary Sign Recognition from Continuous Signing via Video Captioning and Description Retrieval
Sep 3, 2026Santiago Poveda-Gutiérrez, Hideki Nakayama, Mayumi BonoSign Language Recognition Sign

Aug 3, 2026 · cs.CVJ/K move · Enter open · S save

TransSLR: A Lightweight Transformer for Sign Language Recognition

Lucia Yen Wanchi, Samuel Johnny, Victor Tolulope Olufemi, Emmanuel Aaron+1

Carnegie Mellon University Africa

Automated Sign Language Recognition for under-represented languages remains a largely unsolved problem. Central African Sign Language (CASL) exemplifies this gap: the only available bench-mark, CASL-W60, has a best reported accuracy of 69.93%, and we show that the common heuristic of fine-tuning high-resource models fails to close it. This failure stems from two compounding factors: the limited scale of available CASL data and the significant lexical and visual domain gap between CASL and large-scale corpora such as WLASL, which renders pre-trained representations largely uninformative. To address this, we propose TransSLR, a lightweight Temporal Transformer Encoder trained from scratch on 64-frame normalized pose sequences, with average pooling and a classification head. By operating on geometric keypoint representations rather than raw RGB, TransSLR achieves signer-independent generalization without relying on visual appearance. On the CASL-W60 benchmark, TransSLR establishes a new state-of-the-art accuracy of 80.39%, surpassing the prior best by +10.46%. Beyond accuracy, our encoder-only design significantly reduces computational overhead, making deployment feasible in resource-constrained environments. We conduct extensive experiments on the CASL-W60 benchmark, comparing against RGB-based and multimodal baselines, and demonstrate that TransSLR achieves state-of-the-art performance.

Sign Language Recognition 3D Encoder Transformer Architectures Keypoint Detection

Open paper Ask about it PDF

Abstract

Explore similar work

TransSLR: A Lightweight Transformer for Sign Language Recognition

SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval

A Reverse Sign Language Dictionary: Open-Vocabulary Sign Recognition from Continuous Signing via Video Captioning and Description Retrieval