Skeleton-based isolated sign language recognition (ISLR) demands fine-grained understanding of articulated motion across multiple spatial scales, from subtle finger movements to global body dynamics. Existing approaches typically rely on deep feed-forward architectures, which increase model capacity but lack mechanisms for recurrent refinement and structured representation. We propose LA-Sign, a looped transformer framework with geometry-aware alignment for ISLR. Instead of stacking deeper layers, LA-Sign derives its depth from recurrence, repeatedly revisiting latent representations to progressively refine motion understanding under shared parameters. To further regularise this refinement process, we present a geometry-aware contrastive objective that projects skeletal and textual features into an adaptive hyperbolic space, encouraging multi-scale semantic organisation. We study three looping designs and multiple geometric manifolds, demonstrating that encoder-decoder looping combined with adaptive Poincare alignment yields the strongest performance. Extensive experiments on WLASL and MSASL benchmarks show that LA-Sign achieves state-of-the-art results while using fewer unique layers, highlighting the effectiveness of recurrent latent refinement and geometry-aware representation learning for sign language recognition.
Figures & tables
Figure 1: Overview of LA-Sign. Motion sequences are first processed by a part-wise ST-GCN encoder to extract sign features, which are then fed into a looped transformer for recurrent refinement. We study three looping variants: (a) encoder–decoder, (b) encoder-focused, and (c) decoder-focused, to assess how modality interaction patterns affect refinement. To regularise the latent space, we present a GA objective that maps sign and text features onto non-Euclidean manifolds. While traditional geometric approaches apply this alignment as a static, final objective, we integrate it as a recurrent signal computed at each looping iteration. This allows for the continuous minimisation of geodesic distances between paired embeddings throughout the refinement process (see Section 3.4 ). The refined representation is finally projected to text tokens for recognition.
Figure 2: Architectural details of the three recurrent looping variants. (a) Encoder-decoder looping: the initial sign representation S is concatenated with the previous cross-modal state Hi−1s2t before passing through the shared encoder-decoder block. (b) Encoder-focused looping: the visual representation His is iteratively refined via residual updates with S . (c) Decoder-focused looping: the encoder processes S into a fixed visual representation Hs , which the looped decoder repeatedly references to refine the cross-modal interpretation His2t . In all variants, T provides the linguistic context or decoder prefix. Features highlighted in green and blue denote intermediate and final representations, respectively; both are utilised in our GA alignment.
Method
P-I
P-C
Base ( 12×1 )
63.55
61.35
Base ( 2×1 )
0.13
0.13
Loop ( 2×6 )
33.81
33.70
Base ( 4×1 )
7.91
7.57
Loop ( 4×3 )
35.44
36.87
Base ( 6×1 )
23.17
20.12
Table 1: Performance analysis and benchmark comparison. (a) Base vs. Loop architectures. (b) Looping designs. (c) Geometric manifolds. (d) Comparison with SOTA methods on WLASL and MSASL. Configurations in (a) are denoted as (U×L) ( U unique layers, L recurrent depthes) keeping effective computational depth constant. Best and second-best results are highlighted. ∗ indicates additional modalities. † denotes multi-crop inference. ‡ denotes optical flow fusion.
Figure 3: UMAP visualisations of learned embeddings. Compared to the Euclidean baseline (a), the hyperbolic space (b) exhibits clearer radial organisation and improved semantic separation.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Representation refinement across loop iterations. We visualise the embedding distributions of three sign classes (“before”, “chair”, and “go”) across successive loop iterations. At early iterations (i = 1), the embeddings exhibit substantial overlap, indicating weak semantic separation. As the number of loops increases, the representations progressively form clearer cluster structures, with reduced inter-class overlap. This behaviour suggests that iterative looping refines the latent representation and enhances semantic organisation in the embedding space.
Table 2: Additional ablation studies on geometry-aware alignment and latent refinement mechanisms. (a) Impact of initial curvature in the adaptive Poincaré ball. (b) Impact of initial projection scale. (c) Effect of injecting auxiliary features during looping. (d) Comparison of different input fusion strategies for recurrent refinement. Best and second-best results are highlighted. Results show that adaptive hyperbolic geometry consistently improves recognition, while excessive external injection degrades performance. Concatenation-based fusion and learnable geometric parameters yield the most stable and accurate representations, highlighting the importance of preserving latent continuity during iterative refinement.
Sign Language Recognition (SLR) presents complex challenges due to variable signing speeds and spatial intricacies. Specifically, Indian Sign Language (ISL) requires precise modeling of its inherently bimanual nature and is bottlenecked by the scarcity of large-vocabulary benchmarks and the inability of standard models to handle irregular signing velocities. To address this, a structurally-aware hybrid framework for efficient, gloss ISL recognition is proposed. To minimize computational redundancy, an entropy-driven SVD sampler that dynamically extracts informative keyframes is used. Because this subsampling creates irregular physical time gaps, the proposed architecture employs a Continuous Dual Positional Encoding paired with a distance-aware Elastic Transformer to preserve natural signing rhythm. Spatially, a Kinematically-Conditioned Adaptive GCN explicitly models bimanual geometry using unified bone-joint features. To validate our framework, a 120-class, 9,453-video native ISL dataset is created. Augmented by Physical Node Dropout and ArcFace optimization to mitigate subject-specific memorization, our framework achieves a competitive 89.8% accuracy, establishing a strong baseline for complex sign language modeling.
Pushp Garg, Himangshu Sarma, Mrinmoy Ghorai
Human Computer Interaction Lab, IIIT SriCity, India
Automated sign language recognition for underrepresented languages remains a largely unsolved problem. Central African Sign Language (CASL) exemplifies this challenge: the CASL-W60 benchmark contains 60 isolated signs, and the best published result is 69.93%. We present TransSLR, a lightweight Temporal Transformer Encoder trained from scratch on normalized 64-frame pose sequences. By operating on geometric keypoint representations rather than raw RGB, TransSLR achieves 80.39% top-1 accuracy and 91.07% top-5 accuracy on a signer-independent evaluation split. This result is 10.46 percentage points higher than the published 69.93% result. TransSLR has 8.67 million trainable parameters. We also evaluate zero-shot transfer from a high-resource sign-language model, which achieves 0.00% exact-match accuracy under our manual gloss-matching protocol. Our experiments compare pose-based, RGB-based, and multimodal approaches to sign-language recognition for a low-resource language.
Lucia Yen Wanchi, Samuel Johnny, Victor Tolulope Olufemi +2
Sign language recognition from monocular video or 2D pose sequences is challenging, both because 3D information must be inferred from 2D observations and because the signal is inherently spatiotemporal. Moreover, the large and continually growing vocabulary of signs in production settings makes conventional closed-set classification impractical: adding a class requires new labeled data and retraining. We propose a contrastive Transformer-based model that learns rich representations of body key-point sequences, enabling direct comparison between embedding vectors. These representations support one-shot and few-shot tasks such as classification of signs never seen during training. On the LSA64 dataset, using only 48 classes for representation learning, the model reaches 88.4% accuracy on 16 held-out classes with as few as eight reference examples per class, and its accuracy improves consistently with the number of training classes and support examples.
Silvan Ferreira, Esdras Costa, Marcio Dahia +1
CESAR · Universidade Federal do Rio Grande do Norte (UFRN), Natal, Brazil · Lenovo