Skeleton-based isolated sign language recognition (ISLR) demands fine-grained understanding of articulated motion across multiple spatial scales, from subtle finger movements to global body dynamics. Existing approaches typically rely on deep feed-forward architectures, which increase model capacity but lack mechanisms for recurrent refinement and structured representation. We propose LA-Sign, a looped transformer framework with geometry-aware alignment for ISLR. Instead of stacking deeper layers, LA-Sign derives its depth from recurrence, repeatedly revisiting latent representations to progressively refine motion understanding under shared parameters. To further regularise this refinement process, we present a geometry-aware contrastive objective that projects skeletal and textual features into an adaptive hyperbolic space, encouraging multi-scale semantic organisation. We study three looping designs and multiple geometric manifolds, demonstrating that encoder-decoder looping combined with adaptive Poincare alignment yields the strongest performance. Extensive experiments on WLASL and MSASL benchmarks show that LA-Sign achieves state-of-the-art results while using fewer unique layers, highlighting the effectiveness of recurrent latent refinement and geometry-aware representation learning for sign language recognition.
Figures & tables
Figure 1: Overview of LA-Sign. Motion sequences are first processed by a part-wise ST-GCN encoder to extract sign features, which are then fed into a looped transformer for recurrent refinement. We study three looping variants: (a) encoder–decoder, (b) encoder-focused, and (c) decoder-focused, to assess how modality interaction patterns affect refinement. To regularise the latent space, we present a GA objective that maps sign and text features onto non-Euclidean manifolds. While traditional geometric approaches apply this alignment as a static, final objective, we integrate it as a recurrent signal computed at each looping iteration. This allows for the continuous minimisation of geodesic distances between paired embeddings throughout the refinement process (see Section 3.4 ). The refined representation is finally projected to text tokens for recognition.
Figure 2: Architectural details of the three recurrent looping variants. (a) Encoder-decoder looping: the initial sign representation S is concatenated with the previous cross-modal state Hi−1s2t before passing through the shared encoder-decoder block. (b) Encoder-focused looping: the visual representation His is iteratively refined via residual updates with S . (c) Decoder-focused looping: the encoder processes S into a fixed visual representation Hs , which the looped decoder repeatedly references to refine the cross-modal interpretation His2t . In all variants, T provides the linguistic context or decoder prefix. Features highlighted in green and blue denote intermediate and final representations, respectively; both are utilised in our GA alignment.
Method
P-I
P-C
Base ( 12×1 )
63.55
61.35
Base ( 2×1 )
0.13
0.13
Loop ( 2×6 )
33.81
33.70
Base ( 4×1 )
7.91
7.57
Loop ( 4×3 )
35.44
36.87
Base ( 6×1 )
23.17
20.12
Table 1: Performance analysis and benchmark comparison. (a) Base vs. Loop architectures. (b) Looping designs. (c) Geometric manifolds. (d) Comparison with SOTA methods on WLASL and MSASL. Configurations in (a) are denoted as (U×L) ( U unique layers, L recurrent depthes) keeping effective computational depth constant. Best and second-best results are highlighted. ∗ indicates additional modalities. † denotes multi-crop inference. ‡ denotes optical flow fusion.
Figure 3: UMAP visualisations of learned embeddings. Compared to the Euclidean baseline (a), the hyperbolic space (b) exhibits clearer radial organisation and improved semantic separation.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Representation refinement across loop iterations. We visualise the embedding distributions of three sign classes (“before”, “chair”, and “go”) across successive loop iterations. At early iterations (i = 1), the embeddings exhibit substantial overlap, indicating weak semantic separation. As the number of loops increases, the representations progressively form clearer cluster structures, with reduced inter-class overlap. This behaviour suggests that iterative looping refines the latent representation and enhances semantic organisation in the embedding space.
Table 2: Additional ablation studies on geometry-aware alignment and latent refinement mechanisms. (a) Impact of initial curvature in the adaptive Poincaré ball. (b) Impact of initial projection scale. (c) Effect of injecting auxiliary features during looping. (d) Comparison of different input fusion strategies for recurrent refinement. Best and second-best results are highlighted. Results show that adaptive hyperbolic geometry consistently improves recognition, while excessive external injection degrades performance. Concatenation-based fusion and learnable geometric parameters yield the most stable and accurate representations, highlighting the importance of preserving latent continuity during iterative refinement.