Sign language recognition suffers from catastrophic scaling failure: models achieving high accuracy on small vocabularies collapse at realistic sizes. Existing architectures treat signs as atomic visual patterns, learning flat representations that cannot exploit the compositional structure of sign languages-systematically organized from discrete phonological parameters (handshape, location, movement, orientation) reused across the vocabulary. We introduce PHONSSM, enforcing phonological decomposition through anatomically-grounded graph attention, explicit factorization into orthogonal subspaces, and prototypical classification enabling few-shot transfer. Using skeleton data alone on the largest ASL dataset ever assembled (5,565 signs), PHONSSM achieves 72.1% on WLASL2000 (+18.4pp over skeleton SOTA), surpassing most RGB methods without video input. Gains are most dramatic in the few-shot regime (+225% relative), and the model transfers zero-shot to ASL Citizen, exceeding supervised RGB baselines. The vocabulary scaling bottleneck is fundamentally a representation learning problem, solvable through compositional inductive biases mirroring linguistic structure.
Figures & tables
Figure 1: PhonSSM architecture. Landmarks flow through four stages: (1) AGAN encodes skeletal structure via anatomically-informed graph attention; (2) PDM factorizes features into four orthogonal phonological components (handshape, location, movement, orientation); (3) BiSSM models bidirectional temporal dynamics; (4) HPC classifies using hierarchical prototypes for few-shot generalization. Input is T×N×3 where N=21 (dominant hand) or N=75 (pose+hands). Total: 3.2M parameters.
Dataset Statistics
Top-1 Accuracy (%)
Dataset
Input
Signs
Train
Test
DSTA-SLR †
Pose-TGCN
I3D
PhonSSM
Standard Benchmarks (pose + both hands, 75 landmarks)
WLASL100
Pose+Hands
100
1,442
774
83.56
74.19
65.89
88.37 ± 0.42
WLASL300
Pose+Hands
300
3,912
2,005
80.00
–
56.14
74.41 ± 0.58
WLASL1000
Pose+Hands
1,000
11,246
5,628
67.81
–
47.33
62.90 ± 0.71
WLASL2000
Pose+Hands
2,000
17,272
8,634
53.70
–
32.48
72.08 ± 0.65
Table 1: Dataset statistics and main results. We train separate PhonSSM models for each row: four models for WLASL (one per split, using 75 pose+hand landmarks) and one model for Merged-5565 (using 21 dominant-hand landmarks). Results are mean ± std over 3 seeds. Bold : best skeleton; underline : second-best skeleton. † Results from Hu et al. (2024) . ‡ Baselines trained by us with dominant-hand input for Merged-5565.
Figure 2: Main results. (a,b,d) WLASL evaluation using pose+hands input (75 landmarks). (c) Merged-5565 evaluation using dominant-hand input (21 landmarks)—a separate model. Specifically: (a) WLASL-2000: 72.1% vs baselines. (b) Vocabulary scaling (separate models per split). (c) Few-shot accuracy by training samples; gains largest for rare signs. (d) Ablation: PDM removal causes largest drop ( − 11.9pp).
Table 2: Few-shot performance on Merged-5565 by training samples per sign.
Figure 4: Phonological analysis. (a) Minimal pair error rates by component type. (b) Per-component accuracy across datasets. (c) Error rate by training frequency—gains are largest for rare signs.
Configuration
Top-1 (%)
Δ
Full PhonSSM
88.37
–
w/o PDM
76.49
− 11.9
AGAN → MLP
79.84
− 8.5
BiSSM → LSTM
82.17
− 6.2
Table 3: Ablation summary on WLASL100. PDM removal causes largest drop.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Architecture
Model dimension D
128
Component dimension Dc
32
GAT heads
4
GAT layers
3
SSM layers
4
Appendix
Table 4: Full hyperparameter configuration.
Source Dataset
Samples
Signs
Type
ASL Citizen ( Desai et al., 2023 )
83,399
2,731
Isolated
WLASL ( Li et al., 2020 )
21,083
2,000
Isolated
MVP/Kaggle ASL-Signs a
94,477
250
Isolated
ASL Alphabet b
27,455
29
Fingerspell
ASL MNIST b
27,455
26
Fingerspell
ChicagoFSWild ( Shi et al., 2019a )
5,846
26
Fingerspell
Appendix
Table 5: Composition of the Merged-5565 dataset.
Category
# Signs
Bi-LSTM
PhonSSM
Δ
One-handed signs
62
71.2
89.4
+18.2
Two-handed signs
38
68.9
86.8
+17.9
Static (no movement)
15
73.4
91.2
+17.8
Dynamic (with movement)
85
69.8
87.9
+18.1
Face-region location
28
65.2
85.6
+20.4
Body-region location
31
71.8
89.1
+17.3
Appendix
Table 6: Performance breakdown by phonological characteristics on WLASL100. Δ : improvement over Bi-LSTM.
Table 8: Error analysis by phonological distance (number of differing components).
Split
Signs
Near-Minimal Pairs
Density
PhonSSM Δ
WLASL100
100
891
18.0%
+4.8pp
WLASL300
300
15,283
34.1%
− 5.6pp
WLASL1000
1,000
142,450
28.5%
− 4.9pp
WLASL2000
2,000
279,314
14.0%
+18.4pp
Appendix
Table 9: Phonological similarity density across WLASL vocabulary splits.
Components Shared
DSTA-SLR
PhonSSM
Δ
WLASL300
0–1 (distinct)
72.4
78.9
+6.5
2 (moderate)
81.2
76.3
− 4.9
3–4 (minimal pair)
84.1
71.8
− 12.3
WLASL2000
0–1 (distinct)
48.2
74.6
+26.4
Appendix
Table 10: Accuracy (%) stratified by phonological distance to nearest training neighbor.
Prediction Target
PDM Branch
Handshape
Location
Movement
Orientation
Handshape
78.4
31.2
24.8
38.1
Location
29.6
71.8
22.4
35.7
Movement
26.3
28.9
68.2
31.4
Orientation
33.8
32.1
25.6
74.6
Random baseline
3.3
8.3
6.7
12.5
Appendix
Table 11: Complete linear probe results. Each row shows one PDM branch; each column shows one phonological prediction task. Diagonal entries (bold) indicate intended correspondences.
Linguistic Category
Proto #1
Proto #2
Proto #3
“1” (index point)
P7 (0.89)
P12 (0.42)
P3 (0.18)
“5” (spread hand)
P2 (0.91)
P15 (0.38)
P8 (0.21)
“A” (fist)
P19 (0.87)
P4 (0.45)
P11 (0.19)
“B” (flat hand)
P2 (0.72)
P8 (0.51)
P15 (0.32)
“C” (curved)
P23 (0.83)
P7 (0.28)
P12 (0.24)
Appendix
Table 12: Top-3 handshape prototypes activated by each major handshape category.
Configuration
Top-1 (%)
Δ
Full PhonSSM
88.37 ± 0.42
–
Architecture ablations
w/o PDM (no factorization)
76.49 ± 0.83
− 11.9
w/o AGAN (MLP encoder)
79.84 ± 0.71
− 8.5
w/o BiSSM (LSTM temporal)
82.17 ± 0.65
− 6.2
w/o HPC (linear classifier)
84.11 ± 0.58
− 4.3
Appendix
Table 13: Detailed ablation on WLASL100 (mean ± std over 3 seeds).
Configuration
Params
Top-1 (%)
Throughput
Model dimension D
D=64
0.9M
85.21
412/s
D=128 (default)
3.2M
88.37
260/s
D=256
11.8M
89.02
98/s
Number of BiSSM layers
2 layers
2.1M
86.45
318/s
Appendix
Table 14: Architecture scaling on WLASL100.
λortho
λdiv
Top-1 (%)
0.01
0.01
86.42
0.05
0.01
87.63
0.1
0.01
88.37
0.2
0.01
87.91
0.5
0.01
86.18
0.1
0.001
87.82
Appendix
Table 15: Sensitivity to loss weights on WLASL100.
Learning Rate
Top-1 (%)
Convergence
1×10−4
86.82
95 epochs
2×10−4
87.91
78 epochs
3×10−4
88.37
65 epochs
5×10−4
87.63
52 epochs
1×10−3
85.41
41 epochs
Appendix
Table 16: Learning rate sensitivity on WLASL100.
Nh
Nl
Nm
No
Top-1 (%)
15
8
5
4
86.12
20
10
8
6
87.45
30
15
10
8
88.37
40
20
15
10
88.52
50
25
20
12
88.48
Appendix
Table 17: Effect of prototype counts on WLASL100.
Metric
Epoch 25
Epoch 50
Epoch 100
Train Loss
2.14
0.89
0.42
Val Loss
2.31
1.12
0.68
Train Acc (%)
52.3
78.6
94.2
Val Acc (%)
48.1
74.2
88.4
Lortho
0.42
0.18
0.08
Ldiv
0.31
0.12
0.05
Appendix
Table 18: Training convergence metrics on WLASL100.
Sign languages are compositional systems where meaning arises by combining sublexical phonological parameters, such as handshape, location, and movement. While deep learning models for Sign Language Recognition (SLR) have achieved increased performance on translation benchmarks, it remains unclear whether these models distinguish abstract phonological features or merely rely on low-level statistical correlations. This work evaluates the phonological perception of SLR models trained on American Sign Language (ASL) by probing phonological sensitivity using minimal pairs and evaluating representational alignment with human behavioral data. Our results reveal that SLR models exhibit emergent phonological sensitivity, but with clear architectural trade-offs: pose-based models are sensitive to handshape contrasts, while pixel-based models better capture location changes. Furthermore, pose-based models learn latent representations that correlate with human perceptual similarity judgments (r~0.49). These findings suggest that while SLR models exhibit emergent phonology, current training paradigms are insufficient to scale them beyond their architectural inductive biases.
Kayo Yin, Jessica Carter, Alex Xijie Lu +1
University of California, Berkeley · Johns Hopkins University · Microsoft Research
Sign language processing advances rapidly for high-resource languages such as American Sign Language (ASL), yet most of the world's sign languages lack the phonological annotations new methods require. We present the first zero-shot cross-lingual framework for handshape recognition, transferring from ASL to Catalan Sign Language (LSC). Our approach leverages the decomposition of handshapes into five phonological features -- selected fingers, flexion, spread, thumb position, and thumb contact -- shared across both languages, to decode LSC handshapes from predicted features via a composite phonological distance metric. We evaluate three architectures (MLP, SL-GCN, SHuBERT) trained on two ASL corpora (PopSign, Sem-Lex) against a 37-handshape, single-signer LSC benchmark. Zero-shot transfer proves viable once recording-format disparities are harmonized, reaching 80.0% phonological feature accuracy and 54.5% expected handshape accuracy. Phonological decomposition thus offers a bridge for extending sign language technologies to low-resource languages without any target-language video training labels.
Marcel Granero-Moya, Carolina del Corral Farrarós, Gloria Haro +2
Recent Vision Language Models (VLMs) have demonstrated strong performance across a wide range of multimodal reasoning tasks. This raises the question of whether such general-purpose models can also address specialized visual recognition problems such as isolated sign language recognition (ISLR) without task-specific training. In this work, we investigate the capability of modern VLMs to perform ISLR in a zero-shot setting. We evaluate several open-source and proprietary VLMs on the WLASL300 benchmark. Our experiments show that, under prompt-only zero-shot inference, current open-source VLMs remain far behind classic supervised ISLR classifiers by a wide margin. However, follow-up experiments reveal that these models capture partial visual-semantic alignment between signs and text descriptions. Larger proprietary models achieve substantially higher accuracy, highlighting the importance of model scale and training data diversity. All our code is publicly available on GitHub.
Vaclav Javorek, Jakub Honzik, Ivan Gruber +2
University of West Bohemia, Czech Republic · Eindhoven University of Technology, The Netherlands