State Space Models are Effective Sign Language Learners: Exploiting Phonological Compositionality for Vocabulary-Scale Recognition

Bryan Cheng; Austin Jin; Jasper Zhang

cs.CVApr 9, 2026

State Space Models are Effective Sign Language Learners: Exploiting Phonological Compositionality for Vocabulary-Scale Recognition

Authors: Bryan Cheng, Austin Jin, Jasper Zhang

Organizations: William A. Shine Great Neck South High School

Abstract

Sign language recognition suffers from catastrophic scaling failure: models achieving high accuracy on small vocabularies collapse at realistic sizes. Existing architectures treat signs as atomic visual patterns, learning flat representations that cannot exploit the compositional structure of sign languages-systematically organized from discrete phonological parameters (handshape, location, movement, orientation) reused across the vocabulary. We introduce PHONSSM, enforcing phonological decomposition through anatomically-grounded graph attention, explicit factorization into orthogonal subspaces, and prototypical classification enabling few-shot transfer. Using skeleton data alone on the largest ASL dataset ever assembled (5,565 signs), PHONSSM achieves 72.1% on WLASL2000 (+18.4pp over skeleton SOTA), surpassing most RGB methods without video input. Gains are most dramatic in the few-shot regime (+225% relative), and the model transfers zero-shot to ASL Citizen, exceeding supervised RGB baselines. The vocabulary scaling bottleneck is fundamentally a representation learning problem, solvable through compositional inductive biases mirroring linguistic structure.

Figures & tables

Figure 1: PhonSSM architecture. Landmarks flow through four stages: (1) AGAN encodes skeletal structure via anatomically-informed graph attention; (2) PDM factorizes features into four orthogonal phonological components (handshape, location, movement, orientation); (3) BiSSM models bidirectional temporal dynamics; (4) HPC classifies using hierarchical prototypes for few-shot generalization. Input is $T\times N\times 3$ where $N=21$ (dominant hand) or $N=75$ (pose+hands). Total: 3.2M parameters.

		Dataset Statistics			Top-1 Accuracy (%)
Dataset	Input	Signs	Train	Test	DSTA-SLR †	Pose-TGCN	I3D	PhonSSM
Standard Benchmarks (pose + both hands, 75 landmarks)
WLASL100	Pose+Hands	100	1,442	774	83.56	74.19	65.89	88.37 $\pm$ 0.42
WLASL300	Pose+Hands	300	3,912	2,005	80.00	–	56.14	74.41 $\pm$ 0.58
WLASL1000	Pose+Hands	1,000	11,246	5,628	67.81	–	47.33	62.90 $\pm$ 0.71
WLASL2000	Pose+Hands	2,000	17,272	8,634	53.70	–	32.48	72.08 $\pm$ 0.65

Table 1: Dataset statistics and main results. We train separate PhonSSM models for each row: four models for WLASL (one per split, using 75 pose+hand landmarks) and one model for Merged-5565 (using 21 dominant-hand landmarks). Results are mean

\pm

std over 3 seeds. Bold : best skeleton; underline : second-best skeleton. † Results from Hu et al. (2024) . ‡ Baselines trained by us with dominant-hand input for Merged-5565.

Figure 2: Main results. (a,b,d) WLASL evaluation using pose+hands input (75 landmarks). (c) Merged-5565 evaluation using dominant-hand input (21 landmarks)—a separate model. Specifically: (a) WLASL-2000: 72.1% vs baselines. (b) Vocabulary scaling (separate models per split). (c) Few-shot accuracy by training samples; gains largest for rare signs. (d) Ablation: PDM removal causes largest drop (

-

11.9pp).

Figure 3: Component factorization. Cosine similarity matrix.

Samples/Sign	Bi-LSTM	PhonSSM	Gain
1–5	4.08	13.27	+225%
6–20	4.50	10.99	+144%
21–100	5.83	25.03	+329%
101+	52.66	92.82	+76%

Table 2: Few-shot performance on Merged-5565 by training samples per sign.

Figure 4: Phonological analysis. (a) Minimal pair error rates by component type. (b) Per-component accuracy across datasets. (c) Error rate by training frequency—gains are largest for rare signs.

Configuration	Top-1 (%)	$\Delta$
Full PhonSSM	88.37	–
w/o PDM	76.49	$-$ 11.9
AGAN $\rightarrow$ MLP	79.84	$-$ 8.5
BiSSM $\rightarrow$ LSTM	82.17	$-$ 6.2

Table 3: Ablation summary on WLASL100. PDM removal causes largest drop.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Architecture
Hyperparameter	Value
Model dimension $D$	128
Component dimension $D_{c}$	32
GAT heads	4
GAT layers	3
SSM layers	4

Appendix

Table 4: Full hyperparameter configuration.

Source Dataset	Samples	Signs	Type
ASL Citizen ( Desai et al., 2023 )	83,399	2,731	Isolated
WLASL ( Li et al., 2020 )	21,083	2,000	Isolated
MVP/Kaggle ASL-Signs a	94,477	250	Isolated
ASL Alphabet b	27,455	29	Fingerspell
ASL MNIST b	27,455	26	Fingerspell
ChicagoFSWild ( Shi et al., 2019a )	5,846	26	Fingerspell

Appendix

Table 5: Composition of the Merged-5565 dataset.

Category	# Signs	Bi-LSTM	PhonSSM	$\Delta$
One-handed signs	62	71.2	89.4	+18.2
Two-handed signs	38	68.9	86.8	+17.9
Static (no movement)	15	73.4	91.2	+17.8
Dynamic (with movement)	85	69.8	87.9	+18.1
Face-region location	28	65.2	85.6	+20.4
Body-region location	31	71.8	89.1	+17.3

Appendix

Table 6: Performance breakdown by phonological characteristics on WLASL100.

\Delta

: improvement over Bi-LSTM.

WLASL100
Comparison	$\Delta$ Acc.	$t$ -statistic	$p$ -value
vs. DSTA-SLR	+4.81	8.7	$<$ 0.01
vs. Pose-TGCN	+14.18	28.4	$<$ 0.001
vs. Bi-LSTM	+18.21	35.2	$<$ 0.001
WLASL2000
vs. DSTA-SLR	+18.38	22.1	$<$ 0.001

Appendix

Table 7: Statistical significance tests comparing PhonSSM to baselines (paired t-tests, 3 seeds).

Components Shared	% of Errors	Bi-LSTM	PhonSSM
4 (identical)	–	–	–
3 (minimal pair)	31.2%	28.4%	38.7%
2	40.8%	35.1%	33.2%
1	19.7%	22.8%	18.4%
0 (unrelated)	8.3%	13.7%	9.7%

Appendix

Table 8: Error analysis by phonological distance (number of differing components).

Split	Signs	Near-Minimal Pairs	Density	PhonSSM $\Delta$
WLASL100	100	891	18.0%	+4.8pp
WLASL300	300	15,283	34.1%	$-$ 5.6pp
WLASL1000	1,000	142,450	28.5%	$-$ 4.9pp
WLASL2000	2,000	279,314	14.0%	+18.4pp

Appendix

Table 9: Phonological similarity density across WLASL vocabulary splits.

WLASL300
Components Shared	DSTA-SLR	PhonSSM	$\Delta$
0–1 (distinct)	72.4	78.9	+6.5
2 (moderate)	81.2	76.3	$-$ 4.9
3–4 (minimal pair)	84.1	71.8	$-$ 12.3
WLASL2000
0–1 (distinct)	48.2	74.6	+26.4

Appendix

Table 10: Accuracy (%) stratified by phonological distance to nearest training neighbor.

	Prediction Target
PDM Branch	Handshape	Location	Movement	Orientation
Handshape	78.4	31.2	24.8	38.1
Location	29.6	71.8	22.4	35.7
Movement	26.3	28.9	68.2	31.4
Orientation	33.8	32.1	25.6	74.6
Random baseline	3.3	8.3	6.7	12.5

Appendix

Table 11: Complete linear probe results. Each row shows one PDM branch; each column shows one phonological prediction task. Diagonal entries (bold) indicate intended correspondences.

Linguistic Category	Proto #1	Proto #2	Proto #3
“1” (index point)	P7 (0.89)	P12 (0.42)	P3 (0.18)
“5” (spread hand)	P2 (0.91)	P15 (0.38)	P8 (0.21)
“A” (fist)	P19 (0.87)	P4 (0.45)	P11 (0.19)
“B” (flat hand)	P2 (0.72)	P8 (0.51)	P15 (0.32)
“C” (curved)	P23 (0.83)	P7 (0.28)	P12 (0.24)

Appendix

Table 12: Top-3 handshape prototypes activated by each major handshape category.

Architecture ablations
Configuration	Top-1 (%)	$\Delta$
Full PhonSSM	88.37 $\pm$ 0.42	–
w/o PDM (no factorization)	76.49 $\pm$ 0.83	$-$ 11.9
w/o AGAN (MLP encoder)	79.84 $\pm$ 0.71	$-$ 8.5
w/o BiSSM (LSTM temporal)	82.17 $\pm$ 0.65	$-$ 6.2
w/o HPC (linear classifier)	84.11 $\pm$ 0.58	$-$ 4.3

Appendix

Table 13: Detailed ablation on WLASL100 (mean

\pm

std over 3 seeds).

Model dimension $D$
Configuration	Params	Top-1 (%)	Throughput
$D=64$	0.9M	85.21	412/s
$D=128$ (default)	3.2M	88.37	260/s
$D=256$	11.8M	89.02	98/s
Number of BiSSM layers
2 layers	2.1M	86.45	318/s

Appendix

Table 14: Architecture scaling on WLASL100.

$\lambda_{\text{ortho}}$	$\lambda_{\text{div}}$	Top-1 (%)
0.01	0.01	86.42
0.05	0.01	87.63
0.1	0.01	88.37
0.2	0.01	87.91
0.5	0.01	86.18
0.1	0.001	87.82

Appendix

Table 15: Sensitivity to loss weights on WLASL100.

Learning Rate	Top-1 (%)	Convergence
$1\times 10^{-4}$	86.82	95 epochs
$2\times 10^{-4}$	87.91	78 epochs
$\mathbf{3\times 10^{-4}}$	88.37	65 epochs
$5\times 10^{-4}$	87.63	52 epochs
$1\times 10^{-3}$	85.41	41 epochs

Appendix

Table 16: Learning rate sensitivity on WLASL100.

$N_{h}$	$N_{l}$	$N_{m}$	$N_{o}$	Top-1 (%)
15	8	5	4	86.12
20	10	8	6	87.45
30	15	10	8	88.37
40	20	15	10	88.52
50	25	20	12	88.48

Appendix

Table 17: Effect of prototype counts on WLASL100.

Metric	Epoch 25	Epoch 50	Epoch 100
Train Loss	2.14	0.89	0.42
Val Loss	2.31	1.12	0.68
Train Acc (%)	52.3	78.6	94.2
Val Acc (%)	48.1	74.2	88.4
$\mathcal{L}_{\text{ortho}}$	0.42	0.18	0.08
$\mathcal{L}_{\text{div}}$	0.31	0.12	0.05

Appendix

Table 18: Training convergence metrics on WLASL100.

Sign Language Recognition

Sign language recognition uses video, images, or sensor data to identify signs, fingerspelling, and continuous signing across people and settings.

See all 56 papers

Phonological Perception of Sign Language Models
Jun 27, 2026Kayo Yin, Jessica Carter, Alex Xijie Lu +1Sign Language Recognition Computational Psycholinguistics
University of California, Berkeley · Johns Hopkins University · Microsoft Research
Zero-Shot Cross-Lingual Recognition of Sign Language Handshapes
Sep 16, 2026Marcel Granero-Moya, Carolina del Corral Farrarós, Gloria Haro +2Zero-Shot Learning Sign Language Recognition
Universitat Pompeu Fabra
Sign Language Recognition in the Age of LLMs
Apr 13, 2026Vaclav Javorek, Jakub Honzik, Ivan Gruber +2Sign Language Recognition VLM Evaluation
University of West Bohemia, Czech Republic · Eindhoven University of Technology, The Netherlands

Jun 27, 2026J/K move · Enter open · S save

Phonological Perception of Sign Language Models

Kayo Yin, Jessica Carter, Alex Xijie Lu, Annemarie Kocab

University of California, Berkeley · Johns Hopkins University · Microsoft Research

Sign Language Recognition Computational Psycholinguistics

Open paper AskPDF arXiv

Sign languages are compositional systems where meaning arises by combining sublexical phonological parameters, such as handshape, location, and movement. While deep learning models for Sign Language Recognition (SLR) have achieved increased performance on translation benchmarks, it remains unclear whether these models distinguish abstract phonological features or merely rely on low-level statistical correlations. This work evaluates the phonological perception of SLR models trained on American Sign Language (ASL) by probing phonological sensitivity using minimal pairs and evaluating representational alignment with human behavioral data. Our results reveal that SLR models exhibit emergent phonological sensitivity, but with clear architectural trade-offs: pose-based models are sensitive to handshape contrasts, while pixel-based models better capture location changes. Furthermore, pose-based models learn latent representations that correlate with human perceptual similarity judgments (r~0.49). These findings suggest that while SLR models exhibit emergent phonology, current training paradigms are insufficient to scale them beyond their architectural inductive biases.

Abstract

Sign Language Recognition

Explore similar work

Phonological Perception of Sign Language Models

Zero-Shot Cross-Lingual Recognition of Sign Language Handshapes

Sign Language Recognition in the Age of LLMs