Studies on Sign Language Retrieval in Low-Resource Dictionary Settings
Organizations: Computer & Informatics Group, Variable Energy Cyclotron Centre, Kolkata, India.
Abstract
Most sign languages have no annotated corpus, only small dictionaries with one or a few example videos per sign. This study examines how well a pose encoder trained on a single sign language serves such languages without any target-language training. An encoder trained with metric learning on American Sign Language (ASL-Citizen) is evaluated frozen on six datasets of Turkish, Russian, Indian, Argentinian, Pakistani and Sri Lankan sign languages, with one or four clips per sign enrolled and, where signer identities are available, the query's signer excluded from the gallery. With one enrolled clip, the encoder retrieves the correct sign first in 36.4-80.7 % of queries, against 16.0-69.6 % for the best learning-free method (angle-based dynamic time warping) and a six-target mean of 39.3 % for an ASL-trained OpenHands graph network. In a crossed ablation of the encoder (embedding geometry, aggregation of body and hand embeddings, training objective, and attention between articulators), two choices matter: a per-pair InfoNCE objective outperforms supervised contrastive and cross-entropy training, and blocking attention between body, left hand and right hand inside the encoder raises accuracy by up to 5.1 points. The best configuration reaches 68.0 % one-shot accuracy (59.0 % for the reference encoder). On CISLR, an Indian Sign Language dictionary of 4,765 words with one video per word, the encoders find the correct word first for 17.3-19.0 % of queries, above the 16.8 % of the original RGB baseline. Over eight seeds with the joint encoder, spherical, Euclidean and hyperbolic embedding spaces differ by at most 1.6 points, except the hyperbolic token-attention head trained with InfoNCE, which is 1.4-2.0 points worse, so the geometry is a minor factor.
Figures & tables
| Dataset | Language | Clips | Signs | Signers | Role |
|---|---|---|---|---|---|
| ASL-Citizen | American SL | 83,399 | 2,731 | 52 | source |
| AUTSL | Turkish SL | 36,291 | 226 | 43 | target |
| Slovo | Russian SL | 20,000 | 1,000 | 194 | target |
| INCLUDE | Indian SL | 4,284 | 262 | 10 | target |
| LSA64 | Argentinian SL | 3,200 | 64 | 10 (11 IDs) | target |
| WLPSL | Pakistani SL | 248 | 31 | n/a | target |
| Experiment | Factor or change | Levels | Seeds | Runs |
|---|---|---|---|---|
| Main grid | Embedding geometry | S (sphere), E (Euclidean), H (Poincaré ball) | 0–2; 0–7 for the joint encoder | 264 |
| Aggregation (head) | G (global), P (product), F (barycentre), T (token attention) | |||
| Objective | SupCon, InfoNCE | |||
| Attention between articulators | joint, separated | |||
| Objective ablation | ProxyNCA++ | all 12 joint configurations | 0–2 | 36 |
| Cross-entropy classifier | S-G, E-G (joint) | 0–2 | 6 |
| System | ASL-Cit. test | AUTSL | Slovo | INCLUDE | LSA64 | WLPSL | SSL | Mean |
|---|---|---|---|---|---|---|---|---|
| 1+1: one enrolled clip per sign | ||||||||
| Raw keypoints, cosine | 5.3 | 10.5 | 7.0 | 6.6 | 51.1 | 23.5 | 17.0 | 19.3 |
| nDTWpp [ 18 ] | – | 8.8 | 5.3 | 13.0 | 49.6 | 30.4 | 17.1 | 20.7 |
| Angle DTW [ 17 ] | – | 21.1 | 16.0 | 21.5 | 69.6 | 58.8 | 29.3 | 36.0 |
| Untrained encoder | 4.2 | 10.7 | 6.7 | 13.5 | 45.2 | 38.2 | 23.7 | 23.0 |
| OpenHands [ 8 ] SL-GCN [ 36 ] , WLASL [ 38 ] | 11.3 | 19.1 | 13.8 | 35.3 | 64.3 | 70.6 | 32.8 | 39.3 |
| Protocol | System | R@1 | R@10 | mAP |
|---|---|---|---|---|
| AUTSL, 4 gallery clips + 1 query per sign [ 14 ] | Transformer, WLASL [ 38 ] , published | 3.5 | 29.2 | 9.1 |
| OpenHands [ 8 ] SL-GCN [ 36 ] , WLASL [ 38 ] | 28.7 | 63.0 | 21.6 | |
| OpenHands [ 8 ] ST-GCN [ 37 ] , WLASL [ 38 ] | 28.9 | 65.8 | 20.7 | |
| Untrained encoder | 17.8 | 52.3 | 13.7 | |
| This work | ||||
| S-G, SupCon | 66.1 0.8 | 92.1 0.5 | 55.4 0.8 |
| Work | Model, input | Trained on ASL-Citizen | Ranked set | R@1 | R@5 | R@10 | MRR |
| Desai et al. [ 28 ] | I3D classifier, RGB | yes | 2,731 classes by classifier score | 63.1 | 86.1 | 90.9 | 0.73 |
| Desai et al. [ 28 ] | ST-GCN classifier, pose | yes | 2,731 classes by classifier score | 59.5 | 82.7 | 88.1 | 0.70 |
| Jiang et al. [ 18 ] | nDTWpp, learning-free, pose | no | 10 nearest training clips | 19 | – | – | – |
| Wong et al. [ 44 ] | SignMatch, pose | no | dictionary entries | 57.3 | 83.7 | – | – |
| Wong et al. [ 44 ] | SignMatch, RGB | no | dictionary entries | 67.1 | 91.8 | – | – |
| This work | |||||||
| System | Input | Seeds | Hit@1 | Hit@5 | Hit@10 |
|---|---|---|---|---|---|
| I3D, WLASL [ 3 ] , published | RGB | 1 | 16.8 | 20.0 | 22.6 |
| Raw keypoints, all landmarks | pose | – | 11.9 | 15.8 | 18.1 |
| Raw keypoints, hands | pose | – | 12.3 | 16.5 | 18.8 |
| Untrained encoder | pose | 1 | 13.0 | 16.2 | 17.9 |
| This work | |||||
| S-G, SupCon | pose | 8 | 17.3 0.2 | 23.0 0.4 | 26.0 0.6 |
Cross-Lingual Transfer
Cross-lingual transfer enables models to apply knowledge or capabilities learned in one language to tasks in another.
See all 268 papers