Geometry-Aware Metric Learning for Cross-Lingual Few-Shot Sign Language Recognition on Static Hand Keypoints

Chayanin Chamachot; Kanokphan Lertniponphan

cs.CVMar 10, 2026

Geometry-Aware Metric Learning for Cross-Lingual Few-Shot Sign Language Recognition on Static Hand Keypoints

Authors: Chayanin Chamachot, Kanokphan Lertniponphan

Organizations: Department of Computer Engineering Faculty of Engineering, Chulalongkorn University

Abstract

Sign language recognition (SLR) systems typically require large labeled corpora for each language, yet the majority of the world's 300+ sign languages lack sufficient annotated data. Cross-lingual few-shot transfer, pretraining on a data-rich source language and adapting with only a handful of target-language examples, offers a scalable alternative, but conventional coordinate-based keypoint representations are susceptible to domain shift arising from differences in camera viewpoint, hand scale, and recording conditions. This shift is particularly detrimental in the few-shot regime, where class prototypes estimated from only K examples are highly sensitive to extrinsic variance. We propose a geometry-aware metric-learning framework centered on a compact 20-dimensional inter-joint angle descriptor derived from MediaPipe static hand keypoints. These angles are invariant to SO(3) rotation, translation, and isotropic scaling, eliminating the dominant sources of cross-dataset shift and yielding tighter, more stable class prototypes. Evaluated on four fingerspelling alphabets spanning typologically diverse sign languages, ASL, LIBRAS, Arabic Sign Language, and Thai Sign Language, the proposed angle features improve over normalized-coordinate baselines by up to 25 percentage points within-domain and enable frozen cross-lingual transfer that frequently exceeds within-domain accuracy, using a lightweight MLP encoder with about 10^5 parameters. These findings demonstrate that invariant hand-geometry descriptors provide a portable and effective foundation for cross-lingual few-shot SLR in low-resource settings.

Figures & tables

Figure 1: Pipeline overview. A hand image is processed by MediaPipe Hands into 21 keypoints (63-D when flattened), converted to one of three representations— raw (63-D), angle (20-D), or raw_angle (83-D)—and encoded into a 128-D embedding for Prototypical Network classification. In the cross-lingual setting, the encoder is either frozen or undergoes target-supervised adaptation (last-layer fine-tuning on the target train split).

Figure 2: MediaPipe Hands keypoint topology ( $i\in\{0,\ldots,20\}$ ). Keypoint 0 is the wrist (root); each finger forms a four-joint kinematic chain following the official ordering [ 29 ] : Thumb (1–4), Index (5–8), Middle (9–12), Ring (13–16), Pinky (17–20).

Encoder	Repr	Input Dim	Parameters
MLP	raw	63	116,096
MLP	angle	20	105,088
MLP	raw_angle	83	121,216
Transformer	raw	63	282,240
Transformer	angle	20	284,416
Transformer	raw_angle	83	292,480

Table 1: Encoder parameter counts by representation. All encoders output 128-D embeddings.

Dataset	Classes	Train	Test	Min $n_{c}^{\text{test}}$	Elig. ( $K{=}5$ )
ASL Alphabet [ 20 ]	29	44 498	19 093	1	28/29
LIBRAS Alphabet [ 11 ]	21	23 993	10 291	447	21/21
Arabic SL [ 1 ]	31	4 952	2 141	56	31/31
Thai Finger. [ 14 ]	42	1 998	863	12	27/42

Table 2: Dataset statistics after MediaPipe extraction and 70/30 splitting (seed 42). Eligible = classes with

\geq K{+}Q

test samples at

K{=}5

Q{=}15

		MLP Encoder			Transformer Encoder
Dataset	Repr.	1-shot	3-shot	5-shot	1-shot	3-shot	5-shot
ASL	raw	88.7 $\pm$ 0.7	93.8 $\pm$ 0.5	94.9 $\pm$ 0.4	74.2 $\pm$ 1.0	80.3 $\pm$ 0.9	82.1 $\pm$ 0.9
	angle	81.5 $\pm$ 0.9	86.8 $\pm$ 0.7	88.4 $\pm$ 0.6	80.8 $\pm$ 0.9	86.2 $\pm$ 0.7	87.7 $\pm$ 0.7
	raw_angle	90.4 $\pm$ 0.7	94.5 $\pm$ 0.5	95.4 $\pm$ 0.4	90.8 $\pm$ 0.7	94.7 $\pm$ 0.4	95.4 $\pm$ 0.4
LIBRAS	raw	69.7 $\pm$ 1.0	78.9 $\pm$ 0.8	81.2 $\pm$ 0.8	61.1 $\pm$ 0.9	65.9 $\pm$ 0.9	67.9 $\pm$ 0.9
	angle	89.2 $\pm$ 0.7	92.9 $\pm$ 0.5	94.1 $\pm$ 0.5	86.9 $\pm$ 0.8	91.1 $\pm$ 0.6	92.5 $\pm$ 0.5

Table 3: Within-domain few-shot accuracy (%) on the test split . 5-way

K

-shot,

Q{=}15

, ProtoNet (Euclidean), 600 episodes, seed 42. Bold = best accuracy per dataset across all representations and encoders.

		MLP Encoder			Transformer Encoder
Target	Mode	raw	angle	raw_angle	raw	angle	raw_angle
ASL	Frozen	97.0 $\pm$ 0.4	91.7 $\pm$ 0.5	97.1 $\pm$ 0.4	92.7 $\pm$ 0.6	87.5 $\pm$ 0.7	97.1 $\pm$ 0.3
ASL	Target-sup.	97.0 $\pm$ 0.4	91.7 $\pm$ 0.5	97.1 $\pm$ 0.4	92.7 $\pm$ 0.6	87.5 $\pm$ 0.7	97.1 $\pm$ 0.3
LIBRAS	Frozen	86.5 $\pm$ 0.8	95.0 $\pm$ 0.4	87.8 $\pm$ 0.7	81.3 $\pm$ 0.8	93.2 $\pm$ 0.5	88.3 $\pm$ 0.7
LIBRAS	Target-sup.	94.2 $\pm$ 0.5	96.0 $\pm$ 0.4	95.2 $\pm$ 0.5	81.4 $\pm$ 0.8	93.2 $\pm$ 0.5	88.4 $\pm$ 0.7
Arabic	Frozen	74.2 $\pm$ 0.8	91.3 $\pm$ 0.5	76.6 $\pm$ 0.9	66.1 $\pm$ 0.9	87.4 $\pm$ 0.7	75.9 $\pm$ 0.9

Table 4: Effect of ASL pretraining on few-shot accuracy (%): encoder pretrained on ASL (SupCon), evaluated on each target’s test split . 5-way 5-shot,

Q{=}15

, 600 episodes, seed 42. Bold = best accuracy within each (target, mode) pair across all representations and encoders. ASL

\to

ASL rows are in-domain references (source=target), not cross-lingual transfer.

\dagger

Direction	Repr	Frozen	Target-sup.
LIBRAS $\to$ Arabic	raw	89.8 $\pm$ 0.6	91.8 $\pm$ 0.6
	angle	91.2 $\pm$ 0.6	92.2 $\pm$ 0.5
	raw_angle	91.7 $\pm$ 0.6	93.8 $\pm$ 0.5
Arabic $\to$ LIBRAS	raw	95.9 $\pm$ 0.5	96.5 $\pm$ 0.4
	angle	94.9 $\pm$ 0.5	95.9 $\pm$ 0.4
	raw_angle	97.1 $\pm$ 0.4	97.4 $\pm$ 0.4

Table 5: Multi-source transfer: LIBRAS

\leftrightarrow

Arabic per-representation breakdown (MLP, 5-way 5-shot,

Q{=}15

, 600 episodes, seed 42). Bold = best representation within each (direction, mode) pair.

$\dagger$ Within-domain (no pretraining); all other cells use frozen cross-lingual transfer.
Target	ASL	LIBRAS	Arabic	Thai
ASL	95.4 $\pm$ 0.4 $\dagger$ ( ra )	93.7 $\pm$ 0.5 ( r )	94.1 $\pm$ 0.4 ( r )	95.1 $\pm$ 0.4 ( ra )
LIBRAS	95.0 $\pm$ 0.4 ( a )	94.1 $\pm$ 0.5 $\dagger$ ( a )	97.1 $\pm$ 0.4 ( ra )	94.6 $\pm$ 0.5 ( a )
Arabic	91.3 $\pm$ 0.5 ( a )	91.7 $\pm$ 0.6 ( ra )	89.8 $\pm$ 0.6 $\dagger$ ( a )	90.2 $\pm$ 0.5 ( a )
Thai	53.2 $\pm$ 0.8 ( a )	52.2 $\pm$ 0.8 ( ra )	54.6 $\pm$ 0.9 ( r )	52.7 $\pm$ 0.8 $\dagger$ ( a )

Table 6: Best accuracy (%) by source language (MLP, 5-way 5-shot,

Q{=}15

, 600 episodes, seed 42). Off-diagonal: frozen-encoder transfer; each cell reports the highest-scoring representation (parenthesised: r = raw , a = angle , ra = raw_angle ). Diagonal

\dagger

: within-domain baseline (no pretraining; from Tab. 3 ). Bold = best cross-lingual source for each target row.

Dataset	Repr	No Norm	Normalised	Diff. (p.p.)
LIBRAS	raw	76.4	81.2	4.8
LIBRAS	angle	94.4	94.1	$-$ 0.3 †
Arabic	raw	59.1	64.5	5.4
Arabic	angle	90.1	89.8	$-$ 0.3 †

Table 7: Effect of removing wrist-centring and scale normalisation. 5-way 5-shot,

Q{=}15

, MLP encoder, 600 episodes, seed 42. “No norm” = MediaPipe raw output.

Setting	1-shot	3-shot	5-shot
No normalisation	41.8	56.3	59.1
+ Wrist-centring & scale	51.3	60.6	64.5
+ Geometry-aware (angle)	81.1	88.2	89.8

Table 8: Cumulative normalisation ablation on Arabic (MLP / raw). 5-way

K

-shot,

Q{=}15

, 600 episodes, seed 42.

Dataset	Repr	Input-Sp.	Ep.-Linear	ProtoNet
ASL	raw	95.1 $\pm$ 0.4	91.9 $\pm$ 0.5	94.9 $\pm$ 0.4
ASL	angle	89.4 $\pm$ 0.6	85.7 $\pm$ 0.7	88.4 $\pm$ 0.6
LIBRAS	raw	81.0 $\pm$ 0.8	80.2 $\pm$ 0.8	81.2 $\pm$ 0.8
LIBRAS	angle	94.2 $\pm$ 0.5	93.1 $\pm$ 0.5	94.1 $\pm$ 0.5
Arabic	raw	65.0 $\pm$ 0.9	63.7 $\pm$ 0.9	64.5 $\pm$ 0.8
Arabic	angle	90.5 $\pm$ 0.6	88.2 $\pm$ 0.6	89.8 $\pm$ 0.6

Table 9: Baseline comparison (5-way 5-shot,

Q{=}15

, 600 episodes, seed 42, MLP encoder where applicable).

Figure 3: Error analysis on Thai (MLP / angle / 5-way 5-shot, 600 episodes). Left: Per-class accuracy (26–94%). Right: Top confused pairs.

Explore similar work

Zero-Shot Cross-Lingual Recognition of Sign Language Handshapes
Sep 16, 2026Marcel Granero-Moya, Carolina del Corral Farrarós, Gloria Haro +2Zero-Shot Learning Sign Language Recognition
Universitat Pompeu Fabra
Isolated Sign Language Recognition for Icelandic Sign Language: Experiments in a Low-resource Setting
Sep 22, 2026Finnur Ágúst Ingimundarson, Guðný Björk Þorvaldsdóttir, Mathias Müller +1Few-Shot Learning Sign Language Recognition
Department of Computational Linguistics, University of Zurich · Communication Centre for the Deaf and Hard of Hearing in Iceland
TransSLR: A Lightweight Transformer for Sign Language Recognition
Aug 3, 2026Lucia Yen Wanchi, Samuel Johnny, Victor Tolulope Olufemi +2Sign Language Recognition
Carnegie Mellon University Africa

Sep 16, 2026J/K move · Enter open · S save

Zero-Shot Cross-Lingual Recognition of Sign Language Handshapes

Marcel Granero-Moya, Carolina del Corral Farrarós, Gloria Haro, Coloma Ballester+1

Universitat Pompeu Fabra

Zero-Shot Learning Sign Language Recognition Low-Resource Language Processing Cross-Lingual Transfer

Open paper AskPDF arXiv

Sign language processing advances rapidly for high-resource languages such as American Sign Language (ASL), yet most of the world's sign languages lack the phonological annotations new methods require. We present the first zero-shot cross-lingual framework for handshape recognition, transferring from ASL to Catalan Sign Language (LSC). Our approach leverages the decomposition of handshapes into five phonological features -- selected fingers, flexion, spread, thumb position, and thumb contact -- shared across both languages, to decode LSC handshapes from predicted features via a composite phonological distance metric. We evaluate three architectures (MLP, SL-GCN, SHuBERT) trained on two ASL corpora (PopSign, Sem-Lex) against a 37-handshape, single-signer LSC benchmark. Zero-shot transfer proves viable once recording-format disparities are harmonized, reaching 80.0% phonological feature accuracy and 54.5% expected handshape accuracy. Phonological decomposition thus offers a bridge for extending sign language technologies to low-resource languages without any target-language video training labels.

Abstract

Explore similar work

Zero-Shot Cross-Lingual Recognition of Sign Language Handshapes

Isolated Sign Language Recognition for Icelandic Sign Language: Experiments in a Low-resource Setting

TransSLR: A Lightweight Transformer for Sign Language Recognition