Cross-Sign Language Transfer Learning Using Domain Adaptation with Multi-scale Temporal Alignment
Authors: Keren Artiaga, Yang Li, Ercan Engin Kuruoglu, Wai Kin, Chan
Organizations: Victor · Tsinghua-Berkeley Shenzhen Insitute, Tsinghua Shenzhen International Graduate School, The University Town, Nanshan District, Shenzhen, 518055, P.R.China.
Sign language serves as a vital means of communication for individuals with hearing impairments, yet recognition resources for the over 100 distinct sign languages are severely lacking. In response, we present our work on sign language recognition using transfer learning and the domain adaptation method TA3N, which utilizes the Temporal Relational Network (TRN) module for aligning multi-scale temporal relations. Our findings highlight the superior performance of Domain Adaptation to neural network-based transfer learning, particularly in improving recognition of American Sign Language (ASL). Our research also identifies the effectiveness of aligning shorter-term temporal features between source and target domains. In addition to using RGB, we conducted experiments using Optical Flow mode for the sign language samples, ultimately determining that RGB outperforms Optical Flow in the majority of cases. Our work aims to improve accessibility and communication for individuals who rely on sign language as their primary mode of communication.
Figures & tables
Figure 1: The process involved two steps: pre-processing and domain adaptation, to generate an SLR model. In the pre-processing step, we convert videos into both Optical Flow and RGB frames. The features of these frames are then fed into the Domain Adaptation network five times, each time utilizing a different N-multiscale temporal relations network (TRN) to determine the optimal timescale for aligning the temporal relations of the sign language domains.
Figure 2: Subsampled RGB frames (starting from left to right) for the gesture Copy in ASL from WLASL dataset
Figure 3: Subsampled optical flow frames (starting from left to right) for the gesture Copy in ASL from WLASL dataset
Figure 4: Subsampled RGB frames (starting from left to right) for the gesture Copy in LSA from LSA64 dataset
Figure 5: Subsampled optical Flow frames (starting from left to right) for the gesture Copy in LSA from LSA64 dataset
Figure 6: TRN showing the process of generating N-frame temporal relations and finally a multiscale temporal relation for an ASL gesture
Figure 7: The architecture of TA3N applied on Sign Language Recognition task.
\toprule CSL and WLASL300 labels
LSA and WLASL300 labels
\midrule deaf
deaf
red
red
green
green
yellow
yellow
colors
colors
son
son
Table 1: List of classes used in all transfer learning experiments.
\toprule
LSA
CSL
ASL for LSA to ASL DA
ASL for CSL to ASL DA
\midrule Training Sample Size
1040
4600
284
257
Testing Sample Size
260
1,150
71
64
\bottomrule
Table 2: Training sample sizes and testing sample sizes video-level 0 0 footnotetext: DA stands for Domain Adaptation
\toprule Number of Parameters
2582844
3895616
5732676
9471306
18323796
\midrule N-multiscale TRN
3
5
7
10
15
\bottomrule
Table 3: Complexity of the Domain Adaptation Models for LSA to ASL. The number of parameters for each N-multiscale TRN is indicated in the first row, while the number of multiscale TRN for each model is listed in the second row.
\toprule Number of Parameters
2580534
3893306
5730366
9468996
18321486
\midrule N-multiscale TRN
3
5
7
10
15
\bottomrule
Table 4: Complexity of the Domain Adaptation Models for CSL to ASL. The number of parameters for each N-multiscale TRN is indicated in the first row, while the number of multiscale TRN for each model is listed in the second row.
Figure 8: Bar chart showing the Full-scale Domain Adaptation experiment between LSA to ASL across different N-multiscale TRNs
Figure 9: Bar chart showing the Full-scale Domain Adaptation experiment between CSL to ASL across different N-multiscale TRNs
\toprule N-multiscale TRN
3
5
7
10
15
\midrule RGB
ASL with LSA classes
Non-transfer learning
mean
8.32%
8.1%
9.364%
9.174%
8.308%
std.
1.399
2.502
1.26
1.716
1.696
LSA to ASL
mean
11.75%
12.93%
9.564%
9.318%
9.024%
std.
1.501
1.821
1.91
2.583
2.571
ASL with CSL classes
Non-transfer learning
mean
9.524%
9.418%
9.55%
7.59%
6.294%
std.
1.64
1.261
4.922
0.396
0.5
Table 5: Overall results of the Full-scale Domain Adaptation experiment. The second column of the table corresponds to the relevant subsets of ASL, which were selected based on their shared classes with LSA and CSL, as shown in Table 1 . The third column specifies the type of domain adaptation that was performed where Non-transfer learning refers to the baseline. 0 0 footnotetext: Standard deviations correspond to the mean of the 5-fold cross-validation results of each N-multiscale TRN domain adaptation
Figure 10: Bar chart showing the Few-shot Domain Adaptation experiment between LSA to ASL across different N-multiscale TRNs
Figure 11: Bar chart showing the Few-shot Domain Adaptation experiment between CSL to ASL across different N-multiscale TRNs
\toprule N-multiscale TRN
3
5
7
10
15
\midrule RGB
ASL with LSA classes
Non-transfer learning
mean
6.4%
6.386%
6.078%
5.88%
6.32%
std.
0.998
0.929
0.77
0.543
0.48
LSA to ASL
mean
6.85%
6.764%
6.24%
6.022%
6.702%
std.
0.821
0.686
0.537
0.588
0.267
ASL with CSL classes
Non-transfer learning
mean
7.816%
7.184%
8.96%
8.792%
8.958%
std.
1.54
1.3
2.234
2.099
1.998
Table 6: Overall results of the Few-shot Domain Adaptation experiment. The second column of the table corresponds to the relevant subsets of ASL, which were selected based on their shared classes with LSA and CSL, as shown in Table 1 . The third column specifies the type of domain adaptation that was performed where Non-transfer learning refers to the baseline. 0 0 footnotetext: Standard deviations correspond to the mean of the 5-fold cross-validation results of each N-multiscale TRN domain adaptation
\toprule
ASL Baseline for LSA to ASL
LSA to ASL
ASL Baseline for CSL to ASL
CSL to ASL
\midrule RGB
mean
6.926%
10.676%
9.832%
9.518%
std.
2.129
7.209
4.645
4.807
OF
mean
10.344%
6.094%
15.998%
11.748%
std.
7.409
1.601
8.425
7.489
\cmidrule (l)1-6
Table 7: 5-fold classification accuracies of RGB to RGB and optical flow to optical flow of ASL pre-trained with LSA and CSL as well as the ASL baseline in optical flow (OF) and RGB modes in the full-scale learning setting.
Figure 12: Bar chart showing the accuracies of ASL when pre-trained with LSA and CSL in optical flow (OF) and RGB modes in the full-scale learning setting. Baseline results are also indicated
\toprule
ASL Baseline for LSA to ASL
LSA to ASL
ASL Baseline for CSL to ASL
CSL to ASL
\midrule RGB
mean
6.686%
6.602%
7.818%
7.59%
std.
1.422
1.257
2.201
2.322
OF
mean
6.84%
6.086%
7.198%
7.504%
std.
1.826
1.609
1.828
2.615
\cmidrule (l)1-6
Table 8: 5-fold classification accuracies of RGB to RGB and optical flow to optical flow of ASL pre-trained with LSA and CSL as well as the ASL baseline in optical flow (OF) and RGB modes in the few-shot learning setting.
Figure 13: Bar chart showing the accuracies of ASL when pre-trained with LSA and CSL in optical flow (OF) and RGB modes in the few-shot learning setting. Baseline results are also indicated
\toprule Tasks
Accuracies
\midrule Baseline(LSA to ASL) RGB
6.926
8.32
8.1
9.364
9.174
8.308
LSA to ASL RGB
10.676
11.75
12.93
9.564
9.318
9.024
Baseline(CSL to ASL) RGB
9.832
9.524
9.418
9.55
7.59
6.294
CSL to ASL RGB
9.518
10.46
10.55
11.86
8.56
9.906
Baseline(LSA to ASL) OF
10.344
8.754
7.79
5.186
7.618
7.16
LSA to ASL OF
6.094
10.08
9.412
9.958
9.756
8.45
Table 9: Performance matrix of Full-scale Neural network-based Transfer Learning and Domain Adaptation 0 0 footnotetext: NN: Neural Network
\toprule Tasks
Accuracies
\midrule Baseline(LSA to ASL) RGB
6.686
6.4
6.386
6.078
5.88
6.32
LSA to ASL RGB
6.602
6.85
6.764
6.24
6.022
6.702
Baseline(CSL to ASL) RGB
7.818
7.816
7.184
8.96
8.792
8.958
CSL to ASL RGB
7.59
8.056
8.572
9.266
8.842
8.796
Baseline(LSA to ASL) OF
6.84
6.268
6.58
7.108
5.806
5.714
LSA to ASL OF
6.086
7.184
6.654
7.262
7.344
6.738
Table 10: Performance matrix of Few-shot Neural network-based Transfer Learning and Domain Adaptation 0 0 footnotetext: NN: Neural Network
\toprule Study
Source Domain
Target Domain
Level
Dynamic
Two-handed
Signers
Number of times a subject repeat a sign
Classes
Accuracy (Highest Score Achieved)
\midrule Bird et al. Bird et al. (2020)
BSL
ASL
Word
Yes
No
2
2
18
94.44%
Halvardsson et al. Halvardsson et al. (2021)
ImageNet
SSL
Alphabet
No
No
5
8
26
85%
Cayamcela et al. Morocho-Cayamcela and Lim (2019)
ImageNet
ASL
Alphabet
No
No
Implied as many. Exact number not indicated
Not indicated
26
99.39%
Nishat et al. Nishat and Shopon (2020)
ImageNet
BSDL
digits, vowel, consonants
No
Yes
Implied as many. Exact number not indicated
Not indicated
46
96.57%
Farhadi et al. Farhadi et al. (2007)
ASL Avatar
ASL
Word
Yes
Yes
1
3
50
64.17%
Vazquez-Enriquez et al. Vázquez-Enríquez et al. (2021)
ASL
Turkish and Spanish Sign Language
Word
Yes
Yes
43
Not indicated
226
95.24% (Turkish), 93.91% (Spanish)
Table 11: Comparison of our model to other related Sign Language Recognition models. 0 0 footnotetext: The number of signers for our study refers to the average number of signers for each class of our subsets. The number of times a signer repeats a sign refers to the number of times a participant recorded a particular sign. In the classes column, we wrote 23 and 26 as we have two sets of classes, one for LSA to ASL transfer and one for CSL to ASL transfer. The average number of samples for each class of our subsets is 14. WLASL300, the dataset we used for our ASL domain, consists of 109 signers and a mean of 17 samples. However, by the time of this study, several of WLASL300’s source links from YouTube and other educational sign language websites are no longer available.
Automated sign language recognition for underrepresented languages remains a largely unsolved problem. Central African Sign Language (CASL) exemplifies this challenge: the CASL-W60 benchmark contains 60 isolated signs, and the best published result is 69.93%. We present TransSLR, a lightweight Temporal Transformer Encoder trained from scratch on normalized 64-frame pose sequences. By operating on geometric keypoint representations rather than raw RGB, TransSLR achieves 80.39% top-1 accuracy and 91.07% top-5 accuracy on a signer-independent evaluation split. This result is 10.46 percentage points higher than the published 69.93% result. TransSLR has 8.67 million trainable parameters. We also evaluate zero-shot transfer from a high-resource sign-language model, which achieves 0.00% exact-match accuracy under our manual gloss-matching protocol. Our experiments compare pose-based, RGB-based, and multimodal approaches to sign-language recognition for a low-resource language.
Lucia Yen Wanchi, Samuel Johnny, Victor Tolulope Olufemi +2
Sign language translation (SLT) remains challenging due to its high spatio-temporal complexity, long sequences, and the need to model multiple articulators without relying on gloss annotations. Existing approaches are typically tailored to individual datasets or languages and struggle to scale, while overlooking the relationships between sign languages that could inform more effective cross-lingual transfer. We present SIGNET, a framework that enables motion-level knowledge transfer for cross-language sign language translation. Our key insight is that, although sign languages differ in grammar and lexicon, pretrained models capture motion-level visual patterns that can be reused across datasets and languages. SIGNET integrates multiple pretrained sign language backbones through an attention-based, hand-prior aggregation mechanism that guides a gated fusion network in dynamically selecting the most relevant experts. Comprehensive experiments on four benchmarks (How2Sign, Phoenix14T, CSL-Daily, and MeineDGS) demonstrate state-of-the-art translation performance, and SIGNET also surpasses prior methods on WLASL for sign language recognition.
Sobhan Asasi, Ozge Mercanoglu Sincan, Richard Bowden
Continuous sign language recognition (CSLR) aims to recognize gloss sequences from unsegmented sign videos under weak sequence-level supervision. However, existing methods rely on sentence-level gloss annotations, providing limited temporal and semantic guidance for fine-grained representation learning. Conventional video-text alignment also requires large batch sizes, making it inefficient for memory-intensive sign language video training. In this work, we propose SMART, an MLLM-guided temporal alignment framework for joint sign recognition and spotting. SMART uses MLLMgenerated motion descriptions as auxiliary semantic cues and performs stable videotext alignment under small-batch training. To improve temporal representation learning, we introduce a Multi-Scale Temporal Adapter that models temporal interactions during transformer encoding. For dense temporal localization, SMART incorporates CSFormer, a CSLR-guided spotting module that injects recognition-derived gloss evidence into a boundary-aware spotting network. This unified framework enables CSLR features to benefit spotting, while spotting supervision complements weak CTC-based recognition. Experiments on four sign language benchmarks, including PHOENIX14-T, CSL-Daily, Large-scale KSL, and Disaster and Safety KSL datasets, demonstrate the effectiveness of SMART across both recognition and spotting tasks.