cs.AIAug 17, 2026
SaveCross-Sign Language Transfer Learning Using Domain Adaptation with Multi-scale Temporal Alignment
Organizations: Victor · Tsinghua-Berkeley Shenzhen Insitute, Tsinghua Shenzhen International Graduate School, The University Town, Nanshan District, Shenzhen, 518055, P.R.China.
Abstract
Sign language serves as a vital means of communication for individuals with hearing impairments, yet recognition resources for the over 100 distinct sign languages are severely lacking. In response, we present our work on sign language recognition using transfer learning and the domain adaptation method TA3N, which utilizes the Temporal Relational Network (TRN) module for aligning multi-scale temporal relations. Our findings highlight the superior performance of Domain Adaptation to neural network-based transfer learning, particularly in improving recognition of American Sign Language (ASL). Our research also identifies the effectiveness of aligning shorter-term temporal features between source and target domains. In addition to using RGB, we conducted experiments using Optical Flow mode for the sign language samples, ultimately determining that RGB outperforms Optical Flow in the majority of cases. Our work aims to improve accessibility and communication for individuals who rely on sign language as their primary mode of communication.
Figures & tables
Figure 1: The process involved two steps: pre-processing and domain adaptation, to generate an SLR model. In the pre-processing step, we convert videos into both Optical Flow and RGB frames. The features of these frames are then fed into the Domain Adaptation network five times, each time utilizing a different N-multiscale temporal relations network (TRN) to determine the optimal timescale for aligning the temporal relations of the sign language domains.
Figure 2: Subsampled RGB frames (starting from left to right) for the gesture Copy in ASL from WLASL dataset
Figure 3: Subsampled optical flow frames (starting from left to right) for the gesture Copy in ASL from WLASL dataset
Figure 4: Subsampled RGB frames (starting from left to right) for the gesture Copy in LSA from LSA64 dataset
Figure 5: Subsampled optical Flow frames (starting from left to right) for the gesture Copy in LSA from LSA64 dataset
Figure 6: TRN showing the process of generating N-frame temporal relations and finally a multiscale temporal relation for an ASL gesture
Figure 7: The architecture of TA3N applied on Sign Language Recognition task.
| \toprule CSL and WLASL300 labels | LSA and WLASL300 labels |
|---|---|
| \midrule deaf | deaf |
| red | red |
| green | green |
| yellow | yellow |
| colors | colors |
| son | son |
Table 1: List of classes used in all transfer learning experiments.
| \toprule | LSA | CSL | ASL for LSA to ASL DA | ASL for CSL to ASL DA |
|---|---|---|---|---|
| \midrule Training Sample Size | 1040 | 4600 | 284 | 257 |
| Testing Sample Size | 260 | 1,150 | 71 | 64 |
| \bottomrule |
Table 2: Training sample sizes and testing sample sizes video-level 0 0 footnotetext: DA stands for Domain Adaptation
| \toprule Number of Parameters | 2582844 | 3895616 | 5732676 | 9471306 | 18323796 |
| \midrule N-multiscale TRN | 3 | 5 | 7 | 10 | 15 |
| \bottomrule |
Table 3: Complexity of the Domain Adaptation Models for LSA to ASL. The number of parameters for each N-multiscale TRN is indicated in the first row, while the number of multiscale TRN for each model is listed in the second row.
| \toprule Number of Parameters | 2580534 | 3893306 | 5730366 | 9468996 | 18321486 |
| \midrule N-multiscale TRN | 3 | 5 | 7 | 10 | 15 |
| \bottomrule |
Table 4: Complexity of the Domain Adaptation Models for CSL to ASL. The number of parameters for each N-multiscale TRN is indicated in the first row, while the number of multiscale TRN for each model is listed in the second row.
Figure 8: Bar chart showing the Full-scale Domain Adaptation experiment between LSA to ASL across different N-multiscale TRNs
Figure 9: Bar chart showing the Full-scale Domain Adaptation experiment between CSL to ASL across different N-multiscale TRNs
| \toprule N-multiscale TRN | 3 | 5 | 7 | 10 | 15 | |||
|---|---|---|---|---|---|---|---|---|
| \midrule RGB | ASL with LSA classes | Non-transfer learning | mean | 8.32% | 8.1% | 9.364% | 9.174% | 8.308% |
| std. | 1.399 | 2.502 | 1.26 | 1.716 | 1.696 | |||
| LSA to ASL | mean | 11.75% | 12.93% | 9.564% | 9.318% | 9.024% | ||
| std. | 1.501 | 1.821 | 1.91 | 2.583 | 2.571 | |||
| ASL with CSL classes | Non-transfer learning | mean | 9.524% | 9.418% | 9.55% | 7.59% | 6.294% | |
| std. | 1.64 | 1.261 | 4.922 | 0.396 | 0.5 |
Table 5: Overall results of the Full-scale Domain Adaptation experiment. The second column of the table corresponds to the relevant subsets of ASL, which were selected based on their shared classes with LSA and CSL, as shown in Table 1 . The third column specifies the type of domain adaptation that was performed where Non-transfer learning refers to the baseline. 0 0 footnotetext: Standard deviations correspond to the mean of the 5-fold cross-validation results of each N-multiscale TRN domain adaptation
Figure 10: Bar chart showing the Few-shot Domain Adaptation experiment between LSA to ASL across different N-multiscale TRNs
Figure 11: Bar chart showing the Few-shot Domain Adaptation experiment between CSL to ASL across different N-multiscale TRNs
| \toprule N-multiscale TRN | 3 | 5 | 7 | 10 | 15 | |||
|---|---|---|---|---|---|---|---|---|
| \midrule RGB | ASL with LSA classes | Non-transfer learning | mean | 6.4% | 6.386% | 6.078% | 5.88% | 6.32% |
| std. | 0.998 | 0.929 | 0.77 | 0.543 | 0.48 | |||
| LSA to ASL | mean | 6.85% | 6.764% | 6.24% | 6.022% | 6.702% | ||
| std. | 0.821 | 0.686 | 0.537 | 0.588 | 0.267 | |||
| ASL with CSL classes | Non-transfer learning | mean | 7.816% | 7.184% | 8.96% | 8.792% | 8.958% | |
| std. | 1.54 | 1.3 | 2.234 | 2.099 | 1.998 |
Table 6: Overall results of the Few-shot Domain Adaptation experiment. The second column of the table corresponds to the relevant subsets of ASL, which were selected based on their shared classes with LSA and CSL, as shown in Table 1 . The third column specifies the type of domain adaptation that was performed where Non-transfer learning refers to the baseline. 0 0 footnotetext: Standard deviations correspond to the mean of the 5-fold cross-validation results of each N-multiscale TRN domain adaptation
| \toprule | ASL Baseline for LSA to ASL | LSA to ASL | ASL Baseline for CSL to ASL | CSL to ASL | |
|---|---|---|---|---|---|
| \midrule RGB | mean | 6.926% | 10.676% | 9.832% | 9.518% |
| std. | 2.129 | 7.209 | 4.645 | 4.807 | |
| OF | mean | 10.344% | 6.094% | 15.998% | 11.748% |
| std. | 7.409 | 1.601 | 8.425 | 7.489 | |
| \cmidrule (l)1-6 |
Table 7: 5-fold classification accuracies of RGB to RGB and optical flow to optical flow of ASL pre-trained with LSA and CSL as well as the ASL baseline in optical flow (OF) and RGB modes in the full-scale learning setting.
Figure 12: Bar chart showing the accuracies of ASL when pre-trained with LSA and CSL in optical flow (OF) and RGB modes in the full-scale learning setting. Baseline results are also indicated
| \toprule | ASL Baseline for LSA to ASL | LSA to ASL | ASL Baseline for CSL to ASL | CSL to ASL | |
|---|---|---|---|---|---|
| \midrule RGB | mean | 6.686% | 6.602% | 7.818% | 7.59% |
| std. | 1.422 | 1.257 | 2.201 | 2.322 | |
| OF | mean | 6.84% | 6.086% | 7.198% | 7.504% |
| std. | 1.826 | 1.609 | 1.828 | 2.615 | |
| \cmidrule (l)1-6 |
Table 8: 5-fold classification accuracies of RGB to RGB and optical flow to optical flow of ASL pre-trained with LSA and CSL as well as the ASL baseline in optical flow (OF) and RGB modes in the few-shot learning setting.
Figure 13: Bar chart showing the accuracies of ASL when pre-trained with LSA and CSL in optical flow (OF) and RGB modes in the few-shot learning setting. Baseline results are also indicated
| \toprule Tasks | Accuracies | |||||
|---|---|---|---|---|---|---|
| \midrule Baseline(LSA to ASL) RGB | 6.926 | 8.32 | 8.1 | 9.364 | 9.174 | 8.308 |
| LSA to ASL RGB | 10.676 | 11.75 | 12.93 | 9.564 | 9.318 | 9.024 |
| Baseline(CSL to ASL) RGB | 9.832 | 9.524 | 9.418 | 9.55 | 7.59 | 6.294 |
| CSL to ASL RGB | 9.518 | 10.46 | 10.55 | 11.86 | 8.56 | 9.906 |
| Baseline(LSA to ASL) OF | 10.344 | 8.754 | 7.79 | 5.186 | 7.618 | 7.16 |
| LSA to ASL OF | 6.094 | 10.08 | 9.412 | 9.958 | 9.756 | 8.45 |
Table 9: Performance matrix of Full-scale Neural network-based Transfer Learning and Domain Adaptation 0 0 footnotetext: NN: Neural Network
| \toprule Tasks | Accuracies | |||||
|---|---|---|---|---|---|---|
| \midrule Baseline(LSA to ASL) RGB | 6.686 | 6.4 | 6.386 | 6.078 | 5.88 | 6.32 |
| LSA to ASL RGB | 6.602 | 6.85 | 6.764 | 6.24 | 6.022 | 6.702 |
| Baseline(CSL to ASL) RGB | 7.818 | 7.816 | 7.184 | 8.96 | 8.792 | 8.958 |
| CSL to ASL RGB | 7.59 | 8.056 | 8.572 | 9.266 | 8.842 | 8.796 |
| Baseline(LSA to ASL) OF | 6.84 | 6.268 | 6.58 | 7.108 | 5.806 | 5.714 |
| LSA to ASL OF | 6.086 | 7.184 | 6.654 | 7.262 | 7.344 | 6.738 |
Table 10: Performance matrix of Few-shot Neural network-based Transfer Learning and Domain Adaptation 0 0 footnotetext: NN: Neural Network
| \toprule Study | Source Domain | Target Domain | Level | Dynamic | Two-handed | Signers | Number of times a subject repeat a sign | Classes | Accuracy (Highest Score Achieved) |
| \midrule Bird et al. Bird et al. (2020) | BSL | ASL | Word | Yes | No | 2 | 2 | 18 | 94.44% |
| Halvardsson et al. Halvardsson et al. (2021) | ImageNet | SSL | Alphabet | No | No | 5 | 8 | 26 | 85% |
| Cayamcela et al. Morocho-Cayamcela and Lim (2019) | ImageNet | ASL | Alphabet | No | No | Implied as many. Exact number not indicated | Not indicated | 26 | 99.39% |
| Nishat et al. Nishat and Shopon (2020) | ImageNet | BSDL | digits, vowel, consonants | No | Yes | Implied as many. Exact number not indicated | Not indicated | 46 | 96.57% |
| Farhadi et al. Farhadi et al. (2007) | ASL Avatar | ASL | Word | Yes | Yes | 1 | 3 | 50 | 64.17% |
| Vazquez-Enriquez et al. Vázquez-Enríquez et al. (2021) | ASL | Turkish and Spanish Sign Language | Word | Yes | Yes | 43 | Not indicated | 226 | 95.24% (Turkish), 93.91% (Spanish) |
Table 11: Comparison of our model to other related Sign Language Recognition models. 0 0 footnotetext: The number of signers for our study refers to the average number of signers for each class of our subsets. The number of times a signer repeats a sign refers to the number of times a participant recorded a particular sign. In the classes column, we wrote 23 and 26 as we have two sets of classes, one for LSA to ASL transfer and one for CSL to ASL transfer. The average number of samples for each class of our subsets is 14. WLASL300, the dataset we used for our ASL domain, consists of 109 signers and a mean of 17 samples. However, by the time of this study, several of WLASL300’s source links from YouTube and other educational sign language websites are no longer available.