SignFLIP: A Unified Model for Sign Language Translation and Generation via Stage-wise Alignment at Scale

Zhaoyi An; Sihan Tan; Youngbae Hwang; Kazuhiro Nakadai; Rei Kawakami

cs.CLSep 28, 2026

SignFLIP: A Unified Model for Sign Language Translation and Generation via Stage-wise Alignment at Scale

Authors: Zhaoyi An, Sihan Tan, Youngbae Hwang, Kazuhiro Nakadai, Rei Kawakami

Organizations: Institute of Science Tokyo · Chungbuk National University

Abstract

Sign language translation and generation share the goal of bidirectional alignment between text and sign representations. However, existing approaches either treat them as isolated tasks or are only verified on limited datasets, limiting effective modeling between modalities. In this paper, we propose SignFLIP, a unified LLM-centered framework for translation and generation. To enable bidirectional mapping between text and sign, SignFLIP adopts a symmetric architecture together with a stage-wise training strategy built on large-scale data. The shared sign--text representation is progressively refined: pre-alignment facilitates subsequent SLT, while the SLT-adapted representation further benefits SLG. Extensive experiments on multiple benchmarks show that SignFLIP shows competitive performance compared with task-specific models on both translation and generation tasks, as well as strong transferability to sign language recognition.

Figures & tables

Figure 1: Framework of SignFLIP, a unified model for SLT and SLG. A shared encoder learns joint representations from sign features and text inputs, which are then used by task-specific decoders to generate spoken text and sign poses. SignFLIP provides a unified framework for SLT and SLG while achieving competitive performance on both tasks.

Figure 2: SignFLIP is trained in three stages. Stage 1 performs pretraining to align sign and text embeddings through reconstruction and contrastive learning. Stages 2 and 3 focus on downstream SLT and SLG tasks, respectively. In particular, Stage 1 unfreezes the pose encoder and pose decoder. Stage 2 trains the pose encoder together with mT5 for text generation. Stage 3 then optimizes the mT5 encoder, projector, and pose decoder for sign generation.

Gloss-based
Method	Modality		Dev		Test
Method	Pose	RGB	BLEU-4 $\uparrow$	ROUGE $\uparrow$	BLEU-4 $\uparrow$	ROUGE $\uparrow$
SLRT † Camgoz et al. (2020)		✓	11.88	37.96	11.79	36.74
MMTLB Chen et al. (2022a)		✓	24.42	53.38	23.92	53.25
TS-SLT Chen et al. (2022b)	✓	✓	25.76	55.10	25.79	55.72
SLTUNET Zhang et al. (2023)		✓	23.99	53.58	25.01	54.08

Table 1: SLT results on the CSL-Daily dataset.

\dagger

and

\ddagger

denote results reproduced by Zhou et al. (2021) and Zhou et al. (2023) , respectively. Underlined results indicate the best performance among gloss-based SLT methods, Blue and Green denote the best results of previous methods and ours, respectively. SignFLIP achieves competitive performance among pose-only methods and remains competitive with RGB-based and multimodal approaches.

How2Sign
Method	Modality		Test
Method	Pose	RGB	BLEU-4 $\uparrow$	ROUGE $\uparrow$	BLEURT $\uparrow$
GloFE-VN Lin et al. (2023)	✓		2.2	12.6	31.7
YouTube-ASL Uthus et al. (2023)	✓		12.4	–	46.6
MSLU Zhou et al. (2025)	✓		2.4	17.2	–
C 2 RL Chen et al. (2024a)		✓	9.4	27.0	–

Table 2: SLT results on How2Sign. SignFLIP continues to outperform existing pose-only methods on How2Sign dataset.

Method	DTW $\downarrow$	DTW–MJE $\downarrow$	BLEU-4 $\uparrow$
How2Sign
Progressive Transformer
Saunders et al. (2020)	0.397	8.98e–4	1.63
Teach Me Sign
An and Kawakami (2025)	0.345	7.20e–4	3.95
Unified

Table 3: SLG results on the How2Sign and CSL-Daily datasets. Metric scales differ across datasets due to variations in generation difficulty and keypoint ranges.

Figure 3: Qualitative results on both How2Sign (left) and CSL-Daily (right) datasets. While the arm movements of the poses generated by our proposed SignFLIP are closer to the Ground Truth, our generated hand motions are also more natural than those from previous methods, by leveraging the feature distillation loss and hand-specific loss.

Method	How2Sign	CSL-Daily
SignFLIP (Ours)	1.69	1.94
GT motions	3.48	3.44

Table 4: Human evaluation scores for generated poses on How2Sign and CSL-Daily datasets.

Method	Modality		WER $\downarrow$
Method	Pose	RGB	Dev	Test
SignBT ( Zhou et al., 2021 )		✓	33.2	33.2
SEN ( Hu et al., 2023c )		✓	31.1	30.7
CorrNet ( Hu et al., 2023b )		✓	30.6	30.1
MSLU ( Zhou et al., 2025 )	✓		28.6	27.9
Uni-Sign ( Li et al., 2025 )	✓		28.2	27.4

Table 5: CSLR results on the CSL-Daily dataset with WER scores. SignFLIP demonstrates its strong transferability on CSLR task even with encoders frozen.

Method	Modality		WLASL100
Method	Pose	RGB	P-I	P-C
BEST Zhao et al. (2023)	✓		77.91	77.83
SignBERT+ Hu et al. (2023a)	✓		79.84	80.72
MSLU Zhou et al. (2025)	✓		88.76	89.25
NLA-SLR Zuo et al. (2023)	✓	✓	91.47	92.17
SignFLIP (Frozen)	✓		88.87	89.16

Table 6: ISLR results on the WLASL100 dataset under the frozen setting. Without task-specific fine-tuning of SignFLIP encoders, the model achieves promising performance, demonstrating the transferability of its learned shared representations to downstream tasks.

Figure 4: We visualize sign and text embeddings before and after SigFLIP, showing that initially separated feature distributions become closely aligned after training. Black-circled points and connecting lines indicate randomly selected sign–text pairs.

Setting	CSL-News	YouTube-ASL
Setting	BLEU-4	BLEU-4
w/o Pretraining	18.24	4.17
SignFLIP	21.61	8.43

Table 7: Ablation study on stage design for SLT task.

Setting	DTW	DTW-MJE
w/o Pretraining & SLT	0.034	8.44e-5
w/o SLT	0.030	7.37e-5
SignFLIP	0.025	6.15e-5

Table 8: Ablation study on stage design for SLG task.

Setting	CSL-News	CSL-News
Setting	BLEU-4	ROUGE
Qwen3 Yang et al. (2025)	5.19	20.07
mT5 (dec only)	3.10	19.61
mT5 (enc-dec, w/o stage1)	18.24	39.13

Table 9: Ablation study of LLM choices for SLT on CSL-News dataset.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Config	Stage 1	Stage 2	Stage 3
batch size	256	128	64
optimizer	AdamW
learning rate	$3\times 10^{-4}$
weight decay	$1\times 10^{-4}$
optimizer momentum	$\beta_{1},\beta_{2}=0.9,0.999$
learning rate schedule	cosine decay

Appendix

Table 10: Training recipe of each stage.

Name	Language	Samples	Vocab.	Hours	Source
YouTube-ASL Uthus et al. (2023)	ASL	$\sim$ 610K	60K	984	Web
How2Sign Duarte et al. (2021)	ASL	33K	16K	79	Lab
OpenASL Shi et al. (2022)	ASL	97K	33K	288	Web
WLASL Li et al. (2020)	ASL	2,038	100	Web
CSL-News Li et al. (2025)	CSL	$\sim$ 700K	5K	1,985	TV
CSL-Daily Zhou et al. (2021)	CSL	20K	2K	23	Lab

Appendix

Table 11: Selected ASL and CSL datasets.

Method	Gloss	Visual	BLEU-4	ROUGE-L	BLEURT
OpenASL
GloFE-VN Lin et al. (2023)	✓		7.06	21.75	36.35
Conv-GRU † Camgoz et al. (2018)		✓	4.58	16.10	25.65
I3D-transformer Shi et al. (2022)		✓	5.66	18.64	28.82
OpenASL Shi et al. (2022)		✓	8.59	21.02	31.09
C 2 RL Chen et al. (2024a)		✓	13.21	31.36	–

Appendix

Table 12: SignFLIP SLT performance on OpenASL after Stage-2 fine-tuning.

Setting	DTW	DTW-MJE	$\mathcal{L}_{\text{align}}$
16-frame	0.30	6.1e–4	0.17
24-frame	0.12	2.7e–4	0.10
32-frame	0.31	7.2e–4	0.15

Appendix

Table 13: Ablation study with MPM window size on CSL-News dataset.

Setting	DTW	DTW-MJE
w/o $\mathcal{L}_{\text{hand}}$ and $\mathcal{L}_{\text{distill}}$	0.044	1.19e–4
w/o $\mathcal{L}_{\text{hand}}$	0.029	7.84e–5
w/o $\mathcal{L}_{\text{distill}}$	0.037	1.02e–4
Ours	0.025	6.15e–5

Appendix

Table 14: Ablation study with SLG losses on CSL-Daily dataset.

Figure 5: Qualitative results on both How2Sign (left) and CSL-Daily (right) datasets.

Systems	Translation Output
Examples from CSL-Daily
Reference	你和小张什么时候认识的 ?
Reference	(When did you and Zhang meet?)
SignFLIP	你的小张什么时候认识的 ?
SignFLIP	(When did you meet your Zhang?)
Reference	我不去爬山 , 我有事。

Appendix

Table 15: Case study of translation outputs on CSL-Daily, How2Sign, and OpenASL. Examples are taken from the test sets. Sentences in brackets are our approximate English translations.

Explore similar work

Direct Translation between Sign Languages
May 20, 2026Zetian Wu, Bowen Xie, Wuyang Meng +3Gloss-Free Sign Language Translation Sign Language Translation
Oregon State University
Bridging the Gap Between Semantics and Reconstruction:Unifying Sign Language Translation and Production
Aug 10, 2026Xiao Liu, Shiwei Gan, Yafeng Yin +5Sign Language Translation Motion-Language Alignment
State Key Laboratory of Novel Software Technology, Nanjing University, Suzhou, Jiangsu, China · State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, Jiangsu, China
SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translation
Jun 26, 2026Sobhan Asasi, Ozge Mercanoglu Sincan, Richard BowdenGloss-Free Sign Language Translation Sign Language Translation
CVSSP, University of Surrey, United Kingdom

May 20, 2026, cs.CLJ/K move · Enter open · S save

Direct Translation between Sign Languages

Zetian Wu, Bowen Xie, Wuyang Meng, Milan Gautam+2

Oregon State University

Gloss-Free Sign Language Translation Sign Language Translation Sign Speech Translation Text Corpora

Open paper AskPDF arXiv

The field of sign language translation has witnessed significant progress in the translation between sign and spoken languages, but the translation between sign languages remains largely unexplored and out of reach. The latter can help 1.5 billion deaf and hard-of-hearing (DHH) people worldwide communicate across language barriers without relying on hearing interpreters or written-language fluency. The cascade approach composing separate sign-to-text, text-to-text, and text-to-sign systems suffers from error propagation and extra latency as well as the loss of information unique in the visual modality. We aim to develop direct sign-to-sign translation. However, a large-scale open-domain parallel corpus has not been curated between sign languages. To enable direct translation between sign language utterances, we use back-translation to produce synthetic sign-sign pairs from unaligned individual language utterance-sign corpora. Using this data, we jointly train a single MBART-based model for both text->sign (T2S) and sign->sign (S2S). On synthetically generated paired sets between American Sign Language (ASL), Chinese Sign Language (CSL), and German Sign Language (DGS), our direct S2S method outperforms the cascaded baseline on geometric sign error metrics (20% lower DTW-aligned MPJPE) and language matching metrics after predicted sign utterances are translated back to sentences (50% high BLEU-4) while achieving a roughly 2.3* speedup. On a small set of pre-existing cross-lingual sign data, we find similar improvements for our proposed method.

Abstract

Explore similar work

Direct Translation between Sign Languages

Bridging the Gap Between Semantics and Reconstruction:Unifying Sign Language Translation and Production

SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translation