OneSign: Unifying Sign Language Understanding Tasks with One Model

Shiwei Gan; Yafeng Yin; Xiao Liu; Desibieer Tuerdaken; Lei Xie; Sanglu Lu

cs.CLSep 27, 2026

OneSign: Unifying Sign Language Understanding Tasks with One Model

Authors: Shiwei Gan, Yafeng Yin, Xiao Liu, Desibieer Tuerdaken, Lei Xie, Sanglu Lu

Organizations: State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China

Abstract

SLU encompasses a diverse set of tasks, including ISLR, CSLR, and SLT. Although these tasks share basic semantic and linguistic foundations, they are typically addressed with task-specific architectures and training pipelines, which hinders knowledge sharing and requires costly pretraining and finetuning for each task. In this paper, we focus on two aspects of SLU tasks: (1) training and inference pipelines are highly fragmented: most methods rely on pretraining on large-scale SL datasets followed by task- or dataset-specific finetuning, which leads to multiple specialized models rather than a single checkpoint. (2) current LLM-based methods may overlook the inherent modality discrepancy between sign and text tokens, simply concatenating them and processing both modalities with the same decoder layers. In this paper, we present OneSign, a unified framework that addresses multiple SLU tasks within a single model and a single checkpoint. OneSign reformulates ISLR, CSLR, and SLT under a single training paradigm. To accommodate the heterogeneous characteristics of sign and text representations, we introduce a Modality-Adaptive Mixture-of-Experts (MA-MoE) architecture, consisting of a shared expert and modality-specific experts for sign and text tokens. A modality router dynamically activates the corresponding experts, and their outputs are aggregated to form the final token representations. By enabling modality-dependent expert specialization while preserving a shared expert path, MA-MoE can effectively model the modality differences between continuous sign representations and discrete text tokens. Extensive experiments on multiple benchmarks demonstrate that OneSign achieves competitive or state-of-the-art performance on several benchmarks, highlighting its effectiveness as a unified SLU model. Datasets are available at : https://github.com/gswycf/OneSign.

Figures & tables

Figure 1: Overall performance comparison across multiple tasks.

Figure 2: Comparison between previous SLU pipelines and the proposed OneSign framework.

Name	Language	Vocab.	Hours	Source	Topic
KETI ( Ko et al., 2019 )	KVK	419	28	Lab	Emergency Situations
VRT-RAW ( Camgöz et al., 2021 )	VGT	-	100	TV	Broadcast
SWISSTXT ( Camgöz et al., 2021 )	DGS	-	88	TV	News
PHOENIX-2014T ( Camgoz et al., 2018 )	DGS	3K	11	TV	Weather
DGS Corpus ( Hanke et al., 2020 )	DGS	-	50	Lab	Dialogue
BOBSL ( Albanie et al., 2021 )	BSL	77K	1,447	TV	Broadcast

Table 1: Comparison with existing SLT datasets. Vocab : vocabulary size; Hours : total video duration in hours; Topic : main domain of videos ( e.g. , daily life, TV news).

Figure 3: Distribution of topics and data sources.

Figure 4: Distribution of video durations and text lengths.

Figure 5: Left: the OneSign Architecture. Right: The MA-MoE Layer with Modality Adaptive MoE structure.

Model	DEV		TEST
Model	WER	del/ins	WER	del/ins
OneSign ♡	21.1	7.2/3.0	22.1	8.1/2.3
OneSign ♠ W/O MHCL	23.5	11.8/2.1	24.9	11.7/2.5
OneSign ♠	20.0	5.2/3.1	20.1	5.7/2.9

Table 2: Effect of Multi-Head Classification Layer.

Dataset	TEST
Dataset	WER	Rouge	BLEU1	BLEU4
Phoenxi14T	22.1	51.18	52.68	26.22
+CSLR	21.5	51.65	52.45	26.25
+ISLR+CSLR	21.0	52.18	52.65	26.24
+ISLR+CSLR+SLT	20.1	52.01	53.36	26.45

Table 3: Effect of heterogeneous training datasets.

Experts			DEV				TEST
Shared	Sign	Text	WER	Rouge	BLEU1	BLEU4	WER	Rouge	BLEU1	BLEU4
✓			22.3	49.12	49.89	23.45	22.5	48.99	49.72	23.01
✓	✓		21.0	50.12	49.65	23.65	21.4	50.01	49.41	23.47
✓		✓	20.8	50.64	50.12	24.36	21.0	50.03	49.95	24.13
✓	✓	✓	20.0	52.34	53.41	26.98	20.1	52.01	53.36	26.45

Table 4: Effect of MA-MOE.

$N_{s}=N_{t}$	Top- $k$	Params (M)	Activated (M)	Dev-Rouge	Dev-BLEU1	Dev-BLEU4
0	0	1235.8	1235.8	49.45	49.36	22.67
2	1	4457.0	2041.1	52.34	53.41	26.98
4	1	7678.2	2041.1	52.74	53.65	26.78
4	2	7678.2	3651.7	52.99	53.94	26.97

Table 5: Impact of activated expert count.

	Dev				Test
	WER	ROUGE	BLEU1	BLEU4	WER	ROUGE	BLEU1	BLEU4
W/O DPGF	21.6	51.01	52.87	25.83	21.1	51.07	52.17	25.63
W/ Fix	20.6	51.87	52.36	26.36	20.7	51.46	52.48	25.87
W/ DPGF	20.0	52.35	53.41	26.98	20.1	52.01	53.36	26.45

Table 6: Effects of Dynamic Part Graph Fusion.

Model	Training Time	Inference Time (250 frames)
	$\sim$ 10.4 min/epoch	$\sim$ 12.5 samples/s
+MA-MOE	$\sim$ 11.2 min/epoch	$\sim$ 11.8 samples/s

Table 7: Training speed and Inference speed.

Method	Extra			ASL Citizen		WLASL2000		MultiVSL1000
Method	Pose	RGB	FT	Top@1	Top@5	P-I	P-C	Top@1	Top@5
I3D		✓	✓	63.10	86.09	37.91	35.90	40.60	73.03
HMA ( Hu et al., 2021 )		✓	✓	-	-	37.91	35.90	-	-
NLA-SLR ( Zuo et al., 2023 )		✓	✓	-	-	61.05	58.05	-	-
Uni-Sign ( Li et al., 2025 )	✓	✓	✓ ∗	-	-	63.52	61.32	-	-
SHuBERT ( Gueuwou et al., 2025a )		✓	✓ ∗	65.00	87.00	60.90	58.01	-	-

Table 8: ISLR results on various benchmarks. ∗ denotes results reproduced by us using publicly available code, and

\checkmark^{*}

indicates pretraining on large-scale SL datasets.

Method	Extra			CSL-Daily		Phoenix14T
Method	Pose	RGB	FT	Dev	Test	Dev	Test
SignBT ( Zhou et al., 2021a )		✓	✓	33.2	33.2	-	-
CorrNet ( Hu et al., 2023b )		✓	✓	30.6	30.1	18.9	20.5
SEN ( Hu et al., 2023c )		✓	✓	31.1	30.7	19.3	20.7
C2ST ( Zhang et al., 2023 )		✓	✓	25.9	25.8	17.5	17.7
SignGraph ( Gan et al., 2024 )		✓	✓	27.3	26.4	17.8	19.1

Table 9: CSLR results on CSL-Daily and Phoenix14T. “FT” denotes fine-tuning on each dataset.

SLT	Extra			Phoenix14T						CSL-Daily
				DEV			TEST			DEV			TEST
	Pose	RGB	FT	ROUGE	BLEU1	BLEU4	ROUGE	BLEU1	BLEU4	ROUGE	BLEU1	BLEU4	ROUGE	BLEU1	BLEU4
PGG-SLT ( Guo et al., 2025 )		✓	✓	52.01	53.16	27.09	51.85	53.45	26.85	-	-	-	-	-	-
GFSLT-VLP ( Yin et al., 2023 )		✓	✓	43.72	44.08	22.12	42.49	43.71	21.44	36.44	39.20	11.07	36.70	39.37	11.00
SignCL ( Ye et al., 2024 )		✓	✓				49.04	49.76	22.74	-	-	-	48.92	47.47	16.16
Sign2GPT ( Wong et al., 2024 )		✓	✓	-	-		48.90	49.54	22.52	-	-	-	42.36	41.75	15.40

Table 10: Comparison of SLT performance on Phoenix14T and CSL-Daily. ∗ denotes results reproduced by us using publicly available code. We underline pose-based results that outperform our baseline model for clarity.

Gloss-free SLT	Extra			How2Sign						CSL-OpenWorld
				DEV			TEST			DEV			TEST
	Pose	RGB	FT	ROUGE	BLEU1	BLEU4	ROUGE	BLEU1	BLEU4	ROUGE	BLEU1	BLEU4	ROUGE	BLEU1	BLEU4
PGG-SLT (mBART) ( Guo et al., 2025 )	✓	✓	✓	32.6	41.8	15.9	31.50	38.90	13.10	-	-	-	-	-	-
YouTube-SLT ∗ ( Uthus et al., 2023 )		✓	✓	-	-	-	-	14.96	1.22	-	-	-	-	-	-
$C^{2}$ RL ( Chen et al., 2025 )		✓	✓	-	-	-	27.02	29.07	9.37	-	-	-	-	-	-
FLa-LLM ( Chen et al., 2024 )		✓	✓	-	-	-	27.81	29.81	9.66	-	-	-	-	-	-

Table 11: Comparison of gloss-free SLT performance on How2Sign and CSL-OpenWorld.

Dataset	Video Samples			Gloss Vocabulary			Token Vocabulary
Dataset	Train	Test	Validation	Train	Test	Validation	Train	Test	Validation
ASL Citizen ( Desai et al., 2023 )	40154	32941	10304	2731	2731	2731	-	-	-
WLASL2000 ( Li et al., 2020a )	14289	2878	6149	2000	1999	2000	-	-	-
MultiVSL1000 ( Dinh et al., 2025 )	20161	4538	3712	999	999	999	-	-	-
PHOENIX14T ( Camgoz et al., 2018 )	7096	642	519	1085	411	393	2143	976	927
CSL-Daily ( Zhou et al., 2021a )	18401	1176	1077	2000	1345	1358	2342	1358	1344

Table 12: Details of datasets used in our paper.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Task	Dev				TEST
Dataset	Task	WER	Rouge	BLEU1	BLEU4	WER	Rouge	BLEU1	BLEU4
		20.1	51.58	52.12	26.01	20.2	51.59	52.01	25.64
✓		20.0	52.24	53.14	26.76	20.2	51.87	53.16	26.31
	✓	20.0	52.13	53.35	26.65	20.1	51.64	53.07	26.14
✓	✓	20.0	52.34	53.41	26.98	20.1	52.01	53.36	26.45

Appendix

Table 13: Effect of Dataset and Task descriptor.

$\lambda_{sign}=\lambda_{text}$	0.3	0.5	0.7	0.9	1
WER $\downarrow$	22.1	20.1	22.3	23.4	25.1
BLUE4 $\uparrow$	25.75	26.45	25.74	25.15	24.78

Appendix

Table 14: Phoenix14T results with

\lambda

\mathcal{L}_{\mathrm{bal}}

Model	Parameter(M)		Dev				TEST
Model	Total	Activated	WER	Rouge	BLEU1	BLEU4	WER	Rouge	BLEU1	BLEU4
Dense FFN	2041.1	2041.1	22.6	49.63	50.23	23.68	22.8	49.56	49.32	23.67
Sparse MOE	4457.0	2041.1	22.4	49.23	49.36	23.15	22.6	48.65	49.36	22.64
shared+modality-specific FFN	2041.1	2041.1	21.4	51.69	51.60	25.68	21.5	51.52	51.98	25.79
MA-MOE	4457.0	2041.1	20.0	52.34	53.41	26.98	20.1	52.01	53.36	26.45

Appendix

Table 15: Comparison of different experts.

Model	Test				+MA-MOE	TEST
Model	WER	Rouge	BLEU1	BLEU4	+MA-MOE	WER	Rouge	BLEU1	BLEU4
Llama 3.2 1B	22.5	48.99	49.72	23.01	$\rightarrow$	20.1	52.01	53.36	26.45
Gemma3 1b	22.9	49.96	48.04	22.85	$\rightarrow$	20.2	51.32	52.66	26.35
Qwen2.5 1.5B	23.7	49.06	48.81	22.93	$\rightarrow$	20.3	51.17	52.69	26.34

Appendix

Table 16: Effect of MA-MOE on different LLMs.

$N_{s}=N_{t}$	Top- $k$	Params (M)	Activated (M)	Rouge	BLEU1	BLEU4	Rouge	BLEU1	BLEU4
0	0	1235.8	1235.8	49.45	49.36	22.67	48.99	49.72	23.01
2	1	4457.0	2041.1	52.34	53.41	26.98	52.01	53.36	26.45
4	1	7678.2	2041.1	52.74	53.65	26.78	52.14	53.24	26.53
4	2	7678.2	2846.4	52.99	53.94	26.97	52.45	53.47	26.56

Appendix

Table 17: Impact of activated expert count.

example(a)	Phoenix14T dataset
Groundtruth	HEUTE NACHT REGEN DOCH MITTE NORD
	(TODAY NIGHT RAIN HOWEVER MIDDLE NORTH)
OneSign	HEUTE NACHT REGEN DOCH MITTE NORD
	TODAY NIGHT RAIN HOWEVER MIDDLE NORTH
W/O MA-MOE	HEUTE NACHT NEBEL DOCH MITTE NORD
	( TODAY NIGHT FOG HOWEVER MIDDLE NORTH )

Appendix

Table 18: CSLR Qualitative results on Phoenix-2014T and CSL-Daily.

example(a)	Phoenix14T dataset
Groundtruth	und nun die wettervorhersage für morgen sonntag den zwölften juli
	(And now the weather forecast for tomorrow, Sunday, July 11th)
OneSign	und nun die wettervorhersage für morgen sonntag den zwölften juli
	(And now the weather forecast for tomorrow, Sunday, July 11th)
W/O MA-MOE	und nun die wettervorhersage für morgen sonntag den elften juli
	(And now the weather forecast for tomorrow, Sunday, July 12th)

Appendix

Table 19: SLT Qualitative results on Phoenix-2014T, How2Sign , CSL-Daily and CSL-OpenWorld.

Panel of Figure 6 — Figure 6: Visualization of used datasets.

Panel of Figure 7 — Figure 7: Visualization of used datasets. We blurred the facial areas for privacy.

Panel of Figure 8 — Figure 8: More visualizations of the proposed CSL-OpenWorld dataset, illustrating its diversity in backgrounds, signers, and topics. We blurred the facial areas for privacy.

Explore similar work

Aug 10, 2026

Bridging the Gap Between Semantics and Reconstruction:Unifying Sign Language Translation and Production

Recent advances in sign language (SL) research have shown a trend toward unifying multiple sign language understanding (SLU) subtasks, such as isolated sign language recognition (ISLR), continuous sign language recognition (CSLR), and sign language translation (SLT), within a single framework, leading to substantial progress. Meanwhile, sign language production (SLP), which generates sign sequences from text, has also attracted growing attention. This naturally raises an important question: can sign language understanding and production be unified within a single framework? Compared with unifying SLU subtasks, this problem is substantially more challenging. Existing SLU tasks largely share the same direction of mapping, namely from sign inputs to linguistic outputs, whereas SLT and SLP lie in opposite directions of sign-text mapping. A unified framework must therefore address two key challenges: (1) bridging the modality gap between continuous sign motions and discrete text tokens through a shared sign tokenizer that supports both linguistic abstraction and motion reconstruction; and (2) learning a single conditional autoregressive model that can take either sign or text as input and generate the corresponding target sequence in the opposite modality. To this end, we propose Uni-SLTP, a unified framework for SLT and SLP with two key components: (1) a shared sign tokenizer that converts sign sequences into discrete tokens and latent representations, capturing both semantic and reconstructive information; and (2) a unified autoregressive generation model that formulates both tasks as conditional sequence generation. Experiments on widely used public datasets show that Uni-SLTP achieves superior motion accuracy for SLP while maintaining competitive SLT performance.

Xiao Liu, Shiwei Gan, Yafeng Yin +5

State Key Laboratory of Novel Software Technology, Nanjing University, Suzhou, Jiangsu, China · State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, Jiangsu, China

Sign Language Translation Representation Learning Sign Language Generation

Ask PDF

Apr 16, 2026

Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translation

Many SLT systems quietly assume that brief chunks of signing map directly to spoken-language words. That assumption breaks down because signers often create meaning on the fly using context, space, and movement. We revisit SLT and argue that it is mainly a cross-modal reasoning task, not just a straightforward video-to-text conversion. We thus introduce a reasoning-driven SLT framework that uses an ordered sequence of latent thoughts as an explicit middle layer between the video and the generated text. These latent thoughts gradually extract and organize meaning over time. On top of this, we use a plan-then-ground decoding method: the model first decides what it wants to say, and then looks back at the video to find the evidence. This separation improves coherence and faithfulness. We also built and released a new large-scale gloss-free SLT dataset with stronger context dependencies and more realistic meanings. Experiments across several benchmarks show consistent gains over existing gloss-free methods. Our code and data are available at https://github.com/fletcherjiang/SignThought.

Yiyang Jiang, Li Zhang, Xiao-Yong Wei +1

The Hong Kong Polytechnic University · Sichuan University

Sign Language Translation Video-Language Models Latent CoT Reasoning

Ask PDF

Apr 13, 2026

Sign Language Recognition in the Age of LLMs

Recent Vision Language Models (VLMs) have demonstrated strong performance across a wide range of multimodal reasoning tasks. This raises the question of whether such general-purpose models can also address specialized visual recognition problems such as isolated sign language recognition (ISLR) without task-specific training. In this work, we investigate the capability of modern VLMs to perform ISLR in a zero-shot setting. We evaluate several open-source and proprietary VLMs on the WLASL300 benchmark. Our experiments show that, under prompt-only zero-shot inference, current open-source VLMs remain far behind classic supervised ISLR classifiers by a wide margin. However, follow-up experiments reveal that these models capture partial visual-semantic alignment between signs and text descriptions. Larger proprietary models achieve substantially higher accuracy, highlighting the importance of model scale and training data diversity. All our code is publicly available on GitHub.

Vaclav Javorek, Jakub Honzik, Ivan Gruber +2

University of West Bohemia, Czech Republic · Eindhoven University of Technology, The Netherlands

Sign Language Recognition VLM Evaluation Zero-Shot Learning

Ask PDF