SignGPT: Toward LLM-Mediated Sign Language Interaction through Gloss-Free Translation and Generation

Ronghui Li; Jun Dong; Zhongyuan Hu; Zunnan Xu; Jun Zhou; Liyuan Chen; Shuoling Liu; Jiangpeng Yan

cs.CVSep 18, 2026

SignGPT: Toward LLM-Mediated Sign Language Interaction through Gloss-Free Translation and Generation

Authors: Ronghui Li, Jun Dong, Zhongyuan Hu, Zunnan Xu, Jun Zhou, Liyuan Chen, Shuoling Liu, Jiangpeng Yan,

Organizations: Tsinghua University, China · Tencent AI Lab, China

Abstract

Large language models (LLMs) provide limited support for sign language interaction. Unifying sign language translation (SLT) and generation (SLG) to enable sign language as both input and output can reduce switching between separate models during sign-text interaction. We present SignGPT, a unified, pose-based framework for gloss-free SLT and SLG. SignGPT integrates part-aware hierarchical representations of body, hand, and facial motion into a shared language model and employs asymmetric multi-token prediction and progressive training for bidirectional modeling. We evaluate SignGPT on How2Sign (ASL) and Phoenix-2014T (DGS) through benchmark comparisons, qualitative analyses, and component ablations. An exploratory study with 12 Deaf ASL signers assesses an LLM-mediated sign-to-sign response pipeline, highlighting the potential of unified modeling to support sign language conversation (SLC).

Figures & tables

Figure 1. A central SignGPT model connects translation and generation. On the left, a text instruction is converted into a sequence of signing avatars. In the middle, an input signing sequence is translated into text. On the right, a signed question is translated, answered in English, and converted into a signed response, illustrating the one-turn response pipeline. SignGPT jointly models sign-to-text translation and text-to-sign generation within each dataset-specific model, and we explore how these complementary capabilities can be composed in a one-turn sign-to-sign response pipeline.

Figure 2. Overview of SignGPT’s three training stages: (1) PHVQ discretizes continuous body and hand motion, with an optional text-alignment objective; (2) GHMLM is jointly trained on SLG and SLT and uses AMTP to decode text or part-aware motion tokens; and (3) instruction fine-tuning supports prompted translation and generation. A three-stage pipeline. Stage 1 trains PHVQ encoders, codebooks, a projector–reconstructor path, and a decoder; an optional frozen text-embedding branch supplies the alignment target. Stage 2 freezes the tokenizer and trains a heterogeneous motion-aware language model on text-to-sign and sign-to-text directions, using separate motion heads and a standard text head. Stage 3 freezes selected components and instruction-tunes text and motion prompts for translation and generation use. Flame and snowflake icons distinguish trainable and frozen modules.

Figure 3. Overview of the Part-aware Hierarchical VQ-VAE (PHVQ) tokenizer. PHVQ combines bidirectional multi-scale temporal modeling with body-to-hand hierarchical quantization and encodes a full-body sign language motion sequence into three discrete streams: a body–face codebook $B_{Body}$ , a left-hand codebook $B_{LH}$ , and a right-hand codebook $B_{RH}$ . Facial-expression features are encoded jointly with the body stream. A flow diagram separates an input motion into body–face, left-hand, and right-hand features. A body–face encoder and codebook produce global context, which is transformed and passed to the left- and right-hand encoders before their separate codebooks. The three quantized streams are projected, fused, and decoded into a holistic motion sequence. Insets show the bidirectional multi-scale temporal encoder–decoder and the optional frozen text-embedding branch used for cosine alignment. Dashed arrows denote loss paths, and solid arrows denote the training and inference flow.

Method	Bi-T	Gloss	Input		Phoenix-2014T (DTW $\downarrow$ )			How2Sign (DTW $\downarrow$ )
Method	Bi-T	Gloss	Pose	RGB	Body	Hand	Avg	Body	Hand	Avg
MotionGPT ( Jiang et al., 2023 )	✓	$\times$	✓	$\times$	8.97	10.14	9.45	9.43	10.96	9.82
SOKE ( Zuo et al., 2025 )	$\times$	$\times$	✓	$\times$	2.58	5.89	4.26	7.92	3.07	5.49
T2S-GPT ( Yin et al., 2024 )	$\times$	$\times$	✓	$\times$	7.32	9.86	8.28	7.15	11.21	8.49
MoMask ( Guo et al., 2024 )	$\times$	$\times$	✓	$\times$	3.55	6.75	5.38	-	-	-
NSA ( Baltatzis et al., 2024 )	$\times$	$\times$	✓	$\times$	3.09	6.80	5.41	7.83	7.33	7.44

Table 1. Sign language generation (SLG) on Phoenix-2014T and How2Sign. DTW-JPE is reported for the body, hands, and the aggregate labeled “Avg” in each implementation. Bi-T indicates support for both sign-to-text and text-to-sign directions within one dataset-specific model. SignGPT+TSA uses the optional text-embedding alignment objective with paired sentence-level translations during PHVQ training. MotionGPT results are reproduced using our matched sign-language adaptation; all other baseline values are transcribed from the source studies cited for the corresponding results. Bold and underline mark the best and second-best values.

Method	Bi-T	Gloss	Input		Phoenix-2014T (Dev/Test)				How2Sign (Test)
Method	Bi-T	Gloss	Pose	RGB	B4 $\uparrow$	R $\uparrow$	B4 $\uparrow$	R $\uparrow$	B4 $\uparrow$	R $\uparrow$
MotionGPT ( Jiang et al., 2023 )	✓	$\times$	✓	$\times$	11.53	27.14	10.98	27.05	8.72	28.61
SLTCC ( Jang et al., 2025 )	$\times$	$\times$	✓	$\times$	-	-	-	-	11.8	31.1
SLT ( Camgoz et al., 2020 )	$\times$	✓	$\times$	✓	20.69	45.54	20.17	45.34	-	-
CSGCR ( Zhao et al., 2021 )	$\times$	$\times$	$\times$	✓	15.08	38.96	15.18	38.85	-	-
Uni-Sign ( Li et al., 2025b )	$\times$	$\times$	✓	✓	-	-	-	-	14.9	36.0

Table 2. Sign language translation (SLT) on Phoenix-2014T and How2Sign. We report BLEU-4 (B4) and ROUGE-L (R). Bi-T indicates support for both sign-to-text translation and text-to-sign generation within one dataset-specific model. MotionGPT results are reproduced using our matched sign-language adaptation; all other baseline values are transcribed from the source studies cited for the corresponding results.

Figure 4. Qualitative comparison between SignGPT and MotionGPT on SLT and SLG. The upper half shows reference, MotionGPT, and SignGPT avatar frames for text-to-sign generation on How2Sign and Phoenix-2014T; red circles mark visibly incorrect hand or arm configurations. The lower half shows input signing frames followed by reference text and the translations from both models. SignGPT outputs remain closer to the reference motion and text than MotionGPT outputs. A two-dataset qualitative comparison.

Method	M2T	T2T	T2M	Pipeline M2M
Method	B4 $\uparrow$	DR $\uparrow$	B4 $\uparrow$	DR $\uparrow$
MotionGPT ( Jiang et al., 2023 )	10.26	0.681	4.90	0.089
SignGPT (Ours)	21.83	0.714	13.69	0.236

Table 3. Stage-wise automatic proxy metrics for the LLM-mediated single-turn ASL response pipeline on How2Sign. M2T compares the translated question against the reference question; T2T scores the (translated question, English reply) pair; T2M compares the English reply against the text back-translated by a frozen evaluator from its generated pose sequence; M2M scores the (reference question, back-translated reply) pair. B4 denotes BLEU-4 and DR denotes DialogRPT-updown. Both systems use the same frozen mediator, response prompt, sampling configuration, and frozen SLT evaluator.

Method	Response Relevance	Subjective Pilot (1–5)
Method	LLM-AR (%) $\uparrow$	Response Appropriateness $\uparrow$	Motion Smoothness $\uparrow$
MotionGPT ( Jiang et al., 2023 )	15.7	2.71	1.26
SignGPT (Ours)	52.2	3.67	4.35

Table 4. End-to-end response relevance and exploratory subjective ratings for the single-turn ASL response pipeline. For each system, LLM-AR is the percentage of its 1,000 outputs judged relevant by GPT-4o after the shared frozen SLT back-translation, with one binary judgment per output (

n=1{,}000

per system). The subjective results are descriptive means from 12 Deaf participants. Every sample was rated by three different participants; each system–item mean therefore summarizes 1,000 three-rater sample averages (3,000 raw ratings). Response appropriateness examines whether the signed reply plausibly answers the signed question; motion smoothness examines whether the rendered motion is free of frame-to-frame jitter.

Figure 5. Qualitative examples from the LLM-mediated, one-turn ASL response pipeline. GTQ-Text is the reference question; Q-Motion is the input query; Q-Text is the SLT output; A-Text is the response-model output; A-Motion is the SLG output; and BTA-Text is the frozen evaluator’s back-translation. The examples show one semantically related response and one case exhibiting a discrepancy. Two one-turn response examples compare MotionGPT in the left column with SignGPT in the right column. Each example begins with a reference question and input signing frames, followed by the translated question, generated answer text, answer signing frames, and back-translated answer.

Dataset	Method				MPJPE $\downarrow$	PA-MPJPE $\downarrow$	DTW-JPE $\downarrow$	ACCEL $\downarrow$
Dataset	Part-Aware	BM-TCN	Hand-E	TSA	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	DTW-JPE $\downarrow$	ACCEL $\downarrow$
Phoenix-2014T	Vanilla VQ-VAE				45.5	30.9	4.52	39.4
	✓				26.7	19.3	2.42	35.2
	✓	✓			24.5	19.1	2.31	34.8
	✓	✓	✓		21.7	16.9	2.03	32.4
	✓	✓	✓	✓	22.0	17.0	2.07	32.3

Table 5. Ablation study on constructing PHVQ across two datasets. The blank row denotes the vanilla VQ-VAE baseline without any of the following components. Part-Aware: modeling different body parts using independent codebooks (body and face, left hand, right hand); TCN: temporal convolutional network for enhanced temporal modeling; Hand-E: incorporating

\mathcal{L}_{\text{Pos}}

and

\mathcal{L}_{\text{Angle}}

to strengthen fine-grained quantization and reconstruction quality of fingers. (Text-Space Alignment) TSA: adding the optional cosine loss between pooled motion projections and the frozen LLaMA input-embedding target during PHVQ training.

Method				SLG	SLT
PHVQ	PHVQ+TSA	GHMLM	GHMLM IFT	Avg $\downarrow$	B4 $\uparrow$	R $\uparrow$
MotionGPT				9.45	10.98	27.05
✓				8.13	12.54	30.72
✓		✓		4.48	22.18	39.86
✓		✓	✓	4.36	22.95	41.28
	✓	✓	✓	4.32	23.56	42.13

Table 6. Cumulative SignGPT ablations on the Phoenix-2014T test set. The first row is the adapted MotionGPT baseline. PHVQ+TSA uses paired sentence-level translations for text-space alignment during tokenizer training. GHMLM denotes joint LoRA-based SLT–SLG model adaptation, IFT denotes instruction fine-tuning, and Avg is the reported aggregate DTW-JPE.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Input Format			Phoenix-2014T				How2Sign
Pos.	Rot.	Vel.	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	DTW-JPE $\downarrow$	ACCEL $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	DTW-JPE $\downarrow$	ACCEL $\downarrow$
✓	✓	✓	23.1	18.2	2.24	28.5	22.3	17.9	1.97	6.38
✓		✓	23.3	17.9	2.26	28.9	22.9	18.3	2.14	6.46
✓	✓		24.6	18.5	2.42	33.1	23.1	19.4	2.18	6.70
✓			21.7	16.9	2.03	32.4	18.3	15.6	1.66	6.17

Appendix

Table 7. Ablation study on the impact of different motion representations on PHVQ reconstruction quality across two sign language datasets.

Codebook Size		Phoenix-2014T (DTW $\downarrow$ )			How2Sign (DTW $\downarrow$ )
$N_{BF}$	$N_{LH}=N_{RH}$	Body	Hand	Avg	Body	Hand	Avg
128	192	1.89	2.55	2.26	1.43	2.05	1.76
128	320	1.86	2.32	2.14	1.40	1.90	1.65
160	256	1.85	2.49	2.18	1.41	1.95	1.68
96	256	1.97	2.41	2.23	1.48	1.96	1.72
128	256	1.82	2.21	2.03	1.39	1.93	1.66

Appendix

Table 8. PHVQ reconstruction results for different body–face and hand codebook sizes. We use

N_{BF}=128

and

N_{LH}=N_{RH}=256

in the remaining experiments.

Figure 6. Statistics of discrete tokens in the sign language motion vocabularies (codebooks) for different body parts after quantizing the training samples. For the Phoenix-2014T and How2Sign datasets, from left to right the represented body parts are body–face ( $N_{BF}=128$ ), left hand ( $N_{LH}=256$ ), and right hand ( $N_{RH}=256$ ). Six filled line plots show token frequency by codebook index. The top row covers Phoenix-2014T and the bottom row covers How2Sign; columns represent body-face, left-hand, and right-hand codebooks. Frequencies vary substantially across entries, but activity is distributed across the full index ranges rather than concentrated in only a few codes.

Encoder	Phoenix-2014T				How2Sign
Encoder	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	DTW-JPE $\downarrow$	ACCEL $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	DTW-JPE $\downarrow$	ACCEL $\downarrow$
Causal TCN ( Bai et al., 2018 )	30.6	23.9	2.94	43.6	25.8	22.0	2.40	8.30
ResNet ( Zhang et al., 2023 )	23.6	17.4	2.17	33.9	19.9	16.1	1.78	6.45
BM-TCN (Ours)	21.7	16.9	2.03	32.4	18.3	15.6	1.66	6.17

Appendix

Table 9. Ablation study on different encoder designs in PHVQ.

Hierarchical Strategy	Phoenix-2014T				How2Sign
Hierarchical Strategy	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	DTW-JPE $\downarrow$	ACCEL $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	DTW-JPE $\downarrow$	ACCEL $\downarrow$
Hand-to-Body ( Lu et al., 2023 )	23.2	17.5	2.14	33.0	19.6	16.2	1.75	6.30
Independent ( Lee et al., 2025 ; Zuo et al., 2025 )	22.0	17.3	2.04	32.7	18.5	16.0	1.67	6.22
Body-to-Hand (Ours)	21.7	16.9	2.03	32.4	18.3	15.6	1.66	6.17

Appendix

Table 10. Ablation study on the conditioning direction of the hierarchical quantization in PHVQ.

Dataset	$\beta$	SLG	SLT
		Avg-DTW $\downarrow$	B4 $\uparrow$	R $\uparrow$
Phoenix-2014T	0.2	4.65	19.58	36.19
	$1/3$	4.36	22.95	41.28
	0.4	4.38	23.07	41.26
How2Sign	0.2	5.32	13.81	35.27
	$1/3$	4.82	16.42	37.69

Appendix

Table 11. Ablation study of SignGPT using different motion embedding fusion weights

\beta

on two sign language datasets.

Dataset	Init. Scheme	SLG	SLT
		Avg-DTW $\downarrow$	B4 $\uparrow$	R $\uparrow$
Phoenix-2014T	Random Init	10.54	12.90	26.10
	Codebook	4.36	22.95	41.28
How2Sign	Random Init	8.85	9.23	23.83
	Codebook	4.82	16.42	37.69

Appendix

Table 12. Ablation study on the motion embedding initialization scheme of SignGPT. “Random Init” adds new motion tokens with randomly initialized embeddings; “Codebook Reuse” reuses frozen PHVQ codebook features projected by

P^{\uparrow}_{p}

Dataset	Decoding Scheme	SLG	SLT
		Avg-DTW $\downarrow$	B4 $\uparrow$	R $\uparrow$
Phoenix-2014T	Serial-AR	5.48	14.59	28.00
	AMTP (ours)	4.36	22.95	41.28
How2Sign	Serial-AR	5.71	10.44	25.56
	AMTP (ours)	4.82	16.42	37.69

Appendix

Table 13. Comparison between parallel multi-head decoding (AMTP) and serial single-head autoregressive decoding of body-part motion tokens.

Figure 7. Qualitative comparison between SignGPT and MotionGPT for sign language generation on the test sets of Phoenix-2014T (left) and How2Sign (right). Two generation examples for each dataset compare ground-truth signing frames with MotionGPT and SignGPT outputs. For both German weather text and English instructional text, SignGPT reproduces hand positions and upper-body trajectories that more closely follow the reference sequences, whereas MotionGPT exhibits larger pose and hand-shape deviations.

Figure 8. Qualitative comparison between SignGPT and MotionGPT for sign-to-text translation on the test sets of Phoenix-2014T (left) and How2Sign (right). Two translation examples per dataset list the reference sentence followed by MotionGPT and SignGPT outputs. The SignGPT sentences retain more of the reference meaning in both German weather forecasts and English instructional descriptions, while MotionGPT introduces unrelated or contradictory content.

Sign Language Translation

Sign language translation maps continuous signing into written or spoken-language text using visual, pose, and linguistic cues.

See all 50 papers

SignRAG: Unified Retrieval-Augmented Gloss-Free Sign Language Translation
Oct 8, 2026Zhi Rao, Yucheng Zhou, Qianran Sun +8Sign Language Translation Retrieval-Augmented Generation
Faculty of Innovation Engineering, Macau University of Science and Technology, Macau, China · SKL-IOTSC, CIS, University of Macau, Macau, China · MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China +2
Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translation
Apr 16, 2026Yiyang Jiang, Li Zhang, Xiao-Yong Wei +1Sign Language Translation Video-Language Models
The Hong Kong Polytechnic University · Sichuan University
SignFLIP: A Unified Model for Sign Language Translation and Generation via Stage-wise Alignment at Scale
Sep 28, 2026Zhaoyi An, Sihan Tan, Youngbae Hwang +2Sign Language Translation Representation Alignment
Institute of Science Tokyo · Chungbuk National University

Oct 8, 2026J/K move · Enter open · S save

SignRAG: Unified Retrieval-Augmented Gloss-Free Sign Language Translation

Zhi Rao, Yucheng Zhou, Qianran Sun, Yiqing Huang+7

Faculty of Innovation Engineering, Macau University of Science and Technology, Macau, China · SKL-IOTSC, CIS, University of Macau, Macau, China · MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China+2

Sign Language Translation Retrieval-Augmented Generation

Open paper AskPDF arXiv

Contemporary decoder-only large language models (LLMs) have demonstrated strong capabilities across a wide range of domains. However, existing pretraining paradigms for gloss-free sign language translation (SLT) are largely designed around conventional encoder-decoder pretrained language models, which limits their direct applicability to decoder-only LLMs. To address this limitation, we propose SignRAG, a unified framework combining hierarchical pretraining, target-domain retrieval augmentation, and retrieval-aware reinforcement fine-tuning. Hierarchical pretraining first learns linguistically grounded sign representations and then jointly aligns the sign encoder with an LLM, mitigating cross-modal optimization imbalance. For downstream adaptation, SignRAG complements parameter-based fine-tuning with a target-domain retrieval gallery that provides instance-specific translation cues. To ensure that retrieved contexts are used appropriately, we further introduce Retrieval Utility-Guided Reinforcement Fine-Tuning (RUG-RFT), which combines translation-quality and retrieval-utility rewards to encourage beneficial retrieval use while suppressing harmful reliance. Experiments on multiple SLT benchmarks establish new state-of-the-art performance. In particular, to the best of our knowledge, SignRAG is the first gloss-free approach to outperform gloss-supervised methods across all reported metrics on CSL-Daily. Our code has been released at GitHub, together with models of different sizes to support future academic research.

Abstract

Sign Language Translation

Explore similar work

SignRAG: Unified Retrieval-Augmented Gloss-Free Sign Language Translation

Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translation

SignFLIP: A Unified Model for Sign Language Translation and Generation via Stage-wise Alignment at Scale