Sign language generation (SLG) maps written sentences into signing motion while preserving both lexical content and fine-grained articulation. This requires a model to decide what signs should be expressed and to realize them through precise handshape, trajectory, facial expression and body coordination. Existing pose-based systems and reconstruction-only motion tokenizers often entangle these two decisions, leaving the generator without an explicit text-aligned intermediate target for what to sign before refining how to sign it. To address this issue, we introduce Semantic Residual Vector Quantization (SRVQ), a framework for SLG that frames generation as a two-level process moving from semantics to kinematics. The newly introduced SRV Quantizer learns a hierarchy in which the first residual level encodes sign content and subsequent levels refine pose, handshape, trajectory, and body coordination, all within a single quantizer and without separate semantic representations or external planning stages. Through word-aware conditioning for hierarchical code generation, a masked semantic generator predicts semantic codes and a kinematic generator completes the residual codes for full SMPL-X motion synthesis. We evaluate using retrieval and FID in a shared text-motion embedding space, alongside standard geometric metrics. On CSL-Daily and Phoenix-2014T, SRVQ achieves R@1 of 82.3 and 70.3, outperforming the previous SOTA by 30 and 46 points, respectively.
Figures & tables
Figure 1: Comparison of quantization paradigms for SLG. (a) Kinematic-only quantization learns motion codes from motion alone, leaving semantics implicit. (b) Separate semantic-kinematic quantization models semantics and motion separately using additional semantic representations. (c) Our hierarchical semantic residual quantization embeds semantics and kinematics within a single residual quantizer, where the first residual level captures semantics and the remaining levels capture kinematic details.
Figure 2: Overview of the SRVQ framework. Top: The SRV Quantizer learns a hierarchical discrete representation of each SMPL-X motion. The first residual level ( Q0 ) learns a text-aligned semantic codebook, while the remaining residual levels Q1 – QL progressively capture finer kinematic details. Bottom: During generation, word embeddings together with a global sentence token form a word-aware textual representation. A Semantic Generator predicts Q0 from this representation, and a kinematic generator predicts Q1 - QL before the part decoders reconstruct the continuous SMPL-X motion.
Figure 3: Top: Each quantization level encodes the residual from the previous level, with semantic supervision applied only to Q0 . The final representation consists of the semantic code and residual kinematic codes. Bottom: During inference, the Semantic Generator predicts Q0 from the word-aware textual representation, followed by the Kinematic Generator predicting Q1,…,QL .
CSL-Daily
Phoenix-2014T
Method
Retrieval / FID
DTW-JPE
DTW-PA
Retrieval / FID
DTW-JPE
DTW-PA
R@1 ↑
R@5 ↑
R@10 ↑
FID ↓
Body ↓
Hands ↓
Body ↓
Hands ↓
R@1 ↑
R@5 ↑
R@10 ↑
FID ↓
Body ↓
Hands ↓
Body ↓
Hands ↓
Prog. Trans. [ 31 ]
–
–
–
–
16.30
32.63
15.98
12.91
–
–
–
–
15.01
31.77
13.67
11.95
Text2Mesh [ 36 ]
–
–
–
–
13.76
30.37
13.47
12.10
–
–
–
–
14.04
31.64
13.48
12.06
T2S-GPT [ 42 ]
–
–
–
–
12.32
15.43
11.94
5.93
–
–
–
–
11.65
19.09
10.38
6.47
S-MotionGPT [ 17 ]
–
–
–
–
11.58
11.31
10.81
3.78
–
–
–
–
10.42
9.08
9.45
3.41
Table 1: Comparison with state-of-the-art methods on CSL-Daily and Phoenix-2014T. The best and second-best results are highlighted in bold and underline , respectively.
Figure 4: Qualitative comparison of generated sign sequences. We show representative frames from ground truth (GT), SRVQ semantic-only generation, SRVQ sem+kinematic generation, and SOKE. The text above each example is the corresponding input sentence. The first two examples are from CSL-Daily, and the last example is from Phoenix-2014T.
Method
CSL-Daily
Phoenix-2014T
R@1 ↑
R@5 ↑
R@10 ↑
R@1 ↑
R@5 ↑
R@10 ↑
λtext=0 , λgloss=0
72.96
87.24
91.75
52.96
84.27
91.12
λtext=0.3 , λgloss=0.3
80.78
91.16
93.96
74.92
92.83
95.79
No-gloss (λgloss=0.0)
75.60
89.80
93.62
58.88
84.27
90.50
SRVQ (sem+kinematic)
82.31
94.13
96.34
70.25
87.54
92.52
Table 2: Semantic alignment ablation. Bold indicates the best result and underline the second best.
Representation
NN ↑
P@5 ↑
P@10 ↑
Sil. ↑
Reconstruction-only [ 48 ]
56.2
52.8
49.6
0.032
SRVQ ( Q0 )
73.8
73.4
73.2
0.260
Table 3: Quantitative lexical organization of semantic codes.
Figure 5: Lexical organization of motion codes. t-SNE visualization of gloss-span motion features on CSL-Daily. Colors denote gloss identities. SRVQ level-0 forms clearer lexical neighborhoods than reconstruction-only tokenization.
esent
W
CSL-Daily
Phoenix-2014T
R@1 ↑
R@5 ↑
R@10 ↑
R@1 ↑
R@5 ↑
R@10 ↑
✓
✗
75.34
90.14
93.62
41.28
71.81
79.28
✓
✓
82.31
94.13
96.34
70.25
87.54
92.52
Table 4: Text conditioning ablation.
Depth
CSL-Daily
Phoenix-2014T
R@1 ↑
R@5 ↑
R@10 ↑
R@1 ↑
R@5 ↑
R@10 ↑
1 †
75.77
89.54
93.28
61.21
83.49
88.32
2
80.53
93.11
95.24
65.89
87.54
92.37
4
82.48
92.18
94.47
68.85
87.54
92.68
6
82.31
94.13
96.34
70.25
87.54
92.52
Table 5: Number of residuals ablation. † One-level VQ uses semantic-only decoding.
Dataset / Ratio
DTW-B ↓
DTW-H ↓
FID ↓
R@1 ↑
CSL-Daily
1.0
6.787
8.535
0.058
82.31
0.9
6.763
8.532
0.059
80.53
0.8
6.751
8.530
0.062
74.15
0.7
6.613
8.408
0.080
64.37
0.6
6.561
8.365
0.105
55.02
Table 6: Length sensitivity of DTW.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset / Method
BLEU-1
BLEU-2
BLEU-4
ROUGE
Phoenix-2014T
SOKE
24.75
13.65
7.05
25.14
SRVQ (sem+kinematic)
35.92
23.56
13.47
36.05
CSL-Daily
SOKE
21.60
11.62
4.33
22.58
SRVQ (sem+kinematic)
25.81
14.99
5.97
26.93
Appendix
Table 7: Back-translation with the Two-Stream SLT evaluator. The Two-Stream Network [ 5 ] is adapted from keypoint input to SMPL-X input.
Dataset / Method
BLEU-1
BLEU-2
BLEU-4
ROUGE
Phoenix-2014T
SOKE
24.44
14.28
7.90
25.80
SRVQ (sem+kinematic)
39.36
27.40
16.39
40.96
CSL-Daily
SOKE
25.42
14.78
5.50
25.99
SRVQ (sem+kinematic)
29.49
18.02
7.42
30.38
Appendix
Table 8: Back-translation with the MSKA SLT evaluator. MSKA [ 11 ] is adapted from keypoint input to SMPL-X input.
(a) Pose Retrieval
All joints
Hands
Dataset / Method
R@1 ↑
R@5 ↑
R@10 ↑
MRR ↑
MedR ↓
R@1 ↑
R@5 ↑
R@10 ↑
MRR ↑
MedR ↓
CSL-Daily
SOKE [ 48 ]
1.956
5.697
9.014
0.0472
216.0
2.126
5.527
8.844
0.0474
216.0
SRVQ (sem+kinematic)
2.806
7.823
11.650
0.0607
192.0
2.976
7.908
11.310
0.0603
195.5
SRVQ (sem-only)
2.551
6.718
10.714
0.0548
202.0
2.381
6.378
9.779
0.0533
206.5
Appendix
Table 9: Meaningful Pose evaluation on CSL-Daily and Phoenix-2014T. SMPL-X motion sequences are converted to OpenPose keypoints. The official DTW and Aggregated Power Distance metrics reported by the evaluation toolkit correspond to the “All” DTW and MJE columns, respectively. Best results on each dataset are shown in bold .
Conditioning
R@1 ↑
R@5 ↑
R@10 ↑
FID ↓
DTW-B ↓
DTW-H ↓
Word-aware (ours)
82.31
94.13
96.34
0.058
6.79
8.54
Word tokens only
80.87
91.07
95.15
0.057
6.83
8.63
Appendix
Table 10: Word-token-only comparison on CSL-Daily. Both rows use the final generation-validation protocol. DTW-B: DTW-JPE body; DTW-H: DTW-JPE averaged over left and right hands.
Setting
CSL-Daily
Phoenix-2014T
R@1 ↑
R@5 ↑
R@10 ↑
FID ↓
R@1 ↑
R@5 ↑
R@10 ↑
FID ↓
SRVQ (sem+kinematic)
82.31
94.13
96.34
0.0580
70.25
87.54
92.52
0.0590
Gloss span shift ± 4 frames
80.61
92.09
94.81
0.0488
70.40
90.19
94.86
0.0562
Gloss dropout 20%
77.47
91.24
94.22
0.0495
69.78
91.12
94.70
0.0567
Appendix
Table 11: Robustness to gloss perturbations. We report sem+kinematic generation for the final model and perturbation settings.
Dataset / Method
R@1 ↑
FID ↓
DTW-B ↓
DTW-H ↓
CSL-Daily
SRVQ full reconstruction
85.97
0.002
1.78
2.73
SRVQ Q0 reconstruction
74.49
0.018
3.33
4.81
Phoenix-2014T
SRVQ full reconstruction
64.49
0.008
1.56
2.89
SRVQ Q0 reconstruction
40.03
0.067
3.15
5.14
Appendix
Table 12: Tokenizer reconstruction analysis. DTW-B: DTW-JPE body; DTW-H: DTW-JPE averaged over left and right hands.
Dataset / Code combination
R@1
R@5
DTW-B
DTW-H
PA-H
CSL-Daily
Q^0+ zero residual
77.81
89.97
6.296
8.376
1.508
Q^0+Q^1:L
82.31
94.13
6.787
8.535
1.544
Q^0+Q1:LGT
77.98
88.95
5.140
7.519
1.513
Q0GT+Q^1:L
74.15
86.90
4.945
6.430
0.845
Q0GT+Q1:LGT
85.97
93.45
1.779
2.727
0.390
Appendix
Table 13: Controlled residual contribution analysis. Q^ℓ denotes predicted codes at level ℓ and QℓGT denotes codes from the ground-truth motion. DTW-B, DTW-H, and PA-H are lower-is-better; hand metrics average the left and right hands.
Figure 6: Lexical organization of motion codes. t-SNE visualization of gloss-span motion features on Phoenix-2014T. Colors denote gloss identities. SRVQ level-0 forms clearer lexical neighborhoods than reconstruction-only tokenization.
Metric
Dataset
Part
Q0
Q1
Q2
Q3
Q4
Q5
Active codes (%) ↑
CSL-Daily
Body
79.17
96.88
95.83
97.92
97.92
100.00
Left hand
99.61
100.00
100.00
100.00
100.00
100.00
Right hand
100.00
100.00
99.61
99.61
97.66
84.38
Phoenix-2014T
Body
22.92
91.67
94.79
94.79
100.00
97.92
Left hand
98.83
99.22
98.83
99.61
100.00
99.61
Right hand
99.61
99.61
100.00
100.00
98.05
91.41
Appendix
Table 14: Per-level codebook utilization of the final SRVQ tokenizer. Active codes and normalized perplexity are higher-is-better, while Top-1 usage is lower-is-better.
English display text
Dataset
Sample ID
Original annotated sentence
You’re my client, so dinner’s on me.
CSL-Daily
S001423_P0004_T00
你是我的客户,我应该请你吃饭。
Exam tomorrow. Bring a pen, no phone.
CSL-Daily
S001372_P0008_T00
明天考试,带上笔,不要带手机。
Today’s weather will definitely remain wintry.
PHOENIX-2014T
26January_2010_Tuesday_heute-105
in den nächsten tagen geht es auf jeden fall winterlich weiter immer wieder mal mit etwas schnee .
Appendix
Table 15: Qualitative sample metadata. The English display text, dataset, sample ID, and original annotated sentence for the qualitative examples.
Representation
NN ↑
P@5 ↑
P@10 ↑
Sil. ↑
Reconstruction-only [ 48 ]
52.4
43.7
37.4
-0.083
SRVQ Q0
69.4
66.9
65.2
0.199
Appendix
Table 16: Quantitative evaluation of lexical organization on Phoenix-2014T.
Dataset / Type
Codes
NN ↑
P@5 ↑
P@10 ↑
Sil. ↑
Phoenix-2014T
Cumulative
Q0
0.6940
0.6692
0.6520
0.1995
Q0 – Q1
0.7120
0.6888
0.6748
0.1942
Q0 – Q2
0.7160
0.7008
0.6896
0.1984
Q0 – Q3
0.7220
0.7064
0.6880
0.1988
Q0 – Q4
0.7280
0.7132
0.6900
0.1999
Appendix
Table 17: Lexical organization of cumulative and individual SRVQ codes. NN, P@5, P@10, and Sil. measure same-gloss neighborhood quality, with higher values being better. “Cumulative” concatenates the codes from Q0 through the indicated residual level, while “Individual” evaluates each residual level independently.
Seed
R@1 ↑
R@5 ↑
R@10 ↑
FID ↓
DTW-B ↓
DTW-H ↓
0
82.23
93.28
96.00
0.050
6.62
8.40
1
82.91
93.96
95.92
0.050
6.60
8.41
2
83.08
94.05
96.09
0.050
6.58
8.41
Mean ± std
82.74±0.45
93.76±0.42
96.00±0.09
0.050±0.000
6.60±0.02
8.41±0.01
Appendix
Table 18: Seed stability on CSL-Daily. All rows use the same sem+kinematic checkpoint. DTW-B: DTW-JPE body; DTW-H: DTW-JPE averaged over left and right hands.
Dataset
Length
N
R@1 ↑
R@5 ↑
R@10 ↑
DTW-B ↓
DTW-H ↓
CSL-Daily
≤ 100
422
82.94
94.31
96.45
4.61
5.78
101–140
367
80.11
94.28
96.46
6.62
8.32
> 140
387
83.72
93.80
96.12
9.31
11.75
Phoenix-2014T
≤ 76
219
63.01
82.65
89.50
3.14
4.20
77–112
216
75.00
89.35
95.37
5.14
6.70
> 112
207
72.95
90.82
92.75
7.85
10.03
Appendix
Table 19: Performance by ground-truth motion length. Retrieval is evaluated against the full test gallery. DTW-B: DTW-JPE body; DTW-H: DTW-JPE averaged over left and right hands.
Sign language generation creates signed motion or video from text, glosses, or other inputs while preserving linguistic structure and natural movement.