Semantic Residual Vector Quantization for Sign Language Generation
Organizations: University of Trento · Fondazione Bruno Kessler
Abstract
Sign language generation (SLG) maps written sentences into signing motion while preserving both lexical content and fine-grained articulation. This requires a model to decide what signs should be expressed and to realize them through precise handshape, trajectory, facial expression and body coordination. Existing pose-based systems and reconstruction-only motion tokenizers often entangle these two decisions, leaving the generator without an explicit text-aligned intermediate target for what to sign before refining how to sign it. To address this issue, we introduce Semantic Residual Vector Quantization (SRVQ), a framework for SLG that frames generation as a two-level process moving from semantics to kinematics. The newly introduced SRV Quantizer learns a hierarchy in which the first residual level encodes sign content and subsequent levels refine pose, handshape, trajectory, and body coordination, all within a single quantizer and without separate semantic representations or external planning stages. Through word-aware conditioning for hierarchical code generation, a masked semantic generator predicts semantic codes and a kinematic generator completes the residual codes for full SMPL-X motion synthesis. We evaluate using retrieval and FID in a shared text-motion embedding space, alongside standard geometric metrics. On CSL-Daily and Phoenix-2014T, SRVQ achieves R@1 of 82.3 and 70.3, outperforming the previous SOTA by 30 and 46 points, respectively.
Figures & tables
| CSL-Daily | Phoenix-2014T | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Retrieval / FID | DTW-JPE | DTW-PA | Retrieval / FID | DTW-JPE | DTW-PA | ||||||||||
| R@1 | R@5 | R@10 | FID | Body | Hands | Body | Hands | R@1 | R@5 | R@10 | FID | Body | Hands | Body | Hands | |
| Prog. Trans. [ 31 ] | – | – | – | – | 16.30 | 32.63 | 15.98 | 12.91 | – | – | – | – | 15.01 | 31.77 | 13.67 | 11.95 |
| Text2Mesh [ 36 ] | – | – | – | – | 13.76 | 30.37 | 13.47 | 12.10 | – | – | – | – | 14.04 | 31.64 | 13.48 | 12.06 |
| T2S-GPT [ 42 ] | – | – | – | – | 12.32 | 15.43 | 11.94 | 5.93 | – | – | – | – | 11.65 | 19.09 | 10.38 | 6.47 |
| S-MotionGPT [ 17 ] | – | – | – | – | 11.58 | 11.31 | 10.81 | 3.78 | – | – | – | – | 10.42 | 9.08 | 9.45 | 3.41 |
| Method | CSL-Daily | Phoenix-2014T | ||||
|---|---|---|---|---|---|---|
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | |
| , | 72.96 | 87.24 | 91.75 | 52.96 | 84.27 | 91.12 |
| , | 80.78 | 91.16 | 93.96 | 74.92 | 92.83 | 95.79 |
| No-gloss | 75.60 | 89.80 | 93.62 | 58.88 | 84.27 | 90.50 |
| SRVQ (sem+kinematic) | 82.31 | 94.13 | 96.34 | 70.25 | 87.54 | 92.52 |
| Representation | NN | P@5 | P@10 | Sil. |
|---|---|---|---|---|
| Reconstruction-only [ 48 ] | 56.2 | 52.8 | 49.6 | 0.032 |
| SRVQ ( ) | 73.8 | 73.4 | 73.2 | 0.260 |
| CSL-Daily | Phoenix-2014T | ||||||
|---|---|---|---|---|---|---|---|
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | ||
| ✓ | ✗ | 75.34 | 90.14 | 93.62 | 41.28 | 71.81 | 79.28 |
| ✓ | ✓ | 82.31 | 94.13 | 96.34 | 70.25 | 87.54 | 92.52 |
| Depth | CSL-Daily | Phoenix-2014T | ||||
|---|---|---|---|---|---|---|
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | |
| 1 † | 75.77 | 89.54 | 93.28 | 61.21 | 83.49 | 88.32 |
| 2 | 80.53 | 93.11 | 95.24 | 65.89 | 87.54 | 92.37 |
| 4 | 82.48 | 92.18 | 94.47 | 68.85 | 87.54 | 92.68 |
| 6 | 82.31 | 94.13 | 96.34 | 70.25 | 87.54 | 92.52 |
| Dataset / Ratio | DTW-B | DTW-H | FID | R@1 |
|---|---|---|---|---|
| CSL-Daily | ||||
| 1.0 | 6.787 | 8.535 | 0.058 | 82.31 |
| 0.9 | 6.763 | 8.532 | 0.059 | 80.53 |
| 0.8 | 6.751 | 8.530 | 0.062 | 74.15 |
| 0.7 | 6.613 | 8.408 | 0.080 | 64.37 |
| 0.6 | 6.561 | 8.365 | 0.105 | 55.02 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset / Method | BLEU-1 | BLEU-2 | BLEU-4 | ROUGE |
|---|---|---|---|---|
| Phoenix-2014T | ||||
| SOKE | 24.75 | 13.65 | 7.05 | 25.14 |
| SRVQ (sem+kinematic) | 35.92 | 23.56 | 13.47 | 36.05 |
| CSL-Daily | ||||
| SOKE | 21.60 | 11.62 | 4.33 | 22.58 |
| SRVQ (sem+kinematic) | 25.81 | 14.99 | 5.97 | 26.93 |
| Dataset / Method | BLEU-1 | BLEU-2 | BLEU-4 | ROUGE |
|---|---|---|---|---|
| Phoenix-2014T | ||||
| SOKE | 24.44 | 14.28 | 7.90 | 25.80 |
| SRVQ (sem+kinematic) | 39.36 | 27.40 | 16.39 | 40.96 |
| CSL-Daily | ||||
| SOKE | 25.42 | 14.78 | 5.50 | 25.99 |
| SRVQ (sem+kinematic) | 29.49 | 18.02 | 7.42 | 30.38 |
| (a) Pose Retrieval | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| All joints | Hands | |||||||||
| Dataset / Method | R@1 | R@5 | R@10 | MRR | MedR | R@1 | R@5 | R@10 | MRR | MedR |
| CSL-Daily | ||||||||||
| SOKE [ 48 ] | 1.956 | 5.697 | 9.014 | 0.0472 | 216.0 | 2.126 | 5.527 | 8.844 | 0.0474 | 216.0 |
| SRVQ (sem+kinematic) | 2.806 | 7.823 | 11.650 | 0.0607 | 192.0 | 2.976 | 7.908 | 11.310 | 0.0603 | 195.5 |
| SRVQ (sem-only) | 2.551 | 6.718 | 10.714 | 0.0548 | 202.0 | 2.381 | 6.378 | 9.779 | 0.0533 | 206.5 |
| Conditioning | R@1 | R@5 | R@10 | FID | DTW-B | DTW-H |
|---|---|---|---|---|---|---|
| Word-aware (ours) | 82.31 | 94.13 | 96.34 | 0.058 | 6.79 | 8.54 |
| Word tokens only | 80.87 | 91.07 | 95.15 | 0.057 | 6.83 | 8.63 |
| Setting | CSL-Daily | Phoenix-2014T | ||||||
|---|---|---|---|---|---|---|---|---|
| R@1 | R@5 | R@10 | FID | R@1 | R@5 | R@10 | FID | |
| SRVQ (sem+kinematic) | 82.31 | 94.13 | 96.34 | 0.0580 | 70.25 | 87.54 | 92.52 | 0.0590 |
| Gloss span shift 4 frames | 80.61 | 92.09 | 94.81 | 0.0488 | 70.40 | 90.19 | 94.86 | 0.0562 |
| Gloss dropout 20% | 77.47 | 91.24 | 94.22 | 0.0495 | 69.78 | 91.12 | 94.70 | 0.0567 |
| Dataset / Method | R@1 | FID | DTW-B | DTW-H |
|---|---|---|---|---|
| CSL-Daily | ||||
| SRVQ full reconstruction | 85.97 | 0.002 | 1.78 | 2.73 |
| SRVQ reconstruction | 74.49 | 0.018 | 3.33 | 4.81 |
| Phoenix-2014T | ||||
| SRVQ full reconstruction | 64.49 | 0.008 | 1.56 | 2.89 |
| SRVQ reconstruction | 40.03 | 0.067 | 3.15 | 5.14 |
| Dataset / Code combination | R@1 | R@5 | DTW-B | DTW-H | PA-H |
|---|---|---|---|---|---|
| CSL-Daily | |||||
| zero residual | 77.81 | 89.97 | 6.296 | 8.376 | 1.508 |
| 82.31 | 94.13 | 6.787 | 8.535 | 1.544 | |
| 77.98 | 88.95 | 5.140 | 7.519 | 1.513 | |
| 74.15 | 86.90 | 4.945 | 6.430 | 0.845 | |
| 85.97 | 93.45 | 1.779 | 2.727 | 0.390 | |
| Metric | Dataset | Part | Q0 | Q1 | Q2 | Q3 | Q4 | Q5 |
|---|---|---|---|---|---|---|---|---|
| Active codes (%) | CSL-Daily | Body | 79.17 | 96.88 | 95.83 | 97.92 | 97.92 | 100.00 |
| Left hand | 99.61 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | ||
| Right hand | 100.00 | 100.00 | 99.61 | 99.61 | 97.66 | 84.38 | ||
| Phoenix-2014T | Body | 22.92 | 91.67 | 94.79 | 94.79 | 100.00 | 97.92 | |
| Left hand | 98.83 | 99.22 | 98.83 | 99.61 | 100.00 | 99.61 | ||
| Right hand | 99.61 | 99.61 | 100.00 | 100.00 | 98.05 | 91.41 |
| English display text | Dataset | Sample ID | Original annotated sentence |
|---|---|---|---|
| You’re my client, so dinner’s on me. | CSL-Daily | S001423_P0004_T00 | 你是我的客户,我应该请你吃饭。 |
| Exam tomorrow. Bring a pen, no phone. | CSL-Daily | S001372_P0008_T00 | 明天考试,带上笔,不要带手机。 |
| Today’s weather will definitely remain wintry. | PHOENIX-2014T | 26January_2010_Tuesday_heute-105 | in den nächsten tagen geht es auf jeden fall winterlich weiter immer wieder mal mit etwas schnee . |
| Representation | NN | P@5 | P@10 | Sil. |
|---|---|---|---|---|
| Reconstruction-only [ 48 ] | 52.4 | 43.7 | 37.4 | -0.083 |
| SRVQ | 69.4 | 66.9 | 65.2 | 0.199 |
| Dataset / Type | Codes | NN | P@5 | P@10 | Sil. |
|---|---|---|---|---|---|
| Phoenix-2014T | |||||
| Cumulative | 0.6940 | 0.6692 | 0.6520 | 0.1995 | |
| – | 0.7120 | 0.6888 | 0.6748 | 0.1942 | |
| – | 0.7160 | 0.7008 | 0.6896 | 0.1984 | |
| – | 0.7220 | 0.7064 | 0.6880 | 0.1988 | |
| – | 0.7280 | 0.7132 | 0.6900 | 0.1999 |
| Seed | R@1 | R@5 | R@10 | FID | DTW-B | DTW-H |
|---|---|---|---|---|---|---|
| 0 | 82.23 | 93.28 | 96.00 | 0.050 | 6.62 | 8.40 |
| 1 | 82.91 | 93.96 | 95.92 | 0.050 | 6.60 | 8.41 |
| 2 | 83.08 | 94.05 | 96.09 | 0.050 | 6.58 | 8.41 |
| Mean std |
| Dataset | Length | N | R@1 | R@5 | R@10 | DTW-B | DTW-H |
|---|---|---|---|---|---|---|---|
| CSL-Daily | 100 | 422 | 82.94 | 94.31 | 96.45 | 4.61 | 5.78 |
| 101–140 | 367 | 80.11 | 94.28 | 96.46 | 6.62 | 8.32 | |
| 140 | 387 | 83.72 | 93.80 | 96.12 | 9.31 | 11.75 | |
| Phoenix-2014T | 76 | 219 | 63.01 | 82.65 | 89.50 | 3.14 | 4.20 |
| 77–112 | 216 | 75.00 | 89.35 | 95.37 | 5.14 | 6.70 | |
| 112 | 207 | 72.95 | 90.82 | 92.75 | 7.85 | 10.03 |
Sign Language Generation
Sign language generation creates signed motion or video from text, glosses, or other inputs while preserving linguistic structure and natural movement.
See all 38 papers