SIGNTOK: A Text-Aligned Continuous Pose Tokenizer for Sign Language Generation
Organizations: Institute of Science Tokyo, Japan · IIIT Delhi, India · RIKEN, Japan
Abstract
Text-conditioned sign language generation (SLG) depends not only on the generator but also on the space in which signing is generated. Existing approaches generate directly in pose space, in quantized code spaces, or in learned latent spaces trained for reconstruction. We find that these spaces are organized largely by signer identity rather than linguistic content: on CSL-Daily, a test clip's nearest neighbor in raw pose space is the same sentence signed by another person in only 2% of cases. To address this, we propose SIGNTOK, a continuous pose tokenizer whose latent space is shaped by three training-only heads: a kinematics head that predicts motion dynamics, a sentence-level head that aligns signing with the overall text, and a word-level head that aligns short motion windows with content words. Together, these objectives raise the same-sentence cross-signer nearest-neighbor rate to 82%. On three sign language benchmarks, the same generator trained in the SIGNTOK latent instead of pose space raises back-translation BLEU-4 in five of six benchmark-scorer settings at comparable joint accuracy, and word-level alignment gives higher back-translation scores than sentence-level alignment on every benchmark.
Figures & tables
| CSL-Daily | PHOENIX-2014T | How2Sign | ||||||||||||||||
| PA-JPE | JPE | B-T | PA-JPE | JPE | B-T | PA-JPE | JPE | B-T | ||||||||||
| Method | Body | Hand | Body | Hand | S | U | Body | Hand | Body | Hand | S | U | Body | Hand | Body | Hand | S | U |
| Gloss-free text-to-pose, as published | ||||||||||||||||||
| Prog. Transformer † | 15.98 | 12.91 | 16.30 | 32.63 | – | – | 13.67 | 11.95 | 15.01 | 31.77 | – | – | 14.15 | 11.57 | 14.74 | 30.17 | – | – |
| Text2Mesh † | 13.47 | 12.10 | 13.76 | 30.37 | – | – | 13.48 | 12.06 | 14.04 | 31.64 | – | – | 13.99 | 13.47 | 15.50 | 32.97 | – | – |
| T2S-GPT † | 11.94 | 5.93 | 12.32 | 15.43 | – | – | 10.38 | 6.47 | 11.65 | 19.09 | – | – | 11.48 | 6.39 | 12.65 | 18.44 | – | – |
| Nearest neighbour | |||
|---|---|---|---|
| Representation | R@1 | same sent. | same signer |
| No text | |||
| Raw pose | .061 | .023 | .955 |
| SOKE VQ-VAE (released code) | .060 | .090 | .779 |
| Fauré et al. VAE (released code) | .072 | .070 | .852 |
| Masked AE, no text head | .123 | .058 | .937 |
| Body | Hand | B-T | Jerk | |||
| Generation space | PA | JPE | PA | JPE | B-4 | ratio |
| CSL-Daily | ||||||
| Human, same sentence (ceiling) | 36.13 | 34.35 | 5.32 | 47.55 | 6.89 | 1.00 |
| Pose space (SignFlow) | 40.53 | 37.67 | 7.16 | 57.71 | 4.60 | 3.70 |
| Masked AE, no text head | 41.53 | 38.82 | 7.03 | 58.28 | 5.26 | 1.25 |
| + sentence-level | 40.70 | 37.84 | 6.80 | 57.39 | 5.53 | 1.27 |
Sign Language Generation
Sign language generation creates signed motion or video from text, glosses, or other inputs while preserving linguistic structure and natural movement.
See all 39 papers