Text-conditioned sign language generation (SLG) depends not only on the generator but also on the space in which signing is generated. Existing approaches generate directly in pose space, in quantized code spaces, or in learned latent spaces trained for reconstruction. We find that these spaces are organized largely by signer identity rather than linguistic content: on CSL-Daily, a test clip's nearest neighbor in raw pose space is the same sentence signed by another person in only 2% of cases. To address this, we propose SIGNTOK, a continuous pose tokenizer whose latent space is shaped by three training-only heads: a kinematics head that predicts motion dynamics, a sentence-level head that aligns signing with the overall text, and a word-level head that aligns short motion windows with content words. Together, these objectives raise the same-sentence cross-signer nearest-neighbor rate to 82%. On three sign language benchmarks, the same generator trained in the SIGNTOK latent instead of pose space raises back-translation BLEU-4 in five of six benchmark-scorer settings at comparable joint accuracy, and word-level alignment gives higher back-translation scores than sentence-level alignment on every benchmark.
Figures & tables
Fig. 1 : What organizes the space a sign generator works in? t-SNE of mean-pooled CSL-Daily test clips, coloured by signer (ten most frequent), for (a) the pose space itself, (b) the discrete tokenizer of SOKE [ 54 ] , (c) the sign pose VAE of Fauré et al. [ 9 ] , both trained with the released code on the same poses, and (d) SignTok . In (a)-(c) every signer forms clusters: the representations that current generators map text into are organized by who signed, not by what was signed.
Fig. 2 : SignTok overview. (a) Stage 1: the encoder Eφ sees only the visible frames of a masked clip and yields one latent per visible frame; the decoder input is the latent completed with a learned token m at the masked positions, and Dθ reconstructs the clip. Three training-only heads read the latent: kinematics, sentence, and word-level alignment. (b) Stage 2: with the tokenizer frozen and no masking, a flow-matching model transports noise to the per-frame latents under cross-attention to the frozen mBART states of the sentence, and the frozen decoder gives the poses.
Fig. 3 : Kinematics and sentence heads. (a) A shallow Transformer over the latent, with its own mask token at the masked positions, predicts the velocity and acceleration of the masked frames from finite differences of the true poses. (b) An MLP maps the time-pooled latent to the token-pooled state of the frozen mBART encoder under a cosine loss.
Fig. 4 : Word-level alignment head. Non-overlapping windows of the latent are mean-pooled and ℓ2 -normalized without projection. The content pieces of the sentence are looked up in the frozen input embedding table of mBART and projected into the latent space, together with a learnable null vector n . Two late-interaction scores (best window per token; best token or null per window) enter a symmetric InfoNCE over the batch in which the same sentence signed by another person is also a positive.
CSL-Daily
PHOENIX-2014T
How2Sign
PA-JPE ↓
JPE ↓
B-T ↑
PA-JPE ↓
JPE ↓
B-T ↑
PA-JPE ↓
JPE ↓
B-T ↑
Method
Body
Hand
Body
Hand
S
U
Body
Hand
Body
Hand
S
U
Body
Hand
Body
Hand
S
U
Gloss-free text-to-pose, as published
Prog. Transformer †
15.98
12.91
16.30
32.63
–
–
13.67
11.95
15.01
31.77
–
–
14.15
11.57
14.74
30.17
–
–
Text2Mesh †
13.47
12.10
13.76
30.37
–
–
13.48
12.06
14.04
31.64
–
–
13.99
13.47
15.50
32.97
–
–
T2S-GPT †
11.94
5.93
12.32
15.43
–
–
10.38
6.47
11.65
19.09
–
–
11.48
6.39
12.65
18.44
–
–
TABLE I: Text-to-pose results. DTW joint position error in the SOKE convention (accumulated cost in metres, lower is better), Procrustes-aligned (PA-JPE) and unaligned (JPE), for body and hands, and back-translation BLEU-4 (B-T) under two scorers: the model released by Low et al. [ 25 ] (S) and Uni-Sign fine-tuned on the same poses (U). † : as reproduced by Zuo et al. [ 54 ] ; sources of all published rows are listed in the supplementary material. The two latent baselines are the same generator in the VAE latent of Fauré [ 9 ] and in the pre-quantization latent of SOKE, trained with released code on the same poses; they were not run on How2Sign. ⋆ : our run of the released checkpoint in text-only mode (retrieval off). The first two blocks are scored on other pose fits and listed for reference; the last block is a controlled comparison on identical clips, poses, and scorers. Bold: best, underlined: second best in the controlled block.
Nearest neighbour
Representation
R@1 ↑
same sent. ↑
same signer ↓
No text
Raw pose
.061
.023
.955
SOKE VQ-VAE (released code)
.060
.090
.779
Fauré et al. VAE (released code)
.072
.070
.852
Masked AE, no text head
.123
.058
.937
TABLE II : Organization of the latent , CSL-Daily test split, mean-pooled clips, no tokenizer head involved. R@1: a ridge map fitted on the training split maps the sentence feature into the latent and retrieves the clip among 1,162. Nearest neighbour: fraction of clips whose cosine nearest neighbour is the same sentence signed by another person, or any clip by the same signer. The two external tokenizers are trained with their authors’ code on the same poses. Bold: best, underlined: second best.
Fig. 5 : Generated signing on the three benchmarks. One test clip per corpus, four evenly spaced frames per row: the reference video, the released SignSparK model (not available for PHOENIX-2014T, on which it was not trained), the pose-space generator SignFlow, and the same generator in the SignTok latent. Generated clips have their own timing, so frames at the same position do not show the same sign; what to compare is whether the handshapes of the sentence are formed and where the hands move.
Body
Hand
B-T ↑
Jerk
Generation space
PA ↓
JPE ↓
PA ↓
JPE ↓
B-4
ratio
CSL-Daily
Human, same sentence (ceiling)
36.13
34.35
5.32
47.55
6.89
1.00
Pose space (SignFlow)
40.53
37.67
7.16
57.71
4.60
3.70
Masked AE, no text head
41.53
38.82
7.03
58.28
5.26
1.25
+ sentence-level
40.70
37.84
6.80
57.39
5.53
1.27
TABLE III : Ablation of the generation space. Per-frame DTW error in mm (lower is better), back-translation BLEU-4 (S scorer), and jerk relative to real signing ( 1 = natural, >1 jittery, <1 over-smoothed). All tokenizer variants keep the masked autoencoder and the kinematics head. Single runs; seed-to-seed noise is 0.2 - 0.5 mm on body. Bold: best, underlined: second best.
Sign language generation creates signed motion or video from text, glosses, or other inputs while preserving linguistic structure and natural movement.
Sign language generation (SLG), also known as text-to-sign generation, aims to bridge the communication gap between signers and non-signers. Unlike many other generative tasks, SLG must satisfy two fundamental linguistic constraints. First, sign language expresses meaning through a sequence of gestures aligned with word-like units called glosses, and therefore requires correct lexical ordering to preserve intended meaning. Second, each gesture should faithfully reflect the intended gloss (semantic accuracy). Despite recent progress, existing SLG methods frequently produce signs with incorrect lexical order and low semantic accuracy. A common limitation of prior approaches stems from globally fused conditioning strategies, which weaken temporal grounding, the temporal correspondence between glosses and their realized sign segments. This often leads to incorrect lexical order and semantically ambiguous signs. To address this limitation, we propose SIGNER, a SIGN language generation framework with timE-Resolved conditioning to ensure temporal grounding, leveraging a temporal-gloss condition and local temporal fusion (LTF). SIGNER constructs a temporal-gloss condition by estimating a gloss sequence and its durations from input text, and assigning gloss semantics across the temporal dimension. We then introduce LTF, a temporally grounded fusion module that integrates the temporal-gloss condition within a constrained temporal window during denoising. By enforcing temporal locality in condition fusion, LTF preserves temporal grounding, leading to correct lexical ordering and clearer per-gloss semantics. Experiments on Phoenix-2014T and CSL-Daily demonstrate state-of-the-art performance, further supported by motion-smoothness analysis. The project page is available here https://taeryunglee.github.io/projects/signer.
Taeryung Lee, Hyeongjin Nam, Gyeongsik Moon +1
IPAI · Dept. of ECE & ASRI, Seoul National University · Dept. of CSE, Korea University
Sign language generation (SLG) maps written sentences into signing motion while preserving both lexical content and fine-grained articulation. This requires a model to decide what signs should be expressed and to realize them through precise handshape, trajectory, facial expression and body coordination. Existing pose-based systems and reconstruction-only motion tokenizers often entangle these two decisions, leaving the generator without an explicit text-aligned intermediate target for what to sign before refining how to sign it. To address this issue, we introduce Semantic Residual Vector Quantization (SRVQ), a framework for SLG that frames generation as a two-level process moving from semantics to kinematics. The newly introduced SRV Quantizer learns a hierarchy in which the first residual level encodes sign content and subsequent levels refine pose, handshape, trajectory, and body coordination, all within a single quantizer and without separate semantic representations or external planning stages. Through word-aware conditioning for hierarchical code generation, a masked semantic generator predicts semantic codes and a kinematic generator completes the residual codes for full SMPL-X motion synthesis. We evaluate using retrieval and FID in a shared text-motion embedding space, alongside standard geometric metrics. On CSL-Daily and Phoenix-2014T, SRVQ achieves R@1 of 82.3 and 70.3, outperforming the previous SOTA by 30 and 46 points, respectively.
Sign language translation and generation share the goal of bidirectional alignment between text and sign representations. However, existing approaches either treat them as isolated tasks or are only verified on limited datasets, limiting effective modeling between modalities. In this paper, we propose SignFLIP, a unified LLM-centered framework for translation and generation. To enable bidirectional mapping between text and sign, SignFLIP adopts a symmetric architecture together with a stage-wise training strategy built on large-scale data. The shared sign--text representation is progressively refined: pre-alignment facilitates subsequent SLT, while the SLT-adapted representation further benefits SLG. Extensive experiments on multiple benchmarks show that SignFLIP shows competitive performance compared with task-specific models on both translation and generation tasks, as well as strong transferability to sign language recognition.
Zhaoyi An, Sihan Tan, Youngbae Hwang +2
Institute of Science Tokyo · Chungbuk National University