Organizations: Independent Researcher, Australia · Monash University, Australia · University of Trento, Italy · Japan Advanced Institute of Science and Technology, Japan
Sign language production (SLP) aims to generate appropriate sign language motion from spoken language text. However, current 3D SLP methods suffer from noisy supervision and weak text conditioning. Specifically, these methods typically rely on pseudo-3D motion data fitted from single-view frontal videos, which introduces severe self-occlusion and loses fine-grained hand details, ultimately degrading downstream production. To overcome these limitations, we propose a comprehensive framework that jointly integrates multi-view 3D motion data and optimizes the text-conditioning pathway. First, we introduce How2Sign-MV3D, a high-fidelity dataset constructed via a novel cross-view curation pipeline that fuses multi-view estimates to resolve occlusions and filter non-signing segments. Second, we design a robust SLP model incorporating a sign-aware spatio-temporal VQ-VAE (ST-VQVAE) for joint body-hand tokenization, a pseudo-gloss adapter, and keyframe-aware retrieval to enhance text-to-motion alignment. Evaluated on the How2Sign benchmark, our framework reduces body and hand DTW-PA-JPE by 32.8% and 20.9% respectively compared to the state-of-the-art. Finally, we release two new high-quality 3D datasets, ASLCitizen-3D and MM-WAuslan-MV3D, alongside SGNify-MV as a multi-view video benchmark to support broader community research.
Figures & tables
Figure 2: Cross-view motion curation pipeline. We first build per-view SMPL-X [ 41 ] sequences using AiOS [ 56 ] as a stable body backbone and WiLoR [ 43 ] as detailed hand evidence. The frontal and side streams are then aligned, synchronized, cleaned of non-signing frames, and used for cross-view body and finger repair. The side view acts as verifier and repair evidence; the final training target for SLP is the refined frontal stream.
Figure 3: Sign-aware ST-VQVAE. Unlike independent body/hand tokenizers, our encoder processes the 43-joint upper-body and hands skeleton jointly, pools it to three semantic slots only at the bottleneck, and quantizes the slots with body, left, and right hand codebooks. This preserves body-hands coupling while retaining the three-token-stream interface used by autoregressive SLP models.
Figure 4: Keyframe distribution. In the ASLCitizen [ 10 ] dataset, the average and maximum IG [ 57 ] saliency of joints over time follow a rough Gaussian trend overall yet vary widely across signs, motivating per-clip rather than global-envelope selection.
Figure 5: SLP with sign classifier enhanced retrieval. We train an auxiliary isolated-sign classifier and extract joint-time saliency W using Integrated Gradients (IG). After temporal pooling to the ST-VQVAE token rate, we use the saliency scores to select a fixed 10-token window from the encoded isolated-sign sequence Zw . The selected tokens serve as a compact retrieval prompt for autoregressive sign generation.
Training
UBody-F ↓
LHand ↓
RHand ↓
SOKE (3 datasets, reported)
6.82
2.35 (hands avg.)
SOKE (NSA’s H2S, reported)
7.92
3.07 (hands avg.)
SOKE ( H2S-MV3D -only)
4.58
1.92
1.98
Table 1: Quantitative How2Sign-MV3D results. We report DTW-PA-JPE only. Reported rows are taken from prior work and hand values are averages
Method
DTW-PA-JPE ↓
DTW-JPE ↓
Body
Hands
Body
Hands
NAR [ 26 ]
13.94
11.80
–
–
Prog. Trans. [ 49 ]
14.15
11.57
14.74
30.17
Text2Mesh [ 50 ]
13.99
13.47
15.50
32.97
Adv. Train. [ 48 ]
13.78
11.17
–
–
T2S-GPT [ 69 ]
11.48
6.39
12.65
18.44
Table 2: Quantitative SLP results. Following prior SLP work, we report DTW-PA-JPE and DTW-JPE for body and hands. Since each method relies on pseudo-3D supervision, this table compares complete pipelines under prior motion-space protocols, not isolated generator architectures under shared ground-truth. Baseline rows follow the SOKE comparison.
Method
PA-JPE ↓
PA-VPE ↓
Body
Hands
All
Hands
SOKE (VQ-VAE)
19.37
6.65
–
–
SOKE (VQ-VAE, H2S-MV3D)
14.00
6.24
18.12
6.07
Ours (ST-VQVAE)
14.21
5.84
16.59
5.84
Table 3: Tokenizer ablation. Using the same codebook size and dimension, our tokenizer reaches better reconstruction results compared to the SOKE tokenizer. DTW is not used because tokenizer reconstruction is frame-aligned.
Loss
PA-JPE ↓
PA-VPE ↓
Body
Hands
All
Hands
Rotation-only
18.48
6.23
21.93
6.04
Mesh-only
11.61
10.24
13.33
9.96
Rotation + mesh
14.21
5.84
16.59
5.84
Table 4: Tokenizer loss ablation. Mesh supervision improves global geometry, while the hand-weighted combined loss gives the best hand reconstruction. DTW is not used because tokenizer reconstruction is frame-aligned.
Setting
MV3D
Gloss
Adapter
Keyframe
ST-VQVAE
DTW-PA-JPE ↓
Gain ↑
Body
LHand
RHand
Body
Hands
SOKE (reported)
–
–
–
Original
Original
6.82
2.35 avg.
–
–
SOKE (H2S-MV3D)
✓
–
–
–
–
4.58
1.92
1.98
32.8%
17.0%
+ adapter
✓
✓
✓
–
–
4.49
1.93
1.99
34.2%
16.6%
+ retrieval
✓
✓
✓
✓
–
4.52
1.91
1.97
33.7%
17.4%
+ ST-VQVAE
✓
✓
✓
✓
✓
4.58
1.81
1.91
32.8%
20.9%
Table 5: Component ablation. DTW-PA-JPE for the main generation variants and relative gains over the reported SOKE body and hands average. Components provide complementary gains rather than a strictly monotonic improvement in every region. Percentage gains are computed from the displayed scores.
Figure 6: Qualitative How2Sign-MV3D results. The figure compares noisy front-view 3D motions with our cross-view-curated motions, emphasizing wrist and finger repair.
Figure 7: Qualitative SLP results. Comparisons of generated sign sequences from our method (trained with our How2Sign-MV3D pseudo-3D motion) and the state-of-the-art SOKE [ 78 ] (trained with NSA [ 5 ] pseudo-3D motion) on How2Sign [ 12 ] test set.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Timeline illustration of heuristic non-sign pruning and motion-blend transitions. In this frontal sequence (195 source frames), removing two detected idle intervals (23 and 16 frames) leaves 156 active source frames, and inserting two 4-frame synthetic bridges yields a 164-frame continuous output sequence.
Level
Size/Time
Output Nodes
0
43→16
0: mean(0,1,2,3)
8: mean(19,20,21)
T→T/2
1: mean(4,7,9,11)
9: mean(22,23,24)
2: mean(5,8,10,12)
10: mean(25,26,27)
3: mean(9,11,25)
11: mean(28,29,30)
4: mean(10,12,40)
12: mean(31,32,33)
5: mean(3,6)
13: mean(34,35,36)
Appendix
Table 6: Fixed skeleton hierarchy in ST-VQVAE. Each node is produced by average pooling over a fixed joint group. The first two levels also apply temporal average pooling with kernel size 2.
Model
BLEU-4 ↑
GPT-4o
.5789
GPT-5.2
.6589
Qwen2.5-72B-Instruct
.5553
Appendix
Table 7: ASLG-PC12 20-shot reference agreement on the same 200 test sentences. The prompting approach is motivated by Zhang et al . [ 75 ] ; scores are our local evaluations, not a direct reproduction of their reported scores. BLEU-4 is on a 0–1 scale.
Model
Method
BLEU-4 ↑
GPT-5.4 (2026-03-05)
80-shot, API batch
.7709
GPT-5.5 (2026-04-23)
80-shot, API batch
.8077
DeepSeek V4 Flash
80-shot, thinking disabled
.7411
DeepSeek V4 Pro
80-shot, thinking disabled
.8037
Qwen3.5-27B
80-shot, general gloss prompt
.7143
Qwen3.5-27B
80-shot, ASLG-style prompt
.7599
Appendix
Table 8: Expanded ASLG-PC12 comparison on the 200-item test set. All listed configurations achieved 100% completion across the test set. Fine-tuned (QLoRA) models leverage 5,000 in-domain training pairs, whereas API and prompting baselines operate in an 80-shot in-context regime.
Split
Rows
Empty
Multiline
Avg. tokens
Exact coverage
Loose coverage
Rows with match
Train
31,165
0
0
11.90
59.22%
59.84%
98.10%
Val
1,741
0
0
12.00
57.77%
58.39%
97.30%
Test
2,357
0
0
12.17
56.09%
56.68%
97.62%
All
35,263
0
0
11.92
58.94%
59.55%
98.03%
Appendix
Table 9: How2Sign pseudo-gloss formatting and ASLCitizen lexical coverage audit.
English
Dataset reference
Prediction
Structural observation
I would drive across.
DRIVE
IX-1P DRIVE CROSS
Reference is highly compact; prediction explicitly encodes subject pronoun and path.
Does the teacher like chocolate?
TEACH+AGENT LIKE CHOCOLATE
CHOCOLATE REALLY LEARN+AGENT LIKE Q
Prediction applies topic-fronting and an explicit question marker ( Q ), but substitutes LEARN+AGENT .
What did John buy yesterday?
JOHN BUY YESTERDAY
YESTERDAY JOHN BUY WHAT PART HOLD
Prediction fronts temporal adverb and appends explicit WH-question and aspectual markers.
Appendix
Table 10: Qualitative comparison of English inputs, NCSLGR reference glosses, and GPT-5.2 predictions, illustrating differences in compression, lexical choice, and grammatical marking conventions.
Setting
Retrieval prompt
DTW-JPE ↓
DTW-PA-JPE ↓
Body
LHand
RHand
Body
LHand
RHand
E0 fixed crop, τ=.20
10-token crop
4.983
8.197
7.697
4.577
1.814
1.906
E1 fixed crop, τ=.14
10-token crop
5.137
8.192
8.179
4.570
1.983
1.996
E2 saliency scale
full sequence
5.159
8.356
8.434
4.685
1.983
1.998
E3 weight embedding
full sequence
5.261
8.343
8.390
4.744
1.987
1.999
E4 scale + embedding
full sequence
5.189
8.207
8.209
4.659
1.944
1.979
Appendix
Table 11: Retrieval prompt diagnostic on How2Sign-MV3D. E0 is the setting used in the main experiments. Full-sequence weighting does not improve hand reconstruction over the fixed 10-token crop.
Figure 9: SGNify-MV visualization. Examples for two SGNify signs. We show the original SGNify [ 17 ] video, ground-truth mesh sequence, and our rendered low FPS frontal/side videos with controlled temporal mismatch and motion blur.
Setting/Method
Input
UBody-F ↓
LHand ↓
RHand ↓
Real SGNify, DexAvatar protocol
OSX [ 37 ]
Front
47.32
18.34
18.12
Neural Sign Actors [ 5 ]
Front
46.42
16.17
15.23
EVA* [ 23 ]
Front
40.38
13.73
13.68
DexAvatar [ 32 ]
Front
30.13
13.53
13.08
Real SGNify
Appendix
Table 12: TR-V2V results on SGNify and SGNify-MV. We report TR-V2V error (mm) for UBody-F and both hands. The first block follows the DexAvatar [ 32 ] SGNify evaluation protocol. The remaining blocks report our results using DexAvatar’s evaluation code.
Dataset
Wrist
Finger
F/W Ratio
Local Finger
ASLCitizen
0.301
0.408
1.36
0.188
How2Sign
0.252
0.471
1.87
0.308
MM-WLAuslan
0.382
0.547
1.43
0.220
SGNify
0.466
0.535
1.15
0.124
SGNify(central)
0.383
0.510
1.33
0.220
Appendix
Table 13: Hand motion dynamics. We compare wrist and finger motion measured from GT SMPL-X meshes for SGNify and pseudo SMPL-X meshes for the other datasets. F/W Ratio is the average global finger motion divided by the average global wrist motion. Local Finger subtracts the wrist position before measuring finger movement. All motion values are speeds in the SMPL-X coordinate scale, computed as frame-to-frame 3D joint displacement multiplied by frame rate.
Setting/Method
Input
UBody-F ↓
LHand ↓
RHand ↓
Real SGNify front, H2S-Synth3D protocol
OSX [ 37 ]
Front
47.32
18.34
18.12
SGNify [ 17 ]
Front
55.63
19.22
17.50
Neural Sign Actors [ 5 ]
Front
46.42
16.17
15.23
SignAvatars [ 72 ]
Front
38.50
15.56
13.28
SOKE [ 78 ]
Front
46.73
10.55
8.94
Appendix
Table 14: MPVPE-PA results on SGNify and SGNify-MV. We report MPVPE-PA (mm) for UBody-F and both hands. The first block follows the How2Sign-Synth3D [ 61 ] SGNify evaluation protocol. The remaining blocks report our results using SOKE [ 78 ] ’s evaluation code.
Figure 10: MM-WLAuslan-MV3D and ASLCitizen-3D visualization. Examples for two signs for each video dataset. We show the original ASLCitizen [ 10 ] and MM-WLAuslan [ 53 ] videos, with our 3D annotations below.
View
Hand
Frames
No box (%)
Rule rejected (%)
Rejected / finite (%)
front
left
5,508,797
2.28
4.15
1.92
front
right
5,508,797
0.67
2.49
1.83
side
left
5,508,715
8.41
15.39
7.62
side
right
5,508,715
23.06
34.32
14.64
Appendix
Table 15: Raw detection availability and heuristic rejection rates. No box and Rule rejected are normalized by all decoded frames, whereas Rejected / finite uses valid finite fits as the denominator.
Figure 11: Qualitative example of detection absence and cross-view recovery. Six mapped frames showing raw RGB inputs, per-view estimator outputs, and the resulting processed motion.
Figure 12: Stage-by-stage pipeline visualization on clip 1xl3MOjKX84_2-8-rgb . From top to bottom: synchronized frontal and side RGB frames, native-view AiOS-only meshes, and per-view AiOS+WiLoR merged meshes (incorporating single-view reliability filtering and interpolation prior to cross-view synchronization and finger repair). Note the visible torso tilt in the side-view body meshes, which motivates retaining the frontal body backbone. The bottom rows highlight the frontal output sequence with cross-view-refined hands, illustrating local finger configurations before and after cross-view repair across five updated timestamps (with root, body, and shape held fixed for visual comparison).
Cohort Workload
Per-Clip Deleted Source Frames
Cohort
Split
Clips
Edited clips (%)
Frames removed (%)
Bridge frames
Mean / Med (All)
Mean / Med (Edited)
Min (Edited)
Max total
Max single run
New
train
26,791
16,194 (60.45%)
18.72%
91,176
32.69 / 18
54.08 / 37
11
2,400
850
New
val
1,557
966 (62.04%)
18.57%
5,600
33.45 / 21
53.92 / 42
12
377
336
New
test
2,099
1,218 (58.03%)
16.79%
6,672
29.33 / 16
50.54 / 36
12
457
436
New
overall
30,447
18,378 (60.36%)
18.58%
103,448
32.49 / 18
53.83 / 37
11
2,400
850
Reused
overall
4,492
2,769 (61.64%)
18.89%
15,988
34.08 / 19
55.28 / 38
12
738
676
Appendix
Table 16: Motion-side non-sign pruning workload and per-clip source-frame deletion statistics. Real-frame removal is frame-weighted over decoded frontal frames, while 4-frame synthetic bridges are counted separately per view. Per-clip deletion distributions are reported across all clips ( All , where min=0 ) and restricted to pruned clips ( Edited , with ≥1 removed interval).
Sign language generation creates signed motion or video from text, glosses, or other inputs while preserving linguistic structure and natural movement.