SeRV: Semantic-Aligned Residual Vector Quantization for American Sign Language Generation

Hongyu Wu; Xu Wu; Tianhao Wu; Jiawei Yu; Phuc Nguyen; Jian Liu; Yi Wu

cs.CVSep 4, 2026

SeRV: Semantic-Aligned Residual Vector Quantization for American Sign Language Generation

Authors: Hongyu Wu, Xu Wu, Tianhao Wu, Jiawei Yu, Phuc Nguyen, Jian Liu, Yi Wu

Organizations: University of Oklahoma · University of Georgia · University of Massachusetts Amherst

Abstract

American Sign Language (ASL) generation remains challenging due to limited paired text-ASL motion data and the difficulty of learning motion representations both precise for reconstruction and predictable from linguistic input. Existing methods rely on motion tokenizers optimized for reconstruction, without explicit semantic supervision from paired text. As a result, the learned tokens remain limited in supporting semantically consistent and fine-grained ASL motion generation. To address this limitation, we propose SeRV (Semantic-Aligned Residual Vector Quantization), a semantic-aligned RVQ tokenizer for ASL generation. SeRV learns a semantically structured residual token space by combining global-level motion-text alignment with token-level text-conditioned supervision. Building on this tokenizer, a Hierarchical GPT predicts residual motion tokens in a coarse-to-fine manner, generating structurally coherent and semantically aligned 3D ASL motion. We further construct a large-scale reconstructed 3D ASL motion-text benchmark by recovering paired 3D motion from OpenASL videos. Experiments on How2Sign and OpenASL show that SeRV outperforms six state-of-the-art baselines, with user studies assessing reconstruction quality and semantic fidelity. Samples of our reconstructed 3D ASL motion are available on our project page: https://signvideos.github.io/Demo/.

Figures & tables

Figure 1: Left : ASL generated by SeRV from text for two example sentences, rendered as 3D avatars alongside the reference signer. Right: comparison with state-of-the-art methods on six metrics across How2Sign and YouTube-ASL datasets (body/hand DTW-JPE, lower better; SiBLEU-4, higher better; all axes oriented so larger area = better). SeRV achieves the best result on five of six metrics. Note MaDiS Zuo et al. (2026) is concurrent work reporting How2Sign only; its YouTube-ASL axes are drawn at the center to indicate results are unavailable, not zero.

Figure 2: Overview of SeRV. We introduce a global-level loss $\mathcal{L}_{\mathrm{global}}$ on the level-0 motion representation and a token-level loss $\mathcal{L}_{\mathrm{tok}}$ on the fully quantized latent sequence. Together with the standard RVQ tokenizer objective $\mathcal{L}_{\mathrm{RVQ}}$ , these objectives encourage the learned discrete motion tokens to preserve motion fidelity while retaining text-relevant semantics.

Figure 3: Hierarchical GPT for coarse-to-fine ASL motion generation. Given the text feature from the frozen condition encoder, the generator autoregressively predicts SeRV tokens over time and RVQ levels. For each time step, a coarse decoder first predicts the level- $0$ token, while residual decoders progressively predict higher-level tokens conditioned on the temporal history, the generated coarser tokens, and the text feature. The predicted multi-level tokens are then decoded by the shared motion decoder to reconstruct the ASL motion sequence.

Method	How2Sign			YouTube-ASL
Method	Body $\downarrow$	Hands $\downarrow$	SiBLEU-4 $\uparrow$	Body $\downarrow$	Hands $\downarrow$	SiBLEU-4 $\uparrow$
S-MotionGPT ∗ Jiang et al. (2023)	12.41	13.74	2.32	8.95	10.12	1.84
MoMask++ ∗ Guo et al. (2025a)	9.06	11.34	1.53	6.54	8.35	1.67
SOKE † Zuo et al. (2025)	7.75	10.08	2.37	5.42	7.44	2.18
MaDiS Zuo et al. (2026)	6.59	9.45	4.47	–	–	–
Ours	6.51	9.22	4.12	4.98	6.70	3.13

Table 1: Comparison with state-of-the-art methods on How2Sign and YouTube-ASL. Lower is better for DTW-JPE, and higher is better for SiBLEU-4.

Figure 4: Qualitative comparison of text-to-3D ASL generation. For each of six sentences, we show the reference signer video (top) and the 3D motion generated by SOKE and by our method (SeRV). The top three examples are from How2Sign; the bottom three are from YouTube-ASL (faces masked for privacy).

Method	How2Sign(PA-MPJPE $\downarrow$ )		YouTube-ASL(PA-MPJPE $\downarrow$ )
Method	Body $\downarrow$	Hands $\downarrow$	Body $\downarrow$	Hands $\downarrow$
VQ-VAE	24.07	7.96	19.53	4.45
Multi-VAE	19.37	6.65	16.96	3.62
SeRV	16.23	6.13	13.91	3.31

Table 2: Reconstruction performance of different motion tokenizers on How2Sign and YouTube-ASL.

Dataset	Tokenizer	Direction	R@1 $\uparrow$	R@5 $\uparrow$	R@10 $\uparrow$
How2Sign	RVQ	T $\rightarrow$ S	11.61	37.28	59.04
	SeRV	T $\rightarrow$ S	14.52	46.97	73.21
	RVQ	S $\rightarrow$ T	6.73	27.05	47.10
	SeRV	S $\rightarrow$ T	8.42	34.08	58.41
YouTube-ASL	RVQ	T $\rightarrow$ S	7.55	24.23	38.38
	SeRV	T $\rightarrow$ S	9.73	31.50	49.01

Table 3: Per-batch bidirectional text–sign retrieval analysis with batch size 64. Higher is better for all recall metrics.

Tokenizer Objective	How2Sign			YouTube-ASL
Tokenizer Objective	Body $\downarrow$	Hands $\downarrow$	SiB4. $\uparrow$	Body $\downarrow$	Hands $\downarrow$	SiB4. $\uparrow$
Baseline (RVQ)	6.82	9.63	3.20	5.39	7.20	2.56
+ $\mathcal{L}_{\mathrm{global}}$	6.67	9.50	3.53	5.24	7.01	2.62
+ $\mathcal{L}_{\mathrm{tok}}$	6.53	9.31	3.80	5.15	6.85	2.87
+ $\mathcal{L}_{\mathrm{global}}$ + $\mathcal{L}_{\mathrm{tok}}$	6.51	9.22	4.12	4.98	6.70	3.13

Table 4: Effect of SeRV tokenizer objectives on How2Sign and YouTube-ASL. We report DTW-JPE (lower is better) and SiBLEU-4 (higher is better). All variants use the same generator.

Generator	How2Sign (DTW-JPE $\downarrow$ )		YouTube-ASL (DTW-JPE $\downarrow$ )
Generator	Body	Hands	Body	Hands
Flat AR	7.73	10.41	7.59	9.25
Two-stage	7.21	9.86	6.92	8.33
Hierarchical GPT	6.51	9.22	4.98	6.70

Table 5: Ablation on generator architectures on How2Sign and YouTube-ASL. All variants use the same SeRV tokenizer. Lower is better for DTW-JPE.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Tokenizer	H2S Recon. B/H	YT Recon. B/H	H2S Gen. B/H, SiB4	YT Gen. B/H, SiB4
RVQ only	17.65 / 6.85	15.60 / 3.85	6.82 / 9.63, 3.20	5.39 / 7.20, 2.56
RVQ $+\mathcal{L}_{\mathrm{global}}$	17.05 / 6.55	15.05 / 3.62	6.67 / 9.50, 3.53	5.24 / 7.01, 2.62
RVQ $+\mathcal{L}_{\mathrm{tok}}$	16.65 / 6.35	14.45 / 3.45	6.53 / 9.31, 3.80	5.15 / 6.85, 2.87
Full SeRV	16.23 / 6.13	13.91 / 3.31	6.51 / 9.22, 4.12	4.98 / 6.70, 3.13

Appendix

Table 6: Controlled tokenizer-objective ablation. All variants share the same RVQ architecture, reconstruction objective, data split, and training protocol. A separate Hierarchical GPT is trained from scratch for each tokenizer under identical settings. Lower is better for PA-MPJPE and DTW-JPE, while higher is better for SiBLEU-4.

Supervision target	SiBLEU-4	DTW-JPE B/H	T2S R@1
Level 0	3.70	6.62 / 9.45	12.65
Levels 0–1	3.91	6.56 / 9.32	13.80
Full latent	4.12	6.51 / 9.22	14.52

Appendix

Table 7: Effect of the supervision target for

\mathcal{L}_{\mathrm{tok}}

on How2Sign. Applying token-level semantic supervision to progressively more complete cumulative RVQ representations improves both generation and text-to-sign retrieval.

Representation	PA-MPJPE B/H	T2S / S2T R@1
Level 0	20.84 / 8.06	13.61 / 7.79
Levels 0–1	17.42 / 6.71	14.18 / 8.16
Full latent	16.23 / 6.13	14.52 / 8.42

Appendix

Table 8: Cumulative contribution of RVQ levels. Level 0 captures most of the cross-modal retrieval signal, while subsequent residual levels progressively improve geometric reconstruction and provide smaller but consistent semantic gains.

$\lambda_{\mathrm{sem}}$	Recon. PA-MPJPE B/H	Gen. DTW-JPE B/H	SiBLEU-4
0.05	18.71 / 7.02	6.73 / 9.58	3.63
0.005	16.23 / 6.13	6.51 / 9.22	4.12
0.0005	15.77 / 6.05	6.66 / 9.40	3.82

Appendix

Table 9: Sensitivity to the semantic-loss weight on How2Sign. The reported weight of

0.005

provides the best downstream generation performance. A smaller value slightly favors reconstruction, whereas an excessively large value degrades both reconstruction and generation.

Method	Ranked first	Mean rank	SeRV pairwise win
SOKE	23.3%	2.19	72.9%
S-MotionGPT	17.8%	2.28	74.4%
SeRV	58.9%	1.53	–

Appendix

Table 10: Human perceptual evaluation results. SeRV is ranked first most frequently and obtains the lowest mean rank. Pairwise preference rates are derived from the valid rankings.

Processing stage	Number of clips
Metadata records	611,533
Successfully downloaded	561,145
Removed: multiple signers	187,880
Removed: unreliable signer or low confidence	136,728
Retained ASL–text pairs	236,537

Appendix

Table 11: Summary of the YouTube-ASL data processing pipeline.

Rating	Percentage	Number of ratings
Good	85.75%	343
Partially correct	11.25%	45
Poor	2.25%	9
Uncertain	0.75%	3
At least partially correct	97.0%	388

Appendix

Table 12: Human audit of reconstructed YouTube-ASL motion. Four raters independently evaluate 100 randomly sampled clips, producing 400 ratings.

Dataset	Lang.	Clips / Pairs	Duration	Text Paired
How2Sign-3D Duarte et al. (2021)	ASL	35K	80+h	✓
SOKE Curated 3D Poses Zuo et al. (2025)	ASL / CSL / DGS	35K / 20K / 8K	–	✓
SignAvatars Yu et al. (2024)	Multi-SL	70K	117h	✓
Reconstructed YouTube-ASL (Ours)	ASL	236,537	295.1h	✓

Appendix

Table 13: Comparison with existing 3D sign language motion resources. “Text paired” indicates whether spoken-language text is paired with the 3D motion.

Figure 5: Qualitative comparison of tokenizer reconstruction results. Our tokenizer preserves motion details more effectively, especially for hand articulations.

Figure 6: Qualitative comparison of generated 3D ASL motions on two examples. For each example, the top row shows the baseline and the bottom row shows SeRV. With semantic-aligned RVQ, SeRV better captures hand information and produces more faithful signing motions.

Explore similar work

Toward Phonology-Guided Sign Language Motion Generation: A Diffusion Baseline and Conditioning Analysis
Mar 18, 2026Rui Hong, Jana KoseckaSign Language Generation Human Motion Generation
George Mason University
SignRR: Retrieve and Refine Real Motion for Sign Language Production
Aug 28, 2026Fidel Omar Tito Cruz, Angie Sanchez Marquina, Summy Farfan +1Human Motion Generation Motion Representation Learning
University of Central Florida Florida, USA · Universidad Nacional Mayor de San Marcos Lima, Peru · Universidad Catolica San Pablo Arequipa, Peru +1
SeMoCo: A Semantic-First Motion Codec for Motion Language Modeling
Aug 25, 2026Tianlv Huang, Hetian Guo, Ziyi Cai +6Text-to-Motion Generation Human Motion Generation
Jilin University · Frontier Robotics · Harbin Institute of Technology, Shenzhen

Mar 18, 2026J/K move · Enter open · S save

Toward Phonology-Guided Sign Language Motion Generation: A Diffusion Baseline and Conditioning Analysis

Rui Hong, Jana Kosecka

George Mason University

Sign Language Generation Human Motion Generation Diffusion Models for Motion Generation

Open paper AskPDF arXiv

Generating natural, correct, and visually smooth 3D avatar sign language motion conditioned on the text inputs continues to be very challenging. In this work, we train a generative model of 3D body motion and explore the role of phonological attribute conditioning for sign language motion generation, using ASL-LEX 2.0 annotations such as hand shape, hand location and movement. We first establish a strong diffusion baseline using an Human Motion MDM-style diffusion model with SMPL-X representation, which outperforms SignAvatar, a state-of-the-art CVAE method, on gloss discriminability metrics. We then systematically study the role of text conditioning using different text encoders (CLIP vs. T5), conditioning modes (gloss-only vs. gloss+phonological attributes), and attribute notation format (symbolic vs. natural language). Our analysis reveals that translating symbolic ASL-LEX notations to natural language is a necessary condition for effective CLIP-based attribute conditioning, while T5 is largely unaffected by this translation. Furthermore, our best-performing variant (CLIP with mapped attributes) outperforms SignAvatar across all metrics. These findings highlight input representation as a critical factor for text-encoder-based attribute conditioning, and motivate structured conditioning approaches where gloss and phonological attributes are encoded through independent pathways.

Abstract

Explore similar work

Toward Phonology-Guided Sign Language Motion Generation: A Diffusion Baseline and Conditioning Analysis

SignRR: Retrieve and Refine Real Motion for Sign Language Production

SeMoCo: A Semantic-First Motion Codec for Motion Language Modeling