Sign language production (SLP) aims to generate sign pose sequences that faithfully express spoken-language text while preserving the rhythm and coordination patterns of signing. Common SLP methods learn text-to-pose mappings from training data but often suffer from semantic inaccuracies in long or complex sentences. Retrieval augmentation offers a way to introduce realistic signing priors to provide more explicit guidance for text-to-pose generation. However, existing retrieval-based SLP methods typically operate at the gloss or word level, relying on gloss-pose dictionaries or isolated sign segments that require costly annotations and fail to preserve continuous sentence-level dynamics. To bridge this gap, we propose SemRAG, a semantic-guided sentence-level retrieval-augmented framework that retrieves full-sentence signing templates as priors without constructing gloss/word-level sign dictionaries. While such templates preserve local motion patterns and global rhythm, they also introduce a semantic mismatch problem: irrelevant motions in the retrieved sentence may interfere with the target semantics. To address this issue, we introduce an SLT-guided template semantic localization module that identifies target-relevant temporal regions within the retrieved template while suppressing irrelevant signing content. To adaptively incorporate the template features into the generator, we further introduce layer-specific learnable gates that regulate their contribution across generator layers. Extensive experiments show that SemRAG improves semantic consistency and motion quality over baselines under both back-translation and motion accuracy metrics.
Figure 2: Overview of SemRAG. Given an input sentence, SemRAG retrieves sentence-level signing templates and encodes text features (Stage 1), localizes semantically relevant template regions using a frozen SLT model and builds filtered template features (Stage 2), and performs layer-gated template injection into a generator for SLP (Stage 3).
Method
Phoenix14T
CSL-Daily
Dev
Test
Dev
Test
R ↑
B4 ↑
MPJPE ↓
Hand ↓
R ↑
B4 ↑
MPJPE ↓
Hand ↓
R ↑
B4 ↑
MPJPE ↓
Hand ↓
R ↑
B4 ↑
MPJPE ↓
Hand ↓
Without retrieval
PT ∗ ( Saunders et al. 2020b )
11.87
3.88
40.63
10.47
13.17
4.31
50.80
10.26
7.21
0.39
87.92
13.97
7.54
0.41
90.28
13.29
SignPR ( Liu et al. 2026 )
30.84
9.12
23.77
1.90
32.86
9.41
23.04
1.87
15.02
3.36
44.59
2.07
14.43
3.01
44.22
2.09
MoLingo ∗ ( He et al. 2026 )
31.43
9.57
25.28
1.88
32.06
9.83
24.85
1.95
16.33
1.88
23.26
2.06
15.97
1.72
23.03
2.11
Table 1: SLP performance comparison on Phoenix14T and CSL-Daily.
Variant
R ↑
B4 ↑
MPJPE ↓
Hand ↓
Component ablation
Text-only
31.24
9.94
25.15
1.90
+ retrieved template
33.76
11.02
24.08
1.69
+ SLT-filtered
34.32
11.58
22.50
1.57
+ layer gate
35.09
12.29
22.47
1.56
Filtering mechanism analysis
Table 2: Ablation of component and semantic filtering.
Retrieval Strategy
Retrieval Quality (Top-1)
Generation Quality
Text Sim. ↑
Gloss Cov. ↑
R ↑
B4 ↑
MPJPE ↓
Random retrieval
3.83
7.39
30.70
9.68
25.45
Sem. similarity
44.70
39.69
32.95
10.81
24.43
Text overlap
54.54
45.01
34.42
11.43
22.69
Hybrid retrieval
58.87
45.01
35.09
12.29
22.47
Table 3: Analysis of sentence-level retrieval strategies.
Figure 3: Analysis of SemRAG. (a) Number of retrieved templates. (b) Effect of retrieval score. (c) Injection layer analysis. (d) Layer gate visualization.
Figure 4: Effect of semantic template filtering.
Figure 5: Qualitative comparison.
Method
Phoenix14T
CSL-Daily
Naturalness ↑
Motion sim. ↑
Naturalness ↑
Motion sim. ↑
SignPR
3.20
3.10
3.05
2.96
SOKE
3.43
3.25
3.31
3.17
SemRAG
3.80
3.70
3.92
3.84
Table 4: Human perceptual evaluation on Phoenix14T and CSL-Daily. Participants rate motion naturalness and motion similarity to the ground-truth reference on a five-point scale.
Sign language generation creates signed motion or video from text, glosses, or other inputs while preserving linguistic structure and natural movement.
Sign language production (SLP) aims to generate continuous signing motion from spoken language, often through gloss-to-pose generation. Prior work mainly follows two paradigms. Generative models synthesize motion from a learned prior or from noise, without reference to an observed signing instance, making rare hand configurations and signer-specific articulation difficult to preserve. Retrieval-based methods reuse real, well-articulated motion segments, but concatenating segments from different signers and co-articulation contexts can introduce rhythm and style inconsistencies across the full sequence, not only at segment boundaries. These limitations suggest a complementary solution: use retrieval to provide realistic articulation, and use learned refinement to impose the global coherence that retrieval alone lacks. We therefore propose retrieve-and-refine, a paradigm that starts from real retrieved motion and refines it into a globally coherent signing sequence rather than generating motion from scratch. Our framework, SignRR, initializes motion from a dictionary of real sign segments and refines the full sequence with a part-aware Residual VQ-VAE, where residual quantization preserves fine hand articulation and temporal length differences are handled in the latent space. Experiments on PHOENIX14T and CSL-Daily show that SignRR achieves state-of-the-art back-translation performance while maintaining competitive pose quality.
To generate natural and accurate sentence-level sign language, synthesizing the "gloss", the fundamental semantic unit, is essential. However, most current sign-language production (SLP) methods generate entire sequences at once. While this end-to-end approach is often efficient, it is prone to temporal drift and hand motion blur as sentences get longer, and fails to accurately control individual glosses. In this paper, we propose the Context-aware Gloss-wise AutoRegressive Diffusion model (GARD), a gloss-wise diffusion framework that models coarticulation by conditioning on both semantic (linguistic) and kinematic (motion) contexts. To ensure natural continuity between gloss motions, GARD introduces two additional strategies: i) Inter-Gloss Transition Guidance, which applies gradient-based guidance to kinematically align inter-gloss boundaries and ensure seamless pose consistency. ii) Global Motion Harmonizer, refining the entire gloss motion sequence based on the boundary poses adjusted by Inter-Gloss Transition Guidance. Extensive experiments on Phoenix-T and CSL-Daily datasets demonstrate that GARD achieves superior performance over existing SLP methods in terms of both linguistic accuracy and motion similarity.
Sign language generation (SLG) maps written sentences into signing motion while preserving both lexical content and fine-grained articulation. This requires a model to decide what signs should be expressed and to realize them through precise handshape, trajectory, facial expression and body coordination. Existing pose-based systems and reconstruction-only motion tokenizers often entangle these two decisions, leaving the generator without an explicit text-aligned intermediate target for what to sign before refining how to sign it. To address this issue, we introduce Semantic Residual Vector Quantization (SRVQ), a framework for SLG that frames generation as a two-level process moving from semantics to kinematics. The newly introduced SRV Quantizer learns a hierarchy in which the first residual level encodes sign content and subsequent levels refine pose, handshape, trajectory, and body coordination, all within a single quantizer and without separate semantic representations or external planning stages. Through word-aware conditioning for hierarchical code generation, a masked semantic generator predicts semantic codes and a kinematic generator completes the residual codes for full SMPL-X motion synthesis. We evaluate using retrieval and FID in a shared text-motion embedding space, alongside standard geometric metrics. On CSL-Daily and Phoenix-2014T, SRVQ achieves R@1 of 82.3 and 70.3, outperforming the previous SOTA by 30 and 46 points, respectively.