Deep learning models have improved sign language-to-text translation and made it easier for non-signers to understand signed messages. When the goal is spoken communication, a naive approach is to convert signed messages into text and then synthesize speech via Text-to-Speech (TTS). However, this two-stage pipeline inevitably treat text as a bottleneck representation, causing the loss of rich non-verbal information originally conveyed in the signing. To address this limitation, we propose a novel task, Sign-to-Speech Prosody Transfer, which aims to capture the global prosodic nuances expressed in sign language and directly integrate them into synthesized speech. A major challenge is that aligning sign and speech requires expert knowledge, making annotation extremely costly and preventing the construction of large parallel corpora. To overcome this, we introduce SignRecGAN, a scalable training framework that leverages unimodal datasets without cross-modal annotations through adversarial learning and reconstruction losses. Furthermore, we propose S2PFormer, a new model architecture that preserves the expressive power of existing TTS models while enabling the injection of sign-derived prosody into the synthesized speech. Extensive experiments demonstrate that the proposed method can synthesize speech that faithfully reflects the emotional content of sign language, thereby opening new possibilities for more natural sign language communication. Our code will be available upon acceptance.
Figures & tables
Figure 1: In the reference sign language video (left), the first phrase, “many Italians,” is emphasized through rapid hand movements and facial expressions. The two-stage baseline (middle) fails to reflect this prosody, whereas our approach (right) successfully captures the emphasis on “many.”
Figure 2: The proposed learning framework of SignRecGAN.
Figure 3: The architecture of S2PFormer. S2PFormer extends FastSpeech2 by incorporating sign language information through a module called AdaPM. Specifically, the visual backbone converts sign language inputs into feature representations, which are then fed into AdaPM. Conditioned on these representations, the variance predictor estimates speech prosody parameters, which the prosody estimator uses to predict the original sign language prosody.
Figure 4: Adaptive Prosody Mixer
Figure 5: An example of sign language prosody labels. The histograms represent the distribution of hand and face motion information in sign language videos.
Expressiveness
Naturalness
Method
Lnatural
LSignRec
LProMo
Pitch( → )
Energy( → )
WER( ↓ )
UTMOS( ↑ )
Real Speech
28.9
25.4
0.04
4.07
two-stage
17.6
20.1
0.293
3.52
Ours
✓
48.4
44.6
0.212
3.29
✓
✓
44.2
39.7
0.448
2.95
✓
✓
45.3
28.6
0.623
3.03
Table 1: Objective evaluation. The best scores are highlighted in bold , and the second-best scores are underlined .
Short
Medium
Long
Neutral
Total
Prosody
3.12(±0.22)
3.19(±0.16)
3.19(±0.16)
3.00(±0.18)
3.17(±0.11)
Naturalness
2.93(±0.17)
2.97(±0.17)
2.87(±0.15)
2.96(±0.15)
2.92(±0.11)
Table 2: The CMOS of the user study with 95% confidence intervals. The scores with statistically significant improvements are highlighted in green .
Figure 6: An example of input sign language video (left) and synthesized speech (right).
Figure 7: Prominence analysis results. A black line on a spectrogram indicates emphasis, and the corresponding word is labeled on the spectrogram. The size of a labeled word indicates the intensity of the emphasis.
Generative Adversarial Networks (GANs) train competing generators and discriminators to synthesize data, translate images, and model complex distributions.
The field of sign language translation has witnessed significant progress in the translation between sign and spoken languages, but the translation between sign languages remains largely unexplored and out of reach. The latter can help 1.5 billion deaf and hard-of-hearing (DHH) people worldwide communicate across language barriers without relying on hearing interpreters or written-language fluency. The cascade approach composing separate sign-to-text, text-to-text, and text-to-sign systems suffers from error propagation and extra latency as well as the loss of information unique in the visual modality. We aim to develop direct sign-to-sign translation. However, a large-scale open-domain parallel corpus has not been curated between sign languages. To enable direct translation between sign language utterances, we use back-translation to produce synthetic sign-sign pairs from unaligned individual language utterance-sign corpora. Using this data, we jointly train a single MBART-based model for both text->sign (T2S) and sign->sign (S2S). On synthetically generated paired sets between American Sign Language (ASL), Chinese Sign Language (CSL), and German Sign Language (DGS), our direct S2S method outperforms the cascaded baseline on geometric sign error metrics (20% lower DTW-aligned MPJPE) and language matching metrics after predicted sign utterances are translated back to sentences (50% high BLEU-4) while achieving a roughly 2.3* speedup. On a small set of pre-existing cross-lingual sign data, we find similar improvements for our proposed method.
Sign Language Production (SLP) faces a fundamental trade-off: direct text-to-pose models suffer from regression-to-the-mean effects, while dictionary-retrieval methods produce disjointed transitions. To resolve this, we propose a novel training paradigm that leverages sparse keyframes to capture the underlying kinematic distribution of human signing. By generating dense motion from discrete anchors, our approach mitigates regression-to-the-mean while ensuring fluid articulation. To achieve this at scale, we introduce FAST, an ultra-efficient sign segmentation model that automatically mines precise temporal boundaries. We then present SignSparK, a Conditional Flow Matching (CFM) framework that utilizes these temporal anchors to synthesize 3D signing sequences. This keyframe-driven formulation also unlocks Keyframe-to-Pose (KF2P) generation, making precise spatiotemporal editing of signing sequences possible. Furthermore, SignSparK scales across four distinct sign languages, constituting the largest multilingual SLP framework to date, and integrates 3D Gaussian Splatting for photorealistic rendering. Extensive evaluations demonstrate that SignSparK achieves state-of-the-art across diverse SLP tasks and multilingual benchmarks. Our code is available at https://github.com/JianHe0628/SignSparK.
Sign language video generation demands precise hand and facial articulation, yet modern video diffusion models, trained predominantly on spoken-language video, produce artifacts that render signing unintelligible. We propose SignRefine, a sign language video generation model that produces comprehensible signing from 2D keypoint conditioning alone, generalizing across appearances and visual conditions. Our approach builds on a pretrained video diffusion transformer and introduces local adapters with spatial grounding to selectively refine hand and face regions, steering the strong base model's prior toward accurate articulation. To enable this work and support broader sign language research, we present NVSign, a large-scale dataset of video content natively produced in sign language, offering diverse signer appearances, environments, and natural conversational settings. Trained on this data, our model shows up to 30% improvement in hand pose precision metrics over the strongest baseline and is preferred by sign language users for visual quality and comprehensibility in more than 80% of comparisons.
Anton Pelykh, Edward Fish, Ozge Mercanoglu Sincan +1
Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, UK