Simultaneous Translation between Sign Languages

Zetian Wu; Bowen Xie; Stefan Lee; Liang Huang

cs.CVSep 28, 2026

Simultaneous Translation between Sign Languages

Authors: Zetian Wu, Bowen Xie, Stefan Lee, Liang Huang

Organizations: Oregon State University

Abstract

Deaf and hard-of-hearing (DHH) signers cannot converse in real time across different sign languages today: existing sign-to-sign translation systems run offline, requiring the full source clip before any target sign is emitted. Live use cases - e.g. broadcast interpretation and two-way video calls - instead demand simultaneous output, while the source signer is still signing. We present, to our knowledge, the first simultaneous sign-to-sign (S2S) translation system, with two wait-k regimes: test-time wait-k inference applied directly to a full-sentence model, and a trained wait-k model via stochastic multi-path supervision. We further introduce ca-Stream-AL, a computation-aware latency metric for streaming output. Averaged across six S2S directions on both a smaller human-verified test set and a larger synthetic S2S corpus, our streaming system achieves a 38% ca-Stream-AL reduction while staying within a 9% DTW-PA-MPJPE increase and a 2.1 BLEU-4 drop compared to the full-sentence baseline. A word-order case study probes how the streaming model handles word order mismatch between different sign languages - a consequence of simultaneous translation.

Figures & tables

Figure 1: Simultaneous

\text{sign}\!\to\!\text{sign}

translation pipeline, shown for CSL

\to

DGS (the same architecture serves all six

\text{sign}\!\to\!\text{sign}

directions). The source video is incrementally fed to SMPL-X and tokenized by a VQ-VAE encoder into the source sign-token stream (purple, per-token duration

l

); wait-

k

decoding (§ 3.1 ) emits target sign tokens (cyan, per-token duration

l

) after a

k

-token lag, and a VQ-VAE decoder maps them back to SMPL-X for rendering as the target ( DGS ) avatar. We use

l{=}160

ms in this work (4 frames at 25 fps). The vertical dashed line marks the end of source tokens at which target signing has already begun — but under offline

\text{sign}\!\to\!\text{sign}

no target frame would be emitted before this point.

Panel of Figure 2 — Figure 2: From classical AL to Stream-AL . (a) Dropping the $\tau$ cutoff captures the per-token theoretical lag of the target tail, but the metric remains target-centric and implicitly assumes the tail can be batch-emitted at the source-end instant $t=|\mathbf{x}|w_{\text{chunk}}$ (visualized by the targets “dropped down” to a single column). (b) We fix the target-tail truncation issue by removing the source-end cutoff, which means extending the averaging horizon from the previous cutoff point $\tau$ to the full target sequence length $|\mathbf{y}|$ . (c) Stream-AL lays source and target on a shared $w_{\text{chunk}}{=}160$ ms time axis and sums the signed gap against the source-side proportional ideal. Every unit is a real $w_{\text{chunk}}$ slot of wall-clock time, so the metric converts to seconds via $\times w_{\text{chunk}}$ .

Figure 3: In ca-Stream-AL , for source step $i$ , the chunk arrives at $i\times w_{\text{chunk}}$ , but the model completes generation at $t_{i}=\max(t_{i-1},i\times w_{\text{chunk}})+c_{i}$ , where $c_{i}$ includes VQ-VAE encoding( $c_{i}^{(1)}$ )/decoding( $c_{i}^{(3)}$ ), transformer inference( $c_{i}^{(2)}$ ), and SMPL-X rendering( $c_{i}^{(4)}$ ). The wall-clock computation lag $\Delta_{i}=(t_{i}-i\times w_{\text{chunk}})/w_{\text{chunk}}$ shifts the wait- $k$ schedule from $i-k$ to $i-k-\Delta_{i}$ , yielding $\text{ca-Stream-AL}=\text{Stream-AL}+\bar{\Delta}$ .

Panel of Figure 4 — Figure 4: Quality–latency trade-off on CSL $\to$ DGS , a representative direction out of the six $\text{sign}\!\to\!\text{sign}$ directions we evaluate. Test-time wait- $k$ (blue) and trained wait- $k$ (orange) traced over $k\in\{1,3,5,7,9,11,13\}$ ; the full-sentence $k{=}\infty$ baseline (green star) is connected by a dashed segment. Rows: test set (BT, Strict); columns: quality metric ( DTW-PA-MPJPE , BLEU-4 ); x-axis: ca-Stream-AL (seconds).

Direction	$k^{\star}$	AL (s) ↓	DTW ↓	BLEU ↑
ASL $\to$ CSL	7	1.97	3.74	7.5
CSL $\to$ ASL	9	3.71	4.79	10.4
ASL $\to$ DGS	7	1.15	3.85	7.4
DGS $\to$ ASL	9	2.41	3.55	9.0
CSL $\to$ DGS	7	1.34	2.88	7.5
DGS $\to$ CSL	9	3.05	3.27	7.4

Table 1: Best-tradeoff

k^{\star}

per direction on BT under Train+TT,

m{=}5

. AL = ca-Stream-AL (seconds); DTW = DTW-PA-MPJPE ; BLEU = BLEU-4 from the Sign Language Transformer evaluator. AL admits possible negative values for directions with

r<1

Direction	$k^{\star}$	AL (s) ↓	DTW ↓	BLEU ↑
ASL $\to$ CSL	7	2.88	2.53	5.5
CSL $\to$ ASL	9	1.78	2.09	8.5
ASL $\to$ DGS	7	2.11	2.02	5.3
DGS $\to$ ASL	9	1.79	1.52	7.6
CSL $\to$ DGS	7	1.29	2.39	5.6
DGS $\to$ CSL	9	2.94	3.72	6.3

Table 2: Best-tradeoff

k^{\star}

per direction on Strict under Train+TT,

m{=}5

. Columns as in Tab. 1 .

Panel of Figure 5 — Figure 5: CSL $\to$ DGS case study (Train+TT, $k{=}3$ ). (a) When the canonical source and target orders align over content words, the model emits the right DGS sequence and drops the CSL topic-marker glossed “color” ( yanse ) that has no DGS counterpart. (b) When the two languages disagree on the position of the negator, the streaming model emits source-order neg-nichts kalt as soon as the source negator ( bu , “not”) arrives, instead of the canonical DGS order kalt neg-nichts .

Explore similar work

Direct Translation between Sign Languages
May 20, 2026Zetian Wu, Bowen Xie, Wuyang Meng +3Sign Language Translation Machine Translation
Oregon State University
Toward Real-Time Sentence-Level Sign Language Translation
Jul 10, 2026Thanh-Hoang Nguyen DoanSign Language Translation
The University of Danang - University of Science and Technology · The University of Danang – University of Science and Technology
Towards Compact Sign Language Translation: Frame Rate and Model Size Trade-offs
May 10, 2026Kuanwei Chen, Mengfeng TsaiSign Language Translation Small Language Models
Computer Science and Information Engineering, National Central University, Zhongli, Taiwan

May 20, 2026, cs.CLJ/K move · Enter open · S save

Direct Translation between Sign Languages

Zetian Wu, Bowen Xie, Wuyang Meng, Milan Gautam+2

Oregon State University

Sign Language Translation Machine Translation Sign Language Generation

Open paper AskPDF arXiv

The field of sign language translation has witnessed significant progress in the translation between sign and spoken languages, but the translation between sign languages remains largely unexplored and out of reach. The latter can help 1.5 billion deaf and hard-of-hearing (DHH) people worldwide communicate across language barriers without relying on hearing interpreters or written-language fluency. The cascade approach composing separate sign-to-text, text-to-text, and text-to-sign systems suffers from error propagation and extra latency as well as the loss of information unique in the visual modality. We aim to develop direct sign-to-sign translation. However, a large-scale open-domain parallel corpus has not been curated between sign languages. To enable direct translation between sign language utterances, we use back-translation to produce synthetic sign-sign pairs from unaligned individual language utterance-sign corpora. Using this data, we jointly train a single MBART-based model for both text->sign (T2S) and sign->sign (S2S). On synthetically generated paired sets between American Sign Language (ASL), Chinese Sign Language (CSL), and German Sign Language (DGS), our direct S2S method outperforms the cascaded baseline on geometric sign error metrics (20% lower DTW-aligned MPJPE) and language matching metrics after predicted sign utterances are translated back to sentences (50% high BLEU-4) while achieving a roughly 2.3* speedup. On a small set of pre-existing cross-lingual sign data, we find similar improvements for our proposed method.

Abstract

Explore similar work

Direct Translation between Sign Languages

Toward Real-Time Sentence-Level Sign Language Translation

Towards Compact Sign Language Translation: Frame Rate and Model Size Trade-offs