ReSCUE: Re-translation with Sentence Commitment for Unsegmented Long-Form Simultaneous Sign Language Translation

Sihan Ren; Gaozheng Li; Yuanshang Quan; Yiming Qin; Fuyi Yang; Chang Liu; Lan Xu; Minye Wu

cs.CVOct 2, 2026

ReSCUE: Re-translation with Sentence Commitment for Unsegmented Long-Form Simultaneous Sign Language Translation

Authors: Sihan Ren, Gaozheng Li, Yuanshang Quan, Yiming Qin, Fuyi Yang, Chang Liu, Lan Xu, Minye Wu

Organizations: ShanghaiTech University · DGene · University of Derby

Abstract

Simultaneous Sign Language Translation (SLT) is critical for real-time communication, yet existing methods remain largely confined to sentence-level, offline settings that assume pre-segmented inputs. These assumptions hinder deployment in realistic scenarios involving continuous, unsegmented video streams. We present ReSCUE, a unified framework for simultaneous SLT on unsegmented long-form sign language videos that aligns training and inference with realistic streaming conditions. ReSCUE combines inference-aware training to handle partial inputs, non-signing pauses, and multi-sentence contexts, stabilized re-translation to enable low-latency yet revisable predictions with reduced output flicker, and a sentence commitment mechanism for online segmentation and memory management. Experiments on standard sentence-level benchmarks show that ReSCUE achieves lower latency and the best translation quality under low-latency settings. On long-form unsegmented datasets, ReSCUE approaches the translation quality of oracle offline systems that use ground-truth sentence boundaries, while operating at substantially lower latency, demonstrating its practicality for real-world streaming scenarios.

Figures & tables

Figure 1: Overview of the proposed simultaneous SLT pipeline. The unsegmented video stream is processed in fixed-length chunks and accumulated in a dynamic input buffer. At each step, the SLT module performs re-translation over the buffered video segment to produce a revisable hypothesis for the current sentence, while encouraging prefix consistency with the previous hypothesis. Finalized translations of previous sentences are stored in the committed history and concatenated with the current hypothesis to form the user-facing output. A sentence commitment mechanism detects sentence completion, extracts the finalized translation, appends it to the committed history, and advances the input buffer accordingly.

Figure 2: Inference-aware training data construction. From a long-form sign language video containing at least two sentences (top), besides the standard sentence-level clips, we construct three types of augmented training instances to reflect streaming inference conditions: (1) prefix-to-prefix instances with truncated video and text, (2) unsegmented multi-sentence clips, and (3) clips augmented with surrounding non-signing segments represented by the <SIL> token.

Figure 3: Stabilized re-translation with sentence commitment. At each step $t$ , the model re-translates the buffered video input to produce a raw hypothesis $\tilde{\mathbf{Y}}_{t}$ , from which a stabilized hypothesis $\mathbf{Y}_{t}$ is obtained by mask- $k$ (mask- $2$ ). Re-translation corrects premature predictions, including early terminal punctuation, as more context becomes available. When a terminal punctuation (e.g. “!") remains stable in $\mathbf{Y}_{t}$ , it triggers sentence commitment: all preceding tokens are finalized as the translation of the current sentence, appended to the history, and the input buffer is advanced based on the punctuation’s token arrival time.

offline SLT
Method	AL(ms) $\downarrow$	BLEU $\uparrow$	ROUGE $\uparrow$
SLRT [ 5 ]	4259	3.03	19.67
GASLT [ 41 ]	$\dagger$	4.07	20.35
GFSLT-VLP [ 46 ]	$\dagger$	11.00	36.44
FLa-LLM [ 8 ]	$\dagger$	14.20	37.25
Sign2GPT [ 35 ]	$\dagger$	15.40	42.36

Table 1: Comparison on CSL-Daily . Bold values indicate the best offline SLT result; underline indicates the best simultaneous SLT result.

\dagger

denotes the same AL as SLRT; for offline SLT models, AL is determined by the average duration of the full test samples. Zuo et al. [ 49 ] is excluded from latency comparison since its reported latency is not end-to-end video-to-text.

Figure 4: Sentence-Level Translation quality (BLEU) vs. latency (Average Lagging) on CSL-Daily. For CTL++, latency is controlled by parameter $m$ , with points from left to right corresponding to $m\in\{0,2,4,6,8,10,18\}$ . For SimulSLT, latency is adjusted via the wait- $k$ parameter $k_{wait}$ , with points from left to right representing $k_{wait}\in\{1,3,5,7\}$ . For ReSCUE, latency is adjusted via the Mask- $k$ parameter $k$ , with points from left to right representing $k\in\{0,1,2,3,4,5,6\}$ . The top-left region represents the optimal performance.

Table 6

Figure 5: Ablation study on sentence-level Simultaneous SLT. We explore the trade-offs between latency (Average Lagging), stability (NE), and translation quality (BLEU). $\beta$ and $k$ refer to the bias strength in biased search and the parameter for the Mask- $k$ strategy, respectively. In subfigure (a), the bottom-left region represents the optimal performance, while in subfigure (b), the top-left region represents the optimal performance.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	How2Sign-Long	CSL-Story
Mean Abs. Diff.	3.48	0.83
Max Abs. Diff.	21	3
Std. Dev.	3.30	0.90

Appendix

Table 4: Statistics of sentence commitment on the two long-form test sets.

Figure 6: Illustration of the pose partitioning strategy. The skeleton keypoints are grouped into Face, Hands, and Upper Body patches.

Height threshold	Speed threshold
	0.02	0.03	0.04
0.65	0.8115	0.8596	0.8596
0.85	0.7789	0.8396	0.8396
1.05	0.7789	0.8396	0.8396

Appendix

Table 5: Recall of silence detection under different threshold choices on manually annotated CSL-Story videos.

	Predicted Silence	Predicted Sign
Actual Silence	1450	545
Actual Sign	163	11422

Appendix

Table 6: Frame-level confusion matrix of the silence detector on full CSL-Story videos.

Panel of Figure 7 — Figure 7: Examples of active and inactive frames for silence segmentation.

Figure 8: Frame-level detector decisions over time

Figure 9: The user interface of Our Simul-SLT system. The simultaneous SLT result is presented on the right of the screen.

Metric	CV-SLT vs. ReSCUE $(t/p)$	Uni-Sign vs. ReSCUE $(t/p)$	CV-SLT vs. Uni-Sign $(t/p)$
Quality	$-4.011/0.002$	$-2.230/0.048$	$-0.960/0.358$
Fluency	$-3.996/0.002$	$-3.937/0.002$	$-0.992/0.342$
Selection Accuracy	$-4.051/0.005$	$3.919/0.006$	$-6.021/0.001$
Response Time	$4.312/0.004$	$2.981/0.020$	$1.629/0.147$

Appendix

Table 8: Paired statistical tests using participants as the unit of analysis.

Training Set	AL $\downarrow$	BLEU $\uparrow$	ROUGE $\uparrow$
CSL-Daily	2438	24.72	54.02
CSL-Daily + CSL-Story	2032.87	21.91	49.48

Appendix

Table 9: Performance on CSL-Daily for the jointly trained checkpoint used in CSL-Story experiments.

Training Set	AL $\downarrow$	BLEU $\uparrow$	ROUGE $\uparrow$
CSL-Daily + CSL-Story	1588	23.64	41.38
CSL-Story	1830	24.91	44.89

Appendix

Table 10: Performance on CSL-Story with joint training and CSL-Story-only training.

Figure 10: Samples from the CSL-Story Dataset. The dataset features narrative storytelling (e.g., fables) with aligned Chinese subtitles and English translations, introducing vocabulary distinct from daily conversations. The signer’s face is masked to protect privacy.

Metric	How2Sign-Long	CSL-Story
Average	339.2	146.0
Min	15	30
Max	525	270
Std	180.2	72.4

Appendix

Table 11: Buffer length statistics at commitment steps, measured in frames.

Example 1	Success: Mask- $k$ Prevents Premature Commitment
Translation Steps
Step Buffer Hypothesis $68$ $[58:68]$ …The mulch side to keep weeds down where I walk [ . ] $72$ $[58:72]$ …The mulch side to keep weeds down where I walk and now I’m ready to go [ in ] $75$ $[58:75]$ …The mulch side to keep weeds down where I walk and now I’m ready to go into the bead, OK. we’re going to use [ a ]
Reference	…The mulch was there to keep the weeds down where I walk and now it’s all ready to go into the bed. Ok, then we’re going to use a…

Appendix

Analysis. In this example, terminal punctuation is predicted before sufficient visual context is observed at step

68

, but Mask-

k

hides it from the stabilized output and prevents premature commitment. After more chunks arrive, the model revises the hypothesis and commits the correct sentence. Here

k=1

Example 2	Success
Final Translation	Some of you have to push now for a few months but you can’t get off your knees. Hi, I’m Carol Ann with Studio Group X, and I’m going to show you how you can improve your push up when you first start working on your push up. Push you in all four.
Reference	Some of you have been doing push ups now for several months, but you just can’t get off your knees. I am CarolAnn with Studio Group X and I’m going to show you how you can improve your push ups. Well, when you first start working on your push ups you go into all fours.

Appendix

Table 21

Example 3	Success
Final Translation	For our second line, we’re just going to connect the first line that we just did with about ten degrees line, ok. The line goes up, you want to go up about the top of the needle, you don’t want to go any further than that if you want your line, you can go up and high on the line of each nail.
Reference	For our second line, we’re going to want to connect the first line that we just did with about a hundred and ten degree line going up. You only want to go up to about the top quarter of you nail, you don’t want to go all the way to the top.

Appendix

Table 22

Example 4	Error: Premature commitment
Translation Steps
Step Buffer Hypothesis $2$ $[1:2]$ It’s a kick ok. Hi, this [ is ] $3$ $[3:3]$ It’s a kick ok. OK [ . ] $4$ $[3:4]$ It’s a kick ok. Hi, this is Sean [ Doo ]
Reference	The Kick, Ostenado going. So for the Rim clicks we got the two, the and of three…

Appendix

Analysis. The model misinterprets the beginning of a last name starting with “O” as “OK”, which is often followed by terminal punctuation. This causes an early period prediction and triggers premature commitment at step

2

. Once committed, the error cannot be revised and further affects subsequent translation.

Example 5	Error: Delayed Commitment
Translation Steps
Step Buffer Hypothesis $36$ $[15:36]$ …try to move forward, but if you’re set at the basic position, you’re always going to want to move [ . ] $49$ $[15:49]$ …try to move forward, and if you’re going to put your foot on the ground, you always want to move forward, if you [ are ] $50$ $[15:50]$ …try to move forward, and if you’re going to put your foot on the ground, you always want to move forward. There we go [ . ]
Reference	…trying to be moving forward. If you’re in your basic boxing stance and you want to move forward, if you move that back foot first, you’re going to cross those legs…

Appendix

Analysis. From steps 36 to 49, the model does not produce the following sentence, so the terminal punctuation remains in the suffix and is repeatedly hidden by Mask-

k

. As a result, commitment is delayed and the input buffer continues to grow. Although the earlier hypothesis contains partially correct information such as “basic position”, this content is overwritten by later re-translation and disappears from the final committed output.

Example 6	Error: Silence Detection Errors
Translation Steps
Step Buffer Hypothesis $2$ $[1:2]$ <SIL> [ <SIL> ] $3$ $[3:3]$ We’re going to do [ t ] $4$ $[3:4]$ You can see it is a little bit underneath, so [ I ]
Reference	When I lock underneath my legs, I’m going to squeeze in. Let me demonstrate you…

Appendix

Analysis. At step 2, the model incorrectly predicts <SIL> and triggers a commitment, causing meaningful signing frames to be discarded. Subsequent translation therefore starts after the true sentence beginning, leading to incorrect subjects and sentence openings. Although later hypotheses may contain some correct words, the missing initial visual context prevents the full sentence from being recovered.

Method	Dev						Test
Method	AL(ms) $\downarrow$	BLEU1 $\uparrow$	BLEU2 $\uparrow$	BLEU3 $\uparrow$	BLEU4 $\uparrow$	ROUGE $\uparrow$	AL(ms) $\downarrow$	BLEU1 $\uparrow$	BLEU2 $\uparrow$	BLEU3 $\uparrow$	BLEU4 $\uparrow$	ROUGE $\uparrow$
offline SLT
SLRT [ 5 ]	4083	21.03	9.97	5.96	4.04	20.51	4259	20.00	9.11	4.93	3.03	19.67
GASLT [ 41 ]	$\dagger$	-	-	-	-	-	$\dagger$	19.90	9.94	5.98	4.07	20.35
GFSLT-VLP [ 46 ]	$\dagger$	39.20	25.02	16.35	11.07	36.70	$\dagger$	39.37	24.93	16.26	11.00	36.44
FLa-LLM [ 8 ]	$\dagger$	-	-	-	-	-	$\dagger$	37.13	25.12	18.38	14.20	37.25

Appendix

Table 12: Comprehensive SLT performance on CSL-Daily. SLRT method is reproduced by [ 46 ]

Method	Dev						Test
Method	AL(ms) $\downarrow$	BLEU1 $\uparrow$	BLEU2 $\uparrow$	BLEU3 $\uparrow$	BLEU4 $\uparrow$	ROUGE $\uparrow$	AL(ms) $\downarrow$	BLEU1 $\uparrow$	BLEU2 $\uparrow$	BLEU3 $\uparrow$	BLEU4 $\uparrow$	ROUGE $\uparrow$
offline SLT
NSLT [ 4 ]	4299	31.87	19.11	13.16	9.94	31.80	4027	32.24	19.03	12.83	9.58	31.80
GASLT [ 41 ]	$\dagger$	-	-	-	-	-	$\dagger$	39.07	26.74	21.86	15.74	39.86
GFSLT-VLP [ 46 ]	$\dagger$	44.08	33.56	26.74	22.12	43.72	$\dagger$	43.71	33.18	26.11	21.44	42.49
FLa-LLM [ 8 ]	$\dagger$	-	-	-	-	-	$\dagger$	45.27	35.33	28.03	28.03	23.09

Appendix

Table 13: Comprehensive SLT performance on Phoenix-2014-T

Figure 11: Sentence-Level Translation quality (BLEU4) vs. latency (Average Lagging) on Phoenix-2014-T. For CTL++, latency is controlled by parameter $m$ , with points from left to right corresponding to $m\in\{0,2,4,6,8,10,18\}$ . For SimulSLT, latency is adjusted via the wait- $k$ parameter $k_{wait}$ , with points from left to right representing $k_{wait}\in\{1,3,5,7\}$ . For ReSCUE, latency is adjusted via the Mask- $k$ parameter $k$ , with points from left to right representing $k\in\{0,1,2,3,4,5,6\}$ . The top-left region represents the optimal performance.

Method	Test
Method	AL(ms) $\downarrow$	BLEU1 $\uparrow$	BLEU2 $\uparrow$	BLEU3 $\uparrow$	BLEU4 $\uparrow$	ROUGE $\uparrow$
offline SLT
YouTube-ASL [ 34 ]	6659	37.82	24.13	16.92	12.39	-
GloFE-VN [ 21 ]	$\dagger$	14.94	7.27	3.93	2.24	12.61
SSVP-SLT [ 31 ]	$\dagger$	43.2	28.8	20.8	15.5	38.4
C 2 RL [ 7 ]	$\dagger$	29.07	18.56	12.92	9.37	27.02

Appendix

Table 14: Comprehensive SLT performance on How2Sign

Model	AL $\downarrow$	LAAL $\downarrow$	BLEU1 $\uparrow$	BLEU2 $\uparrow$	BLEU3 $\uparrow$	BLEU4 $\uparrow$	Rouge $\uparrow$	NE $\downarrow$
CSL-Story
Uni-Sign	37722	37722	4.22	3.31	2.67	2.10	32.43	0.00
Uni-Sign (Oracle reference)	37722	37722	48.07	38.12	31.54	26.63	44.76	0.00
ReSCUE ( $\beta$ =0.2, $k$ =3)	1588	2725	54.48	39.76	30.39	23.64	41.38	0.15
How2Sign-Long
Uni-Sign	53687	53687	1.93	1.24	0.84	0.59	26.32	0.00

Appendix

Table 15: Comprehensive SLT performance on long-form dataset

Model	AL $\downarrow$	LAAL $\downarrow$	BLEU1 $\uparrow$	BLEU2 $\uparrow$	BLEU3 $\uparrow$	BLEU4 $\uparrow$	Rouge $\uparrow$	NE $\downarrow$
CSL-Story
ReSCUE ( $\beta$ =0.2, $k$ =3)	1588	2725	54.48	39.76	30.39	23.64	41.38	0.15
w/o Sentence Commit	10093	10431	16.16	12.61	10.05	7.91	38.89	1.01
w/o Silence Segments	-473	1699	48.34	33.99	25.29	19.26	37.43	0.32
w/o Prefix-to-Prefix	-2470	1674	47.73	35.57	27.65	21.89	41.21	0.31
w/o Multi-sentence	13525	13663	21.12	16.11	12.74	9.85	38.67	0.76

Appendix

Table 16: Ablation study on long-form dataset. Following [ 28 ] , we prioritize LAAL for latency evaluation, as the standard AL metric may result in negative values for extended long-form inputs.

SimulSLT
$k$	1	3	5	7
AL	1905	2826	3435	3885
BLEU1	45.69	45.89	47.34	47.18
BLEU2	32.66	33.03	35.20	34.85
BLEU3	24.94	25.46	27.74	27.47
BLEU4	20.10	20.70	22.97	22.65

Appendix

Table 17: Numeric results on Phoenix-2014-T

SimulSLT
$k$	1	3	5	7
AL	1427	2054	2756	3370
BLEU1	41.19	42.17	42.09	41.52
BLEU2	27.15	28.45	28.65	28.41
BLEU3	18.09	19.38	19.59	19.60
BLEU4	12.30	13.44	13.55	13.88

Appendix

Table 18: Numeric results on CSL-Daily

ReSCUE( $\beta$ =0.2)
$k$	0	1	2	3
AL	1260	1239	1339	1457
BLEU1	39.18	39.69	39.95	40.07
BLEU2	26.67	27.00	27.17	27.23
BLEU3	19.64	19.86	19.96	19.97
BLEU4	15.03	15.18	15.24	15.22

Appendix

Table 19: Numeric results on How2Sign

$k$	0	1	2	3
CSL-Story
ReSCUE( $\beta$ =0.2)
LAAL	1059	1530	1869	2725
AL	-2041	-201	565	1588
BLEU1	48.85	51.07	51.70	54.48
BLEU2	35.62	37.93	37.69	39.76

Appendix

Table 20: Numeric results on long-form dataset

Explore similar work

Toward Real-Time Sentence-Level Sign Language Translation
Jul 10, 2026Thanh-Hoang Nguyen DoanSign Language Translation
The University of Danang - University of Science and Technology · The University of Danang – University of Science and Technology
Simultaneous Translation between Sign Languages
Sep 28, 2026Zetian Wu, Bowen Xie, Stefan Lee +1Sign Language Translation
Oregon State University
SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translation
Jun 26, 2026Sobhan Asasi, Ozge Mercanoglu Sincan, Richard BowdenSign Language Translation Cross-Lingual Transfer
CVSSP, University of Surrey, United Kingdom

Jul 10, 2026J/K move · Enter open · S save

Toward Real-Time Sentence-Level Sign Language Translation

Thanh-Hoang Nguyen Doan

The University of Danang - University of Science and Technology · The University of Danang – University of Science and Technology

Sign Language Translation

Open paper AskPDF arXiv

Most sign language understanding systems operate at the level of isolated signs, limiting their usefulness in natural communication. We study sentence-level sign language translation (SLT) with the primary goal of real-time deployment rather than proposing a new translation architecture. We fine-tune a SHuBERT-ByT5 translation stack on a uniformly sampled 9,872-example subset of How2Sign, selected because of compute and storage constraints, using QLoRA while keeping SHuBERT frozen. The model obtains a validation BLEU of 16.7 and, on the test split, BLEU 15.9 and BLEURT 44.7. The main contribution is a hardware-aware streaming system: a Raspberry Pi 4B reference client provides camera capture, local text display, and speech output, while compute-intensive perception and translation run on a CPU/GPU backend. The capture protocol remains client-agnostic, so the same backend can serve a browser, phone, or laptop. Chunked ingestion, bounded queues, parallelized perception, temporal reordering, and a sentence-boundary state machine reduce mean post-finalization response latency from 1.873 to 1.354 seconds (27.71%) and P95 latency from 2.919 to 2.130 seconds (27.03%) over the complete 9,872-example working subset.

Abstract

Explore similar work

Toward Real-Time Sentence-Level Sign Language Translation

Simultaneous Translation between Sign Languages

SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translation