Recent advances in sign language understanding have achieved impressive success on short, single-sentence videos, yet their performance drops sharply when applied to long, continuous sign language videos. To bridge this gap, we focus on a challenging and realistic setting: Visual-only Sentence-level Sign Language Segmentation (Vis-SSLS), which aims to partition continuous sign language videos into non-overlapping sentence-level segments without any caption assistance, serving as a crucial prerequisite for downstream recognition and translation tasks. However, sentence transitions in sign language are often smooth and visually ambiguous, lacking explicit pauses or posture resets. As a result, static frame representations may fail to capture the subtle temporal changes that indicate sentence boundaries. To address this challenge, we propose SignShift, a difference-aware segmentation framework that explicitly models frame-to-frame feature variation as semantic cues for sentence boundary detection. First, to model the feature variation, we design a Temporal Difference Module, which incorporates full-frame, facial, and hand cues, and employs inter-frame differencing to learn multi-scale temporal variations that capture both fine-grained local kinematics and global semantic transitions. Second, to mitigate over- and under-segmentation issues, we design a Segment Count Prediction module, which predicts the number of sentences to guide boundary selection. Extensive experiments on benchmark datasets demonstrate that SignShift substantially outperforms existing methods, validating its effectiveness.
Figures & tables
Figure 1. Overview of the Vis-SSLS task. Given a long continuous signing video, the goal is to detect sentence boundaries and segment it into coherent sentence-level clips. (Note: only videos are available, while captions are NOT available.)
Figure 2. Overall framework of our proposed method.
Figure 3. Illustration of the proposed TDM. (a) shows the multi-scale difference extraction process that captures temporal variations at different temporal offsets. (b) presents the overall TDM structure, which fuses the multi-scale difference features to produce enhanced representations.
Model
How2Sign
OpenASL
Dev
Test
Dev
Test
F1@{10,25,50}
SER↓
F1@{10,25,50}
SER↓
F1@{10,25,50}
SER↓
F1@{10,25,50}
SER↓
MS-TCN ( Farha and Gall, 2019 )
64.93 /
62.78 /
45.12
0.51
65.64 /
63.49 /
43.36
0.48
79.96 /
78.61 /
65.06
0.29
79.06 /
77.77 /
65.81
0.29
MS-TCN++ ( Li et al., 2020b )
69.29 /
67.62 /
48.03
0.45
72.91 /
70.16 /
53.12
0.41
76.23 /
74.96 /
62.22
0.31
74.88 /
73.48 /
59.12
0.34
ASFormer ( Yi et al., 2021 )
72.86 /
69.60 /
49.44
0.42
74.56 /
72.93 /
53.80
0.36
76.21 /
74.87 /
61.40
0.33
74.50 /
73.05 /
58.88
0.35
SignBD ( Guo et al., 2025 )
76.40 /
74.39 /
55.51
0.35
76.89 /
73.97 /
55.02
0.36
79.23 /
78.42 /
64.99
0.29
78.95 /
77.56 /
65.03
0.30
Table 1. Comparison with state-of-the-art methods on How2Sign and OpenASL (Dev & Test). Best results are in bold .
Dataset
Modules
TS model: MS-TCN
TS model: ASFormer
TDM
Hands&Face
SCP
F1@10
F1@25
F1@50
SER ↓
F1@10
F1@25
F1@50
SER ↓
How2Sign ( Duarte et al., 2021 )
–
–
–
64.93
62.78
45.12
0.51
72.86
69.60
49.44
0.42
✓
–
–
68.83
66.33
47.60
0.45
75.10
73.77
55.23
0.39
✓
✓
–
71.02
69.69
51.71
0.43
77.99
75.15
53.40
0.35
✓
✓
✓
81.05
79.26
61.82
0.26
83.30
81.73
61.43
0.23
OpenASL ( Shi et al., 2022 )
–
–
–
79.96
78.61
65.06
0.29
76.03
74.29
60.11
0.28
Table 2. Ablation study of SignShift using different TAS models across datasets. Best results are in bold .
Figure 4. Segment count estimation results. (a) Comparison of the average segment count. (b) Mean Absolute Error (MAE) of count estimation.
Duration (s)
Avg Frames
F1 Score (%)
SER ↓
@10
@25
@50
17.9
449.8
85.19
83.55
66.80
0.25
31.9
814.9
84.11
81.83
65.95
0.22
78.71
1887.8
83.65
78.24
51.03
0.13
Table 3. Segmentation results on videos of different durations on How2Sign dataset.
Model
Efficiency
Performance
Params ↓
FLOPs ↓
Time ↓
F1@50 ↑
MS-TCN++
0.89
1.42
18.61
43.36
ASFormer
1.90
2.93
214.45
53.80
SignBD
6.47
2.35
123.10
55.02
DiffAct
2.85
4.94
65.96
55.69
SignShift
5.26
4.25
120.27
62.47
Table 4. Comparison of models in terms of Parameters (M), FLOPs (G), Inference Time (ms), and F1@50.
Figure 5. Qualitative results on OpenASL. The checkmarks ( ✓ ) indicate correctly identified sentence boundaries by SignShift, whereas the crosses ( × ) denote incorrect or missed boundaries predicted by the baseline models.
Figure 6. Qualitative comparison of boundary detection.
Continuous sign language recognition (CSLR) aims to recognize gloss sequences from unsegmented sign videos under weak sequence-level supervision. However, existing methods rely on sentence-level gloss annotations, providing limited temporal and semantic guidance for fine-grained representation learning. Conventional video-text alignment also requires large batch sizes, making it inefficient for memory-intensive sign language video training. In this work, we propose SMART, an MLLM-guided temporal alignment framework for joint sign recognition and spotting. SMART uses MLLMgenerated motion descriptions as auxiliary semantic cues and performs stable videotext alignment under small-batch training. To improve temporal representation learning, we introduce a Multi-Scale Temporal Adapter that models temporal interactions during transformer encoding. For dense temporal localization, SMART incorporates CSFormer, a CSLR-guided spotting module that injects recognition-derived gloss evidence into a boundary-aware spotting network. This unified framework enables CSLR features to benefit spotting, while spotting supervision complements weak CTC-based recognition. Experiments on four sign language benchmarks, including PHOENIX14-T, CSL-Daily, Large-scale KSL, and Disaster and Safety KSL datasets, demonstrate the effectiveness of SMART across both recognition and spotting tasks.
Simultaneous Sign Language Translation (SLT) is critical for real-time communication, yet existing methods remain largely confined to sentence-level, offline settings that assume pre-segmented inputs. These assumptions hinder deployment in realistic scenarios involving continuous, unsegmented video streams. We present ReSCUE, a unified framework for simultaneous SLT on unsegmented long-form sign language videos that aligns training and inference with realistic streaming conditions. ReSCUE combines inference-aware training to handle partial inputs, non-signing pauses, and multi-sentence contexts, stabilized re-translation to enable low-latency yet revisable predictions with reduced output flicker, and a sentence commitment mechanism for online segmentation and memory management. Experiments on standard sentence-level benchmarks show that ReSCUE achieves lower latency and the best translation quality under low-latency settings. On long-form unsegmented datasets, ReSCUE approaches the translation quality of oracle offline systems that use ground-truth sentence boundaries, while operating at substantially lower latency, demonstrating its practicality for real-world streaming scenarios.
Sihan Ren, Gaozheng Li, Yuanshang Quan +5
ShanghaiTech University · DGene · University of Derby
Vision-language models (VLMs) have emerged as a powerful framework for multimodal video understanding. However, they remain limited in the sign language translation task, where we identify a key failure mode of existing VLMbased translators: poor spatial-temporal visual grounding. In particular, we find that standard next-token cross-entropy does not directly provide signal for where and when the model should attend, causing models to overlook sign-relevant regions and frames. To address this challenge, we propose AttnSign, a VLM-based spatial-temporal attention steering framework for sign language translation. AttnSign first introduces spatial attention supervision for sign-relevant regions, such as face and hands, in each frame; then develops an RL-based motion-cadence steering method that encourages the model to explore and focus on sign-level keyframes. Experimental results on How2Sign and OpenASL benchmarks show that our proposed AttnSign consistently outperforms existing methods.
Meibo Hu, Guohao Sun, Annemarie D. Ross +2
Rochester Institute of Technology · National Technical Institute for the Deaf / Rochester Institute of Technology · University of Virginia