Recent advances in sign language understanding have achieved impressive success on short, single-sentence videos, yet their performance drops sharply when applied to long, continuous sign language videos. To bridge this gap, we focus on a challenging and realistic setting: Visual-only Sentence-level Sign Language Segmentation (Vis-SSLS), which aims to partition continuous sign language videos into non-overlapping sentence-level segments without any caption assistance, serving as a crucial prerequisite for downstream recognition and translation tasks. However, sentence transitions in sign language are often smooth and visually ambiguous, lacking explicit pauses or posture resets. As a result, static frame representations may fail to capture the subtle temporal changes that indicate sentence boundaries. To address this challenge, we propose SignShift, a difference-aware segmentation framework that explicitly models frame-to-frame feature variation as semantic cues for sentence boundary detection. First, to model the feature variation, we design a Temporal Difference Module, which incorporates full-frame, facial, and hand cues, and employs inter-frame differencing to learn multi-scale temporal variations that capture both fine-grained local kinematics and global semantic transitions. Second, to mitigate over- and under-segmentation issues, we design a Segment Count Prediction module, which predicts the number of sentences to guide boundary selection. Extensive experiments on benchmark datasets demonstrate that SignShift substantially outperforms existing methods, validating its effectiveness.
Figures & tables
Figure 1. Overview of the Vis-SSLS task. Given a long continuous signing video, the goal is to detect sentence boundaries and segment it into coherent sentence-level clips. (Note: only videos are available, while captions are NOT available.)
Figure 2. Overall framework of our proposed method.
Figure 3. Illustration of the proposed TDM. (a) shows the multi-scale difference extraction process that captures temporal variations at different temporal offsets. (b) presents the overall TDM structure, which fuses the multi-scale difference features to produce enhanced representations.
Model
How2Sign
OpenASL
Dev
Test
Dev
Test
F1@{10,25,50}
SER↓
F1@{10,25,50}
SER↓
F1@{10,25,50}
SER↓
F1@{10,25,50}
SER↓
MS-TCN ( Farha and Gall, 2019 )
64.93 /
62.78 /
45.12
0.51
65.64 /
63.49 /
43.36
0.48
79.96 /
78.61 /
65.06
0.29
79.06 /
77.77 /
65.81
0.29
MS-TCN++ ( Li et al., 2020b )
69.29 /
67.62 /
48.03
0.45
72.91 /
70.16 /
53.12
0.41
76.23 /
74.96 /
62.22
0.31
74.88 /
73.48 /
59.12
0.34
ASFormer ( Yi et al., 2021 )
72.86 /
69.60 /
49.44
0.42
74.56 /
72.93 /
53.80
0.36
76.21 /
74.87 /
61.40
0.33
74.50 /
73.05 /
58.88
0.35
SignBD ( Guo et al., 2025 )
76.40 /
74.39 /
55.51
0.35
76.89 /
73.97 /
55.02
0.36
79.23 /
78.42 /
64.99
0.29
78.95 /
77.56 /
65.03
0.30
Table 1. Comparison with state-of-the-art methods on How2Sign and OpenASL (Dev & Test). Best results are in bold .
Dataset
Modules
TS model: MS-TCN
TS model: ASFormer
TDM
Hands&Face
SCP
F1@10
F1@25
F1@50
SER ↓
F1@10
F1@25
F1@50
SER ↓
How2Sign ( Duarte et al., 2021 )
–
–
–
64.93
62.78
45.12
0.51
72.86
69.60
49.44
0.42
✓
–
–
68.83
66.33
47.60
0.45
75.10
73.77
55.23
0.39
✓
✓
–
71.02
69.69
51.71
0.43
77.99
75.15
53.40
0.35
✓
✓
✓
81.05
79.26
61.82
0.26
83.30
81.73
61.43
0.23
OpenASL ( Shi et al., 2022 )
–
–
–
79.96
78.61
65.06
0.29
76.03
74.29
60.11
0.28
Table 2. Ablation study of SignShift using different TAS models across datasets. Best results are in bold .
Figure 4. Segment count estimation results. (a) Comparison of the average segment count. (b) Mean Absolute Error (MAE) of count estimation.
Duration (s)
Avg Frames
F1 Score (%)
SER ↓
@10
@25
@50
17.9
449.8
85.19
83.55
66.80
0.25
31.9
814.9
84.11
81.83
65.95
0.22
78.71
1887.8
83.65
78.24
51.03
0.13
Table 3. Segmentation results on videos of different durations on How2Sign dataset.
Model
Efficiency
Performance
Params ↓
FLOPs ↓
Time ↓
F1@50 ↑
MS-TCN++
0.89
1.42
18.61
43.36
ASFormer
1.90
2.93
214.45
53.80
SignBD
6.47
2.35
123.10
55.02
DiffAct
2.85
4.94
65.96
55.69
SignShift
5.26
4.25
120.27
62.47
Table 4. Comparison of models in terms of Parameters (M), FLOPs (G), Inference Time (ms), and F1@50.
Figure 5. Qualitative results on OpenASL. The checkmarks ( ✓ ) indicate correctly identified sentence boundaries by SignShift, whereas the crosses ( × ) denote incorrect or missed boundaries predicted by the baseline models.
Figure 6. Qualitative comparison of boundary detection.