Large sign-language corpora benefit from temporal labels at two scales: subtitle spans for subtitle alignment and word spans for sign spotting. Sign spottings already locate words in time at corpus scale, but training subtitle alignment jointly on subtitle and word spans remains underexplored. We present SignAligner, a prompt-conditioned visual-language span generator that maps a sign-language video and a subtitle-sentence or target-word query to a textual start/end frame interval. It is first trained jointly on subtitle spans and word spans from sign spottings, then refined with overlap-based rewards while retaining auxiliary word examples. On the subtitle test set of the British Broadcasting Corporation-Oxford British Sign Language (BOBSL) corpus, the refined model reaches 66.68 [email protected] (F1 with a temporal intersection-over-union threshold of one half), the best among compared systems without external sign-video resources. This overlap-reward refinement outperforms cross-entropy continuation from the same jointly trained model. Joint training with word spans improves subtitle alignment in a pair of our generators matched in code, labels and maximum update budget. The advantage remains after both models of the pair are refined with overlap rewards. For word queries, the refined model follows the queried word, not a fixed position in the video: when the word's sign lies away from the video centre, it is far more accurate than predictions that use position alone. Together, these findings support one shared span generator refined with overlap rewards, aligning subtitles and producing word spans for human review.
Figures & tables
Figure 2: Overview of SignAligner . The prompt selects a subtitle or word query, and the decoder generates the span as text. Stage 1 trains on both span pools, and Stage 2 refines the model with overlap rewards. Drawn spans and rewards are illustrative, on the scale of Eq. ( 3 ).
Split
Use
Labels
Ep.
SA
SS
BOBSL
Train (auto. labels)
Stage 1
SAT + , auto.
1,658
763,976
855,856
Train (manual)
Stage 2
Manual, auto.
16
8,757
7,198
Sent-Val
Selection
Manual
4
1,973
–
Sent-Test
SA, SS tests
Manual
35
20,338
–
Sign-Test
SS test
Verified signs
250
–
24,634
Table 1: Stage 1 trains on automatic labels; every test split is manual or verified. Ep.: episodes. SA, SS: subtitles and word examples or queries ( BOBSL counts after filtering and standard exclusions). Labels: SAT + subtitles [ 2 ] , automatic spottings [ 4 ] (auto.), manual alignment or verified signs. –: not counted.
Method
F1@IoU
FA
0.10
0.25
0.50
Audio priors
Saudio [ 1 , 11 ]
46.4
33.5
14.1
40.1
Saudio+ [ 1 , 11 ]
72.8
64.1
44.6
62.4
BOBSL -only sign-video training
SAT [ 1 ]
74.2
67.0
53.4
71.0
Table 2: SignAligner leads systems without external sign-video resources on BOBSL Sent-Test . Bold marks the best score within each sign-video-resource block. † Mean of three runs.
Figure 3: SignAligner exceeds SA -only on most episodes. Pooled [email protected]: SA -only 62.9, Stage-1 model 64.7, SignAligner (median run) 66.7. SignAligner wins 32 episodes and loses three. SA -only and the Stage-1 model use their own reconstruction settings.
Figure 4: Broadcast subtitle times run ahead of the signing. A selected excerpt of seven consecutive subtitles from one test episode. Arrows link each broadcast start to its reference start (dashed boxes). Bars: final spans of the pair SA -only model (Section 4.1 ) and SignAligner (median run); solid: hit.
Table 3: Overlap-reward refinement improves the Stage-1 model; CE continuation does not on average. All rows use one fixed reconstruction setting. Δ Stage-1 is the [email protected] change from the Stage-1 model. † Mean of three runs; other rows are single runs.
Generator / scoring
Pair SA -only
Pair joint
Llama matched pair
Llama, validation-selected checkpoints
64.05
65.04
Same checkpoints, scored like the refined models
63.99
64.91
After overlap-reward refinement †
65.42
66.85
Qwen pair
Qwen3-4B, 400k maximum updates
63.7
64.1
Table 4: Joint training scores higher for our generator and lower for the decoder-free control. Scores are [email protected]. Bold marks the best score in each comparison.
Figure 5: Off-centre word spans overlap the verified sign. Two selected success cases from Sign-Test , each a word-only query in an off-centre window. The reference is a fixed 0.8-s span around the annotated time of the verified sign.
Table 5: Word spans score far above position-only controls off-centre. Bold marks the best [email protected] within each Sign-Test block. † Mean of three runs.
Crop source
CIDEr
BLEURT ( ×100 )
SAT
56.5
42.9
SAT +
58.8
42.9
SignAligner
59.1
43.9
Table 6: Exploratory transfer and crop-utility checks. (a) Translation trained with alternative subtitle crops. (b) Cross-lingual subtitle alignment. Bold marks the best score per metric in (a).