BEACON: Distilled Begin/End-Aware Structured Decoding for Continuous Sign Language Segmentation

Guohou Zhang; Kalin Stefanov

cs.CVOct 29, 2026

BEACON: Distilled Begin/End-Aware Structured Decoding for Continuous Sign Language Segmentation

Authors: Guohou Zhang, Kalin Stefanov

Organizations: Monash University, Australia · Japan Advanced Institute of Science and Technology (JAIST), Japan

Abstract

In segmentation-based continuous sign language recognition, an untrimmed video is first partitioned into lexical units; segmentation is therefore a means to an end---downstream recognition. However, segmenters are usually judged by local boundary detection rather than by the quality of the lexical segments and the recognition they enable. We present BEACON, a begin/end-aware segmenter that combines four-class Outside/Begin/Inside/End (BIOE) labeling, constrained Viterbi decoding over eight legal transitions, and boundary knowledge distilled from a binary teacher into a shared student. Under strict no-cut evaluation with validation-based model selection, BEACON improves segment-F1 by 14.58/14.59 points and boundary-F1 by 2.17/1.22 points, and reduces the downstream word error rate (WER) by 9.10/4.15 points relative to the state-of-the-art in the PHOENIX-14/BSL Corpus. To examine which segmentation metric better reflects downstream recognition, we evaluated five distinct architecture families. Segment-F1 aligns more consistently with lower WER than boundary-F1 in both corpora (Spearman $ρ=-1.0/-0.9$ versus $-0.9/-0.8$ ). These results support evaluating sign language segmentation in terms of segment quality and downstream recognition utility rather than local boundary accuracy alone.

Figures & tables

Figure 1: BEACON replaces Renz et al. [ 19 ] ’s binary boundary mask—which collapses sign onset/offset—with a 4-class BIOE labeling (explicit Outside/Begin/Inside/End states) and a constrained Viterbi decoder whose eight legal transitions make every output a well-formed segment sequence by construction . This design improves segment quality while retaining boundary accuracy and lowers downstream recognition

\mathrm{WER}

Property	Renz et al. [ 19 ]	Hands-On [ 13 ]	CMPL [ 20 ]	BEACON
Setting	Supervised	Supervised	Source-free DA	Supervised
Input	I3D	HaMeR + 3D angles	I3D	I3D
Temporal model	MS-TCN	Transformer	MS-TCN	TCN + Transformer
Labels	Binary	BIO	Binary	BIOE + IO
Readout	$\mathrm{argmax}$	$\mathrm{argmax}$	$\mathrm{argmax}$	Constrained Viterbi
Boundary cue	GT endpoint cut	—	CP-modulated PL	Distilled IO

Table 1: Positioning of the closest sign-segmentation approaches. DA and PL denote domain adaptation and pseudo-labeling. Matched

\mathrm{WER}

uses a fixed corpus-matched recognizer; Sec. 5 reports our controlled re-evaluation.

Figure 2: Constrained Viterbi decoding [ 24 ] over the BIOE grammar. A framewise $\mathrm{argmax}$ path may contain illegal transitions; the decoder instead returns the highest emission path restricted to the eight legal BIOE moves, so the decoded sequence satisfies the declared grammar. Emission totals are illustrative.

Figure 3: The BEACON pipeline. A frozen I3D [ 3 ] backbone supplies frame features to a shared temporal encoder (Stage 1, standard). BIOE and soft Begin/End predictions are fused and decoded over the eight legal BIOE transitions (inset), producing a preliminary segment path that satisfies the declared grammar (Stage 2, ours). A binary IO branch reads the same shared hidden states $H$ (no extra encoder) and refines endpoints within a validation-selected radius of at most $\pm 3$ frames (Stage 3, ours). During training only, a separately trained, frozen binary teacher (shown dashed, with its own encoder, and discarded after training) distills boundary evidence into this student IO branch via a KL loss; inference uses a single 5M-parameter model with one encoder. The lower strip visualizes the per-frame BIOE labels collapsing into the final segment spans.

Corpus	Segmenter	mF1B $\uparrow$	mF1S $\uparrow$
PHX [ 7 ]	Renz et al. [ 19 ]	69.76	51.12
PHX [ 7 ]	BEACON	71.93 $\pm$ 0.19	65.70 $\pm$ 0.15
BSL [ 21 ]	Renz et al. [ 19 ]	67.28	43.83
	Hands-On [ 13 ] †	62.39 $\pm$ 0.06	47.06 $\pm$ 0.41
	BEACON	68.50 $\pm$ 0.64	58.42 $\pm$ 1.03

Table 2: Strict no-cut segmentation results. BEACON and Hands-On [ 13 ] † are three-seed mean

\pm

SD; Renz et al. [ 19 ] is the public checkpoint. Hands-On † uses full-resolution HaMeR features; all other rows use I3D.

	PHX [ 7 ]				BSL [ 21 ]
Segmenter	WER $\downarrow$	S	D	I	WER $\downarrow$	S	D	I
Renz et al. [ 19 ]	49.73	32.03	5.55	12.15	77.48	57.85	12.13	7.50
Hands-On [ 13 ] †	—	—	—	—	77.52 $\pm$ 0.41	55.55	16.55	5.42
BEACON	40.63 $\pm$ 0.20	27.06	8.90	4.67	73.33 $\pm$ 0.31	55.43	12.68	5.22

Table 3: Downstream recognition under a matched recognizer.

\mathrm{WER}

and substitution/deletion/insertion rates are corpus-level percentages; lower is better. BEACON and Hands-On † use three-seed mean

\pm

SD for

\mathrm{WER}

and three-seed mean for S/D/I. Hands-On † uses HaMeR features; dashes denote an unavailable PHOENIX-14 model. None uses test labels for model/decoder selection or endpoint cutting.

Figure 4: PHOENIX-14 [ 7 ] example (no-cut; seed 42). Enlarged B/E frames anchor each clip to the frame-aligned GT and predictions. Numbers report segment IoU with the corresponding GT sign.

PHX [ 7 ]
Family	mF1B $\uparrow$	mF1S $\uparrow$	WER $\downarrow$
MS-TCN	69.76	51.12	49.73
Local Transformer	48.29 $\pm$ 0.54	37.03 $\pm$ 0.07	50.90 $\pm$ 0.92
BiLSTM-CRF	67.80 $\pm$ 0.10	58.08 $\pm$ 0.26	47.15 $\pm$ 0.75
MS-TCN++	70.60 $\pm$ 0.67	60.10 $\pm$ 1.71	42.54 $\pm$ 0.29
BEACON	71.93 $\pm$ 0.19	65.70 $\pm$ 0.15	40.63 $\pm$ 0.20

Table 4: Architecture families comparison under strict no-cut evaluation (three-seed mean

\pm

SD except Renz et al. [ 19 ] ’s MS-TCN). Local Transformer is an architecture control, not an exact ASFormer [ 25 ] reproduction.

Corpus	Labels/Readout	mF1B $\uparrow$	mF1S $\uparrow$	$\Delta$ mF1S	WER $\downarrow$
PHX [ 7 ]	BIO/argmax	70.78	62.26	—	—
	BIO/grammar	70.68	62.80	—	43.82
	BIOE/argmax	70.56	62.25	—	—
	BIOE/8-edge	70.38	62.76	$-0.04$	43.29
BSL [ 21 ]	BIO/argmax	64.79	54.85	—	—
	BIO/grammar	63.98	54.54	—	73.95

Table 5: Matched label/readout comparison, strict no-cut (three-seed mean). Other factors are fixed; each constrained row uses the matching grammar.

\mathrm{WER}

is reported for constrained readout.

System	mF1B $\uparrow$	mF1S $\uparrow$	WER $\downarrow$
BIOE/8-edge base	70.38/65.48	62.76/55.45	43.29/73.95
IO branch, no KD	71.55/68.10	65.22/58.17	40.72/73.74
Full BEACON	71.93/68.50	65.70/58.42	40.63/73.33

Table 6: Component comparison, strict no-cut (three-seed mean). Each entry reports PHOENIX-14/BSL. The last two rows have the same shared-student architecture and endpoint readout, isolating KD.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Corpus/Seed	Epoch	$\boldsymbol{\alpha}$	$\boldsymbol{r}$	$\boldsymbol{\tau}$	mF1B $\uparrow$	mF1S $\uparrow$
PHX/42	12	.7	2	.5	71.80	65.54
PHX/43	6	.0	1	.3	71.85	65.83
PHX/44	10	.7	3	.3	72.15	65.74
mean $\pm$ SD	—	—	—	—	71.93 $\pm$ .19	65.70 $\pm$ .15
BSL/42	10	.7	3	.5	67.92	57.39
BSL/43	6	.7	1	.3	69.18	58.42

Appendix

Table 7: Distilled BEACON per-seed segmentation results and validation-selected IO refinement settings

(\alpha_{\rm IO},r,\tau_{\rm IO})

, all under strict no-cut evaluation.

Seed	mF1B $\uparrow$	mF1S $\uparrow$
42	62.45	47.44
43	62.38	47.11
44	62.33	46.63
mean $\pm$ SD	62.39 $\pm$ .06	47.06 $\pm$ .41

Appendix

Table 8: Hands-On † reproduction by seed using full-resolution HaMeR features under strict no-cut evaluation. Seed-42 uses the recovered checkpoint; seeds-43 and 44 use fixed epoch-40 checkpoints from the same recipe.

Corpus	Segmenter/Run	WER	S	D	I
PHX	Renz et al.	49.73	32.03	5.55	12.15
	BEACON/42	40.79	27.21	8.87	4.71
	BEACON/43	40.41	26.60	10.13	3.68
	BEACON/44	40.69	27.37	7.71	5.62
	BEACON mean $\pm$ SD	40.63 $\pm$ .20	27.06	8.90	4.67
BSL	Renz et al.	77.48	57.85	12.13	7.50

Appendix

Table 9: Per-run downstream decomposition under the same frozen recognizer per corpus. Rates are corpus-level percentages. Renz et al. is one public checkpoint; Hands-On † and BEACON report all three seeds, with mean

\pm

SD for WER and mean S/D/I.

Item	Chosen by	Test Set Role
Teacher checkpoint	validation mF1B	final reporting only
Student checkpoint	validation balanced score	final reporting only
IO refinement $(\alpha,r,\tau)$	validation grid	final reporting only
B/E fusion and decoder biases	fixed across primary runs	final reporting only
Architecture controls	validation balanced score	final reporting only
Renz et al. checkpoint	released artifact	evaluation only

Appendix

Table 10: Training, selection, and test-use separation for the reported systems.

Corpus	Training	Seed	mF1B $\uparrow$	mF1S $\uparrow$	WER $\downarrow$
PHX	No KD	42	71.74	65.26	40.57
	No KD	43	71.25	65.05	40.68
	No KD	44	71.65	65.34	40.92
	KD	42	71.80	65.54	40.79
	KD	43	71.85	65.83	40.41
	KD	44	72.15	65.74	40.69

Appendix

Table 11: Matched distillation ablation under strict no-cut evaluation. The non-distilled and distilled rows use the same shared-student architecture and readout.

Corpus	Segmenter/Protocol	mF1B $\uparrow$	mF1S $\uparrow$	WER $\downarrow$
PHX	Renz et al./no-cut	69.76	51.12	49.73
	Renz et al./cut	70.28	51.66	—
	BEACON/no-cut	71.93	65.70	40.63
	BEACON/cut	72.08	65.78	40.58
BSL	Renz et al./no-cut	67.28	43.83	77.48
	Renz et al./cut	68.32	45.00	—

Appendix

Table 12: Paired no-cut/cut results from identical predictions. BEACON is a three-seed mean; Renz et al. is one checkpoint.

Corpus	Labels	Readout	mF1B $\uparrow$	mF1S $\uparrow$	WER $\downarrow$
PHX	BIO	argmax	70.78 $\pm$ .14	62.26 $\pm$ .19	—
	BIO	grammar	70.68 $\pm$ .16	62.80 $\pm$ .37	43.82 $\pm$ .15
	BIOE	argmax	70.56 $\pm$ .34	62.25 $\pm$ .41	—
	BIOE	8-edge	70.38 $\pm$ .51	62.76 $\pm$ .27	43.29 $\pm$ 1.01
BSL	BIO	argmax	64.79 $\pm$ .58	54.85 $\pm$ .58	—
	BIO	grammar	63.98 $\pm$ .55	54.54 $\pm$ .75	73.95 $\pm$ .25

Appendix

Table 13: Fair ablation, three-seed mean

\pm

SD. WER is reported for grammar-constrained readout only.

Corpus	mF1B $\uparrow$	mF1S $\uparrow$	WER $\downarrow$	Invalid (%)
PHX	70.13 $\pm$ 1.29	62.78 $\pm$ 2.09	43.77 $\pm$ 2.75	12.65
BSL	66.40 $\pm$ 1.06	56.53 $\pm$ 1.45	73.43 $\pm$ 1.15	12.63

Appendix

Table 14: Learned transition CRF, strict no-cut, three-seed mean

\pm

SD. Invalid is the mean percentage of test sequences containing at least one transition outside the fixed grammar.

PHX
Family	mF1B $\uparrow$	mF1S $\uparrow$	WER $\downarrow$
MS-TCN	69.76	51.12	49.73
Local Transformer	48.29 $\pm$ .54	37.03 $\pm$ .07	50.90 $\pm$ .92
BiLSTM-CRF	67.80 $\pm$ .10	58.08 $\pm$ .26	47.15 $\pm$ .75
MS-TCN++	70.60 $\pm$ .67	60.10 $\pm$ 1.71	42.54 $\pm$ .29
BEACON	71.93 $\pm$ .19	65.70 $\pm$ .15	40.63 $\pm$ .20

Appendix

Table 15: Architecture families comparison (three-seed mean

\pm

SD except the single public Renz et al. checkpoint). Local Transformer is an architecture control, not an exact ASFormer reproduction. Exact permutation

p

-values accompany Spearman

\rho

System	Params	Resident	Peak	Latency
		(MiB)	(MiB)	(ms)
Two encoders	8.82M	34.65	45.25–47.71	1.41–1.43
Single distilled	5.00M	19.59	30.19–32.65	0.89–0.90

Appendix

Table 16: Inference efficiency. Latency ranges cover both corpora and sequence lengths.

Sign Language Recognition

Sign language recognition uses video, images, or sensor data to identify signs, fingerspelling, and continuous signing across people and settings.

See all 57 papers

Seeing Semantic Shift: Difference-Aware Sentence-Level Temporal Segmentation of Sign Language Videos
Sep 25, 2026Bowen Guo, Shiwei Gan, Yafeng Yin +4Boundary-Aware Segmentation Temporal Action Segmentation
State Key Laboratory for Novel Software Technology, Nanjing University, Suzhou, China
TransSLR: A Lightweight Transformer for Sign Language Recognition
Aug 3, 2026Lucia Yen Wanchi, Samuel Johnny, Victor Tolulope Olufemi +2Sign Language Recognition
Carnegie Mellon University Africa
SignMAE: Segmentation-Driven Self-Supervised Learning for Sign Language Recognition
May 3, 2026Kunyuan Xie, Zhixi Cai, Kalin StefanovSelf-Supervised Pre-Training Sign Language Recognition
Monash University, Australia

Sep 25, 2026J/K move · Enter open · S save

Seeing Semantic Shift: Difference-Aware Sentence-Level Temporal Segmentation of Sign Language Videos

Bowen Guo, Shiwei Gan, Yafeng Yin, Xiao Liu+3

State Key Laboratory for Novel Software Technology, Nanjing University, Suzhou, China

Boundary-Aware Segmentation Temporal Action Segmentation Sign Language Recognition

Open paper AskPDF arXiv

Recent advances in sign language understanding have achieved impressive success on short, single-sentence videos, yet their performance drops sharply when applied to long, continuous sign language videos. To bridge this gap, we focus on a challenging and realistic setting: Visual-only Sentence-level Sign Language Segmentation (Vis-SSLS), which aims to partition continuous sign language videos into non-overlapping sentence-level segments without any caption assistance, serving as a crucial prerequisite for downstream recognition and translation tasks. However, sentence transitions in sign language are often smooth and visually ambiguous, lacking explicit pauses or posture resets. As a result, static frame representations may fail to capture the subtle temporal changes that indicate sentence boundaries. To address this challenge, we propose SignShift, a difference-aware segmentation framework that explicitly models frame-to-frame feature variation as semantic cues for sentence boundary detection. First, to model the feature variation, we design a Temporal Difference Module, which incorporates full-frame, facial, and hand cues, and employs inter-frame differencing to learn multi-scale temporal variations that capture both fine-grained local kinematics and global semantic transitions. Second, to mitigate over- and under-segmentation issues, we design a Segment Count Prediction module, which predicts the number of sentences to guide boundary selection. Extensive experiments on benchmark datasets demonstrate that SignShift substantially outperforms existing methods, validating its effectiveness.

Abstract

Sign Language Recognition

Explore similar work

Seeing Semantic Shift: Difference-Aware Sentence-Level Temporal Segmentation of Sign Language Videos

TransSLR: A Lightweight Transformer for Sign Language Recognition

SignMAE: Segmentation-Driven Self-Supervised Learning for Sign Language Recognition