SignDino: Self-Supervised Sign Language Representation Learning via Temporal-Axis Self-Distillation

Junyi Hu; Zhewen He; Haomian Huang; Zhenhua Li; Zhifei Li; Yi Fang

cs.CVSep 5, 2026

SignDino: Self-Supervised Sign Language Representation Learning via Temporal-Axis Self-Distillation

Authors: Junyi Hu, Zhewen He, Haomian Huang, Zhenhua Li, Zhifei Li, Yi Fang

Organizations: New York University Abu Dhabi

Abstract

Self-supervised sign language representation learning must model two properties not central to natural-image SSL: signs are produced by a small set of anatomically distinct articulators, and their meaning depends on the temporal organisation of those articulators. We introduce SignDino, a self-supervised sign-video encoder that moves the DINOv3 student--teacher recipe from the spatial domain of image crops to the temporal domain of tracked sign streams. Each video is decomposed into left-hand, right-hand, and face streams by a detector-first YOLOv8n+ByteTrack pipeline. A frozen DINOv3 ViT-B/16 embeds each per-frame anatomical crop, while lightweight temporal Transformers, not the image backbone, form the student and EMA teacher. They are trained by temporal DINO self-distillation, frame-level masked-token prediction in the style of iBOT, KoLeo feature spreading, and Gram anchoring of the frame-to-frame similarity structure. This design keeps strong image-level visual primitives fixed and learns only how articulator states evolve across time. We evaluate on sign-to-English translation, isolated sign recognition, and fingerspelling detection benchmarks. Across these tasks, SignDino provides a strong public self-supervised representation and shows competitive or state-of-the-art performance under matched downstream evaluation.

Figures & tables

Figure 1: The local–global distillation game, ported from image space to the time axis. Left: image DINOv3 ( Siméoni et al., 2025 ) samples spatial global and local crops in a single image . Right: SignDino samples temporal global ( $T_{g}\!\sim\!\mathcal{U}\{64,\ldots,96\}$ ) and local ( $T_{l}\!\sim\!\mathcal{U}\{10,\ldots,32\}$ ) crops along the tracked $L_{t},R_{t},F_{t}$ rows of a single anatomical stream , with iBOT masks on a few frame slots inside the local crop.

Figure 2: SignDino pipeline. Row A (left-to-right): each source video $V$ passes through a YOLOv8n+ByteTrack detector-tracker yielding synchronised per-frame anatomical streams $x_{t}^{L},x_{t}^{R},x_{t}^{F}$ . Row B (right-to-left, snake): a frozen DINOv3 ViT-B/16 encodes every crop into a $768$ -d CLS token, producing three per-frame embedding sequences $E^{(r)}\!\in\!\mathbb{R}^{T\times 768}$ , each feeding one independent temporal SSL trainer. Row C : stage-2 SignDino teachers serve as per-stream sign encoders $f_{\phi}^{(r)}$ that downstream translation, ISLR and fingerspelling heads consume as fused per-frame features. Figure 4 details the student/teacher loss.

Figure 3: Detector-first pipeline recovers hands YOLOv8n alone misses. Each cell shows the same OpenASL frame under YOLOv8n only (top) and YOLOv8n + ByteTrack with Kalman fill (bottom). Green boxes are active YOLO detections; magenta are tracker predictions for hands YOLOv8n missed at this frame.

Figure 4: Temporal student/teacher game for one anatomical stream (shown for

L

). The frozen DINOv3 ViT-B/16 supplies per-frame embeddings

E^{L}_{1:T}

once; the temporal SSL game runs on top. The teacher sees only global temporal crops of length

T_{g}\!\sim\!\mathcal{U}\{64,\ldots,96\}

, while the student sees both global and local crops of length

T_{l}\!\sim\!\mathcal{U}\{10,\ldots,32\}

with a uniform fraction of frame positions replaced by the iBOT [MASK] token. The teacher is the EMA of the student. The Stage 1 loss combines three terms; Stage 2 adds a Gram anchoring term constraining frame-to-frame structure inside each clip.

Method	SSL	PT hrs	How2Sign		OpenASL		FLEURS-ASL
Method	SSL	PT hrs	BLEU $\uparrow$	BLEURT $\uparrow$	BLEU $\uparrow$	BLEURT $\uparrow$	BLEU $\uparrow$
SSVP-SLT ( Rust et al., 2024 )	✓	1054	15.5	49.6	—	—	—
YouTube-ASL fine-tune ( Uthus et al., 2023 )	$\times$	984	12.4	46.6	—	—	—
YouTube-SL-25 ( Tanzer and Zhang, 2024 )	$\times$	3207	15.4	47.9	—	—	4.4
SignMusketeers ( Gueuwou et al., 2025a )	✓	984	14.3	—	—	—	—
SpaMo ( Hwang et al., 2025 )	$\times$	—	10.11	42.23	—	—	—

Table 1: Sign language translation on How2Sign, OpenASL and FLEURS-ASL under the SHuBERT public-data protocol. SSL

\,=\,

✓ denotes self-supervised pre-training; PT hrs report source pre-training video hours used before benchmark-specific fine-tuning. For SignDino , Phase 1 uses the OpenASL-decontaminated YouTube-ASL split, Phase 2 fine-tunes on How2Sign or OpenASL, and FLEURS-ASL is evaluated zero-shot. SpaMo reports How2Sign BLEU-4/BLEURT under its LLM-based gloss-free protocol and is included as a supervised LLM reference. The “frozen vs live” pair follows SHuBERT ’s frozen/fine-tuned comparison. SignDino entries are mean

\pm

standard deviation over

3

random seeds; baselines are reported as published.

Prior published results
Method	Trainable	ASL Citizen			Sem-Lex			WLASL2000		ASL-STEM-Wiki
Method	Trainable	R@1 $\uparrow$	R@5 $\uparrow$	R@10 $\uparrow$	R@1 $\uparrow$	R@5 $\uparrow$	R@10 $\uparrow$	P-I $\uparrow$	P-C $\uparrow$	mIoU $\uparrow$
ST-GCN ( Desai et al., 2023 )	0.45M	0.60	0.82	0.88	—	—	—	—	—	—
SignCLIP ( Jiang et al., 2024 )	217M	0.60	0.84	0.89	0.30	0.48	0.55	—	—	—
I3D ( Desai et al., 2023 )	25M	0.63	0.86	0.91	—	—	—	—	—	—
SignBERT+ ( Hu et al., 2023 )	—	—	—	—	—	—	—	48.85	46.37	—

Table 2: Isolated sign recognition and fingerspelling detection under the SHuBERT evaluation suite. ASL Citizen and Sem-Lex use Recall@1/5/10; WLASL2000 reports per-instance (P-I) and per-class (P-C) top-1 accuracy; ASL-STEM-Wiki reports mean interval-IoU. Sem-Lex numbers follow the full-test-set protocol of SHuBERT , and are therefore not directly comparable to the original Sem-Lex report, which uses a reduced test set. SignDino entries are mean

\pm

standard deviation over

3

random seeds; baselines are reported as published.

Streams kept at fusion head	BLEU
LH + Face (no RH)	9.8
RH + Face (no LH)	11.2
LH + RH (no Face)	14.6
LH + RH + Face (default)	17.9

Table 3: Per-stream leave-one-out ablation, evaluated on the How2Sign test split with SignDino live fine-tuned (same regime as the SignDino live row of Table 1 ). Each row drops one of the three default streams (LH, RH, Face) at the fusion head while the per-stream SSL stage is unchanged; the bottom row reproduces the SignDino live row of Table 1 .

Ablation	H2S BLEU	ASLC R@1
Full SignDino	17.9	0.704
no Gram anchor (Eq. 9 )	16.5	0.67
Teacher short window only	15.8	0.65
YOLO only (no ByteTrack)	16.4	0.61

Table 4: Ablations specific to SignDino , evaluated on the How2Sign test split (H2S BLEU column) and the ASL Citizen test split (ASLC R@1 column). Each row toggles one factor and reuses the same downstream head; the “Full SignDino ” row reproduces the SignDino live row of Table 1 (H2S BLEU) and the SignDino full fine-tune row of Table 2 (ASLC R@1).

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Default value
Backbone	frozen DINOv3 ViT-B/16
Streams	LH, RH, face (3 streams, independent SSL)
Temporal encoder	$L{=}6$ , $D{=}384$ , 6 heads, MLP ratio 4
Crops (contiguous)	$T_{g}\!\sim\!\mathcal{U}\{64{-}96\}$ , $T_{l}\!\sim\!\mathcal{U}\{10{-}32\}$ ; $N_{g}{=}2$ , $N_{l}{=}8$
Temperatures	$\tau_{T}:0.04\!\to\!0.07$ (30 ep), $\tau_{S}=0.1$
EMA momentum	cosine from $0.994$ to $1.0$

Appendix

Table 5: Default hyperparameter card. Values summarise the default configuration used for the main experiments and ablations.

Figure 5: Sample face clusters from SignDino features on held-out How2Sign and OpenASL frames. Four

k

-means clusters (

K{=}250

); each row shows 10 nearest-to-centroid members. Cluster 176 captures “hand-on-face mouthings” across multiple identities; cluster 104 captures hand-near-mouth gestures with varied skin tones; cluster 160 surfaces side-profile / looking-down poses; cluster 46 collects neutral closed-mouth frontals.

Figure 6: Sample left-hand clusters from SignDino features on held-out How2Sign and OpenASL frames. Five $k$ -means clusters ( $K{=}200$ ). Cluster 75 captures thumb-up / closed-fist handshapes; cluster 137 captures the spread-fingers (“5”) handshape across multiple backgrounds; clusters 6 and 111 capture mid-motion signing gestures; cluster 194 captures two-hand contact poses.

Figure 7: Sample right-hand clusters from SignDino features on held-out How2Sign and OpenASL frames. Five $k$ -means clusters ( $K{=}200$ ). Cluster 145 captures pointing / extended-index handshapes across multiple backgrounds; cluster 150 captures a related forward-pointing pose on dark backgrounds; cluster 96 captures the thumb-up / closed-fist family; cluster 185 captures open-palm / spread-fingers gestures; cluster 194 captures motion-blurred active gesturing.

Phonological feature	Method	Sem-Lex R@1	ASL Citizen R@1
Major Location	SHuBERT	84.77	90.22
	SignDino	94.43	94.96
Minor Location	SHuBERT	71.30	80.00
	SignDino	76.83	85.79
Second Minor Location	SHuBERT	73.28	81.18
	SignDino	82.34	90.73

Appendix

Table 6: Phonological feature recognition on Sem-Lex and ASL Citizen. The table follows the SHuBERT Appendix B layout; R@1 is top-1 accuracy for each feature-specific classifier. For every feature we report the SHuBERT baseline above and SignDino below; SignDino rows are bolded to mark the matched-protocol gain.

Figure 8: Cross-sentence t-SNE of SignDino features after SLT fine-tuning, on a 2400-sentence sample drawn from How2Sign and the three ISLR datasets. Each coloured point is one in-sentence occurrence of the labelled English target word, projected by t-SNE on the post-SLT SignDino stream-fused per-window features. Grey points are all other vocabulary occurrences in the same sample (background). (a) Common nouns: family terms (mother/father/brother/sister/child) concentrate in a central region; weather nouns (rain, snow) sit far from the family cluster. (b) Common verbs: visual-perception verbs (see/look/watch) cluster on the right; motion verbs (walk/run/jump) cluster on the left; “sleep” occupies a distinct region between the two verb groups.

Hyperparameter	Stage 1 (DINO+iBOT+DKoleo)	Stage 2 (Gram refine)
Steps	400K	50K
Batch (videos / GPU)	16	16
Optimiser	AdamW	AdamW
Base LR	$5\!\times\!10^{-4}$	$2\!\times\!10^{-4}$
LR schedule	5% warmup + cosine	1% warmup + cosine
Weight decay	$0.04\to 0.4$	$0.04\to 0.4$

Appendix

Table 7: SignDino pre-training hyperparameters for stage 1 and stage 2.

	Source stage	Target stage
Data	decontaminated YouTube-ASL	How2Sign / OpenASL train
Steps	250K	50K
Optimiser	AdamW ( $\beta_{1}{=}0.9,\beta_{2}{=}0.99$ )	AdamW
LR ByT5 + head	$5\!\times\!10^{-4}$	$1\!\times\!10^{-4}$
LR SignDINO (live)	$5\!\times\!10^{-5}$	$1\!\times\!10^{-5}$
Warmup	10K cosine	5K cosine

Appendix

Table 8: Translation training hyperparameters. The source stage follows SHuBERT §4.2; the target stage fine-tunes on the original How2Sign/OpenASL training split and evaluates on the corresponding validation/test splits.

SSL checkpoint	LH	RH	Face
init	1.000	1.000	1.000
Stage 1 100K	0.962	0.958	0.971
Stage 1 200K	0.945	0.941	0.957
Stage 1 300K	0.933	0.929	0.945
Stage 1 400K (end S1)	0.927	0.922	0.940
Stage 2 50K (end S2)	0.921	0.917	0.935

Appendix

Table 9: Normalised prototype entropy

H(p_{\mathrm{teacher}})/\log K

per stream across the two-stage SSL schedule, evaluated on a held-out batch of

1024

frames. A value near

1

indicates near-uniform prototype use (no collapse); values stay above 0.91 throughout, far from the regime in which the teacher would concentrate mass on a small subset of prototypes.

Task	Stream	$L{=}0$	$L{=}1$	$L{=}2$	$L{=}3$	$L{=}4$	$L{=}5$	$L{=}6$
SLT (H2S)	LH	0.06	0.09	0.12	0.16	0.18	0.21	0.18
	RH	0.05	0.08	0.11	0.15	0.19	0.22	0.20
	Face	0.07	0.11	0.14	0.17	0.19	0.18	0.14
ISLR (ASLC)	LH	0.04	0.07	0.10	0.14	0.18	0.24	0.23
	RH	0.04	0.07	0.10	0.14	0.18	0.24	0.23
	Face	0.09	0.12	0.15	0.17	0.17	0.16	0.14

Appendix

Table 10: Converged layer-weighted-sum softmax weights per task and per stream. Each row is a probability distribution over the seven encoder layers (

L{=}0

: post-stem feature,

L{=}1{..}6

: transformer blocks) and sums to

1.0

. Values are averaged over three random seeds of the target-stage / downstream training.

Task	Query	$\to$ LH	$\to$ RH	$\to$ Face
SLT (H2S)	LH	0.45	0.30	0.25
	RH	0.31	0.44	0.25
	Face	0.21	0.22	0.57
ISLR (ASLC)	LH	0.50	0.34	0.16
	RH	0.34	0.51	0.15
	Face	0.25	0.26	0.49

Appendix

Table 11: Cross-stream attention fractions at the most-active downstream fusion layer, per task. Rows are query streams, columns are key streams, and each row sums to

1.0

Stage	Per-stream GPU-h	Per-stream wall-clock	Total GPU-h
Stage 1 (400K steps)	240	${\approx}30$ h	720
Stage 2 (50K steps, Gram refine)	40	${\approx}5$ h	120
SSL subtotal (3 streams)	280	${\approx}35$ h	840
Embedding cache (one-off)	—	—	${\approx}300$
SHuBERT upstream (ref.)	—	—	1344

Appendix

Table 12: SignDino pre-training compute budget on

8\,\times\,

A100 80 GB. Streams are trained sequentially, so the wall-clock for the full three-stream upstream is approximately

3{\times}

the per-stream wall-clock plus negligible scheduling overhead. The embedding-cache row is the one-off preprocessing cost discussed in § 4.1 . The SHuBERT row reports the upstream-only figure of

1344

GPU-hours quoted by Gueuwou et al. (2025b) (

8\,\times\,

A6000

\times\,7

days).

Pre-train hours	BLEU-1	BLEU	BLEURT
98 (10%)	33.6	13.5	46.1
984 (100%)	37.9	16.8	50.4

Appendix

Table 13: Pre-training data scale ( SHuBERT Tab. 5 analogue), evaluated on the How2Sign test split with the SignDino encoders frozen and the same downstream head as Table 15 . Two-point scan on 10% vs 100% of the decontaminated YouTube-ASL source corpus; PT hours follow the unique source-corpus accounting used in Table 1 . Table 14 extends this to a five-point sweep.

Pre-train hours	H2S BLEU	H2S BLEURT
98 (10%)	13.5	46.1
246 (25%)	14.8	47.9
492 (50%)	16.0	49.4
738 (75%)	16.5	50.0
984 (100%)	16.8	50.4

Appendix

Table 14: SignDino SSL data-scaling curve. Frozen-features H2S BLEU and BLEURT as a function of the source-corpus pre-training budget. The

10\%

and

100\%

rows reproduce the corresponding rows of Table 13 . The curve is monotonic and not yet flat at

100\%

of the public corpus, consistent with the data-scaling regime reported by SHuBERT ( Gueuwou et al., 2025b ) ; doubling pre-training hours yields a roughly

1

BLEU improvement throughout the scanned range.

Strategy	BLEU-1	BLEU	BLEURT
Channel masking	35.2	14.4	47.6
Time masking	36.1	14.9	48.1
Random masking (default)	37.0	15.7	48.9
Frame mask only ( SignDino )	37.9	16.8	50.4
Frame + stream mask	37.3	16.3	49.7

Appendix

Table 15: Masking strategies ( SHuBERT Tab. 4 + SignDino analogues), with the SignDino encoders frozen at 100K steps and the same downstream head. We follow SHuBERT ’s BLEURT-driven default.

Layer of SignDino	BLEU-1	BLEU	BLEURT
None (raw DINOv3 )	24.3	9.4	41.2
Last layer	34.6	14.7	47.8
Weighted sum (default)	37.9	16.8	50.4

Appendix

Table 16: Layer pooling ( SHuBERT Tab. 6 analogue), with the SignDino encoders frozen and the same downstream head as Table 15 . The “None” row feeds the raw DINOv3 per-frame embeddings to ByT5 without the SignDino encoders.

Fine-tune SignDINO?	BLEU-1	BLEU	BLEURT
$\times$ frozen	37.9	16.8	50.4
✓ live fine-tune	39.4	17.9	51.3

Appendix

Table 17: Frozen vs. fine-tune ( SHuBERT Tab. 7 analogue). This isolates whether downstream supervision benefits from updating the temporal SignDino encoder beyond the frozen-feature setting; the two rows reproduce the corresponding “frozen” and “live fine-tune” rows of Table 1 .

$T_{l}\,/\,T_{g}$ sampler	H2S BLEU	$\Delta$
Fixed $16\,/\,64$ (low-FLOP control)	$17.3_{\pm 0.15}$	$-0.6$
Fixed $20\,/\,80$ (iso-FLOP, global-heavy)	$17.5_{\pm 0.15}$	$-0.4$
Fixed $24\,/\,72$ (iso-FLOP, local-heavy)	$17.4_{\pm 0.15}$	$-0.5$
Random $\mathcal{U}\{10{-}32\}\,/\,\mathcal{U}\{64{-}96\}$ (default)	$\mathbf{17.9_{\pm 0.15}}$	—

Appendix

Table 18: Window-size ablation with the SignDino live-fine-tune setup (same regime as the SignDino live row of Table 1 ). Each row re-runs SSL pre-training with a different crop-length sampler for the same 400K SSL steps. The two iso-FLOP rows match the random sampler’s expected per-iter frame count to within

\pm 2.5\%

but allocate compute differently, so the

-0.4

-0.5

BLEU gaps cannot be explained by per-iter compute. The last row reproduces the “Full SignDino ” / SignDino live entry of Tables 4 and 1 . Numbers are mean

\pm

std over

3

downstream seeds with the SSL backbone trained once and shared.

Frozen per-frame backbone	Params	H2S BLEU	BLEURT
DINOv2 ViT-S/14 ( Oquab et al., 2024 )	21M	16.5	49.4
DINOv2 ViT-L/14 ( Oquab et al., 2024 )	300M	17.4	50.5
DINOv3 ViT-S/16 ( Siméoni et al., 2025 )	22M	17.2	50.1
DINOv3 ViT-B/16 (default) ( Siméoni et al., 2025 )	86M	17.9	51.3
DINOv3 ViT-L/16 ( Siméoni et al., 2025 )	304M	18.0	51.4

Appendix

Table 19: Visual backbone ablation with SignDino live fine-tuned (same regime as the SignDino live row of Table 1 ). The backbone is always frozen; only the per-frame CLS token is propagated to the SignDino temporal encoder. SSL pre-training is re-run per row; downstream translation training (Table 8 ) is identical across rows.

Translation decoder	Params	SHuBERT BLEU	SignDino BLEU
ByT5-Small ( Xue et al., 2022 )	300M	15.0	16.4
ByT5-Base (default) ( Xue et al., 2022 )	582M	16.2	17.9
ByT5-Large ( Xue et al., 2022 )	1.23B	15.9	17.5
mT5-Base ( Xue et al., 2021 )	580M	15.5	17.1
mBART-large ( Liu et al., 2020 )	610M	15.1	16.8

Appendix

Table 20: Translation decoder ablation with both upstreams ( SHuBERT and SignDino ) at live fine-tune. All rows use the same source-stage + target-stage recipe (Table 8 ); only the decoder backbone and its

d_{\mathrm{model}}

projection change. The ByT5-Base default cells reproduce the corresponding SHuBERT ( 16.2 ) and SignDino live ( 17.9 ) rows of Table 1 .

#	Source	Sentence
(1)	Reference	And that’s a great vital point technique for women’s self defense.
	SHuBERT	This is a really great point for self defense. (verbatim from Gueuwou et al. (2025b) Tab. 11)
	SignDino	And that’s a great vital technique for women’s self defense.
(2)	Reference	In this clip I’m going to show you how to tape your cables down.
	SHuBERT	In this clip I’m going to show you how to brand out the cable strings.
	SignDino	In this clip I’m going to show you how to tape down your cables.

Appendix

Table 21: Compact qualitative translation comparison on How2Sign test. SHuBERT hypotheses are verbatim from Gueuwou et al. (2025b) Tables 11 and 13 (deduplicated and shortened).

Explore similar work

SignMAE: Segmentation-Driven Self-Supervised Learning for Sign Language Recognition
May 3, 2026Kunyuan Xie, Zhixi Cai, Kalin StefanovSelf-Supervised Pre-Training Sign Language Recognition
Monash University, Australia
SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval
Sep 3, 2026Sobhan Asasi, Ozge Mercanoglu Sincan, Richard BowdenContrastive Learning Representation Learning
CVSSP, University of Surrey
Attention-Steered Vision-Language Models for Sign Language Translation
Jul 31, 2026Meibo Hu, Guohao Sun, Annemarie D. Ross +2Sign Language Translation Vision-Language Models
Rochester Institute of Technology · National Technical Institute for the Deaf / Rochester Institute of Technology · University of Virginia

May 3, 2026J/K move · Enter open · S save

SignMAE: Segmentation-Driven Self-Supervised Learning for Sign Language Recognition

Kunyuan Xie, Zhixi Cai, Kalin Stefanov

Monash University, Australia

Self-Supervised Pre-Training Sign Language Recognition Self-Supervised Visual Representation Learning

Open paper AskPDF arXiv

Subtle hand differences make sign language recognition challenging, yet many existing methods rely on encoders pretrained on generic action datasets that poorly capture such fine-grained cues. We propose a self-supervised pretraining method for sign language recognition that uses segmentation-based masking to adapt to the presence and motion of key body parts, rather than treating hand poses as static visual tokens. The resulting mask-and-reconstruct objective improves fine-grained sign representation learning. On WLASL, NMFs-CSL, and Slovo, our encoder achieves state-of-the-art performance, improving per-instance and per-class Top-1 accuracy while using fewer input frames and modalities than comparable encoders.

Abstract

Explore similar work

SignMAE: Segmentation-Driven Self-Supervised Learning for Sign Language Recognition

SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval

Attention-Steered Vision-Language Models for Sign Language Translation