SignTrace: Describe a Sign, Find the Word

Zengji Tu; Xingye Zhu; Ningjing Wang; Tingyi Huang; Yangjunfeng Zhu; Dai Wan

cs.CLSep 13, 2026

SignTrace: Describe a Sign, Find the Word

Authors: Zengji Tu, Xingye Zhu, Ningjing Wang, Tingyi Huang, Yangjunfeng Zhu, Dai Wan

Organizations: Peking University, Beijing 100871, China · Sign Language Department, Loving Heart Society, Peking University, Beijing 100871, China · Institute of Population Research, Peking University, Beijing 100871, China

Abstract

Identifying an unfamiliar sign is difficult when a learner remembers its movement but does not know its meaning or formal feature codes. SignTrace addresses this longstanding reverse-lookup problem through natural-language access to a Chinese sign-language dictionary. The system integrates LLM-based dictionary enrichment, action extraction, dictionary-style rewriting, seven-channel retrieval, and candidate reranking over 6,699 entries. It has been deployed for user trials and has received positive informal feedback. Evaluation on a dictionary-derived benchmark of 500 movement-description queries yields 94.0% Hit@1, 97.4% Hit@9, and a mean reciprocal rank of 0.9540. Reranking increases Hit@1 from 71.8% to 94.0%, while component analyses show the contribution of enriched entry descriptions. Median query-processing time is 13.37 seconds with six concurrent queries. By connecting everyday movement descriptions to documented signs and meanings, SignTrace provides a practical tool for identifying unfamiliar signs. Dictionary-derived wording and prior selection within the benchmark limit generalization to descriptions independently produced by users.

Figures & tables

Figure 1: The SignTrace architecture: dictionary enrichment, parallel query transformations, seven-channel candidate retrieval, and action-based reranking. The dashed path supplies the extracted observation directly to the reranker.

Figure 2: Complementary retrieval representations for example query 81. The input and selected fields are translated from the recorded Chinese query and extraction. Hand count, fingertip state, movement, location, and contact support complementary query representations; the unreported orientation remains unspecified.

Figure 3: Recorded target votes for query 490. Bars show contributions after channel cutoffs; an absent vote is zero. The target moves from recall rank 13 to final rank 1.

Property	Value
Query source	Dictionary action descriptions and annotations
Queries and target entries	500 / 500
Query identifiers	1–500
Description length, including punctuation	13–64 characters; median 35
Attempted / scored / excluded	500 / 500 / 0

Table 1: Table 1. Characteristics of the evaluation dataset.

ID	Movement description	Target meaning	Rank
1	Hold the left thumb upright. Stroke its back with the right hand, then hold the right hand across its front and lower both hands together.	Care for; protect	1
81	Gather the fingertips of one hand against the mouth, then draw the hand downward while spreading the fingers.	Experience; try	1
201	Raise the left thumb. Move the curved fingers of the right hand from the left shoulder onto the thumb. Then form a ring with the left thumb and index finger and flick the bent right index and middle fingers outward twice at its opening.	Supported care	1
343	Hold the left hand open and upright, palm outward. Bend the right index and middle fingers and tap their tips against the left palm.	Numeral; classifier	13
489	Point both hands downward and rotate the wrists in front of the body. Then curve the fingers slightly and cup one hand over each ear.	Audiometry	1
490	Hold the index, middle, and ring fingers apart and upright, pressing the bent little finger with the thumb. Then hold the index finger horizontally in front of the mouth and rotate it back and forth twice.	Predicate	1

Table 2: Table 2. Example evaluation queries and final target ranks, including a retrieval error and a successful fallback case. English descriptions and glosses translate the Chinese queries and reference meanings. Only movement descriptions are submitted as input.

Metric	Hits / total	Rate (%)	95% Wilson CI (%)
Hit@1	470/500	94.0	91.6–95.8
Hit@9	487/500	97.4	95.6–98.5
Pool@60	497/500	99.4	98.3–99.8

Table 3: Table 3. Final performance on the complete dictionary-derived benchmark. All proportions use a denominator of 500; no attempted query is excluded.

Figure 4: Retrieval hit rates on the evaluation dataset (n = 500). All curves use the same maximum depth of 60. Line styles and markers distinguish raw-query BM25, seven-channel recall, and the final ranking. Vertical guides mark the nine-result cutoff and the 20-candidate reranking window.

Method	Hit@1	Hit@9	Hit@20	Pool@60	MRR
Raw-query BM25	331	445	465	485	0.7446
Seven-channel recall	359	475	488	497	0.8002
SignTrace + Gemini	470	487	488	497	0.9540

Table 4: Table 4. Stage comparison on the same 500 inputs and enriched dictionary. All hit columns are counts. Only the SignTrace candidate lists received Gemini reranking.

Comparison	k	Gain, pp [95% CI]	Wins/losses	p	Holm p
Final vs. recall	1	22.2 [16.8, 27.2]	114/3	3.21e-30	1.29e-29
Final vs. recall	9	2.4 [0.3, 4.4]	12/0	4.88e-04	9.77e-04
Recall vs. BM25	20	4.6 [1.6, 7.4]	24/1	1.55e-06	4.65e-06
Recall vs. BM25	60	2.4 [0.3, 4.4]	12/0	4.88e-04	9.77e-04

Table 5: Table 5. Paired comparisons on the evaluation dataset (n = 500). Gains are percentage points. Difference intervals are conservative simultaneous-binomial intervals; exact p-values use discordant pairs. Holm correction applies to the four displayed comparisons.

Lexical document layers	Hit@1	Hit@9	Hit@20	Pool@60
1: Source description	249	369	403	434
1+2: Visual re-description	265	390	419	451
1+2+3: Structured summaries	333	446	465	485
1+2+3+4: Sense strings	331	445	465	485

Table 6: Table 6. Raw-query BM25 with cumulative document enrichment on the evaluation dataset (n = 500). Cells are first-stage hit counts. The query itself is unchanged.

First-stage configuration	Hit@1	Hit@9	Hit@20	Pool@60
All seven channels	359	475	488	497
Without T	353	469	481	494
Without Td	357	474	484	495
Without Tmv	359	472	488	498
Without Traw	360	474	485	498
Without Tq2d	324	459	479	494

Table 7: Table 7. Retrieval-channel and fusion ablations on the evaluation dataset (n = 500). Cells are hit counts. Removing Tq2d retains its meaning guesses in G. All configurations share the same model outputs; changed candidate lists are not reranked.

Figure 5: Component analyses on the evaluation dataset (n = 500). Left: first-result accuracy of unmodified-query BM25 as entry text is enriched. Right: changes in first-20 hit counts after removing individual channels, relative to 488 hits with all seven channels. The shared query representations are fixed, and no altered candidate list is reranked.

Timed scope	Median (s)	90th percentile (s)	95th percentile (s)
Parallel preparation and recall	2.69	3.30	3.56
Gemini reranking	10.68	14.32	16.39
Complete query processing	13.37	17.14	19.36

Table 8: Table 8. Processing durations for 500 evaluation queries, including the three parsing-fallback cases. Each row summarizes independently measured per-query intervals. Batch scheduling wait and browser/network overhead outside the experimental pipeline are excluded.

Figure 6: Empirical cumulative distributions of processing time for 500 evaluation queries. No observations are filtered or smoothed. Preparation includes parallel extraction and rewriting plus local recall; the full interval also includes reranking and local recording.

Component	Recorded setting
Execution date	2026-09-13; 500 queries evaluated
Dictionary	6,699 entries; 8,687 listed meanings
Text retrieval	BM25; Chinese characters and adjacent character pairs; $k_{1}=1.5$ , $b=0.75$
Fusion	T, Td, Tmv, Traw, Tq2d, S, G; $c=300$ , $m=3$ , $w=1.5$
Channel voting limits	Positive scores for T/Td/S; first 20 for Tmv/Traw/Tq2d; first 10 for G
Candidate pool and window	60 retrieved entries; first 20 reranked

Table 9: Table B1. Experimental configuration for the 500-query evaluation. Unspecified model sampling parameters retain the endpoint defaults. The requested and returned strings are recorded identifiers, not independent verification of underlying model weights.

Model role	Calls	Input tokens	Output tokens	Total tokens
DeepSeek: two query transformations	1000	638,068	225,223	863,291
Gemini: candidate reranking	500	2,242,676	723,172	2,965,848

Table 10: Table B2. API-reported token usage for all 1,500 calls. Output counts already include any reasoning tokens counted within completion usage. No monetary conversion is applied.

Stage	Hit@1	Hit@3	Hit@5	Hit@9	Hit@20	Pool@60
Recall	359 (71.8)	427 (85.4)	453 (90.6)	475 (95.0)	488 (97.6)	497 (99.4)
Final	470 (94.0)	482 (96.4)	484 (96.8)	487 (97.4)	488 (97.6)	497 (99.4)

Table 11: Table C1. Recall and final target-hit counts and percentages at all recorded cutoffs. Every cell uses the complete 500-query denominator.

ID	Target meaning	Recall rank	Final rank
14	Cauliflower	27	27
17	Dismantle	49	49
18	Exceed a production target	50	50
38	Regular; periodic	56	56
58	Bright; light	Absent	Absent
114	Shallow	33	33

Table 12: Table C2. All 13 queries whose final target rank exceeds nine. Absent means the target does not occur in the 60-entry candidate pool. Targets beyond the 20-entry window retain their recall rank.

Characters	n	Hit@1	95% Wilson CI (%)	Hit@9
13–24	45	40 (88.9)	76.5–95.2	45 (100.0)
25–40	307	286 (93.2)	89.8–95.5	297 (96.7)
41–64	148	144 (97.3)	93.3–98.9	145 (98.0)

Table 13: Table C3. Descriptive length strata partitioning all 500 evaluation queries. Length counts Unicode characters including punctuation. Hit columns give counts and percentages; intervals refer to Hit@1. These post hoc strata compare different queries.

Lexical Retrieval

Lexical retrieval ranks documents by query-term overlap using methods such as TF-IDF and BM25.

See all 27 papers

SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval
Sep 3, 2026Sobhan Asasi, Ozge Mercanoglu Sincan, Richard BowdenContrastive Learning Representation Learning
CVSSP, University of Surrey
SignMatch: Matching Dictionary Signs to Continuous Sign Language Video
Sep 1, 2026Ryan Wong, Youngjoon Jang, Liliane Momeni +2Sign Language Recognition
Visual Geometry Group Department of Engineering Science University of Oxford UK · LIGM, École des Ponts, IP Paris, Univ Gustave Eiffel, CNRS France
A Reverse Sign Language Dictionary: Open-Vocabulary Sign Recognition from Continuous Signing via Video Captioning and Description Retrieval
Sep 3, 2026Santiago Poveda-Gutiérrez, Hideki Nakayama, Mayumi BonoVideo Captioning Vision-Language Retrieval
Grad. School of IST The University of Tokyo Tokyo, Japan · Information and Society Research Division National Institute of Informatics Tokyo, Japan

Sep 3, 2026J/K move · Enter open · S save

SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval

Sobhan Asasi, Ozge Mercanoglu Sincan, Richard Bowden

CVSSP, University of Surrey

Contrastive Learning Representation Learning Sign Language Recognition

Open paper AskPDF arXiv

Sign language dictionaries are essential resources for sign language learners, yet automatically retrieving a sign from a dictionary, given only a query video, remains a challenging problem due to the natural variability between signers. Existing sign representation learning methods are built for closed-set recognition, producing embeddings that do not generalise to the open-set, signer-independent setting that retrieval demands. SignSeek closes this gap by contrastively learning sign representations with saliency-guided articulator masking. A contrastive objective aligns same-gloss signs across signers, while our Articulator Saliency-Guided Masking (ASGM) pinpoints the single most critical articulator per sign. This drives two complementary objectives, a masked contrastive alignment (MAC) loss that sees the sign through a single articulator and a masked prediction (MAP) loss that reconstructs it in latent space from the surrounding spatio-temporal context. Pretrained on 266K samples (

\sim

5,700 glosses) across multiple sign languages, SignSeek sets a new state-of-the-art performance in cross-corpus retrieval on ASL-Citizen, WLASL, and NMFs-CSL without any downstream fine-tuning. Strikingly, it achieves zero-shot generalisation to an entirely unseen British Sign Language (BSL), surpassing methods explicitly trained on BSL, and transfers seamlessly to isolated sign recognition and subtitle alignment, outperforming prior skeleton-based methods.

Abstract

Lexical Retrieval

Explore similar work

SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval

SignMatch: Matching Dictionary Signs to Continuous Sign Language Video

A Reverse Sign Language Dictionary: Open-Vocabulary Sign Recognition from Continuous Signing via Video Captioning and Description Retrieval