Recent Vision Language Models (VLMs) have demonstrated strong performance across a wide range of multimodal reasoning tasks. This raises the question of whether such general-purpose models can also address specialized visual recognition problems such as isolated sign language recognition (ISLR) without task-specific training. In this work, we investigate the capability of modern VLMs to perform ISLR in a zero-shot setting. We evaluate several open-source and proprietary VLMs on the WLASL300 benchmark. Our experiments show that, under prompt-only zero-shot inference, current open-source VLMs remain far behind classic supervised ISLR classifiers by a wide margin. However, follow-up experiments reveal that these models capture partial visual-semantic alignment between signs and text descriptions. Larger proprietary models achieve substantially higher accuracy, highlighting the importance of model scale and training data diversity. All our code is publicly available on GitHub.
Figures & tables
Figure 1: Evaluation paradigms for zero-shot ISLR with VLMs. Top: Standard closed-set classification with defined target labels. Middle: Zero-shot prediction instructing the model to behave as an ASL expert capable of categorizing the input videos into an open set. Bottom: Zero-shot binary classification instructing the model to think about the input video to conclude whether the performed sign is the one the user thinks of, with optional knowledge about the data.
Model
Input Format
Top-1
Top-1 + Syn.
Time / vid (s)
Total (h)
Open-source
LLaVA-NeXT-Video-7B [ 42 ]
25 fps
0.30
0.45
0.55
0.10
InternVL3.5-14B [ 16 ]
32 frames/clip
0.30
1.20
11.81
2.19
64 frames/clip
0.00
0.17
26.00
4.22
Qwen3-VL-30B-A3B-Instruct [ 32 ]
4 fps
1.35
1.80
0.76
0.14
25 fps
1.20
1.35
3.22
0.60
Qwen3-VL-32B-Instruct [ 32 ]
4 fps
0.45
1.35
0.59
0.11
Table 1: WLASL300 (668 test split videos): Top-1 accuracy % (exact match) and synonym-aware Top-1 accuracy, together with average per-video processing time and total runtime, measured in the OpenAI API for GPT-5, the Google AI API for Gemini, and on a single NVIDIA H100 GPU for the remaining models. Reminder: SOTA on WLASL300 with a specialized closed-set classification model is 89.97%.
True
False
Model
Input
Positive
Negative
Positive
Negative
F1
Qwen3-VL-30B-A3B-Instruct
Nonskeptical
425
116
380
73
0.65
Skeptical
166
429
69
332
0.45
No description
299
232
263
198
0.56
Qwen3-VL-32B-Instruct
Nonskeptical
347
335
163
151
0.69
Skeptical
197
430
68
300
0.52
Table 2: Comparison of instruction variants for binary sign detection , where the model judges whether an input video matches a provided gloss description. Nonskeptical : the model describes the video, compares it to the description, and decides. Skeptical : the model first argues why the video does not match, then why it does , and finally decides. No description : the model is given only the gloss name (no textual description).
Model
Knowledge
Top-1
Top-1 + Syn.
PID
Qwen3-VL-30B-A3B-Instruct
It’s ASL
1.20
1.35
56.59
It’s WLASL300
2.10
2.10
48.35
All glosses listed
2.25
2.54
88.92
Qwen3-VL-32B-Instruct
It’s ASL
0.45
1.20
16.47
It’s WLASL300
0.45
1.20
27.84
All glosses listed
1.50
1.80
87.13
Table 3: Comparison of dataset-knowledge prompting scenarios . It’s ASL : the prompt specifies only that the inputs are ASL videos. It’s WLASL300 : the prompt additionally states that the videos come from the WLASL300 dataset. All glosses listed : the prompt includes the full list of 300 WLASL300 candidate glosses. Results are reported for Qwen3-VL-30B-A3B-Instruct (4 fps), Qwen3-VL-32B-Instruct (4 fps) and Gemini-2.5-pro (8 frames/clip), including Top-1 accuracy, synonym-aware Top-1 accuracy, and PID (Predictions in Distribution), which denotes the percentage of predictions that fall within the WLASL300 label set (i.e., the predicted gloss is in-distribution).