SafeGesture: Evaluating Fine-Grained Hand Gesture Understanding in Vision-Language Models through Scenario-Conditioned Safety Interpretation

Taegang Kim; Saleh Afroogh; Junfeng Jiao

cs.CVAug 17, 2026

SafeGesture: Evaluating Fine-Grained Hand Gesture Understanding in Vision-Language Models through Scenario-Conditioned Safety Interpretation

Authors: Taegang Kim, Saleh Afroogh, Junfeng Jiao

Organizations: Department of Computer Science, The University of Texas at Austin · Urban Information Lab, The University of Texas at Austin

Abstract

Open-weight and frontier vision-language models (VLMs) perform well on general image understanding, but their ability to interpret fine-grained hand gestures in safety-critical operational contexts remains largely unexamined. We introduce SafeGesture, a benchmark that evaluates whether a model can infer scenario-appropriate safety actions from hand gestures. It pairs six HaGRID gestures with eight operational scenarios for 4,800 items and evaluates Qwen2.5-VL-7B, LLaVA-NeXT-7B, InternVL2-8B, Phi-3.5-Vision, and GPT-4o. Results reveal a perception-reasoning decoupling: GPT-4o achieves 98.4% gesture accuracy but 53.3% safety accuracy, while Qwen2.5-VL reaches 84.9% and 39.5%, yielding gaps of 45.0 and 45.4 percentage points. Four of five models rarely or never use the uncertainty label, and failure directions differ substantially across models. Accuracy also obscures label bias: a scenario-majority policy with no visual input reaches 58.3%, above every evaluated model, while only GPT-4o exceeds this prior under macro-F1. Visual input improves safety accuracy by 11.2 to 30.2 percentage points, but providing the ground-truth gesture as text improves performance by only 0.4 to 3.2 points, and no model exceeds 56.2%. These results indicate that the main bottleneck is scenario-conditioned safety reasoning rather than gesture recognition.

Figures & tables

Scenario ID	Description
construction_crane	A construction site where a crane is moving a suspended load. The operator can see this person.
factory_robot	A factory floor, within the working radius of a robot arm running an automatic cycle.
traffic_control	On a lane of a road with vehicles in motion.
pedestrian_crosswalk	A marked crosswalk. One vehicle is stopped at the stop line and another person is on the far side.
medical_ward	Beside an occupied bed in a hospital ward. An IV pump and monitor are running, and no one else is in the room.
elderly_care_home	A room in a care facility. A call button and a walker are within reach, and no one else is in the room.

Table 1: The eight operational scenarios.

System	GA $\uparrow$	SA $\uparrow$	macro-F1 $\uparrow$	DS $\downarrow$	Under-Safety $\downarrow$	Over-Safety $\downarrow$
GPT-4o	98.4 [97.8, 98.9]	53.3 [40.3, 66.1]	52.6	45.0 [32.4, 58.1]	20.4 [10.7, 31.3]	2.9 [0.1, 6.6]
Qwen2.5-VL-7B	84.9 [78.9, 90.6]	39.5 [27.7, 51.6]	28.6	45.4 [31.9, 58.9]	1.9 [0.0, 5.4]	33.1 [21.3, 45.3]
InternVL2-8B	64.2 [53.9, 73.5]	46.8 [35.3, 58.2]	32.4	17.5 [5.0, 29.4]	0.6 [0.1, 1.1]	18.3 [10.6, 26.6]
Phi-3.5-Vision	50.5 [40.5, 60.3]	32.4 [22.9, 42.2]	26.7	18.1 [6.3, 30.4]	0.1 [0.0, 0.2]	15.4 [7.7, 24.1]
LLaVA-NeXT-7B	45.8 [34.5, 57.3]	32.9 [21.8, 44.5]	23.3	12.9 [2.3, 24.1]	5.6 [0.6, 11.9]	4.6 [0.3, 10.8]
Baseline: gesture-majority	—	62.5	51.0	—	—	—

Table 2: Main benchmark results with label-prior baselines. Brackets give 95% confidence intervals from a bootstrap over the 48 gesture-scenario combinations. The Decoupling Score is computed from unrounded values and can differ from the rounded difference of the two preceding columns by up to 0.1 pp. Baselines use no visual input, so gesture accuracy and the Decoupling Score do not apply to them.

	CRITICAL_STOP	WARNING_ATTENTION	HELP_DISTRESS	SAFE_NEUTRAL	AMBIGUOUS_VERIFY
Ground truth	500	1,700	200	1,200	1,200
GPT-4o	672	634	207	2,011	1,276
Qwen2.5-VL	1,362	2,593	0	829	16
InternVL2	881	2,179	0	1,740	0
LLaVA-NeXT	2,970	372	0	1,458	0
Phi-3.5	1,803	1,345	218	1,434	0

Table 3: Predicted label distribution against the ground-truth distribution (4,800 items).

Table 5

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Model	HuggingFace ID / API snapshot	Commit
Qwen2.5-VL-7B-Instruct	Qwen/Qwen2.5-VL-7B-Instruct	cc59489
LLaVA-NeXT-7B (Mistral)	llava-hf/llava-v1.6-mistral-7b-hf	2424fdd
InternVL2-8B	OpenGVLab/InternVL2-8B	6fb9ad6
Phi-3.5-Vision-Instruct	microsoft/Phi-3.5-vision-instruct	12b77fb
GPT-4o	gpt-4o-2024-08-06	—

Appendix

Table 6: Exact model versions used in the evaluation.

Scenario	CRIT	WARN	HELP	SAFE	AMBIG	Total
construction_crane	100	400	0	0	100	600
factory_robot	200	200	0	100	100	600
traffic_control	100	400	0	0	100	600
pedestrian_crosswalk	0	200	0	200	200	600
medical_ward	100	200	100	100	100	600
elderly_care_home	0	300	100	100	100	600

Appendix

Table 7: Ground-truth safety label counts by scenario.

Gesture	CRIT	WARN	HELP	SAFE	AMBIG	Total	Distinct
stop	400	200	0	100	100	800	4
fist	100	400	0	100	200	800	4
point	0	600	0	100	100	800	3
call	0	300	200	200	100	800	4
like	0	200	0	600	0	800	2
mute	0	0	0	100	700	800	2

Appendix

Table 8: Ground-truth safety label counts by gesture. The last column gives the number of distinct labels the gesture takes across the eight scenarios.

Gesture	crane	factory	traffic	crosswalk	medical	elderly	home	no_ctx
stop	CRIT	CRIT	CRIT	WARN	CRIT	WARN	SAFE	AMBIG
fist	WARN	CRIT	WARN	AMBIG	WARN	WARN	SAFE	AMBIG
point	WARN	WARN	WARN	WARN	WARN	WARN	SAFE	AMBIG
call	WARN	WARN	WARN	SAFE	HELP	HELP	SAFE	AMBIG
like	WARN	SAFE	WARN	SAFE	SAFE	SAFE	SAFE	SAFE
mute	AMBIG	AMBIG	AMBIG	AMBIG	AMBIG	AMBIG	SAFE	AMBIG

Appendix

Table 9: The complete 48-cell ground-truth assignment.

Model	crane	factory	traffic	crosswalk	medical	elderly	home	no_ctx
GPT-4o	49.0	75.5	47.7	55.5	48.0	17.2	99.7	34.0
Qwen2.5-VL	39.7	52.2	42.7	14.2	50.8	65.8	34.3	16.7
InternVL2	64.5	64.8	25.5	40.2	45.8	57.2	59.3	16.7
LLaVA-NeXT	32.5	60.2	21.8	28.0	44.5	18.8	41.5	15.8
Phi-3.5	35.2	32.3	39.2	6.3	32.8	44.5	54.7	14.2
Mean	44.2	57.0	35.4	28.8	44.4	40.7	57.9	19.5

Appendix

Table 10: Safety accuracy (%) by scenario and model.

Model	CORRECT	DECOUPLING	UNDER_SAFETY	OVER_SAFETY	LUCKY_GUESS	MISPERCEPTION
GPT-4o	53.0	22.1	20.4	2.9	0.4	1.3
Qwen2.5-VL	34.8	15.2	1.9	33.1	4.8	10.3
InternVL2	36.5	8.9	0.6	18.3	10.2	25.5
Phi-3.5	19.4	15.6	0.1	15.4	13.0	36.5
LLaVA-NeXT	28.6	7.0	5.6	4.6	4.3	49.9

Appendix

Table 11: Failure type distribution by model (%). Rows may not sum to exactly 100, and the identities of Section 3.7 of the main paper may not reproduce its Table 2 exactly, because of rounding at one decimal place.

Gesture	GPT-4o	Qwen2.5-VL	InternVL2	Phi-3.5	LLaVA-NeXT
stop	.999 / .507 / .492	.974 / .510 / .464	.610 / .461 / .149	.562 / .449 / .113	.999 / .510 / .489
fist	.998 / .351 / .647	1.000 / .381 / .619	.919 / .432 / .487	.575 / .305 / .270	.079 / .161 / $-$ .082
point	.962 / .691 / .271	.451 / .490 / $-$ .039	.832 / .604 / .228	.928 / .229 / .699	.458 / .349 / .109
call	.954 / .295 / .659	.704 / .265 / .439	.001 / .419 / $-$ .418	.109 / .279 / $-$ .170	.262 / .240 / .022
like	.989 / .745 / .244	.990 / .716 / .274	.900 / .872 / .028	.851 / .576 / .275	.948 / .712 / .236
mute	1.000 / .609 / .391	.978 / .010 / .968	.590 / .016 / .574	.006 / .106 / $-$ .100	.000 / .001 / $-$ .001

Appendix

Table 12: Per-gesture gesture accuracy, safety accuracy, and decoupling gap. A negative decoupling gap points to lucky guessing. InternVL2 on call recognizes the gesture in 0.1% of items while getting 41.9% of the safety labels right, and all of those correct answers come from guessing on scenario priors.

GPT-4o
GT $\backslash$ Pred	CRIT	WARN	HELP	SAFE	AMBIG
CRITICAL_STOP	391	99	0	0	10
WARNING_ATTENTION	277	402	107	250	664
HELP_DISTRESS	0	0	98	102	0
SAFE_NEUTRAL	1	1	0	1,132	66
AMBIGUOUS_VERIFY	3	132	2	527	536

Appendix

Table 13: Safety label confusion matrices for the five evaluated models.

System	CRIT	WARN	HELP	SAFE	AMBIG	macro-F1
GPT-4o	66.7	34.4	48.2	70.5	43.3	52.6
InternVL2	49.5	52.4	0.0	60.3	0.0	32.4
Qwen2.5-VL	43.9	43.0	0.0	55.4	0.8	28.6
Phi-3.5	32.4	39.2	20.6	41.2	0.0	26.7
LLaVA-NeXT	28.6	29.8	0.0	58.2	0.0	23.3
Baseline: gesture-majority	61.5	63.4	0.0	60.0	70.0	51.0

Appendix

Table 14: Per-class F1 (%).

System	CRIT	WARN	HELP	SAFE	AMBIG
GPT-4o	58.2	63.4	47.3	56.3	42.0
InternVL2	38.8	46.6	0.0	50.9	0.0
Qwen2.5-VL	30.0	35.6	0.0	67.8	31.3
Phi-3.5	20.7	44.4	19.7	37.8	0.0
LLaVA-NeXT	16.7	83.1	0.0	53.0	0.0

Appendix

Table 15: Per-class precision (%).

System	CRIT	WARN	HELP	SAFE	AMBIG
GPT-4o	78.2	23.6	49.0	94.3	44.7
InternVL2	68.4	59.8	0.0	73.8	0.0
Qwen2.5-VL	81.8	54.2	0.0	46.8	0.4
Phi-3.5	74.6	35.1	21.5	45.2	0.0
LLaVA-NeXT	99.4	18.2	0.0	64.4	0.0

Appendix

Table 16: Per-class recall (%).

	GPT-4o	InternVL2	Qwen2.5-VL	LLaVA-NeXT	Phi-3.5
Balanced acc.	58.0	40.4	36.7	36.4	35.3

Appendix

Table 17: Balanced accuracy (%), the mean of the five recalls. For reference, the baselines score 53.0 (gesture-majority), 38.3 (scenario-majority), 44.0 (alternate tie-break), and 20.0 (always-WARNING).

Model	GA	SA	DS	Under-Safety	Over-Safety
GPT-4o	98.4 [97.8, 98.9]	53.3 [40.3, 66.1]	45.0 [32.4, 58.1]	20.4 [10.7, 31.3]	2.9 [0.1, 6.6]
Qwen2.5-VL	84.9 [78.9, 90.6]	39.5 [27.7, 51.6]	45.4 [31.9, 58.9]	1.9 [0.0, 5.4]	33.1 [21.3, 45.3]
InternVL2	64.2 [53.9, 73.5]	46.8 [35.3, 58.2]	17.5 [5.0, 29.4]	0.6 [0.1, 1.1]	18.3 [10.6, 26.6]
Phi-3.5	50.5 [40.5, 60.3]	32.4 [22.9, 42.2]	18.1 [6.3, 30.4]	0.1 [0.0, 0.2]	15.4 [7.7, 24.1]
LLaVA-NeXT	45.8 [34.5, 57.3]	32.9 [21.8, 44.5]	12.9 [2.3, 24.1]	5.6 [0.6, 11.9]	4.6 [0.3, 10.8]

Appendix

Table 18: Point estimates with 95% confidence intervals.

Pair	Diff. (pp)	95% CI	Above zero
GPT-4o $-$ InternVL2	27.6	[10.8, 45.4]	99.96%
GPT-4o $-$ LLaVA-NeXT	32.2	[19.4, 45.2]	100.00%
GPT-4o $-$ Phi-3.5	26.9	[8.8, 44.8]	99.80%
Qwen2.5-VL $-$ InternVL2	27.9	[15.5, 40.5]	100.00%
Qwen2.5-VL $-$ LLaVA-NeXT	32.5	[16.5, 47.8]	99.99%
Qwen2.5-VL $-$ Phi-3.5	27.2	[8.6, 45.3]	99.80%

Appendix

Table 19: Paired Decoupling Score differences. The last column is the fraction of resamples in which the difference exceeded zero. It is not a

p

-value.

Model	Mean	Median	Min
GPT-4o	90.4	99.0	50.0
LLaVA-NeXT	86.5	95.0	42.0
Qwen2.5-VL	86.4	96.0	50.0
InternVL2	77.4	78.0	46.0
Phi-3.5	73.7	76.0	43.0

Appendix

Table 20: Within-combination modal response share (%). Consistency is not reliability: LLaVA-NeXT ranks second here mainly because it assigns CRITICAL_STOP to 61.9% of all items.

Scenario	Policy used	Correct	Alternate
construction_crane	WARN	4 / 6	WARN
factory_robot	WARN	2 / 6	CRIT
traffic_control	WARN	4 / 6	WARN
pedestrian_crosswalk	WARN	2 / 6	WARN
medical_ward	WARN	2 / 6	WARN
elderly_care_home	WARN	3 / 6	WARN

Appendix

Table 21: The scenario-majority policy.

Gesture	Majority label	Correct
stop	CRITICAL_STOP	4 / 8
fist	WARNING_ATTENTION	4 / 8
point	WARNING_ATTENTION	6 / 8
call	WARNING_ATTENTION	3 / 8
like	SAFE_NEUTRAL	6 / 8
mute	AMBIGUOUS_VERIFY	7 / 8

Appendix

Table 22: The gesture-majority policy.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 986 papers

Visual Grounding Safety in Vision-Language Models
Oct 5, 2026Erfan Shayegani, Kundan Krishna, Yue Dong +3VLM Evaluation VLM Safety Alignment
University of California, Riverside · Work done during internship at Apple. · Apple
From Local Evidence to Safety Verdicts: Causal Tracing in Vision-Language Models
Oct 5, 2026Faezeh Dehghan Tarzjani, Mevan Wijewardena, Alexander Romanus +1Cultural Safety Evaluation of VLMs VLM Safety
Department of Electrical and Computer Engineering University of Southern California, Los Angeles, CA, USA · Department of Computer Science University of Southern California, Los Angeles, CA, USA
EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards
Jun 30, 2026Siddhant Panpatil, Arth Singh, Mijin Koo +3VLM Robustness Egocentric Video Understanding
AIM Intelligence · Seoul National University

Oct 5, 2026J/K move · Enter open · S save

Visual Grounding Safety in Vision-Language Models

Erfan Shayegani, Kundan Krishna, Yue Dong, Nael Abu-Ghazaleh+2

University of California, Riverside · Work done during internship at Apple. · Apple

VLM Evaluation VLM Safety Alignment Vision-Language Grounding Visual Grounding VLM Safety

Open paper AskPDF arXiv

Vision-language models (VLMs) are increasingly trained to generate structured outputs like points and bounding boxes that downstream interfaces, agents, and robots can act on, yet safety alignment of this output channel has not been systematically analyzed. We study visual grounding safety by repurposing three safety benchmarks spanning direct harm (VLSU), social bias (BBQ-V), and situational safety (Asimov-2.0) into 15,401 matched pairs of harmful requests that differ only in the requested output: a free-text answer (VQA) or a grounding (point or bounding box). Across five VLMs, models that refuse a harmful request posed as a question often comply when the same request asks for a grounding: averaged over models, grounding refusal trails VQA refusal by 31-59 percentage points, depending on the domain, and safety system prompts do not close this gap. We propose a fine-tuning approach that combines grounding-form refusals with capability grounding data and self-distilled benign data to counter over-refusal. For Qwen3-VL-8B and VisionReasoner-7B, it improves grounding refusal by 77-95 percentage points on VLSU and BBQ-V and by 64-85 points on the held-out Asimov-2.0 domain, while also improving VQA refusal, preserving grounding capability, and keeping over-refusal limited. Representation analysis shows that fine-tuning moves harmful requests toward each model's refusal direction, most strongly for grounding, while leaving benign requests near the harmless reference.

Abstract

VLM Evaluation

Explore similar work

Visual Grounding Safety in Vision-Language Models

From Local Evidence to Safety Verdicts: Causal Tracing in Vision-Language Models

EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards