HandVQA: Diagnosing and Improving Fine-Grained Spatial Reasoning about Hands in Vision-Language Models

MD Khalequzzaman Chowdhury Sayem; Mubarrat Tajoar Chowdhury; Yihalem Yimolal Tiruneh; Muneeb A. Khan; Muhammad Salman Ali; Binod Bhattarai; Seungryul Baek

Figure 1: Overview of HandVQA’s transfer effect. Fine-tuning a base Vision-Language Model (VLM) on HandVQA teaches it explicit 3D hand geometry and joint-level spatial reasoning. The resulting Spatial-Aware VLM exhibits zero-shot generalization to novel downstream tasks: both image-based gesture recognition and video-based hand-object interaction recognition. Spatial-Aware VLM achieves consistent accuracy gains without task-specific training.

Figure 2: Overview of HandVQA Question Format. This figure illustrates the structure of our benchmark, which divides hand pose estimation into five sub-tasks: Angle, Distance, and Relative Position along X, Y, and Z axes. A hand image with annotated joint indices (top left) supports multiple-choice questions per task type, derived from 3D joint coordinates and the correct answers are shown in green .

Figure 3: Overview of the HandVQA pipeline. The pipeline converts normalized 3D hand joints into interpretable VQA pairs through three deterministic stages: ( 1 )

\mathcal{F}_{\text{pose}}

computes continuous pose descriptors: angles (

\theta

), distances (

d

), and relative positions (

\Delta_{x},\Delta_{y},\Delta_{z}

) and categorizes them into discrete pose descriptors (

\Gamma

); ( 2 )

\mathcal{F}_{\text{text}}

fills deterministic sentence templates using

\Gamma

and filters correct and incorrect options to form candidate answer options

(\mathcal{O})

and the correct label

y^{\star}

; ( 3 )

\mathcal{F}_{\text{mcq}}

constructs multiple-choice questions (MCQ) by pairing each image with

\mathcal{O}

and its correct label

y^{\star}

.

Table 1: Pose descriptor categorization with conditions and illustration.

Model	Tuned	Eval	Angle		Distance
Model	Tuned	Eval	Accuracy $\uparrow$	MAE $\downarrow$	Accuracy $\uparrow$	MAE $\downarrow$
Base model (no tuning)
DeepSeek Janus Pro 7B	–	InterHand2.6M	34.10	0.883	45.55	0.657
DeepSeek Janus Pro 7B	–	FreiHAND	35.31	0.830	44.15	0.668
DeepSeek Janus Pro 7B	–	FPHA	26.46	0.991	39.02	0.819
Finetuned Models

Table 2: Angle and Distance Results for all three models. The best, second-best, and third-best models for each dataset in each metric are highlighted as Gold , Silver , and Bronze , respectively.

Model	Tuned	Eval	Rel. Pos. X	Rel. Pos. Y	Rel. Pos. Z
Model	Tuned	Eval	Accuracy $\uparrow$	Accuracy $\uparrow$	Accuracy $\uparrow$
Base model (no tuning)
DeepSeek Janus Pro 7B	–	InterHand2.6M	50.41	52.46	51.16
DeepSeek Janus Pro 7B	–	FreiHAND	49.80	51.55	50.03
DeepSeek Janus Pro 7B	–	FPHA	43.02	52.64	61.73
Finetuned Models

Table 3: Relative Position Results for all three models. The best, second-best, and third-best models for each dataset are highlighted as Gold , Silver and Bronze , respectively.

Setup	Interaction Recognition	Gesture Recognition
LLaVA Mistral 7B [ 36 ]	-	57.42
Qwen 2.5 VL 7B [ 3 ]	80.26	71.86
LLaVA Mistral 7B finetuned	-	69.58
Qwen 2.5 VL 7B finetuned	82.89	82.19

Table 4: Zero-shot generalization results. HandVQA fine-tuning strengthens 3D spatial reasoning, yielding higher accuracy (%) on gesture and hand–object interaction recognition.

Angle Pose Descriptors	Distance Pose Descriptors	Relative Position Pose Descriptors (XYZ)
Thumb-MCP	Thumb-MCP vs. Index-PIP	Thumb-MCP vs. Index-PIP
Index-PIP	Index-PIP vs. Middle-PIP	Index-PIP vs. Middle-PIP
Middle-PIP	Middle-PIP vs. Ring-PIP	Middle-PIP vs. Ring-PIP
Ring-PIP	Ring-PIP vs. Little-PIP	Ring-PIP vs. Little-PIP
Little-PIP	Thumb-Tip vs. Index-Tip	Thumb-Tip vs. Index-Tip
Thumb-IP	Index-Tip vs. Middle-Tip	Index-Tip vs. Middle-Tip

Table 5: List of joints/joint-pairs on which angles, distances, and relative positions in X,Y, and Z axes are calculated. A total of 107 different joints/joint-pairs across all pose descriptors are considered.

Figure 4: The map of the hand skeleton used in our HandVQA benchmark generation pipeline.

Figure 5: Possible location of the ‘aligned’ Little Finger Proximal Interphalangeal (PIP) joint and Ring Finger Proximal Interphalangeal (PIP) joint underneath the index and middle finger. The relationship along the x-axis for the two PIP joints is ambiguous, making it necessary to drop the relative position X information of the two joints.

Figure 6: Word cloud representation of the most frequently used terms in the caption options extracted from our dataset. Prominent anatomical terms like “tip joint”, “interphalangeal joint”, and “metacarpophalangeal joint” highlight the fine-grained spatial and anatomical focus of the hand-centric question-answer pairs.

Figure 7: Breakdown of question types across the training (left) and evaluation (right) splits for each dataset in the HandVQA benchmark. Each dataset contains a balanced distribution of all five spatial reasoning tasks: angle, distance, and relative positions along the X, Y, and Z axes. This uniformity supports fair evaluation across all pose subtasks.

Pose Descriptors	Label frequency (Count)
Angle	bent slightly inward (130,993), bent inward (107,143), straight (74,007), bent completely inward (21,622)
Distance	spread wide from (222,429), spread from (109,353), close to (1,983)
Rel. Pos. (X)	at the left of (198,988), at the right of (133,109)
Rel. Pos. (Y)	above (215,725), below (112,984)
Rel. Pos. (Z)	in front of (177,216), behind (152,481)

Table 6: Label frequency grouped by pose descriptors.

Figure 8: Angle confusion matrix across three VLMs. All models frequently predict “bent slightly inward” regardless of ground truth, revealing a strong prediction bias. Each model also follows a consistent preference ordering in its outputs, indicating difficulty in distinguishing fine-grained joint angles.

Figure 9: Angle confusion matrix across three fine-tuned VLMs. Fine-tuning significantly reduces the dominant bias toward “bent slightly inward” observed in the base models, resulting in stronger alignment along the diagonal. The models exhibit improved discrimination across angle categories, particularly for “bent inward” and “straight” , although some residual confusion remains between adjacent angle classes, indicating that fine-grained distinctions are still challenging.

Figure 10: Distance confusion matrix comparison between base and fine-tuned VLMs. The base models exhibit a strong bias toward predicting “close to” across multiple ground-truth categories, leading to a skewed prediction distribution. In contrast, fine-tuning produces a more balanced distribution with increased alignment along the diagonal, indicating improved discrimination between distance categories. However, residual confusion persists between “spread from” and “spread wide from” , suggesting continued difficulty in distinguishing fine-grained spatial separations.

Figure 11: Relative Position X confusion matrix comparison between base and fine-tuned VLMs. The base models exhibit near-random predictions between “at the left of” and “at the right of” , indicating a lack of reliable directional understanding. In contrast, fine-tuning leads to a strong concentration along the diagonal, demonstrating significantly improved left–right discrimination. This suggests that relative positional reasoning along the horizontal axis is effectively learned after fine-tuning, with minimal residual confusion.

Figure 12: Relative Position Y confusion matrix comparison between base and fine-tuned VLMs. The base models exhibit inconsistent predictions between “below” and “above” , reflecting weak vertical positional understanding. In contrast, fine-tuning results in a strong alignment along the diagonal, indicating substantially improved discrimination of vertical relationships. While minor confusion remains, the fine-tuned models demonstrate a clear and consistent understanding of relative position along the vertical axis.

Figure 13: Relative Position Z confusion matrix comparison between base and fine-tuned VLMs. The base models exhibit strong bias toward predicting “in front of” , leading to highly skewed outputs and poor discrimination of depth relationships. In contrast, fine-tuning significantly improves alignment with the diagonal, indicating enhanced understanding of front–back spatial relations. While minor confusion persists, the fine-tuned models demonstrate substantially improved depth-aware reasoning compared to the base models.

Model	Tuned	Eval	Angle		Distance
Model	Tuned	Eval	Accuracy $\uparrow$	MAE $\downarrow$	Accuracy $\uparrow$	MAE $\downarrow$
Finetuned Models on Individual Datasets
LLaVA Mistral 7B	InterHand2.6M	InterHand2.6M	74.35	0.263	90.79	0.094
LLaVA Mistral 7B	FreiHAND	FreiHAND	62.91	0.382	86.19	0.141
LLaVA Mistral 7B	FPHA	FPHA	68.37	0.401	83.99	0.161
Finetuned Model on HandVQA

Table 7: Angle and Distance results of LLaVA Mistral 7B finetuned models on Individual Datasets vs. finetuned on the Unified HandVQA Benchmark.

Model	Tuned	Eval	Rel. Pos. X	Rel. Pos. Y	Rel. Pos. Z
Model	Tuned	Eval	Accuracy $\uparrow$	Accuracy $\uparrow$	Accuracy $\uparrow$
Finetuned Models on Individual Datasets
LLaVA Mistral 7B	InterHand2.6M	InterHand2.6M	97.14	98.77	96.82
LLaVA Mistral 7B	FreiHAND	FreiHAND	92.60	93.20	88.17
LLaVA Mistral 7B	FPHA	FPHA	93.81	92.80	90.25
Finetuned Model on HandVQA

Table 8: Relative Position accuracies of LLaVA Mistral 7B finetuned models on Individual Datasets vs. finetuned on the Unified HandVQA Benchmark.

Model	Tuned	Eval	Angle		Distance
Model	Tuned	Eval	Accuracy $\uparrow$	MAE $\downarrow$	Accuracy $\uparrow$	MAE $\downarrow$
Base model (no tuning)
DeepSeek Janus Pro 7B	–	InterHand2.6M	34.10	0.883	45.55	0.657
DeepSeek Janus Pro 7B	–	FPHA	26.46	0.991	39.02	0.819
Finetuned Models
DeepSeek Janus Pro 7B	FreiHAND	InterHand2.6M	56.15	0.456	82.70	0.175

Table 9: Angle and Distance results of Base Models vs Out-of-Distribution Finetuned Models.

Model	Tuned	Eval	Rel. Pos. X	Rel. Pos. Y	Rel. Pos. Z
Model	Tuned	Eval	Accuracy $\uparrow$	Accuracy $\uparrow$	Accuracy $\uparrow$
Base model (no tuning)
DeepSeek Janus Pro 7B	–	InterHand2.6M	50.41	52.46	51.16
DeepSeek Janus Pro 7B	–	FPHA	43.02	52.64	61.73
Finetuned Models
DeepSeek Janus Pro 7B	FreiHAND	InterHand2.6M	77.32	89.80	75.33

Table 10: Relative-Position Results of Base Models vs Out-of-Distribution Finetuned Models.

Figure 24

Figure 25

Model	FreiHAND test		InterHand2.6M test
Model	Angle	Distance	Angle	Distance
	Accuracy % (MAE)	Accuracy % (MAE)	Accuracy % (MAE)	Accuracy % (MAE)
HaMeR	59.53 (0.428)	88.86 (0.113)	78.82 (0.218)	88.11 (0.119)
LLaVA Mistral 7B	42.48 (0.678)	13.18 (1.342)	40.08 (0.739)	16.20 (1.293)
LLaVA Mistral 7B finetuned	64.35 (0.367)	86.71 (0.136)	72.26 (0.283)	90.27 (0.099)

Table 11: Pure vision model (HaMeR [ 49 ] ) vs. LLaVA Mistral 7B.

Difficulty	Angle	Distance	R. Pos.(X)	R. Pos.(Y)	R. Pos.(Z)
	Accuracy % (MAE)	Accuracy % (MAE)	Accuracy %	Accuracy %	Accuracy %
Easy	75.92 (0.255)	95.01 (0.08)	97.20	96.19	98.11
Hard	55.1 (0.454)	78.01 (0.183)	82.11	81.72	78.92

Table 12: LLaVA Mistral 7B finetuned on FreiHAND: Easy (225 QA), Hard (223 QA).

Figure 18: Easy vs Hard Examples Performance of LLaVA Mistral 7B finetuned . Correct( ✓ )/wrong( X ) QA samples for “Which is the distance between thumb tip and index fingertip?”.

Models	Overall Accuracy
LLaVA Mistral 7B	41.96%
DeepSeek Janus Pro 7B	45.82%
Qwen 2.5 VL 7B	41.97%
Humans	80.94%

Table 13: Human vs VLMs Accuracy on a small subset of HandVQA.

Base Label	Description 1	Description 2
one	A single index finger is extended, representing the number one.	The hand is held up with only the index finger pointing upwards.
two_up	The index and middle fingers are extended upwards (peace sign), representing two.	A “V” sign, often used for “two” or “peace”.
two_up_inverted	An inverted “V” sign, with the palm facing inward.	The gesture for “two” or “peace” but inverted.
three	Three fingers are extended to represent the number three.	The hand gesture indicates a quantity of three.
four	Four fingers are extended to show the number four.	The hand gesture indicates a quantity of four.
fist	The hand is closed tightly into a fist.	All fingers are curled inward with the thumb wrapped around them.

Table 14: Gemini-generated descriptions for HaGRID gesture labels.

Figure 19: Qualitative Results on Zero-shot Gesture Recognition on HaGRID dataset [ 29 ] .

Figure 20: Qualitative Results on Zero-shot Hand-Object Interaction Recognition on H2O dataset [ 31 ] .

Figure 21: Qualitative Comparison on FreiHAND [ 70 ] . Examples comparing LLaVA (base) and LLaVA fine-tuned on FreiHAND.

Figure 22: Qualitative Comparison on InterHand2.6M [ 46 ] . Examples comparing LLaVA (base) and LLaVA fine-tuned on InterHand2.6M.

Figure 23: Qualitative Comparison on FPHA [ 20 ] . Examples comparing LLaVA (base) and LLaVA fine-tuned on FPHA.

Figure 24: Qualitative Results on In-the-Wild Images. We evaluate spatial reasoning on challenging questions using in-the-wild images. The fine-tuned LLaVA outperforms the base model on tasks involving occlusion, depth, and inter-finger relationships, demonstrating improved generalization beyond the training data.

HandVQA: Diagnosing and Improving Fine-Grained Spatial Reasoning about Hands in Vision-Language Models

Abstract

Figures & tables

VLM Reasoning

FineBench: Benchmarking and Enhancing Vision-Language Models for Fine-grained Human Activity Understanding

SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models

MultihopSpatial: Multi-hop Compositional Spatial Reasoning Benchmark for Vision-Language Model

Abstract

VLM Reasoning

Explore similar work

FineBench: Benchmarking and Enhancing Vision-Language Models for Fine-grained Human Activity Understanding

SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models

MultihopSpatial: Multi-hop Compositional Spatial Reasoning Benchmark for Vision-Language Model