Understanding the fine-grained articulation of human hands is critical in high-stakes settings such as robot-assisted surgery, chip manufacturing, and AR/VR-based human-AI interaction. Despite achieving near-human performance on general vision-language benchmarks, current vision-language models (VLMs) struggle with fine-grained spatial reasoning, especially in interpreting complex and articulated hand poses. We introduce HandVQA, a large-scale diagnostic benchmark designed to evaluate VLMs' understanding of detailed hand anatomy through visual question answering. Built upon high-quality 3D hand datasets (FreiHAND, InterHand2.6M, FPHA), our benchmark includes over 1.6M controlled multiple-choice questions that probe spatial relationships between hand joints, such as angles, distances, and relative positions. We evaluate several state-of-the-art VLMs (LLaVA, DeepSeek and Qwen-VL) in both base and fine-tuned settings, using lightweight fine-tuning via LoRA. Our findings reveal systematic limitations in current models, including hallucinated finger parts, incorrect geometric interpretations, and poor generalization. HandVQA not only exposes these critical reasoning gaps but provides a validated path to improvement. We demonstrate that the 3D-grounded spatial knowledge learned from our benchmark transfers in a zero-shot setting, significantly improving accuracy of model on novel downstream tasks like hand gesture recognition (+10.33%) and hand-object interaction (+2.63%).
Figures & tables
Figure 1: Overview of HandVQA’s transfer effect. Fine-tuning a base Vision-Language Model (VLM) on HandVQA teaches it explicit 3D hand geometry and joint-level spatial reasoning. The resulting Spatial-Aware VLM exhibits zero-shot generalization to novel downstream tasks: both image-based gesture recognition and video-based hand-object interaction recognition. Spatial-Aware VLM achieves consistent accuracy gains without task-specific training.
Figure 2: Overview of HandVQA Question Format. This figure illustrates the structure of our benchmark, which divides hand pose estimation into five sub-tasks: Angle, Distance, and Relative Position along X, Y, and Z axes. A hand image with annotated joint indices (top left) supports multiple-choice questions per task type, derived from 3D joint coordinates and the correct answers are shown in green .
Figure 3: Overview of the HandVQA pipeline. The pipeline converts normalized 3D hand joints into interpretable VQA pairs through three deterministic stages: ( 1 ) Fpose computes continuous pose descriptors: angles ( θ ), distances ( d ), and relative positions ( Δx,Δy,Δz ) and categorizes them into discrete pose descriptors ( Γ ); ( 2 ) Ftext fills deterministic sentence templates using Γ and filters correct and incorrect options to form candidate answer options (O) and the correct label y⋆ ; ( 3 ) Fmcq constructs multiple-choice questions (MCQ) by pairing each image with O and its correct label y⋆ .
Table 1: Pose descriptor categorization with conditions and illustration.
Model
Tuned
Eval
Angle
Distance
Accuracy ↑
MAE ↓
Accuracy ↑
MAE ↓
Base model (no tuning)
DeepSeek Janus Pro 7B
–
InterHand2.6M
34.10
0.883
45.55
0.657
DeepSeek Janus Pro 7B
–
FreiHAND
35.31
0.830
44.15
0.668
DeepSeek Janus Pro 7B
–
FPHA
26.46
0.991
39.02
0.819
Finetuned Models
Table 2: Angle and Distance Results for all three models. The best, second-best, and third-best models for each dataset in each metric are highlighted as Gold , Silver , and Bronze , respectively.
Model
Tuned
Eval
Rel. Pos. X
Rel. Pos. Y
Rel. Pos. Z
Accuracy ↑
Accuracy ↑
Accuracy ↑
Base model (no tuning)
DeepSeek Janus Pro 7B
–
InterHand2.6M
50.41
52.46
51.16
DeepSeek Janus Pro 7B
–
FreiHAND
49.80
51.55
50.03
DeepSeek Janus Pro 7B
–
FPHA
43.02
52.64
61.73
Finetuned Models
Table 3: Relative Position Results for all three models. The best, second-best, and third-best models for each dataset are highlighted as Gold , Silver and Bronze , respectively.
Setup
Interaction Recognition
Gesture Recognition
LLaVA Mistral 7B [ 36 ]
-
57.42
Qwen 2.5 VL 7B [ 3 ]
80.26
71.86
LLaVA Mistral 7B finetuned
-
69.58
Qwen 2.5 VL 7B finetuned
82.89
82.19
Table 4: Zero-shot generalization results. HandVQA fine-tuning strengthens 3D spatial reasoning, yielding higher accuracy (%) on gesture and hand–object interaction recognition.
Angle Pose Descriptors
Distance Pose Descriptors
Relative Position Pose Descriptors (XYZ)
Thumb-MCP
Thumb-MCP vs. Index-PIP
Thumb-MCP vs. Index-PIP
Index-PIP
Index-PIP vs. Middle-PIP
Index-PIP vs. Middle-PIP
Middle-PIP
Middle-PIP vs. Ring-PIP
Middle-PIP vs. Ring-PIP
Ring-PIP
Ring-PIP vs. Little-PIP
Ring-PIP vs. Little-PIP
Little-PIP
Thumb-Tip vs. Index-Tip
Thumb-Tip vs. Index-Tip
Thumb-IP
Index-Tip vs. Middle-Tip
Index-Tip vs. Middle-Tip
Table 5: List of joints/joint-pairs on which angles, distances, and relative positions in X,Y, and Z axes are calculated. A total of 107 different joints/joint-pairs across all pose descriptors are considered.
Figure 4: The map of the hand skeleton used in our HandVQA benchmark generation pipeline.
Figure 5: Possible location of the ‘aligned’ Little Finger Proximal Interphalangeal (PIP) joint and Ring Finger Proximal Interphalangeal (PIP) joint underneath the index and middle finger. The relationship along the x-axis for the two PIP joints is ambiguous, making it necessary to drop the relative position X information of the two joints.
Figure 6: Word cloud representation of the most frequently used terms in the caption options extracted from our dataset. Prominent anatomical terms like “tip joint”, “interphalangeal joint”, and “metacarpophalangeal joint” highlight the fine-grained spatial and anatomical focus of the hand-centric question-answer pairs.
Figure 7: Breakdown of question types across the training (left) and evaluation (right) splits for each dataset in the HandVQA benchmark. Each dataset contains a balanced distribution of all five spatial reasoning tasks: angle, distance, and relative positions along the X, Y, and Z axes. This uniformity supports fair evaluation across all pose subtasks.
Pose Descriptors
Label frequency (Count)
Angle
bent slightly inward (130,993), bent inward (107,143), straight (74,007), bent completely inward (21,622)
Distance
spread wide from (222,429), spread from (109,353), close to (1,983)
Rel. Pos. (X)
at the left of (198,988), at the right of (133,109)
Rel. Pos. (Y)
above (215,725), below (112,984)
Rel. Pos. (Z)
in front of (177,216), behind (152,481)
Table 6: Label frequency grouped by pose descriptors.
Figure 8: Angle confusion matrix across three VLMs. All models frequently predict “bent slightly inward” regardless of ground truth, revealing a strong prediction bias. Each model also follows a consistent preference ordering in its outputs, indicating difficulty in distinguishing fine-grained joint angles.
Figure 9: Angle confusion matrix across three fine-tuned VLMs. Fine-tuning significantly reduces the dominant bias toward “bent slightly inward” observed in the base models, resulting in stronger alignment along the diagonal. The models exhibit improved discrimination across angle categories, particularly for “bent inward” and “straight” , although some residual confusion remains between adjacent angle classes, indicating that fine-grained distinctions are still challenging.
Figure 10: Distance confusion matrix comparison between base and fine-tuned VLMs. The base models exhibit a strong bias toward predicting “close to” across multiple ground-truth categories, leading to a skewed prediction distribution. In contrast, fine-tuning produces a more balanced distribution with increased alignment along the diagonal, indicating improved discrimination between distance categories. However, residual confusion persists between “spread from” and “spread wide from” , suggesting continued difficulty in distinguishing fine-grained spatial separations.
Figure 11: Relative Position X confusion matrix comparison between base and fine-tuned VLMs. The base models exhibit near-random predictions between “at the left of” and “at the right of” , indicating a lack of reliable directional understanding. In contrast, fine-tuning leads to a strong concentration along the diagonal, demonstrating significantly improved left–right discrimination. This suggests that relative positional reasoning along the horizontal axis is effectively learned after fine-tuning, with minimal residual confusion.
Figure 12: Relative Position Y confusion matrix comparison between base and fine-tuned VLMs. The base models exhibit inconsistent predictions between “below” and “above” , reflecting weak vertical positional understanding. In contrast, fine-tuning results in a strong alignment along the diagonal, indicating substantially improved discrimination of vertical relationships. While minor confusion remains, the fine-tuned models demonstrate a clear and consistent understanding of relative position along the vertical axis.
Figure 13: Relative Position Z confusion matrix comparison between base and fine-tuned VLMs. The base models exhibit strong bias toward predicting “in front of” , leading to highly skewed outputs and poor discrimination of depth relationships. In contrast, fine-tuning significantly improves alignment with the diagonal, indicating enhanced understanding of front–back spatial relations. While minor confusion persists, the fine-tuned models demonstrate substantially improved depth-aware reasoning compared to the base models.
Model
Tuned
Eval
Angle
Distance
Accuracy ↑
MAE ↓
Accuracy ↑
MAE ↓
Finetuned Models on Individual Datasets
LLaVA Mistral 7B
InterHand2.6M
InterHand2.6M
74.35
0.263
90.79
0.094
LLaVA Mistral 7B
FreiHAND
FreiHAND
62.91
0.382
86.19
0.141
LLaVA Mistral 7B
FPHA
FPHA
68.37
0.401
83.99
0.161
Finetuned Model on HandVQA
Table 7: Angle and Distance results of LLaVA Mistral 7B finetuned models on Individual Datasets vs. finetuned on the Unified HandVQA Benchmark.
Model
Tuned
Eval
Rel. Pos. X
Rel. Pos. Y
Rel. Pos. Z
Accuracy ↑
Accuracy ↑
Accuracy ↑
Finetuned Models on Individual Datasets
LLaVA Mistral 7B
InterHand2.6M
InterHand2.6M
97.14
98.77
96.82
LLaVA Mistral 7B
FreiHAND
FreiHAND
92.60
93.20
88.17
LLaVA Mistral 7B
FPHA
FPHA
93.81
92.80
90.25
Finetuned Model on HandVQA
Table 8: Relative Position accuracies of LLaVA Mistral 7B finetuned models on Individual Datasets vs. finetuned on the Unified HandVQA Benchmark.
Model
Tuned
Eval
Angle
Distance
Accuracy ↑
MAE ↓
Accuracy ↑
MAE ↓
Base model (no tuning)
DeepSeek Janus Pro 7B
–
InterHand2.6M
34.10
0.883
45.55
0.657
DeepSeek Janus Pro 7B
–
FPHA
26.46
0.991
39.02
0.819
Finetuned Models
DeepSeek Janus Pro 7B
FreiHAND
InterHand2.6M
56.15
0.456
82.70
0.175
Table 9: Angle and Distance results of Base Models vs Out-of-Distribution Finetuned Models.
Model
Tuned
Eval
Rel. Pos. X
Rel. Pos. Y
Rel. Pos. Z
Accuracy ↑
Accuracy ↑
Accuracy ↑
Base model (no tuning)
DeepSeek Janus Pro 7B
–
InterHand2.6M
50.41
52.46
51.16
DeepSeek Janus Pro 7B
–
FPHA
43.02
52.64
61.73
Finetuned Models
DeepSeek Janus Pro 7B
FreiHAND
InterHand2.6M
77.32
89.80
75.33
Table 10: Relative-Position Results of Base Models vs Out-of-Distribution Finetuned Models.
Figure 24Figure 25
Model
FreiHAND test
InterHand2.6M test
Angle
Distance
Angle
Distance
Accuracy % (MAE)
Accuracy % (MAE)
Accuracy % (MAE)
Accuracy % (MAE)
HaMeR
59.53 (0.428)
88.86 (0.113)
78.82 (0.218)
88.11 (0.119)
LLaVA Mistral 7B
42.48 (0.678)
13.18 (1.342)
40.08 (0.739)
16.20 (1.293)
LLaVA Mistral 7B finetuned
64.35 (0.367)
86.71 (0.136)
72.26 (0.283)
90.27 (0.099)
Table 11: Pure vision model (HaMeR [ 49 ] ) vs. LLaVA Mistral 7B.
Difficulty
Angle
Distance
R. Pos.(X)
R. Pos.(Y)
R. Pos.(Z)
Accuracy % (MAE)
Accuracy % (MAE)
Accuracy %
Accuracy %
Accuracy %
Easy
75.92 (0.255)
95.01 (0.08)
97.20
96.19
98.11
Hard
55.1 (0.454)
78.01 (0.183)
82.11
81.72
78.92
Table 12: LLaVA Mistral 7B finetuned on FreiHAND: Easy (225 QA), Hard (223 QA).
Figure 18: Easy vs Hard Examples Performance of LLaVA Mistral 7B finetuned . Correct( ✓ )/wrong( X ) QA samples for “Which is the distance between thumb tip and index fingertip?”.
Models
Overall Accuracy
LLaVA Mistral 7B
41.96%
DeepSeek Janus Pro 7B
45.82%
Qwen 2.5 VL 7B
41.97%
Humans
80.94%
Table 13: Human vs VLMs Accuracy on a small subset of HandVQA.
Base Label
Description 1
Description 2
one
A single index finger is extended, representing the number one.
The hand is held up with only the index finger pointing upwards.
two_up
The index and middle fingers are extended upwards (peace sign), representing two.
A “V” sign, often used for “two” or “peace”.
two_up_inverted
An inverted “V” sign, with the palm facing inward.
The gesture for “two” or “peace” but inverted.
three
Three fingers are extended to represent the number three.
The hand gesture indicates a quantity of three.
four
Four fingers are extended to show the number four.
The hand gesture indicates a quantity of four.
fist
The hand is closed tightly into a fist.
All fingers are curled inward with the thumb wrapped around them.
Table 14: Gemini-generated descriptions for HaGRID gesture labels.
Figure 19: Qualitative Results on Zero-shot Gesture Recognition on HaGRID dataset [ 29 ] .
Figure 20: Qualitative Results on Zero-shot Hand-Object Interaction Recognition on H2O dataset [ 31 ] .
Figure 21: Qualitative Comparison on FreiHAND [ 70 ] . Examples comparing LLaVA (base) and LLaVA fine-tuned on FreiHAND.
Figure 22: Qualitative Comparison on InterHand2.6M [ 46 ] . Examples comparing LLaVA (base) and LLaVA fine-tuned on InterHand2.6M.
Figure 23: Qualitative Comparison on FPHA [ 20 ] . Examples comparing LLaVA (base) and LLaVA fine-tuned on FPHA.
Figure 24: Qualitative Results on In-the-Wild Images. We evaluate spatial reasoning on challenging questions using in-the-wild images. The fine-tuned LLaVA outperforms the base model on tasks involving occlusion, depth, and inter-finger relationships, demonstrating improved generalization beyond the training data.
Vision-Language Models (VLMs) have demonstrated remarkable capabilities in general video understanding, yet they often struggle with the fine-grained comprehension crucial for real-world applications requiring nuanced interpretation of human actions and interactions. While some recent human-centric benchmarks evaluate aspects of model behaviour such as fairness/ethics, emotion perception, and broader human-centric metrics, they do not combine long-form videos, very dense QA coverage, and frame-level spatial/temporal grounding at scale. To bridge this gap, we introduce FineBench, a human-centric video question answering (VQA) benchmark specifically designed to assess fine-grained understanding. FineBench comprises 199,420 multiple-choice QA pairs densely annotated across 64 long-form videos (15 minutes each), focusing on detailed person movement, person interaction, and object manipulation, including compositional actions. Our extensive evaluation reveals that while proprietary models like GPT-5 achieve respectable performance, current open-source VLMs significantly underperform, struggling particularly with spatial reasoning in multi-person scenes and distinguishing subtle differences in human movements and interactions. To address these identified weaknesses, we propose FineAgent, a modular framework that enhances VLMs by leveraging a Localizer and a Descriptor. Experiments show that FineAgent consistently improves the performance of various open VLMs on FineBench. FineBench provides a rigorous testbed for future research into fine-grained human-centric video understanding, while FineAgent offers a practical approach to enhance such reasoning in current VLMs. Project page and code at https://joslefaure.github.io/assets/html/finebench.html.
Vision-Language Models (VLMs) have been increasingly applied in real-world scenarios due to their outstanding understanding and reasoning capabilities. Although VLMs have already demonstrated impressive capabilities in common visual question answering and logical reasoning, they still lack the ability to make reasonable decisions in complex real-world environments. We define this ability as spatial logical reasoning, which not only requires understanding the spatial relationships among objects in complex scenes, but also the logical dependencies between steps in multi-step tasks. To bridge this gap, we introduce Spatial Logical Question Answering (SpatiaLQA), a benchmark designed to evaluate the spatial logical reasoning capabilities of VLMs. SpatiaLQA consists of 9,605 question answer pairs derived from 241 real-world indoor scenes. We conduct extensive experiments on 41 mainstream VLMs, and the results show that even the most advanced models still struggle with spatial logical reasoning. To address this issue, we propose a method called recursive scene graph assisted reasoning, which leverages visual foundation models to progressively decompose complex scenes into task-relevant scene graphs, thereby enhancing the spatial logical reasoning ability of VLMs, outperforming all previous methods. Code and dataset are available at https://github.com/xieyc99/SpatiaLQA.
Yuechen Xie, Xiaoyan Zhang, Yicheng Shan +6
Zhejiang University · The University of Sydney · ManyCore +2
Spatial reasoning is foundational for Vision-Language Models (VLMs), particularly when deployed as Vision-Language-Action (VLA) agents in physical environments. However, existing benchmarks predominantly focus on elementary, single-hop relations, neglecting the multi-hop compositional reasoning and precise visual grounding essential for real-world scenarios. To address this, we introduce MultihopSpatial, offering three key contributions: (1) A comprehensive benchmark designed for multi-hop and compositional spatial reasoning, featuring 1- to 3-hop complex queries across diverse spatial perspectives. (2) Acc@50IoU, a complementary metric that simultaneously evaluates reasoning and visual grounding by requiring both answer selection and precise bounding box prediction - capabilities vital for robust VLA deployment. (3) MultihopSpatial-Train, a dedicated large-scale training corpus to foster spatial intelligence. Extensive evaluation of 37 state-of-the-art VLMs yields eight key insights, revealing that compositional spatial reasoning remains a formidable challenge. Finally, we demonstrate that reinforcement learning post-training on our corpus enhances both intrinsic VLM spatial reasoning and downstream embodied manipulation performance.
Youngwan Lee, Soojin Jang, Yoorhim Cho +3
Electronics and Telecommunications Research Institute, South Korea · Korea Advanced Institute of Science and Technology, South Korea · Sungkyunkwan University, South Korea +1