Pointing-based methods decompose complex tasks as sequential grounding and reasoning steps. Given a query, the model first grounds the relevant objects by generating their coordinates, and then predicts an answer conditioned on these points. While this approach has been shown to increase the performance of Large Vision-Language Models (LVLMs), it remains unclear why and how it improves the models' visual reasoning. In this work, we evaluate pointing-based methods in the task of zero-shot counting in visual scenes. We experiment with multiple fine-tuning and training-free approaches on state-of-the-art LVLMs, and compare them with Point-then-Count (PtC), where models first generate point coordinates for the target objects and then predict their count. Our results show that PtC achieves the highest accuracy among the evaluated approaches, with predicted points correctly grounded in the image in more than 94% of cases (based on F1-score). Mechanistic analyses show that gains arise from spatial information encoded in the predicted coordinates. Nevertheless, grounding performance varies across image regions, revealing spatial biases. Finally, the results indicate that PtC improves out-of-distribution generalization on both synthetic and real data, suggesting the potential of coordinates to help LVLMs improve their counting skills.
Figures & tables
Model
Synthetic
Real-World
DC
PtC
DC
PtC
Qwen2.5-VL 3B
98.11 ( ↑ 31.33 )
99.94 ( ↑ 33.15 )
70.00 ( ↑ 27.62 )
81.90 ( ↑ 39.52 )
Qwen2.5-VL 7B
93.02 ( ↑ 15.33 )
99.88 ( ↑ 22.19 )
73.57 ( ↑ 19.76 )
76.19 ( ↑ 22.38 )
LLaVA-OneVision
99.81 ( ↑ 12.61 )
99.76 ( ↑ 12.56 )
69.05 ( ↑ 29.53 )
77.14 ( ↑ 37.62 )
InternVL3.5
99.81 ( ↑ 13.20 )
100.00 ( ↑ 13.39 )
74.52 ( ↑ 19.52 )
80.24 ( ↑ 25.24 )
Table 1: Accuracy (%) on Synthetic and Real-World in-distribution ( ID ) data after DC or PtC fine-tuning. Values in parentheses denote absolute improvements over the corresponding pre-trained model. Bold values indicate the best result on synthetic and real-world, respectively. Overall, models benefit more from point supervision, suggesting PtC as a more robust approach across object counts. Notation: DC : Direct Counting, PtC : Point-then-Count
Model
Training-Free
Fine-Tuning
DC
PtC
# Coord.
LtC
Reasoning
DC
PtC
# Coord.
Qwen2.5-VL 3B
20.67
2.98
2.26
3.30
–
30.86
19.44
80.20
Qwen2.5-VL 7B
23.41
14.04
12.96
6.38
–
32.00
46.19
94.96
LLaVA-OneVision 8B
19.19
9.31
7.51
2.78
14.76
30.20
72.22
92.18
InternVL3.5 8B
46.76
24.38
16.04
13.28
20.32
45.52
97.33
97.94
Gemma 4 E4B
22.48
16.56
16.20
22.89
21.34
–
–
–
Table 2: Accuracy (%) on the synthetic out-of-distribution ( OOD ) setting across training-free and fine-tuning approaches. Bold values indicate the highest accuracy among fine-tuned approaches, while underlined indicates the highest among training-free ones. Overall, computing the count from the coordinates predicted by fine-tuned models yields the highest accuracy. Notation: DC : Direct Counting, PtC : Point-then-Count, LtC : List-then-Count
Model
Fine-tuning
DC
PtC
# Coord.
DC
Qwen2.5-VL 3B
2.15
0.98
31.18
2.35
Qwen2.5-VL 7B
8.82
25.68
33.53
6.47
LLaVA-OneVision
12.55
15.69
16.27
1.76
InternVL3.5
18.82
25.88
30.98
8.24
Table 3: Accuracy (%) on the real-world out-of-distribution ( OOD ) setting for fine-tuning approaches as well as training-free direct counting ( DC ). Similar to its synthetic counterpart, using the coordinates to count yields the highest accuracy.
Model
F1
EM
Cons.
Qwen2.5-VL 3B
94.52
76.07
20.56
Qwen2.5-VL 7B
98.59
88.05
48.79
LLaVA-OneVision
96.91
80.88
77.72
InternVL3.5
99.89
98.06
99.27
Table 4: Coordinate reliability (%) on the synthetic out-of-distribution ( OOD ) setting, reported using F1-score, exact match (EM), and consistency (Cons.) Consistency varies substantially across models, showing that, despite achieving high F1-scores, the number of predicted coordinates does not always match the model’s final count.
Figure 1: Cell-level F1-score (%) for each model across our 9×9 grid on the synthetic out-of-distribution ( OOD ) setting. Except for InternVL3.5, the other models exhibit a higher F1-score for the left part of the image, showing the presence of spatial biases.
Table 5: Absolute change in accuracy under different ablations w.r.t. the PtC baseline. (a) reports the change on the synthetic out-of-distribution ( OOD ) setting ( Table 2 ) after X-FT, where each coordinate is replaced with an “X” during PtC training. (b) reports the change on the synthetic in-distribution ( ID ) setting ( Table 1 ) when ablating the input image or coordinates. Results highlight the importance of spatial information for generalization and the dominant role of textual coordinates.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Prompt used for training-free Direct Counting
Figure 3: Prompt used for training-free Point-then-Count for LLaVA-OneVision, InternVL3.5, and Qwen2.5-VL
Figure 4: Prompt template used for training-free Point-then-Count for Molmo.
Figure 5: Prompt used for List-then-Count
Figure 6: Prompt used for Reasoning models.
Figure 7: Visual overview of our synthetic datasets. In the main paper, we consider Base for training and ID and OOD for evaluation. Section D.2 discuss the results for Noisy TR and Noisy TS
Model
DC
PtC
Qwen2.5-VL 3B
66.79
45.71
Qwen2.5-VL 7B
77.69
78.42
LLaVA-OneVision 8B
87.20
62.91
InternVL3.5 8B
86.61
77.42
Molmo 7B
53.06
99.95
Appendix
Table 6: Accuracy (%) of each on the synthetic ID setting for training-free DC and PtC. Values in bold indicate the higher result between the two approaches. Overall, all models achieve more than 66% accuracy, suggesting that our images are compatible with their (pre-)training distribution. On PtC, only Molmo achieves near-perfect accuracy, while other models either degrade or show little change.
Figure 8: Accuracy (%) as a function of the number of distractors after fine-tuning on Noisy TR for DC or PtC. PtC maintains near-perfect accuracy for InternVL3.5 and Qwen2.5-VL 7B, while for Qwen2.5-VL 3B and LLaVA-OneVision PtC degrades faster than DC as distractors increase.
Model
ID
OOD
Noisy TS
F1
EM
Cons.
F1
EM
Cons.
F1
EM
Cons.
Qwen2.5-VL 3B
99.91
99.59
100.00
94.52
76.07
20.56
93.59
69.76
99.56
Qwen2.5-VL 7B
99.89
99.55
100.00
98.59
88.05
48.79
99.19
95.77
99.75
LLaVA-OneVision 8B
99.52
98.31
99.98
96.91
80.88
77.72
89.59
70.57
99.82
InternVL3.5 8B
100.00
100.00
100.00
99.89
98.06
99.27
99.84
99.05
98.51
Appendix
Table 10: Coordinate reliability (%) across three synthetic settings, reported using F1-score, exact match (EM), and consistency (Cons.). Results for Noisy TS are computed only on images containing 9 distractors.
Figure 9: Cell-level F1-score (%) for each model across our 9×9 grid on the synthetic ID setting.
Figure 10: Cell-level F1-score (%) for each model across our 9×9 grid on the Noisy TS setting.
Model
ID
OOD
Noisy TS
Qwen2.5-VL 3B
0.00
0.00
0.00
Qwen2.5-VL 7B
0.00
0.09
0.03
LLaVA-OneVision 8B
0.01
0.08
0.20
InternVL3.5 8B
0.00
0.00
0.00
Appendix
Table 11: Percentage (%) of out-of-bounds coordinates predicted by each model across the synthetic ID , OOD , and Noisy TS settings. Results for Noisy TS are computed only on images containing 9 distractors.
Model
X-FT
Qwen2.5-VL 3B
98.11 ( ↓ 1.83 )
Qwen2.5-VL 7B
97.37 ( ↓ 2.51 )
LLaVA-OneVision 8B
99.71 ( ↓ 0.05 )
InternVL3.5 8B
99.87 ( ↓ 0.13 )
Appendix
Table 12: Accuracy (%) of each model on the synthetic ID setting after X-FT, where each coordinate is replaced with an “X” during PtC training. Values in parentheses indicate the absolute change with respect to the corresponding PtC baseline ( Table 1 ).
Figure 11: F1-score as a function of the number of target objects under the synthetic ID setting for models fine-tuned with DC or PtC. While LLaVA-OneVision and InternVL3.5 show similar performance, Qwen2.5-VL models benefit more from point supervision, suggesting PtC as a more robust choice across object counts.
Figure 12: Layer-wise activation patching for Qwen2.5-VL 7B when fine-tuned for DC and PtC. Each cell shows the percentage of source answers after activation patching at layer L (y-axis) for images containing N objects (x-axis). We perform activation patching by substituting the hidden representations of the source image with those of a target image. Low values (blue) indicate that the patched visual representation changes the model’s prediction, while high values (red) indicate that the model is unaffected by the patching and outputs the source answer.
Figure 13: Layer-wise activation patching for LLaVA-OneVision when fine-tuned for DC and PtC. Each cell shows the percentage of source answers after activation patching at layer L (y-axis) for images containing N objects (x-axis). We perform activation patching by substituting the hidden representations of the source image with those of a target image. Low values (blue) indicate that the patched visual representation changes the model’s prediction, while high values (red) indicate that the model is unaffected by the patching and outputs the source answer.
Figure 14: Layer-wise activation patching for InternVL3.5 when fine-tuned for DC and PtC. Each cell shows the percentage of source answers after activation patching at layer L (y-axis) for images containing N objects (x-axis). We perform activation patching by substituting the hidden representations of the source image with those of a target image. Low values (blue) indicate that the patched visual representation changes the model’s prediction, while high values (red) indicate that the model is unaffected by the patching and outputs the source answer.
Figure 15: Layer-wise activation patching for Qwen2.5-VL 3B when fine-tuned for DC and PtC. Each cell shows the percentage of source answers after activation patching at layer L (y-axis) for images containing N objects (x-axis). We perform activation patching by substituting the hidden representations of the source image with those of a target image. Low values (blue) indicate that the patched visual representation changes the model’s prediction, while high values (red) indicate that the model is unaffected by the patching and outputs the source answer.
Grounding has become a fundamental capability of vision-language models (VLMs). Most existing VLMs point by generating coordinates as part of their text output, which requires learning a complicated coordinate system and results in a high token count. Instead, we propose a more intuitive pointing mechanism that directly selects the visual tokens that contain the target concept. Our model generates a special pointing token that cross-attends to the input image or video tokens and selects the appropriate one. To make this model more fine-grained, we follow these pointing tokens with an additional special token that selects a fine-grained subpatch within the initially selected region, and then a third token that specifies a location within that subpatch. We further show that performance improves by generating points sequentially in a consistent order, encoding the relative position of the previously selected point, and including a special no-more-points class when selecting visual tokens. Using this method, we set a new state-of-the-art on image pointing (70.7% on PointBench), set a new state-of-the-art among fully open models on GUI pointing (61.1% on ScreenSpotPro), and improve video pointing (59.1% human preference win rate vs. a text coordinate baseline) and tracking (+6.3% gain on Molmo2Track). We additionally show that our method achieves much higher sample efficiency and discuss the qualitative differences that emerge from this design change.
Counting serves as a simple but powerful test of a Large Vision-Language Model's (LVLM's) reasoning; it forces the model to identify each individual object and then add them all up. In this study, we investigate how LVLMs implement counting using controlled synthetic and real-world benchmarks, combined with mechanistic analyses. Our results show that LVLMs display a human-like counting behavior, with precise performance on small numerosities and noisy estimation for larger quantities. We introduce two novel interpretability methods, Visual Activation Patching and HeadLens, and use them to uncover a structured "counting circuit" that is largely shared across a variety of visual reasoning tasks. Building on these insights, we propose a lightweight intervention strategy that exploits simple and abundantly available synthetic images to fine-tune arbitrary pretrained LVLMs exclusively on counting. Despite the narrow scope of this fine-tuning, the intervention not only enhances counting accuracy on in-distribution synthetic data, but also yields an average improvement of +8.36% on out-of-distribution counting benchmarks and an average gain of +1.54% on complex, general visual reasoning tasks for Qwen2.5-VL. These findings highlight the central, influential role of counting in visual reasoning and suggest a potential pathway for improving overall visual reasoning capabilities through targeted enhancement of counting mechanisms.
Liwei Che, Zhiyu Xue, Yihao Quan +7
Rutgers University · UC Santa Barbara · University of Washington +1
Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks, yet they still fail on simple grounding skills such as object counting. Existing evaluations mostly assess only final outputs, offering limited insight into where these failures arise inside the model. In this work, we present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis. We introduce COUNTINGTRICKS, a controlled evaluation suite of simple shape-based counting cases designed to expose vulnerabilities under different patchification layouts and adversarial prompting conditions. Using attention analysis and component-wise probing, we show that count-relevant visual evidence is strongest in the modality projection stage but degrades substantially in later language layers, where models become more susceptible to text priors. Motivated by this finding, we further evaluate Modality Attention Share (MAS), a lightweight intervention that encourages a minimum budget of visual attention during answer generation. Our results suggest that counting failures in VLMs stem not only from visual perception limits, but also from the underuse of visual evidence during language-stage reasoning. Code and dataset will be released at https://github.com/leduy99/-CVPRW26-Modality-Attention-Share.
Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van Vo