Getting to the Point: Pointing Improves LVLMs at Counting

Simone Alghisi; Massimo Rizzoli; Seyed Mahed Mousavi; Giuseppe Riccardi

cs.CVMar 23, 2026

Getting to the Point: Pointing Improves LVLMs at Counting

Authors: Simone Alghisi, Massimo Rizzoli, Seyed Mahed Mousavi, Giuseppe Riccardi

Organizations: Signals and Interactive Systems Lab, University of Trento, Italy

Abstract

Pointing-based methods decompose complex tasks as sequential grounding and reasoning steps. Given a query, the model first grounds the relevant objects by generating their coordinates, and then predicts an answer conditioned on these points. While this approach has been shown to increase the performance of Large Vision-Language Models (LVLMs), it remains unclear why and how it improves the models' visual reasoning. In this work, we evaluate pointing-based methods in the task of zero-shot counting in visual scenes. We experiment with multiple fine-tuning and training-free approaches on state-of-the-art LVLMs, and compare them with Point-then-Count (PtC), where models first generate point coordinates for the target objects and then predict their count. Our results show that PtC achieves the highest accuracy among the evaluated approaches, with predicted points correctly grounded in the image in more than 94% of cases (based on F1-score). Mechanistic analyses show that gains arise from spatial information encoded in the predicted coordinates. Nevertheless, grounding performance varies across image regions, revealing spatial biases. Finally, the results indicate that PtC improves out-of-distribution generalization on both synthetic and real data, suggesting the potential of coordinates to help LVLMs improve their counting skills.

Figures & tables

Model	Synthetic		Real-World
Model	DC	PtC	DC	PtC
Qwen2.5-VL 3B	98.11 ( $\uparrow$ 31.33 )	99.94 ( $\uparrow$ 33.15 )	70.00 ( $\uparrow$ 27.62 )	81.90 ( $\uparrow$ 39.52 )
Qwen2.5-VL 7B	93.02 ( $\uparrow$ 15.33 )	99.88 ( $\uparrow$ 22.19 )	73.57 ( $\uparrow$ 19.76 )	76.19 ( $\uparrow$ 22.38 )
LLaVA-OneVision	99.81 ( $\uparrow$ 12.61 )	99.76 ( $\uparrow$ 12.56 )	69.05 ( $\uparrow$ 29.53 )	77.14 ( $\uparrow$ 37.62 )
InternVL3.5	99.81 ( $\uparrow$ 13.20 )	100.00 ( $\uparrow$ 13.39 )	74.52 ( $\uparrow$ 19.52 )	80.24 ( $\uparrow$ 25.24 )

Table 1: Accuracy (%) on Synthetic and Real-World in-distribution ( ID ) data after DC or PtC fine-tuning. Values in parentheses denote absolute improvements over the corresponding pre-trained model. Bold values indicate the best result on synthetic and real-world, respectively. Overall, models benefit more from point supervision, suggesting PtC as a more robust approach across object counts. Notation: DC : Direct Counting, PtC : Point-then-Count

Model	Training-Free					Fine-Tuning
Model	DC	PtC	# Coord.	LtC	Reasoning	DC	PtC	# Coord.
Qwen2.5-VL 3B	20.67	2.98	2.26	3.30	–	30.86	19.44	80.20
Qwen2.5-VL 7B	23.41	14.04	12.96	6.38	–	32.00	46.19	94.96
LLaVA-OneVision 8B	19.19	9.31	7.51	2.78	14.76	30.20	72.22	92.18
InternVL3.5 8B	46.76	24.38	16.04	13.28	20.32	45.52	97.33	97.94
Gemma 4 E4B	22.48	16.56	16.20	22.89	21.34	–	–	–

Table 2: Accuracy (%) on the synthetic out-of-distribution ( OOD ) setting across training-free and fine-tuning approaches. Bold values indicate the highest accuracy among fine-tuned approaches, while underlined indicates the highest among training-free ones. Overall, computing the count from the coordinates predicted by fine-tuned models yields the highest accuracy. Notation: DC : Direct Counting, PtC : Point-then-Count, LtC : List-then-Count

Model	DC	PtC	# Coord.	DC
Model	Fine-tuning
Qwen2.5-VL 3B	2.15	0.98	31.18	2.35
Qwen2.5-VL 7B	8.82	25.68	33.53	6.47
LLaVA-OneVision	12.55	15.69	16.27	1.76
InternVL3.5	18.82	25.88	30.98	8.24

Table 3: Accuracy (%) on the real-world out-of-distribution ( OOD ) setting for fine-tuning approaches as well as training-free direct counting ( DC ). Similar to its synthetic counterpart, using the coordinates to count yields the highest accuracy.

Model	F1	EM	Cons.
Qwen2.5-VL 3B	94.52	76.07	20.56
Qwen2.5-VL 7B	98.59	88.05	48.79
LLaVA-OneVision	96.91	80.88	77.72
InternVL3.5	99.89	98.06	99.27

Table 4: Coordinate reliability (%) on the synthetic out-of-distribution ( OOD ) setting, reported using F1-score, exact match (EM), and consistency (Cons.) Consistency varies substantially across models, showing that, despite achieving high F1-scores, the number of predicted coordinates does not always match the model’s final count.

Figure 1: Cell-level F1-score (%) for each model across our $9\times 9$ grid on the synthetic out-of-distribution ( OOD ) setting. Except for InternVL3.5, the other models exhibit a higher F1-score for the left part of the image, showing the presence of spatial biases.

Table 5: Absolute change in accuracy under different ablations w.r.t. the PtC baseline. (a) reports the change on the synthetic out-of-distribution ( OOD ) setting ( Table 2 ) after X-FT, where each coordinate is replaced with an “X” during PtC training. (b) reports the change on the synthetic in-distribution ( ID ) setting ( Table 1 ) when ablating the input image or coordinates. Results highlight the importance of spatial information for generalization and the dominant role of textual coordinates.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Prompt used for training-free Direct Counting

Panel of Figure 3 — Figure 3: Prompt used for training-free Point-then-Count for LLaVA-OneVision, InternVL3.5, and Qwen2.5-VL

Panel of Figure 4 — Figure 4: Prompt template used for training-free Point-then-Count for Molmo.

Panel of Figure 5 — Figure 5: Prompt used for List-then-Count

Panel of Figure 6 — Figure 6: Prompt used for Reasoning models.

Figure 7: Visual overview of our synthetic datasets. In the main paper, we consider Base for training and ID and OOD for evaluation. Section D.2 discuss the results for Noisy TR and Noisy TS

Model	DC	PtC
Qwen2.5-VL 3B	66.79	45.71
Qwen2.5-VL 7B	77.69	78.42
LLaVA-OneVision 8B	87.20	62.91
InternVL3.5 8B	86.61	77.42
Molmo 7B	53.06	99.95

Appendix

Table 6: Accuracy (%) of each on the synthetic ID setting for training-free DC and PtC. Values in bold indicate the higher result between the two approaches. Overall, all models achieve more than 66% accuracy, suggesting that our images are compatible with their (pre-)training distribution. On PtC, only Molmo achieves near-perfect accuracy, while other models either degrade or show little change.

Figure 8: Accuracy (%) as a function of the number of distractors after fine-tuning on Noisy TR for DC or PtC. PtC maintains near-perfect accuracy for InternVL3.5 and Qwen2.5-VL 7B, while for Qwen2.5-VL 3B and LLaVA-OneVision PtC degrades faster than DC as distractors increase.

Model	ID			OOD			Noisy TS
Model	F1	EM	Cons.	F1	EM	Cons.	F1	EM	Cons.
Qwen2.5-VL 3B	99.91	99.59	100.00	94.52	76.07	20.56	93.59	69.76	99.56
Qwen2.5-VL 7B	99.89	99.55	100.00	98.59	88.05	48.79	99.19	95.77	99.75
LLaVA-OneVision 8B	99.52	98.31	99.98	96.91	80.88	77.72	89.59	70.57	99.82
InternVL3.5 8B	100.00	100.00	100.00	99.89	98.06	99.27	99.84	99.05	98.51

Appendix

Table 10: Coordinate reliability (%) across three synthetic settings, reported using F1-score, exact match (EM), and consistency (Cons.). Results for Noisy TS are computed only on images containing 9 distractors.

Figure 9: Cell-level F1-score (%) for each model across our $9\times 9$ grid on the synthetic ID setting.

Figure 10: Cell-level F1-score (%) for each model across our $9\times 9$ grid on the Noisy TS setting.

Model	ID	OOD	Noisy TS
Qwen2.5-VL 3B	0.00	0.00	0.00
Qwen2.5-VL 7B	0.00	0.09	0.03
LLaVA-OneVision 8B	0.01	0.08	0.20
InternVL3.5 8B	0.00	0.00	0.00

Appendix

Table 11: Percentage (%) of out-of-bounds coordinates predicted by each model across the synthetic ID , OOD , and Noisy TS settings. Results for Noisy TS are computed only on images containing 9 distractors.

Model	X-FT
Qwen2.5-VL 3B	98.11 ( $\downarrow$ 1.83 )
Qwen2.5-VL 7B	97.37 ( $\downarrow$ 2.51 )
LLaVA-OneVision 8B	99.71 ( $\downarrow$ 0.05 )
InternVL3.5 8B	99.87 ( $\downarrow$ 0.13 )

Appendix

Table 12: Accuracy (%) of each model on the synthetic ID setting after X-FT, where each coordinate is replaced with an “X” during PtC training. Values in parentheses indicate the absolute change with respect to the corresponding PtC baseline ( Table 1 ).

Figure 11: F1-score as a function of the number of target objects under the synthetic ID setting for models fine-tuned with DC or PtC. While LLaVA-OneVision and InternVL3.5 show similar performance, Qwen2.5-VL models benefit more from point supervision, suggesting PtC as a more robust choice across object counts.

Figure 12: Layer-wise activation patching for Qwen2.5-VL 7B when fine-tuned for DC and PtC. Each cell shows the percentage of source answers after activation patching at layer $L$ (y-axis) for images containing $N$ objects (x-axis). We perform activation patching by substituting the hidden representations of the source image with those of a target image. Low values (blue) indicate that the patched visual representation changes the model’s prediction, while high values (red) indicate that the model is unaffected by the patching and outputs the source answer.

Figure 13: Layer-wise activation patching for LLaVA-OneVision when fine-tuned for DC and PtC. Each cell shows the percentage of source answers after activation patching at layer $L$ (y-axis) for images containing $N$ objects (x-axis). We perform activation patching by substituting the hidden representations of the source image with those of a target image. Low values (blue) indicate that the patched visual representation changes the model’s prediction, while high values (red) indicate that the model is unaffected by the patching and outputs the source answer.

Figure 14: Layer-wise activation patching for InternVL3.5 when fine-tuned for DC and PtC. Each cell shows the percentage of source answers after activation patching at layer $L$ (y-axis) for images containing $N$ objects (x-axis). We perform activation patching by substituting the hidden representations of the source image with those of a target image. Low values (blue) indicate that the patched visual representation changes the model’s prediction, while high values (red) indicate that the model is unaffected by the patching and outputs the source answer.

Figure 15: Layer-wise activation patching for Qwen2.5-VL 3B when fine-tuned for DC and PtC. Each cell shows the percentage of source answers after activation patching at layer $L$ (y-axis) for images containing $N$ objects (x-axis). We perform activation patching by substituting the hidden representations of the source image with those of a target image. Low values (blue) indicate that the patched visual representation changes the model’s prediction, while high values (red) indicate that the model is unaffected by the patching and outputs the source answer.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

MolmoPoint: Better Pointing for VLMs with Grounding Tokens
Mar 30, 2026Christopher Clark, Yue Yang, Jae Sung Park +8Vision-Language Grounding Large Vision-Language Models
Allen Institute for AI · University of Washington
Counting Circuits: Mechanistic Interpretability of Visual Reasoning in Large Vision-Language Models
Mar 19, 2026Liwei Che, Zhiyu Xue, Yihao Quan +7VLM Reasoning Large Vision-Language Models
Rutgers University · UC Santa Barbara · University of Washington +1
Counting to Four is still a Chore for VLMs
Apr 11, 2026Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van VoAttention Intervention in VLMs Vision-Language Models
MBZUAI

Mar 30, 2026J/K move · Enter open · S save

MolmoPoint: Better Pointing for VLMs with Grounding Tokens

Christopher Clark, Yue Yang, Jae Sung Park, Zixian Ma+7

Allen Institute for AI · University of Washington

Vision-Language Grounding Large Vision-Language Models

Open paper AskPDF arXiv

Grounding has become a fundamental capability of vision-language models (VLMs). Most existing VLMs point by generating coordinates as part of their text output, which requires learning a complicated coordinate system and results in a high token count. Instead, we propose a more intuitive pointing mechanism that directly selects the visual tokens that contain the target concept. Our model generates a special pointing token that cross-attends to the input image or video tokens and selects the appropriate one. To make this model more fine-grained, we follow these pointing tokens with an additional special token that selects a fine-grained subpatch within the initially selected region, and then a third token that specifies a location within that subpatch. We further show that performance improves by generating points sequentially in a consistent order, encoding the relative position of the previously selected point, and including a special no-more-points class when selecting visual tokens. Using this method, we set a new state-of-the-art on image pointing (70.7% on PointBench), set a new state-of-the-art among fully open models on GUI pointing (61.1% on ScreenSpotPro), and improve video pointing (59.1% human preference win rate vs. a text coordinate baseline) and tracking (+6.3% gain on Molmo2Track). We additionally show that our method achieves much higher sample efficiency and discuss the qualitative differences that emerge from this design change.

Abstract

VLM Reasoning

Explore similar work

MolmoPoint: Better Pointing for VLMs with Grounding Tokens

Counting Circuits: Mechanistic Interpretability of Visual Reasoning in Large Vision-Language Models

Counting to Four is still a Chore for VLMs