Vision Language Models (VLMs) exhibit persistent hallucinations in counting tasks, with accuracy substantially lower than other visual reasoning tasks (excluding sentiment). This phenomenon persists even in state-of-the-art reasoning-capable VLMs. Conversely, CNN-based object detection models (ODMs) such as YOLO excel at spatial localization and instance counting with minimal computational overhead. We propose GroundCount, a framework that augments VLMs with explicit spatial grounding from ODMs to mitigate counting hallucinations. In the best case, our prompt-based augmentation strategy achieves 81.3% counting accuracy on the best-performing model (Ovis2.5-2B) - a 6.6pp improvement - while reducing inference time by 22% through elimination of hallucination-driven reasoning loops for stronger models. We conduct comprehensive ablation studies demonstrating that positional encoding is a critical component, being beneficial for stronger models but detrimental for weaker ones. Confidence scores, by contrast, introduce noise for most architectures and their removal improves performance in four of five evaluated models. We further evaluate feature-level fusion architectures, finding that explicit symbolic grounding via structured prompts outperforms implicit feature fusion despite sophisticated cross-attention mechanisms. Our approach yields consistent improvements across four of five evaluated VLM architectures (6.2--7.5pp), with one architecture exhibiting degraded performance due to incompatibility between its iterative reflection mechanisms and structured prompts. These results suggest that counting failures stem from fundamental spatial-semantic integration limitations rather than architecture-specific deficiencies, while highlighting the importance of architectural compatibility in augmentation strategies.
Figures & tables
VLM
Molmo2-4B
Ovis2.5-2B
R-4B
Qwen3-VL-2B
InternVL3.5-1B
Task
base
sec
icc
ccs
base
sec
icc
ccs
base
sec
icc
ccs
base
sec
icc
ccs
base
sec
icc
ccs
object
89.6
79.2
65.9
89.2
88.5
85.1
84.0
84.6
87.9
85.3
82.1
90.4
87.7
84.3
83.1
83.1
70.3
68.3
67.9
82.0
attribute
85.2
44.7
15.1
93.9
86.5
76.8
75.3
89.2
86.5
75.5
64.5
92.7
86.5
77.8
74.2
88.1
83.2
75.2
73.6
80.9
positional
80.8
66.4
47.3
89.1
80.9
73.0
65.3
88.6
80.6
71.5
54.5
94.1
78.3
71.3
63.6
86.8
70.2
61.3
49.2
85.9
counting
67.4
62.9
33.1
87.9
74.7
67.6
52.0
79.8
73.7
69.4
44.2
87.9
69.4
63.5
53.3
83.1
64.0
54.0
41.3
85.5
sentiment
67.3
52.9
32.0
93.6
68.4
59.6
59.2
94.9
68.1
57.9
50.0
96.2
68.1
50.9
50.3
91.0
66.2
57.7
59.7
83.3
TABLE I: Correctness rates (%) of different VLMs on the PhD benchmark using greedy decoding, with explicit thinking mode enabled when available (i.e., the output begins with a prior reasoning segment enclosed by special tokens such as "<think>...</think>" ). The PhD benchmark consists of questions from five task types listed in the Task column on the left. For each subset (excluding sentiment), the task with the lowest correctness rate is highlighted with a light-red background. Results across all four evaluation settings are shown; baseline corresponds to standard VQA without adversarial context.
Fig. 1: Structural overview of three strategies in our proposed fusion framework - A, B and C. In GroundCount A , we run inference with ODM on the image, and then include its output in the VLM prompt. In GroundCount B , we fuse the VLM and ODM on the visual patch latent vector using a light-weight network. To ensure correct information delivery, we finetune the network with our original counting task mutation from COCO. The fusion block is required to be trained; Other modules - VLM, ODM, and language transformer - are optionally frozen. GroundCount C incorporates both plans by including both prompt-level information and architectural-level integration. The training data also includes ODM detections in the textual input.
Fig. 2: Our pipeline of converting ODM outputs to descriptive text. The image is #000000000077.jpg from COCO-train2017, showing 5 young people skateboarding. The bounding boxes (bbox) come from YOLOv13x’s detection: yellow ones are person objects; orange ones are skateboard objects. The location of each object is determined by the center of their corresponding bbox. Two skateboard objects were not included due to low confidence.
Fig. 3: Illustration with real evaluation example for ODM prompt augmentation in GroundCount A. The image is #000000189241.jpg from COCO-val2014, which is used for a counting question in our selected benchmark, PhD. The tested VLM is Qwen3-VL-2B-Thinking. The question asks for a correctness judgment on the number of bowls in the image. There are indeed 2 bowls in the image, with one bowl partially covered by another. The baseline VLM, whose output is showcased in the left frame, fails to find the second bowl and re-thinks iteratively, exhibiting behaviors of hallucination. On the other hand, with information from the object detection model (ODM) appended in the prompt, the VLM successfully finds the second bowl with a double-check.
Fig. 4: Results of GroundCount A across all model families and including ablation studies. Each block contains the accuracy and average inference time for that group of experiment on the PhD counting subset. The bottom row marks the result of running the object detection model only. The right-most column records the special pointing mode for Molmo2 model only.
Training Scheme
Acc(%)
Time(s)
Best Steps
Baseline
74.7
10.0
NA
Plan A - ODM Prompt Augmentation
81.3
7.8
NA
Plan B.1 - Fusion only
75.2
17.8
10k
Plan B.2 - Fusion + VLM, ODM
72.7
14.6
5k
Plan B.3 - Fusion + LangTrans
71.4
4.3
5k
Plan B.4 - Fusion + VLM, ODM
78.0
7.7
5k
TABLE II: Performance comparison of different GroundCount schemes - prompt augmentation (Plan A), architectural fusion with training (Plan B), and both (Plan C). Plan A gains the highest accuracy; Plan B.4 and Plan C have the lowest inference time among schemes that exceed baseline accuracy, due to training on language transformer.
Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks, yet they still fail on simple grounding skills such as object counting. Existing evaluations mostly assess only final outputs, offering limited insight into where these failures arise inside the model. In this work, we present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis. We introduce COUNTINGTRICKS, a controlled evaluation suite of simple shape-based counting cases designed to expose vulnerabilities under different patchification layouts and adversarial prompting conditions. Using attention analysis and component-wise probing, we show that count-relevant visual evidence is strongest in the modality projection stage but degrades substantially in later language layers, where models become more susceptible to text priors. Motivated by this finding, we further evaluate Modality Attention Share (MAS), a lightweight intervention that encourages a minimum budget of visual attention during answer generation. Our results suggest that counting failures in VLMs stem not only from visual perception limits, but also from the underuse of visual evidence during language-stage reasoning. Code and dataset will be released at https://github.com/leduy99/-CVPRW26-Modality-Attention-Share.
Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van Vo
Vision-Language Models (VLMs) excel at multimodal reasoning, yet it remains unclear whether their answers are grounded in visual evidence or driven by learned language and world priors. Counting provides a precise testbed: when visual evidence conflicts with canonical object knowledge, a model must rely on the image rather than a prototypical count. We introduce CounterCount, a diagnostic framework for counterfactual counting in VLMs, consisting of paired factual and counterfactual images with edited count-relevant attributes, verified answers, and localized evidence annotations. Evaluating recent VLMs, we find strong performance on factual images but consistent degradation under counterfactual attribute changes, indicating reliance on object-level priors even when contradictory visual evidence is present. Using localized annotations, we show that these failures are not solely due to missing or ambiguous visual evidence, but to models underweighting attention to count-relevant visual tokens. We introduce a unified inference-time attention modulation strategy that reweights selected visual tokens, improving counterfactual counting accuracy by up to 8% across multiple VLMs. Overall, CounterCount exposes prior-driven counting failures and provides diagnostic insights for designing future VLMs.
Reem Alzahrani, Hassan Alshanqiti, Bushra Bin Hemid +3
King Abdullah University of Science and Technology, Thuwal, Saudi Arabia. · University of Edinburgh · KAUST
Despite strong performance on many multimodal tasks, vision-language models (VLMs) still struggle with basic object counting. We investigate whether this reflects missing internal knowledge or a gap between internal representations and verbalized outputs. Training simple probes on activations from four VLMs across five counting datasets reveals that nonlinear probes can reliably detect counting errors, suggesting that VLMs often encode the correct count even when they output the wrong answer. SVCCA analysis shows that probes trained on ground-truth counts and probes trained on model outputs occupy a partially shared activation subspace but read out along misaligned directions. We further validate our findings using a causal steering intervention, proving that strengthening the direction of count-identified probes does improve model counting performance. Motivated by this result, we propose a detector-guided self-correction method that selectively re-prompts the model only when an internal error detector predicts failure. This simple inference-time intervention improves counting accuracy by up to 15.6 absolute percentage points, without any parameter updates. Our results establish activation-based error probing as both a practical tool for improving VLM counting and a mechanistic lens on the gap between internal knowledge and model outputs.
Ahmed Oumar El-Shangiti, Abzal Nurgazy, Hilal AlQuabeh +2