Counting to Four is still a Chore for VLMs

Duy Le Dinh Anh; Patrick Amadeus Irawan; Tuan Van Vo

cs.CVApr 11, 2026

Counting to Four is still a Chore for VLMs

Authors: Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van Vo

Organizations: MBZUAI

Abstract

Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks, yet they still fail on simple grounding skills such as object counting. Existing evaluations mostly assess only final outputs, offering limited insight into where these failures arise inside the model. In this work, we present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis. We introduce COUNTINGTRICKS, a controlled evaluation suite of simple shape-based counting cases designed to expose vulnerabilities under different patchification layouts and adversarial prompting conditions. Using attention analysis and component-wise probing, we show that count-relevant visual evidence is strongest in the modality projection stage but degrades substantially in later language layers, where models become more susceptible to text priors. Motivated by this finding, we further evaluate Modality Attention Share (MAS), a lightweight intervention that encourages a minimum budget of visual attention during answer generation. Our results suggest that counting failures in VLMs stem not only from visual perception limits, but also from the underuse of visual evidence during language-stage reasoning. Code and dataset will be released at https://github.com/leduy99/-CVPRW26-Modality-Attention-Share.

Figures & tables

Figure 1: Diagnosing the grounding gap in VLM counting. VLMs can answer complex multimodal questions, yet often fail at a minimal grounding skill like counting. The figure illustrates our diagnosis: counting evidence is present in early visual representations but becomes under-used as generation relies more on textual priors. We also test an attention-budget constraint (MAS) as a lightweight step toward improving grounding.

Figure 2: The Toy-32 Visual Suite. We craft multiple patchification cases to probe perception failures under different geometric alignments and isolate Vision Encoder blind spots. Object Counting Benchmark Data: 1A–B (fixed/varied sizes), 2A–D (vertical grid alignment), 3A–D (horizontal grid alignment), 4A–D (grid intersection alignment), 5A–8A (circle diameters 2.5–4 patch sizes).

Model	Year	Backbone	# Params
LLaVA-1.5-7B	2023	Vicuna-7B	7B
LLaVA-1.6-Vicuna-7B	2024	Vicuna-7B	7B
InternVL3-8B	2025	InternViT	8B
Llama-3.2-11B	2024	Llama-3.2	11B
Gemma-3-4B	2025	Gemma-3	4B
Qwen2.5-VL-3B	2025	Qwen2.5	3B

Table 1: Key Details of the Evaluated Vision-Language Models. We focus on recent efficient open-weights models (3B–11B parameters) to test the accessibility of counting capabilities.

Model	Avg Accuracy (%)
LLaVA-1.5-7B	11.82%
LLaVA-1.6-Vicuna-7B	14.20%
InternVL3-8B	21.72%
Llama-3.2-11B	24.00%
Gemma-3-4B	27.37%
Qwen2.5-VL-3B	36.01%

Table 2: Baseline Evaluation on Counting-Tricks. Models are ranked by average accuracy. Despite improvements in recent architectures (e.g., Qwen2.5-VL), most models struggle to exceed 50% accuracy on this diagnostic suite. Each test case is evaluated using 1000 samples, evenly distributed across the respective count range.

Panel of Figure 3 — Figure 3: Evidence of Text-Prior Dominance. Our probing analysis reveals that models suffer from a “visual attention sink,” where computation drifts away from image tokens toward system prompts and instructions.

Tap	Bottleneck	YOLO Head	Total Trainable
Encoder (1024ch)	$\sim 0.52$ M	$\sim 1.19$ M	$\sim\mathbf{1.71}$ M
Projector (2048ch)	$\sim 1.05$ M	$\sim 1.19$ M	$\sim\mathbf{2.24}$ M
LLM (2048ch)	$\sim 1.05$ M	$\sim 1.19$ M	$\sim\mathbf{2.24}$ M

Table 3: Probe Architecture Parameters. We maintain negligible capacity differences between taps to ensure a fair apple-to-apples comparison of feature utility.

Figure 4: Spatial evidence is strong in the vision stack but fades in the language layers. Detection performance (AP@0.5) peaks at the projector tap (green) and drops at the LLM tap (red) across modern architectures, quantifying the loss of spatial information. — Figure 4: Spatial evidence is strong in the vision stack but fades in the language layers. Detection performance ([email protected]) peaks at the projector tap (green) and drops at the LLM tap (red) across modern architectures, quantifying the loss of spatial information.

Panel of Figure 5 — Figure 5: Training Dynamics (YOLO Probing Head). The Projector probe (center) reaches higher AP sooner, while the Encoder plateaus lower and LLM curves exhibit higher volatility. Since the schedule and seed are identical across taps, these differences reflect purely tap utility rather than optimization quirks.

Figure 6: Vision Encoder Heatmaps. Activations in the early visual backbone are sharp and distinct, perfectly localizing the object instances. This confirms that the counting signal is present upstream, however, fading when entering the language space.

Backbone	Training	Circles (Syn)	FSC-Val (Real)	FSC-Test (Real)
Ovis 2.5	Pretrained	53.8	6.3	3.9
Ovis 2.5	SFT	84.9	17.5	16.6
Ovis 2.5	SFT + MAS (Ours)	85.2	17.7	16.1
Qwen3-VL	Pretrained	48.1	7.6	5.5
Qwen3-VL	SFT	18.2	20.3	14.0
Qwen3-VL	SFT + MAS	30.4	18.2	13.4

Table 4: Impact of MAS Regularization. Exact-match accuracy (%). MAS improves validation accuracy for some backbones, but its effect is backbone- and dataset-dependent.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Accuracy vs. Ground Truth Count. Left: Scatter plots show a consistent negative correlation ( $r\approx-0.78$ ) between count magnitude and accuracy across diverse geometric cases. Right: Detailed breakdown for LLaVA-1.5 and Qwen2.5-VL reveals specific “blind spots” (circled in red) where models achieve 0% accuracy for specific numbers (e.g., 7, 11), evidencing strong linguistic priors.

Figure 8: Fused Token Heatmaps. Averaged over deep LLM layers (15–25), the attention becomes diffuse and misaligned. The signal “washes out,” failing to retain the instance-level separation required for accurate counting.

Explore similar work

CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models
May 18, 2026Reem Alzahrani, Hassan Alshanqiti, Bushra Bin Hemid +3VLM Evaluation Vision-Language Models
King Abdullah University of Science and Technology, Thuwal, Saudi Arabia. · University of Edinburgh · KAUST
The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs
Jul 10, 2026Ahmed Oumar El-Shangiti, Abzal Nurgazy, Hilal AlQuabeh +2Vision-Language Models VLM Interpretability
MBZUAI · Imperial College London
GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinations
Mar 11, 2026Boyuan Chen, Minghao Shao, Siddharth Garg +2Vision-Language Models VLM Hallucination Mitigation
Tandon School of Engineering, New York University, NY, USA · eBRAIN Lab, Division of Engineering, New York University Abu Dhabi, UAE

May 18, 2026J/K move · Enter open · S save

CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models

Reem Alzahrani, Hassan Alshanqiti, Bushra Bin Hemid, Zaid Alyafeai+2

King Abdullah University of Science and Technology, Thuwal, Saudi Arabia. · University of Edinburgh · KAUST

VLM Evaluation Vision-Language Models Counterfactual Evaluation of VLMs Object Counting Visually Grounded Reasoning Attention Mechanisms

Open paper AskPDF arXiv

Vision-Language Models (VLMs) excel at multimodal reasoning, yet it remains unclear whether their answers are grounded in visual evidence or driven by learned language and world priors. Counting provides a precise testbed: when visual evidence conflicts with canonical object knowledge, a model must rely on the image rather than a prototypical count. We introduce CounterCount, a diagnostic framework for counterfactual counting in VLMs, consisting of paired factual and counterfactual images with edited count-relevant attributes, verified answers, and localized evidence annotations. Evaluating recent VLMs, we find strong performance on factual images but consistent degradation under counterfactual attribute changes, indicating reliance on object-level priors even when contradictory visual evidence is present. Using localized annotations, we show that these failures are not solely due to missing or ambiguous visual evidence, but to models underweighting attention to count-relevant visual tokens. We introduce a unified inference-time attention modulation strategy that reweights selected visual tokens, improving counterfactual counting accuracy by up to 8% across multiple VLMs. Overall, CounterCount exposes prior-driven counting failures and provides diagnostic insights for designing future VLMs.

Abstract

Explore similar work

CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models

The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs

GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinations