NumerosityVLM: A Cognitively Inspired Benchmark for Interpreting Numerosity Representations in Vision-Language Models

Yiming Fu; Fangjun Li; Xiujin Liu; Ruidong Ma; Hang Yu; Zhichen Lu; Kanwei He; Alessandro Di Nuovo

cs.CVAug 15, 2026

NumerosityVLM: A Cognitively Inspired Benchmark for Interpreting Numerosity Representations in Vision-Language Models

Authors: Yiming Fu, Fangjun Li, Xiujin Liu, Ruidong Ma, Hang Yu, Zhichen Lu, Kanwei He, Alessandro Di Nuovo, +2 more

Organizations: Cognitive Robotics Lab, Department of Computer Science, The University of Manchester, Manchester M13 9PL, UK · University of Michigan, MI 48109, USA · Sheffield Hallam University, Sheffield S1 1WB, UK · Tufts University, MA 02155, USA · ENSTA, Institut Polytechnique de Paris, Palaiseau 91120, France

Abstract

Vision-language models (VLMs) achieve strong performance on high-level multimodal tasks, yet numerosity perception, a cognitive ability that emerges in human infants before language acquisition, remains poorly understood in current models, as existing counting benchmarks entangle numerosity with correlated visual factors. We introduce a cognitively inspired diagnostic benchmark, NumerosityVLM, comprising 10,800 synthetic images across six controlled conditions. The benchmark orthogonally manipulates object size, spatial arrangement, and numerosity, while progressively ablating texture, shape, and color. Evaluating seven VLMs in a zero-shot setting, multi-factor analysis reveals that model architecture explains the largest proportion of performance variance (partial $ω^{2}=0.325$ ), far exceeding visual conditions. Layer-wise probing further shows that linearly separable numerosity signals consistently emerge at early stages of the vision encoder, while performance differences across evaluated models are primarily associated with the language model component. Code and data are publicly available at https://github.com/fuy3/NumerosityVLM-Benchmark, and https://huggingface.co/datasets/fuy3/NumerosityVLM.

Figures & tables

Figure 1: Overview of the proposed NumerosityVLM benchmark framework. Left: The diagnostic benchmark comprises three object categories, six controlled conditions, and 12 numerosity levels. Right: Layer-wise probing is conducted across the full VLM pipeline to localize the source of numerosity perception failures.

Figure 2: Representative dataset stimuli under the six controlled conditions. Columns show the designed controlled conditions. Rows illustrate numerosities of 1, 11, and 100 using examples from the apple, butterfly, and human-figure categories.

Model	Version	Year	Input Size	Vision Model	Language Model
CLIP	CLIP-ViT-B/32	2021	224 $\times$ 224	ViT-B/32	Transformer
LLaVA	LLaVA-1.5-7B-hf	2023	336 $\times$ 336	ViT-L/14	Vicuna-7B
LLaVA	LLaVA-OneVision- Qwen2-7B-ov-hf	2024	384 $\times$ 384	SigLIP	Qwen2-7B
InternVL	InternVL3-1B-hf	2025	448 $\times$ 448	InternViT	Qwen2.5-0.5B
InternVL	InternVL3-8B-hf	2025	448 $\times$ 448	InternViT	Qwen2.5-7B
QwenVL	Qwen3VL-4B-Instruct	2025	Dynamic	Qwen3-ViT	Qwen3-4B

Table 1: Overview of the seven evaluated open-source vision-language models, including the backbone architectures of their vision and language components.

Model	Accuracy (%) $\uparrow$	MAE error $\downarrow$	NAE error $\downarrow$
InternVL3-1B	15.56	21.40	2.24
CLIP-ViT-B/32	20.39	24.06	2.97
LLaVA-1.5-7B	27.24	32.23	1.95
LLaVA-OneVision-Qwen2-7B	71.28	18.14	0.35
InternVL3-8B	77.21	18.47	0.35
Qwen3VL-4B-Instruct	77.73	24.97	0.36

Table 2: Overall performance comparison across different VLMs.

Figure 3: Comprehensive evaluation of numerosity understanding in VLMs. (a–b) Accuracy across conditions and mean residuals indicating underestimation at higher numerosity. (c–d) Output-layer and linear-probe confusion matrices for selected models.

Group	Comparison	Raw $p$ -value	Corrected $p$ -value	Sig.
Tier-1	Baseline vs Size-incongruent	$1.96\times 10^{-3}$	$2.75\times 10^{-3}$	∗∗
Tier-1	Baseline vs Space-incongruent	$8.31\times 10^{-1}$	$8.49\times 10^{-1}$	$-$
Tier-1	Baseline vs Texture-ablated	$1.63\times 10^{-29}$	$5.72\times 10^{-29}$	∗∗∗
Tier-1	Texture-ablated vs Shape-ablated	$3.87\times 10^{-40}$	$2.71\times 10^{-39}$	∗∗∗
Tier-1	Shape-ablated vs Color-ablated	$1.79\times 10^{-14}$	$4.17\times 10^{-14}$	∗∗∗
Tier-2	Baseline vs Size-incongruent	$2.53\times 10^{-52}$	$1.77\times 10^{-51}$	∗∗∗

Table 4: Pairwise statistical comparisons across experimental conditions. Reported

p

-values are shown before and after multiple-comparison correction. Significance levels:

{}^{*}p<0.05

{}^{**}p<0.01

{}^{***}p<0.001

;

-

indicates no significant difference.

Figure 4: Layer-wise probing results. Numerosity signals emerge early in the vision encoder, while performance differences primarily arise during language modeling.

Figure 5: PC1 projections across evaluated models. Tier-1 models (a–d) show clear monotonic organization, whereas Tier-2 models (e–g) exhibit weaker or inconsistent scaling.

Explore similar work

The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models
Jun 5, 2026Lujun Li, Lama Sleem, Niccolo Gentile +4VLM Evaluation Large Vision-Language Models
University of Luxembourg · Foyer S.A. · Université Paris-Saclay
OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Models
Mar 8, 2026Yusuke Tozaki, Hisashi MiyamoriVLM Evaluation VLM Reasoning
Division of Frontier Informatics, Kyoto Sangyo University, Kyoto, Japan
Counting to Four is still a Chore for VLMs
Apr 11, 2026Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van VoVision-Language Models VLM Evaluation
MBZUAI

Jun 5, 2026J/K move · Enter open · S save

The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models

Lujun Li, Lama Sleem, Niccolo Gentile, Yangjie Xu+3

University of Luxembourg · Foyer S.A. · Université Paris-Saclay

VLM Evaluation Large Vision-Language Models VLM Reasoning Fine-Grained Visual Perception

Open paper AskPDF arXiv

Recent vision-language models (VLMs) excel at multimodal understanding and reasoning, yet their fine-grained visual perception remains underexplored. A natural extension of ``How many r are there in Strawberry?'' asks: how small a visual pattern can a VLM reliably perceive? As such, we introduce FineSightBench, a new benchmark that systematically probes this limit by separating perception tasks (pixel-level recognition of letters, shapes, objects) from reasoning tasks (spatial reasoning, counting, ordering over small targets) across controlled scales of 4--48px. Through comprehensive experiments and detailed failure mode analysis on state-of-the-art models, we reveal a sharp dissociation: perception saturates around 12px, while reasoning remains limited even at larger scales, with persistent numeracy and sequence errors. These findings expose fundamental deficiencies in VLMs' fine-scale visual reasoning that demand more rigorous evaluation.

Abstract

Explore similar work

The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models

OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Models

Counting to Four is still a Chore for VLMs