Vision-Language Models (VLMs) have advanced across multimodal benchmarks but still show clear gaps in ordinal number understanding, i.e., the ability to track relative positions and generalize to large indices. We present OrdinalBench, a diagnostic benchmark that standardizes ordinal number understanding as an evaluation task for VLMs. The core task is N-th object identification, defined by a starting reference and traversal rule. Task difficulty is controlled along three axes: (i) ordinal magnitude, from small numbers to extreme cases up to 300; (ii) arrangement complexity, from single loops to maze-like paths; and (iii) object count. The benchmark provides 39,000 question-answer pairs, each annotated with a ground-truth reasoning trajectory and balanced across difficulty levels for controlled large-scale testing. Beyond answer-only evaluation, our framework requires models to generate structured stepwise traces of the counting process and provides an open evaluation toolkit that measures both final accuracy and step-level path consistency. Zero-shot evaluations of GPT-5, Gemini 2.5 Flash Lite, Qwen2.5-VL, InternVL3.5, and Molmo reveal sharp degradation under large-ordinal and complex-path conditions, highlighting weak generalization despite strong scores on standard multimodal tasks. By framing ordinal number understanding as a core target, OrdinalBench provides a reproducible benchmark and diagnostic framework for developing VLMs with stronger sequential reasoning. All data and code are available at https://ordinalbench.github.io/
Figures & tables
Figure 1: Overview of OrdinalBench . We illustrate representative instances of Single-Loop (2D/3D) and Maze-Loop (2D), where models follow explicit traversal rules (start position, facing direction, stride, and blocked-turn policy) to count along a path and predict the identifier at the N -th position. Each example includes the input image and question, as well as a step-wise trace output, enabling evaluation of procedural execution beyond the final answer. For brevity, the instruction prompts shown are abridged examples of the actual inputs used in our evaluation.
2D Single-Loop
3D Single-Loop
2D Maze-Loop
Model
Acc@N
STA
nLCP
Cov.
Acc@N
STA
nLCP
Cov.
Acc@N
STA
nLCP
Cov.
Proprietary models
GPT-5
15.64
18.70
17.07
93.80
13.80
12.05
8.78
100.00
11.04
17.33
15.45
99.40
Gemini 2.5 Flash Lite
21.68
21.13
18.11
100.00
10.93
9.36
6.75
100.00
9.28
16.79
14.66
99.48
Open-source models
Qwen2.5-VL-72B
31.16
31.89
24.38
100.00
31.33
33.25
22.82
100.00
5.07
11.47
9.67
99.54
Table 1: Overall performance of evaluated VLMs on OrdinalBench . We report Final Accuracy (Acc@N), Stepwise Trace Accuracy (STA), Normalized Longest Correct Prefix (nLCP), and Trace Coverage (Cov.) in percentages (%). All metrics are macro-averaged across QA instances; the highest score in each column is highlighted in bold . Note that results for GPT-5 and Gemini 2.5 are based on stratified subsets sampled from public_test (2,500/1,500; seed = 0), while others are based on the full 15,000/9,000-sample test set.
Figure 2: Performance degradation with increasing cognitive load in 2D Single-Loop. (Left) Accuracy consistently declines as the ordinal level increases from Within to Large Scale. (Right) Skip counting (stride >1 ) causes a sharp drop, pushing many models toward chance level.
Figure 3: Average Acc@N performance heatmaps across all models in the three task categories. Each heatmap displays the average accuracy of all evaluated models using color, with the y-axis representing the Ordinal Level and the x-axis representing either the number of objects or grid size.
Vision-language models (VLMs) achieve strong performance on high-level multimodal tasks, yet numerosity perception, a cognitive ability that emerges in human infants before language acquisition, remains poorly understood in current models, as existing counting benchmarks entangle numerosity with correlated visual factors. We introduce a cognitively inspired diagnostic benchmark, NumerosityVLM, comprising 10,800 synthetic images across six controlled conditions. The benchmark orthogonally manipulates object size, spatial arrangement, and numerosity, while progressively ablating texture, shape, and color. Evaluating seven VLMs in a zero-shot setting, multi-factor analysis reveals that model architecture explains the largest proportion of performance variance (partial ω2=0.325), far exceeding visual conditions. Layer-wise probing further shows that linearly separable numerosity signals consistently emerge at early stages of the vision encoder, while performance differences across evaluated models are primarily associated with the language model component. Code and data are publicly available at https://github.com/fuy3/NumerosityVLM-Benchmark, and https://huggingface.co/datasets/fuy3/NumerosityVLM.
Yiming Fu, Fangjun Li, Xiujin Liu +7
Cognitive Robotics Lab, Department of Computer Science, The University of Manchester, Manchester M13 9PL, UK · University of Michigan, MI 48109, USA · Sheffield Hallam University, Sheffield S1 1WB, UK +2
Multimodal LLMs apply the language model interface to visual inputs, where ordinal regression tasks such as age estimation, image quality assessment, and disease grading require autoregressive decisions over ordered class labels. We ask whether MLLMs reliably convert internal ordinal evidence into ordered digit-token outputs. Across four ordinal benchmarks and four MLLM backbones, ordinal labels are linearly recoverable from hidden states with Spearman correlation up to 0.938, and a task-designed prompt further sharpens this structure. Yet native digit-token outputs weakly expose it: the unembedding matrix filters the ordinal direction, and the digit-token row space retains below 1.15% across all 16 model-dataset combinations, with a 16 to 77 absolute-point accuracy gap between linear-probe and native outputs. We introduce Ordinal Lens Alignment (OLA), a frozen-backbone inference-time method that trains lightweight W_S-anchored lenses on mid-to-deep decoder layers, fuses them into an ordinal distribution, and corrects only digit-token logits at generation. OLA outperforms the SOTA LoRA-tuned OrderChain baseline in most settings while keeping the MLLM frozen, surpasses discriminative ordinal baselines in most cells, and improves over an offline lens in every setting.
Haiming Li, Yingsheng Liu, Jingmin Zhu +5
Faculty of Information Technology, Monash University, Melbourne, Australia · Monash University, Melbourne, Victoria, Australia
Recent vision-language models (VLMs) excel at multimodal understanding and reasoning, yet their fine-grained visual perception remains underexplored. A natural extension of ``How many r are there in Strawberry?'' asks: how small a visual pattern can a VLM reliably perceive? As such, we introduce FineSightBench, a new benchmark that systematically probes this limit by separating perception tasks (pixel-level recognition of letters, shapes, objects) from reasoning tasks (spatial reasoning, counting, ordering over small targets) across controlled scales of 4--48px. Through comprehensive experiments and detailed failure mode analysis on state-of-the-art models, we reveal a sharp dissociation: perception saturates around 12px, while reasoning remains limited even at larger scales, with persistent numeracy and sequence errors. These findings expose fundamental deficiencies in VLMs' fine-scale visual reasoning that demand more rigorous evaluation.
Lujun Li, Lama Sleem, Niccolo Gentile +4
University of Luxembourg · Foyer S.A. · Université Paris-Saclay