cs.CLFeb 26, 2026

Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning

Authors: Amita Kamath, Jack Hessel, Khyathi Chandu, Jena D. Hwang, Kai-Wei Chang, Ranjay Krishna

Organizations: University of Washington · University of California, Los Angeles · Samaya AI · Mistral AI · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The lack of reasoning capabilities in Vision-Language Models (VLMs) has remained at the forefront of research discourse. We posit that this behavior stems from a reporting bias in their training data. That is, how people communicate about visual content by default omits tacit information needed to supervise some types of reasoning; e.g., "at the game today!" is a more likely caption than "a photo of 37 people standing behind a field". We investigate the data underlying the popular VLMs OpenCLIP, LLaVA-1.5 and Molmo through the lens of theories from pragmatics, and find that reporting bias results in insufficient representation of four reasoning skills (spatial, temporal, negation, and counting), despite the corpora being of web-scale, and/or synthetically generated. With a set of curated benchmarks, we demonstrate that: (i) VLMs perform poorly on the aforementioned types of reasoning suppressed in the training data by reporting bias; (ii) contrary to popular belief, scaling data size, model size, and to multiple languages does not result in emergence of these skills by default; but, promisingly, (iii) incorporating annotations specifically collected to obtain tacit information is effective. Our findings highlight the need for more intentional training data curation methods, rather than counting on scale for emergence of reasoning capabilities.

Figures & tables

 SpatialCountingNegationTemporal
DataOccurr.Est. True Occurr.Occurr.Est. True Occurr.Occurr.Est. True Occurr.Occurr.Est. True Occurr.
LAION-2B0.30.18.81.70.80.10.90.2
COCO3.73.710.810.40.20.10.20.1
LLAVA-1.5 (train)5.84.712.46.05.21.41.70.6
Molmo (train)3.32.228.816.86.03.22.90.3
Table 1: Percentage Occurrences and Estimated True Occurrences of reasoning-related keywords in popular open-source image-text corpora and training datasets of open-source VLMs.
Figure 1: Examples from LAION-2B of data points that contain reasoning-related keywords that do and do not operationalize the reasoning capability itself.
Figure 2: Examples from our four benchmarks for contrastive and generative evaluations. The generative evaluation is in MCQ format but for counting, for which a free form output with a given range yielded higher scores.
 ModelSpatialNegationCountingTemporal
 CLIP ViT-B/3230.611.543.458.5
 + ML Div.27.415.523.351.5
 CLIP ViT-B/1627.712.748.155.0
(a)CLIP ViT-L/1428.412.364.152.0
 CLIP ViT-g/1428.412.759.052.0
 CLIP ViT-H/1426.013.260.059.0
Table 2: Results on our benchmarks of: (a) Contrastive VLMs, (b) Open-Source Generative VLMs, (c) Closed-Data Generative VLMs. All models fall far behind human performance on multiple types of reasoning.
Figure 3: Scaling laws for OpenCLIP models on ImageNet (top left) compared to our benchmarks on spatial, counting, negation and temporal tasks respectively. Note the log-log plots and differing yy axes across graphs.
InstructionsSpatialCountingNegationTemporal
COCO82322
LLAVA-1.5173830
PixMo2143121
Ours14395244
Table 3: Percentage True Occurrences (manually calculated) of reasoning-related keywords in each set of 100 captions collected with different instructions for the controlled study.
Model% Count. DataCounting Perf.
LLaVA-1.5-13b6.049.8
FT on LLaVA-IT (26K)6.950.7
FT on LLaVA-IT + TallyQA (26K)39.054.4
Table 4: Estimated amount of counting data and Counting performance of LLaVA-1.5-13b and finetuned versions. Our instructions (Section 6.2 ) elicit 39% counting data (Table 3 ), which is sufficient to improve model counting over simply increasing the data.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Instructions provided for the COCO (top left), LLaVA-1.5 (top right), PixMo (bottom left) and our (bottom right) sets of instructions.
More in this topic

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 846 papers

Explore similar work

CardsList
Oct 2, 2026

Evaluating VQA in Vision Language Models using Cooperative Principles

We evaluate the performance of Vision Language Models in Visual Question Answering (VQA) when questions violate Grice's maxims. To do this, we use VLMs to generate question modifiers that add non-essential, ambiguous or false information and show that in the presence of such violations, the VLMs that we evaluate (ChatGPT, Claude, Gemini and Llava) show diminished performance. Further, we empirically show the difference between how humans reason pragmatically compared to VLMs, and the difference in VLM reasoning when it resolves violations that are human-induced compared to those that are AI-generated. Finally, we show that human cognitive effort (measured through time-on-task in an experiment) is lower for resolving VLM-induced violations, but VLMs themselves perform less accurately in such cases.

Monika Shah, Sudarshan Balaji, Somdeb Sarkhel +2

University of Memphis, TN, USA · Adobe Research, San Jose, CA, USA · University of Maryland Baltimore County, USA

VLM EvaluationVisual Question AnsweringVLM Reasoning
AskPDF
Sep 28, 2026

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models

Vision-language models (VLMs) can answer simple visual questions, but often struggle when one question requires several visual judgments. We study this gap with controlled tasks for feature binding, numerosity, spatial relations, and amodal completion, together with a Composite task that combines them. Matched counterfactual image pairs isolate changes in the visual evidence needed to answer. Across four models, direct answers, hidden-state readouts, and state interventions show that the individual judgments can be made without explicit reasoning and that intervening on the corresponding states can affect the answer. During reasoning, the Composite answer becomes decodable from hidden states and usable from shortened traces, often before the model stops on its own. We train a small detector to predict this readiness and stop reasoning at that point. On MMStar and RealWorldQA, this reduces mean reasoning tokens by 79.1% and 74.5%, while average accuracy rises by 3.13 and 3.30 percentage points, respectively. These findings connect the internal development of answer readiness to a practical rule for allocating reasoning computation.

Rong Yu Xu, Prayag Tiwari, Shaolei Zhang

Shenzhen College of International Education · Halmstad University, Sweden · Renmin University of China

Efficient VLM InferenceVLM InterpretabilityLatent Visual ReasoningVLM ReasoningReasoning for VLMsCompositional Reasoning in VLMs
AskPDF
Jun 11, 2026

Position: The Systemic Lack of Agency in Visual Reasoning

This paper argues that a systemic lack of Agency constrains the implicit reasoning capabilities of current Vision-Language Models (VLMs). Implicit reasoning refers to the ability to autonomously discover and utilize hidden visual evidence to bridge information gaps, rather than merely relying on explicitly specified targets. This capacity underlies human visual understanding and everyday reasoning. We argue that this limitation arises from a tendency to approach visual reasoning primarily as passive semantic retrieval, rather than as active, situated reasoning that depends on autonomous visual exploration. As a result, most existing benchmarks primarily assess Passive Capacity, leaving this aspect of reasoning largely unmeasured. To address this gap, we introduce the Visual Implicit Reasoning Diagnosing Benchmark (V-IRD), which targets this missing quadrant by requiring models to derive answers strictly through autonomous visual analysis. Our results show that, despite strong retrieval abilities, prominent VLMs struggle to utilize reference objects and to attend to visual evidence that requires self-directed inquiry. Simply put, strong semantic recognition does not equate to active visual exploration, revealing a critical gap in current VLMs. More information can be found at https://haoychen.github.io/Implicit-Reasoning/

Yizhao Huang, Haoyang Chen, Shiqin Wang +6

National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University · Hubei Key Laboratory of Multimedia and Network Communication Engineering · Zhongguancun Academy, Beijing, China. +2

VLM EvaluationVLM Reasoning
AskPDF