Evaluating VQA in Vision Language Models using Cooperative Principles

Monika Shah; Sudarshan Balaji; Somdeb Sarkhel; Sanorita Dey; Deepak Venugopal

cs.CLOct 2, 2026

Evaluating VQA in Vision Language Models using Cooperative Principles

Authors: Monika Shah, Sudarshan Balaji, Somdeb Sarkhel, Sanorita Dey, Deepak Venugopal

Organizations: University of Memphis, TN, USA · Adobe Research, San Jose, CA, USA · University of Maryland Baltimore County, USA

Abstract

We evaluate the performance of Vision Language Models in Visual Question Answering (VQA) when questions violate Grice's maxims. To do this, we use VLMs to generate question modifiers that add non-essential, ambiguous or false information and show that in the presence of such violations, the VLMs that we evaluate (ChatGPT, Claude, Gemini and Llava) show diminished performance. Further, we empirically show the difference between how humans reason pragmatically compared to VLMs, and the difference in VLM reasoning when it resolves violations that are human-induced compared to those that are AI-generated. Finally, we show that human cognitive effort (measured through time-on-task in an experiment) is lower for resolving VLM-induced violations, but VLMs themselves perform less accurately in such cases.

Figures & tables

Figure 1: Illustrating violations of Grice’s maxims in VQA

Figure 2: Illustration of the evaluation process. A VLM modifies the question which is verified by a human. A VLM answers the original and modified questions and a human compares their responses. For all the human tasks, we used Amazon Mechanical Turk (paid workers between $5-10$ cents per individual response depending on the task complexity.)

Response-Model / Modification-Model	GPT	Gemini	Claude	LLava	$Average\pm std$
GPT	-3.13%	38.08%	2.97%	12.10%	$12.51\pm 18.16\%$
Gemini	10.05%	8.60%	9.07%	12.50%	$10.06\pm 1.74\%$
Claude	2.65%	2.72%	2.43%	11.48%	$4.82\pm 4.44\%$
Llava	15.56%	9.50%	10.51%	17.56%	$13.28\pm 3.89\%$
$Average\pm std$	$6.28\pm 8.21$ %	$14.73\pm 15.86\%$	$6.25\pm 4.14\%$	$13.41\pm 2.80\%$

Table 1:

\%

decrease in response accuracy for modified questions compared to original questions for binary and quantitative responses (smaller is better). Rows indicate the VLM providing the response and columns the VLM performing the modification.

Response-Model / Modification-Model	GPT		Gemini		Claude		LLava
Response-Model / Modification-Model	Statistic	P-value	Statistic	P-value	Statistic	P-value	Statistic	P-value
GPT	4.84	0.028	15.05	0.00	3.45	0.063	9.25	0.002
Gemini	13.78	0.00	9.84	0.002	11.22	0.001	9.25	0.002
Claude	1.66	0.197	1.515	0.218	1.80	0.18	8.89	0.003
Llava	27.50	0.00	14.45	0.00	20.57	0.00	15.29	0.00

Table 2: Mcnemar’s test for assessing significance of the modifiers added to binary and quantitative questions.

Response-Model / Modification-Model	GPT-4o	Gemini-1.5-Flash	Claude-3.5-Sonnet	LLava-7B	$Average\pm std$
GPT4o	7.68%	15.55%	12.32%	15.96%	$12.88\pm 3.32$ %
Gemini	16.15%	18.46%	16.15%	20.0%	$17.69\pm 1.63\%$
Claude	21.53%	20.76 %	13.07%	16.92%	$18.07\pm 3.37$ %
Llava	25.38%	20.00%	20.00%	18.46%	$20.96\pm 2.63$ %
$Average\pm std$	$17.69\pm 6.64$ %	$18.69\pm 1.99$ %	$15.39\pm 3.03$ %	$17.84\pm 1.54$ %

Table 3: Comparing open-ended responses before and after modifications. The

\%

of questions where human users in AMT chose that the response after modification is worse than the response before modification (for each question, we have 3 workers and we use the majority choice).

Figure 3: Comparing the $\%$ of questions where the response to a question is unchanged after a hallucinated modifier is added. Results shown for binary and quantitative response types. The y-axis shows the average (and standard deviation) over 4 hallucinated modifications each added using one of the VLMs (graph created with Claude).

Response Type	GPT	Gemini	Claude	Llava
Binary & Quantitative	83.53%	80.83%	83.87%	77.44%
Open-ended	0.874	0.864	0.864	0.865

Table 4: Comparing VLM and human pragmatic reasoning. For binary & quantitative responses, we show the

\%

of cases where the VLM and human answers match for hallucinated modifications. For open-ended responses, we show the BERTScore (F1-score) between human and VLM responses.

Human Response	GPT	Gemini	Claude	LLava
Negative Response	88.79%	89.47%	92.44%	86.14%
Positive Response	62.79%	48.35%	51.11%	45.05%

Table 5: Comparing binary responses of humans with VLMs. We show the % of VLM responses aligned with the positive and negative responses of humans.

Figure 4: Comparing the accuracy of models on the original questions, questions modified by humans, and questions modified by VLMs (the average accuracy and standard deviation is shown in this case since we have modifications by 4 models). Results shown for binary responses (graph created with Claude).

Figure 5: Comparing open-ended responses of models using human evaluation. Results show the $\%$ of cases where AMT-workers chose that the model’s response to the AI-modified question was worse than its response for the human-modified question (graph created with Claude).

Response Type	Human	GPT	Gemini	Claude	LLava
Binary & Quantitative	0.612	0.641	0.658	0.639	0.670
Open-ended	0.645	0.657	0.644	0.654	0.675

Table 6: CLIPScores of human and AI modifications

Figure 6: Distribution of time-on-task in seconds measured for the experiment where AMT workers were asked to answer modified questions and reflect on the modification by simplifying the question. Results are shown for human and AI (VLMs) modified questions (graph created with ChatGPT).

Question Type	#Questions
Is the	75
Is this	24
Are the	24
Is there a	18
Is this a	24
Is the girl	13

Table 7: Total number of Open-ended-response questions types to evaluate VLM accuracy.

Question Type	#Questions
What	351
Which	21
Who	38
Why	41
Where	63

Table 8: Total number of Open-ended-response questions types to evaluate VLM accuracy.

Question Type	#Questions
Is the	10
Is this	57
Are the	25
Is there a	54
Is this a	41
Is the girl	1

Table 9: Total number of binary-response and quantitative-response question types in the dataset used for comparing human and AI modifications.

Question Type	#Questions
Which	41
Who	16
Why	20
Where	51

Table 10: Total number of Open-ended-response questions types in the dataset used for comparing human and AI modifications.

Figure 7: AMT Task for verifying if humans can pragmatically infer VLM-modified questions. If they choose that the modification is correct, they need to demonstrate their understanding of the question by answering a sub-question based on the salient object in the question.

Figure 8: AMT Task for verifying if humans can pragmatically infer VLM-modified questions. If they choose that the modification is incorrect, then they need to correct the modification using a textbox as shown in the example.

Figure 9: AMT Task for evaluating open-ended VLM responses by comparing answers to original and modified questions.

Figure 10: AMT design to measure time-on-task for humans to answer a modified question and to reflect on the answer by simplifying the question.

Figure 11: Examples showing human and VLM responses on hallucinated modifications

Figure 12: Examples showing human and VLM responses on human and VLM modifications

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 957 papers

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models
Sep 28, 2026Rong Yu Xu, Prayag Tiwari, Shaolei ZhangEfficient VLM Inference VLM Interpretability
Shenzhen College of International Education · Halmstad University, Sweden · Renmin University of China
Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning
Feb 26, 2026Amita Kamath, Jack Hessel, Khyathi Chandu +3VLM Reasoning Reasoning for VLMs
University of Washington · University of California, Los Angeles · Samaya AI +2
CaVe-VLM-CoT: An Interpretable Vision-Language Model Framework
Jun 16, 2026Sneha Rao, Shaina Raza, Dhanesh RamachandramVLM Evaluation Vision-Language Grounding
Vector Institute, Toronto, ON M5G 0C6, Canada

Sep 28, 2026J/K move · Enter open · S save

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models

Rong Yu Xu, Prayag Tiwari, Shaolei Zhang

Shenzhen College of International Education · Halmstad University, Sweden · Renmin University of China

Efficient VLM Inference VLM Interpretability Latent Visual Reasoning VLM Reasoning Reasoning for VLMs Compositional Reasoning in VLMs

Open paper AskPDF arXiv

Vision-language models (VLMs) can answer simple visual questions, but often struggle when one question requires several visual judgments. We study this gap with controlled tasks for feature binding, numerosity, spatial relations, and amodal completion, together with a Composite task that combines them. Matched counterfactual image pairs isolate changes in the visual evidence needed to answer. Across four models, direct answers, hidden-state readouts, and state interventions show that the individual judgments can be made without explicit reasoning and that intervening on the corresponding states can affect the answer. During reasoning, the Composite answer becomes decodable from hidden states and usable from shortened traces, often before the model stops on its own. We train a small detector to predict this readiness and stop reasoning at that point. On MMStar and RealWorldQA, this reduces mean reasoning tokens by 79.1% and 74.5%, while average accuracy rises by 3.13 and 3.30 percentage points, respectively. These findings connect the internal development of answer readiness to a practical rule for allocating reasoning computation.

Abstract

VLM Evaluation

Explore similar work

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models

Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning

CaVe-VLM-CoT: An Interpretable Vision-Language Model Framework