Composition, Not Conversation: VLMs Lose the Scene, Not the Thread

L. D. M. S. Sai Teja; Ufaq Khan; N. Siva Gopala Krishna; Satyajit Tourani; Ashshak Sharifdeen; Fida Mohammad Thoker; Bernard Ghanem; Muhammad Haris Khan

cs.CVSep 29, 2026

Composition, Not Conversation: VLMs Lose the Scene, Not the Thread

Authors: L. D. M. S. Sai Teja, Ufaq Khan, N. Siva Gopala Krishna, Satyajit Tourani, Ashshak Sharifdeen, Fida Mohammad Thoker, Bernard Ghanem, Muhammad Haris Khan

Organizations: NIT Silchar · MBZUAI · BML Munjal University · KAUST

Abstract

Vision-language models (VLMs) increasingly reason over visual evidence that is cropped, segmented, retrieved, or revealed over time. Yet most VQA benchmarks present the complete image and question at once. We ask what models lose when the same information is fragmented. We introduce Layered-VQA, with 93 scenes and 300 questions. Each image is decomposed into ordered RGBA layers that exactly recompose the original scene, and each question is annotated with supporting, minimal-sufficient, and distractor layers. We evaluate eleven open-weight VLMs from 3B to 32B parameters and two proprietary models with a scale of 187,200 conversations, graded by 1.74M open-model cross-judgments. We find three consistent failures. Loss in Composition: fragmenting the question has a small effect, but fragmenting the scene substantially reduces accuracy; recomposing the same layers largely restores performance. Oracle Inversion: even oracle-selected sufficient evidence can perform worse than the complete scene. Loss in Grounding: as more evidence is required, grounding degrades much faster than answer accuracy. Together, these results show that having the right visual evidence is not enough. How that evidence is composed and presented determines whether models can use and ground it. The right evidence is not enough: VLMs need the scene it came from.

Figures & tables

Figure 1: Vision-language models lose the scene, not the thread. Left: Scene fragmentation hurts accuracy, while recomposition recovers it. Middle: Oracle-selected minimal evidence can perform worse than the complete scene. Right: As evidence demand grows, grounding falls much faster than answer accuracy.

Figure 2: Layered-VQA construction. Scenes are decomposed into RGBA layers, manually annotated, expanded into question shards and evidence mappings with GPT-5.6-Luna, and verified before finalization.

Figure 3: Controlled delivery conditions. The sample is presented under different question, visual, alignment, and evidence schedules.

Figure 4: Self-preference after cross-judging. Boxed cells correspond to models judging their own outputs.

base

question in text

question in layers

alignment

order

retained %

Model

full

concat

sharded

recap

snowball

concat

sharded

recap

snowball

aligned

misaligned

ev-first

ev-last

shuffled

Text

Visual

Layered

Qwen2.5-VL-3B

.408

\pm

.02

.334

\pm

.01

.142

\pm

.01

.439

\pm

.01

.213

\pm

.05

.250

\pm

.02

.150

\pm

.02

.373

\pm

.01

.224

\pm

.03

.201

\pm

.03

.170

\pm

.01

.097

\pm

.02

.193

\pm

.03

.333

\pm

.03

Ministral-3-8B

.463

\pm

.03

.433

\pm

.02

.199

\pm

.04

.402

\pm

.01

.200

\pm

.03

.247

\pm

.02

.180

\pm

.01

.523

\pm

.03

.272

\pm

.01

.143

\pm

.02

.176

\pm

.01

.113

\pm

.02

.298

\pm

.02

.454

\pm

.03

Qwen2.5-VL-7B

.556

\pm

.02

.496

\pm

.02

.439

\pm

.03

.672

\pm

.01

.497

\pm

.01

.340

\pm

.02

.208

\pm

.03

.649

\pm

.03

.280

\pm

.01

.309

\pm

.01

.321

\pm

.03

.114

\pm

.01

.329

\pm

.01

.478

\pm

.04

Qwen2.5-VL-32B

.559

\pm

.01

.549

\pm

.03

.533

\pm

.00

.711

\pm

.04

.630

\pm

.03

.426

\pm

.00

.327

\pm

.02

.708

\pm

.03

.433

\pm

.00

.337

\pm

.02

.367

\pm

.02

.274

\pm

.04

.432

\pm

.02

.557

\pm

.02

108

Pixtral-12B

.572

\pm

.01

.556

\pm

.01

.393

\pm

.02

.577

\pm

.02

.471

\pm

.04

.360

\pm

.01

.233

\pm

.02

.579

\pm

.01

.392

\pm

.03

.271

\pm

.01

.299

\pm

.00

.154

\pm

.01

.271

\pm

.02

.502

\pm

.01

Table 1: What it costs to deliver the evidence across turns. Consensus final-turn accuracy.

	answer accuracy			evidence selection					error modes			joint
Model	rgb	all-lyr	oracle	Precision	Recall	F1-score	Ex Match	Comp.	Exc.	Dist.	Inv.	Acc.
Q2.5-VL-3B	.408 $\pm$ .02	.371 $\pm$ .03	.303 $\pm$ .01	.561 $\pm$ .01	.522 $\pm$ .01	.503 $\pm$ .01	.101 $\pm$ .01	.279 $\pm$ .00	.387 $\pm$ .02	.520 $\pm$ .02	.137 $\pm$ .01	.048 $\pm$ .01
Min-3-8B	.463 $\pm$ .03	.420 $\pm$ .02	.267 $\pm$ .02	.617 $\pm$ .01	.565 $\pm$ .01	.562 $\pm$ .01	.174 $\pm$ .00	.295 $\pm$ .01	.318 $\pm$ .01	.409 $\pm$ .02	.159 $\pm$ .02	.080 $\pm$ .01
Q2.5-VL-7B	.556 $\pm$ .02	.463 $\pm$ .02	.372 $\pm$ .01	.469 $\pm$ .01	.364 $\pm$ .02	.385 $\pm$ .02	.105 $\pm$ .01	.154 $\pm$ .02	.395 $\pm$ .02	.327 $\pm$ .02	.251 $\pm$ .01	.059 $\pm$ .01
Q2.5-VL-32B	.559 $\pm$ .01	.629 $\pm$ .01	.474 $\pm$ .02	.519 $\pm$ .01	.437 $\pm$ .02	.447 $\pm$ .01	.126 $\pm$ .02	.205 $\pm$ .03	.455 $\pm$ .01	.314 $\pm$ .02	.413 $\pm$ .03	.083 $\pm$ .02
Pixtral-12B	.572 $\pm$ .01	.478 $\pm$ .02	.396 $\pm$ .03	.649 $\pm$ .03	.494 $\pm$ .01	.531 $\pm$ .02	.168 $\pm$ .02	.230 $\pm$ .01	.324 $\pm$ .03	.383 $\pm$ .02	.092 $\pm$ .02	.087 $\pm$ .02

Table 2: What models can do when evidence is handed to them.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Question type	$n$	full	concat-q	sharded-q	concat-v	sharded-v	recap-v	all-lyr	oracle
Direct ask	83	.663	.632	.513	.538	.376	.693	.650	.573
Attribute binding	65	.601	.565	.434	.507	.398	.682	.604	.549
Occlusion	17	.594	.590	.412	.346	.248	.624	.560	.443
Counting	44	.642	.539	.492	.298	.235	.641	.604	.342
Spatial	54	.524	.513	.527	.280	.252	.658	.496	.305
Multi-hop	37	.550	.495	.293	.372	.282	.615	.546	.394

Appendix

Table 3: Accuracy by question type and delivery condition.

Figure 6: Response length and accuracy across successive answer attempts. The dashed line is the single-turn baseline for the same questions. Attempts get terser, and accuracy peaks mid-conversation.

Figure 7: Aptitude against unreliability. Hollow markers deliver the evidence in one turn, filled markers shard it. An arrow joins the two states of one model.

	first answer attempt					commits
Model	earliest	early	midway	late	latest	share
	0–20%	20–40%	40–60%	60–80%	80–100%
Q2.5-VL-3B	.206	.220	.232	.261	.203	51%
Q3-VL-4B	.417	.561	.545	.514	.500	63%
Q2.5-VL-7B	.332	.393	.398	.390	.333	53%
Q3-VL-8B	.502	.626	.657	.646	.574	45%

Appendix

Table 4: When models first attempt an answer.

	evidence in one turn					evidence across turns
Model	shortest	short	median	long	longest	shortest	short	median	long	longest
Q2.5-VL-3B	.311	.365	.349	.369	.253	.270	.249	.227	.236	.218
Q3-VL-4B	.649	.706	.715	.647	.451	.592	.569	.564	.534	.474
Q2.5-VL-7B	.478	.565	.554	.403	.336	.412	.412	.415	.381	.327
Q3-VL-8B	.696	.676	.732	.694	.521	.649	.657	.623	.622	.557
IVL3.5-8B	.524	.468	.512	.546	.407	.458	.431	.385	.426	.448

Appendix

Table 5: Verbosity: Accuracy by evidence length and delivery mode. Models are grouped by the length of the evidence they receive, either in one turn or spread across multiple turns.

	ungrounded share			accuracy
Model	Visual	Layered	Order	as scored	grounded only
Q2.5-VL-3B	.603	.556	.606	.223	.090
Q3-VL-4B	.397	.369	.274	.482	.312
Q2.5-VL-7B	.454	.404	.376	.338	.196
Q3-VL-8B	.326	.291	.221	.573	.411
IVL3.5-8B	.478	.260	.377	.403	.243

Appendix

Table 6: Ungrounded answers remain common across delivery settings. Left : reports the share of responses produced before the required evidence was available; Right : compares standard accuracy with accuracy restricted to grounded answers.

Layers	Questions	full	$\Delta$ concat-v	$\Delta$ oracle
2	11	.606	$-.050$	$-.003$
3	100	.597	$-.149$	$-.128$
4	110	.611	$-.193$	$-.155$
5	51	.594	$-.224$	$-.156$
6	28	.616	$-.269$	$-.229$

Appendix

Table 7: Loss by scene depth. Both losses grow with the number of layers, and neither depends on a particular granularity.

Condition	Relational	Single-object	Extra loss	$95\%$ CI
concat-v	$-.259$	$-.126$	$-.133$	$[-.189,-.074]$
oracle	$-.228$	$-.081$	$-.147$	$[-.202,-.089]$

Appendix

Table 8: Relational questions lose more under visual decomposition. Extra loss is the difference between relational and single-object degradation.

Figure 8: The loss is concentrated, and it scales with depth. Left: questions ranked by loss against the running total of that loss. Right: both losses by the number of layers a scene decomposes into, with

95\%

scene-level bootstrap intervals.

Figure 9: Restoring a fragmented stream overshoots the baseline. Left: the cost of fragmenting each stream and the recovery when it is restored, against full . Right: the two overshoots per model. Points on the diagonal mean the final turn is worth the same either way.

Panel of Figure 10 — Figure 10: The scene and its four layers. Left to right: the composed scene, then layer $0$ (background, not needed), layer $1$ (the worker, the reference landmark), layer $2$ (the bottle, the answer), and layer $3$ (the man taking a selfie, the relational bridge). Every layer render is used exactly as shown.

Shard	Text	Supported by
1	can you help me identify a particular thing in this image?	—
2	The answer should name an object.	2
3	The target object is positioned to the right of the man.	2, 3
4	The referenced man is taking a selfie.	3
5	The target object is directly in front of the worker.	1, 2
6	The referenced worker is wearing a white helmet and an orange-red	1

Appendix

Table 9: The six question shards, and the layers each one depends on. A shard with no supporting layer is asking about the question rather than the image.

Condition	Turns	What the model receives
The question arrives in parts, the scene stays whole
full	1	$Q$ with $I$
concat-q	1	$q_{1}q_{2}q_{3}q_{4}q_{5}q_{6}$ with $I$
sharded-q	6	$q_{1}$ with $I$ ; then $q_{2}$ , $q_{3}$ , $q_{4}$ , $q_{5}$ , $q_{6}$ , one per turn
recap-q	7	As sharded-q , then a final turn repeating $Q$ with $I$
snowball-q	6	$q_{1}$ with $I$ ; then $q_{1}q_{2}$ ; then $q_{1}q_{2}q_{3}$ ; and so on

Appendix

Table 10: Sixteen delivery schedules for one question.

2 1 + bg	BG background L1 barn	DIRECT ASK 2 layers 5 shards What natural landscape is visible rising behind the barns in the distance? Answer A mountain range. 1 can you help me figure out what is being identified in this image? 2 The answer should name a natural landscape. BG 3 The target is the natural landscape that is visible and rising. 4 It is behind the barns. 5 The barns are in the distance.
3 2 + bg	BG background L1 bird L2 bird	SPATIAL 3 layers 6 shards Where is the brown hawk positioned relative to the white bird with the yellow-and-black beak on the rocky shoreline? Answer The brown hawk is to the left of the white bird. 1 can you help me figure out a spatial property of a particular thing? 2 The answer should report where the thing is located. 3 The target thing is the brown hawk. L2 4 Determine the brown hawk’s position relative to a comparison entity. L1 5 Use the white bird with the yellow-and-black beak as the comparison entity. 6 The white bird with the yellow-and-black beak is on the rocky shoreline. BG
4 3 + bg	BG background L1 dog L2 dog L3 dog	COUNTING 4 layers 6 shards How many dogs in the fenced dirt area are facing generally toward the camera or frontward, and how many are turned backward or away? Answer Two dogs are facing frontward, and one dog is facing backward. 1 can you help me figure out the counts for two groups of subjects? L1 L2 L3 2 Each requested result is a count. 3 The subjects to count are dogs. 4 The dogs are in the fenced dirt area. BG 5 Count the dogs that are facing generally toward the camera or frontward. 6 Include a count of the dogs that are turned backward or away.
5 4 + bg	BG background L1 symbol L2 woman L3 toy L4 man	MULTI-HOP REFERENCE 5 layers 5 shards What object is being held by the person standing to the right of the man who is pointing toward her? Answer A teddy bear. 1 can you help me identify a particular thing in this image? 2 The answer should name an object. L3 3 The target object is being held by a person. L2 4 The person holding the target object is standing to the right of a man. L4 5 The referenced man is pointing toward the person holding the target object.
6 5 + bg	BG background L1 lion L2 lion L3 lion L4 lion L5 lion	COUNTING 6 layers 5 shards How many lions are running across the grassy field in this scene? Answer There are five lions visible in the field… 1 can you help me figure out a quantity for particular animals? L1 L2 L3 L4 L5 2 The requested quantity is a count. 3 The animals to count are lions. 4 Count lions that are running across the grassy field. BG 5 Restrict the count to the lions in this scene.

Appendix

Table 11: Scene depth across the benchmark. One scene per depth, from a single object over a background up to five. Depth is a property of the scene and shard count is a property of the question.

Explore similar work

Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
May 21, 2026Zixuan Lan, Luzhe Sun, Matthew R. Walter +1Weak Visual Grounding Visual Evidence
University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University
Seeing and Solving Are Not Enough for Vision-Language Models
Sep 27, 2026Ziheng Wang, Mingxuan Xie, Yilin Liu +3Multimodal Query State-Tracking
Sun Yat-sen University · Zhejiang University · The Hong Kong University of Science and Technology +2
Ill-Posed by Design: Probing Evidence Use in VLMs
Jun 23, 2026Boaz Meivar, Shaked Perek, Shani Shvartzman +2Prior Knowledge Integration Monocular Depth Estimation
1Tel Aviv University · 2IBM Research

May 21, 2026, cs.CVJ/K move · Enter open · S save

Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?

Zixuan Lan, Luzhe Sun, Matthew R. Walter, Jiawei Zhou

University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University

Weak Visual Grounding Visual Evidence

Open paper AskPDF arXiv

Benchmark accuracy is often implicitly assumed to reflect grounded visual understanding in vision-language models (VLMs), yet it remains unclear to what extent such scores truly reflect reliance on visual evidence. Motivated by a surprising observation that removing a substantial fraction of image tokens only degrades model performance very slightly on a widely used hallucination benchmark, we systematically investigate this mismatch in a set of open-source VLMs. Our analysis spans multiple levels of granularity, spanning global visual degradation, localized occlusion, question reformulation, answer-space expansion, and decision-level analyses beyond standard accuracy. We further complement these behavioral results with a layer-wise analysis of vision-token geometry. Throughout the experiments, we find that although VLMs do incorporate visual input, their predictions are less sensitive to the loss of fine-grained visual evidence that standard accuracy should have suggested. Even when the final prediction remains unchanged, the model's internal support for the correct answer may already be weakened. We further complement a representation-level analysis, which shows increasing similarity among visual tokens in deeper layers, providing a possible explanation for our findings. Together, these results suggest that current benchmarks are not sufficient to reliably evaluate fine-grained visual grounding in VLMs.

Abstract

Explore similar work

Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?

Seeing and Solving Are Not Enough for Vision-Language Models

Ill-Posed by Design: Probing Evidence Use in VLMs