Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysis

Mona Gandhi; Cenk Merih Olcay; Kuan-Chieh Lo; Santiago Castro; Christopher W. Myers; Srinivasan Parthasarathy

cs.CVSep 25, 2026

Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysis

Authors: Mona Gandhi, Cenk Merih Olcay, Kuan-Chieh Lo, Santiago Castro, Christopher W. Myers, Srinivasan Parthasarathy

Organizations: The Ohio State University · Netflix Research

Abstract

Vision-language models (VLMs) often struggle with compositional reasoning tasks, but the reasons for this underperformance remain unclear. A common hypothesis is that models struggle to integrate multiple components, leading to training interventions to improve compositional binding. However, this assumption has never been directly quantified. Existing benchmarks evaluate captions only in their composed form, making it impossible to separate the cost of joint reasoning from the cost of recognizing individual components under increasing load. We introduce COMPASS (COMPositional Analysis of SkillS), a controlled evaluation framework designed to isolate and measure the distinct factors underlying compositional failure. By comparing performance on composed captions with their decomposed counterparts , we directly quantify the cost of compositional integration across 87K image-caption pairs. Across multiple VLMs, this gap is real but partial, accounting for only part of the observed degradation. This motivates a finer-grained investigation into what additional factors govern model behavior. We analyze performance at the level of individual skills: object detection, attribute binding, and relation reasoning, using skill-targeted perturbations across 274K image-caption pairs. We find a consistent skill-specific pattern: each skill degrades primarily with the count of its own primitive type (self-load), while cross-load effects are predominantly positive, suggesting that primitives of different types provide useful grounding context. This pattern holds across standard contrastive encoders, explicitly trained compositional reasoning models, and non-contrastive architectures. These findings show that compositional degradation reflects multiple separable factors that cannot be reduced to joint reasoning alone.

Figures & tables

Structural Level	Composed	Composed/Decomposed	Object	Attribute	Relation
		Evaluation Set
	Ground Truth	Compositional Integration	Skill Load
L3 (OAR)	47K	24K	45K	26K	34K
L2 (OA)	46K	30K	39K	37K	-
L2 (OR)	45K	33K	43K	-	34K

Table 1: COMPASS Statistics. We summarize the total size of our compositionality testbed across complexity levels with ground truth captions and evaluation set sizes for both compositional integration and skill load.

Figure 1: Structured levels in COMPASS. With a subgraph of the scene graph of an input image, we construct captions organized into structured levels based on combinations of primitives (P): L1 (O: objects), L2 (OA/OR: objects with attributes or relations), and L3 (OAR: objects, attributes, and relations). This organization enables controlled evaluation of compositional reasoning by systematically varying object (O), attribute (A), and relation (R) components across levels.

Figure 2: We construct two complementary negative structures to analyze compositional integration and skill load. Left: Composed vs decomposed captions. A composed caption with complexity

N

is paired with

N

hard negatives and decomposed into

N

primitive captions. To enable controlled comparison between joint and independent reasoning, we apply the same perturbation to create negatives for both composed and decomposed. Right: Skill-targeted negatives. To analyze skill load, we isolate one skill at a time and create four hard negatives by modifying that primitive type. The figure shows an example from L3 with complexity

N=5

; this construction generalizes across all structural levels and complexities.

Type	OpenCLIP	SigLIPv2	PE-CLIP	NegCLIP	CE-CLIP	BLIP-L	Qwen3
OAR	0.83 $\pm$ 0.70	2.61 $\pm$ 1.82	1.97 $\pm$ 0.59	3.75 $\pm$ 0.64	-5.66 $\pm$ 3.45	1.38 $\pm$ 1.03	5.34 $\pm$ 4.42
OA	2.39 $\pm$ 1.23	12.65 $\pm$ 2.53	0.94 $\pm$ 1.96	-2.70 $\pm$ 1.83	-20.27 $\pm$ 5.17	1.19 $\pm$ 2.17	11.94 $\pm$ 4.81
OR	17.56 $\pm$ 1.98	8.64 $\pm$ 5.04	2.69 $\pm$ 0.70	5.54 $\pm$ 1.44	-4.96 $\pm$ 3.93	2.30 $\pm$ 2.93	7.31 $\pm$ 5.16

Table 2: Compositional Integration (Aggregated). We report the mean

\pm

standard deviation of the difference between decomposed and composed accuracy (

\Delta

) aggregated across all complexities. Positive values indicate better independent than joint performance, highlighting compositional binding difficulty. Shaded cells indicate models for which composed captions are easier to retrieve than their decomposed counterparts.

Figure 3: Performance on CREPE

Type	Model	Object Negatives			Attribute Negatives			Relation Negatives
		$\beta_{O}$	$\beta_{A}$	$\beta_{R}$	$\beta_{O}$	$\beta_{A}$	$\beta_{R}$	$\beta_{O}$	$\beta_{A}$	$\beta_{R}$
OAR	OpenCLIP	-2.50**	+1.21**	+2.32**	+1.76**	-3.32**	-1.84	+0.60	+1.17**	-2.86**
	SigLIP 2	-2.25**	+0.65*	+1.82	+2.60**	-2.85**	-3.07	+0.70	-0.77**	-1.65*
	PE-CLIP	-2.45**	+0.96**	+1.39	+1.07	-4.13**	-0.50	+0.40	-0.09	-1.91*
	NegCLIP	-1.762**	+0.69	+0.81**	+1.51**	-3.38**	-2.45	+2.31**	+0.13	-7.13**
	CE-CLIP	-1.77**	+0.68**	+2.23*	+0.20**	-2.57**	-2.81	-1.08	+2.31**	-4.89**

Table 3: Skill Load. Values represent change in R@1 (percentage points) per unit increase in primitive count (Eq. 4 ). Negative values indicate degradation, positive values indicate improvement. Shaded cells indicate the dominant source of degradation within each setting. * and ** denote statistical significance at p < 0.005 and p < 0.001, respectively. Object, attribute, and relation self-load are consistently negative across models while cross-load effects are predominantly positive.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: GPT-4o mini prompts used for caption and negative generation. (A) Few-shot prompt for generating natural language captions from scene subgraphs, instructing the model to combine primitives into fluent sentences without introducing new objects or relations. (B-D) Few-shot prompts for generating hard negative candidates for object (B), attribute (C), and relation (D) primitives respectively, eliciting semantically plausible alternatives that are neither synonyms nor identical to the original primitive.

Level	Negative Type	Effect size $r$	Interpretation
L3 (OAR)	Object	0.096	Negligible
L3 (OAR)	Attribute	0.072	Negligible
L3 (OAR)	Relation	0.259	Small
L2 (OA)	Object	0.062	Negligible
L2 (OA)	Attribute	0.099	Negligible
L2 (OR)	Object	0.099	Negligible

Appendix

Table 4: Perplexity effect sizes (rank-biserial

r

) measuring the distributional difference between ground-truth and negative caption perplexities for skill-targeted negatives, evaluated on the hardest negative subset.

Level	Effect size $r$	Interpretation
L3 (OAR)	0.060	Negligible
L2 (OA)	0.152	Small
L2 (OR)	0.071	Negligible

Appendix

Table 5: Perplexity effect sizes (rank-biserial

r

) measuring the distributional difference between ground-truth and negative caption perplexities for composed captions, evaluated on the hardest negative subset.

	OpenCLIP			SigLIP 2			PE-CLIP			NegCLIP			CE-CLIP			BLIP-L			Qwen3
Complexity Level	C	D	$\Delta$	C	D	$\Delta$	C	D	$\Delta$	C	D	$\Delta$	C	D	$\Delta$	C	D	$\Delta$	C	D	$\Delta$
2	55.69	55.26	-0.42	58.70	65.66	6.96	65.48	62.20	-3.28	62.08	56.61	-5.47	62.90	56.14	-6.75	63.51	64.37	0.85	50.38	63.87	13.49
3	43.42	47.55	4.13	41.33	56.19	14.86	54.79	54.23	-0.56	50.40	46.69	-3.71	57.98	42.51	-15.46	47.94	53.14	5.19	34.7	53.43	18.72
4	35.13	37.01	1.87	34.04	47.78	13.75	45.61	44.12	-1.49	39.57	35.47	-4.11	51.76	31.58	-20.17	39.86	44.89	5.02	26.43	43.29	16.86
5	29.61	31.77	2.16	28.77	44.00	15.23	38.23	39.83	1.60	33.12	31.37	-1.75	48.68	26.08	-22.60	34.47	36.72	2.25	19.71	36.64	16.93
6	22.71	25.51	2.79	21.94	36.91	14.96	30.64	31.95	1.31	28.75	25.18	-3.57	44.18	20.54	-23.64	29.01	30.46	1.44	17.29	31.45	14.16

Appendix

Table 6: Compositional Integration L2 Object-Attribute Accuracy (Raw Results). We report the difference between decomposed (

D

) and composed (

C

) accuracy per complexity level, along with the mean and standard deviation of

\Delta

for each model in OA setting. Positive

\Delta

values indicate that model performs better independently than jointly, highlighting compositional binding difficulty.

	OpenCLIP			SigLIP 2			PE-CLIP			NegCLIP			CE-CLIP			BLIP-L			Qwen3
Complexity Level	C	D	$\Delta$	C	D	$\Delta$	C	D	$\Delta$	C	D	$\Delta$	C	D	$\Delta$	C	D	$\Delta$	C	D	$\Delta$
3	28.79	41.34	12.54	18.89	33.73	14.84	34.71	36.30	1.59	41.46	46.17	4.71	49.90	38.49	-11.41	44.23	39.55	-4.69	33.11	45.4	12.29
4	21.39	37.81	16.42	12.71	28.69	15.99	28.00	30.60	2.61	35.01	39.12	4.11	41.24	31.86	-9.38	29.07	34.04	4.97	24.61	40.46	15.84
5	17.80	34.75	16.95	9.21	24.66	15.46	22.85	26.94	4.09	27.79	33.14	5.35	33.55	23.58	-9.96	22.68	28.70	6.02	20.68	33.55	12.87
6	10.13	28.03	17.89	5.85	14.68	8.83	14.53	18.07	3.53	19.63	25.19	5.56	23.89	18.36	-5.53	16.97	19.77	2.80	16.86	24.46	7.59
7	7.50	26.42	18.92	4.76	13.16	8.40	13.10	16.02	2.92	15.12	21.87	6.75	19.94	15.62	-4.32	13.22	17.58	4.36	13.56	22.98	9.41

Appendix

Table 7: Compositional Integration L2 Object-Relation Accuracy (Raw Results). We report the difference between decomposed (

D

) and composed (

C

) accuracy per complexity level, along with the mean and standard deviation of

\Delta

for each model in OR setting. Positive

\Delta

values indicate that model performs better independently than jointly, highlighting compositional binding difficulty.

	OpenCLIP			SigLIP 2			PE-CLIP			NegCLIP			CE-CLIP			BLIP-L			Qwen3
Complexity Level	C	D	$\Delta$	C	D	$\Delta$	C	D	$\Delta$	C	D	$\Delta$	C	D	$\Delta$	C	D	$\Delta$	C	D	$\Delta$
4	20.54	19.85	-0.69	18.99	25.48	6.48	25.56	28.06	2.49	33.78	36.66	2.88	41.03	31.15	-9.88	33.47	33.08	-0.38	25.1	35.98	10.87
5	16.83	18.34	1.51	17.60	22.05	4.44	23.79	26.31	2.52	27.14	29.94	2.79	34.80	23.10	-11.69	24.85	27.14	2.29	19.44	31.51	12.06
6	10.53	12.31	1.78	12.98	15.63	2.64	16.69	18.80	2.11	20.77	24.86	4.08	25.06	17.21	-7.84	19.19	20.68	1.49	16.21	23.96	7.74
7	7.72	8.59	0.86	9.11	11.46	2.35	11.51	13.29	1.77	15.06	19.52	4.46	20.10	15.45	-4.65	14.58	16.69	2.11	14.21	21.36	7.15
8	5.73	6.52	0.79	6.71	8.69	1.97	9.04	11.70	2.66	12.05	16.05	4.00	16.79	12.64	-4.15	11.16	14.03	2.86	12.7	17.79	5.09

Appendix

Table 8: Compositional Integration L3 Object-Attribute-Relation (Raw Results). We report the difference between decomposed (

D

) and composed (

C

) accuracy per complexity level, along with the mean and standard deviation of

\Delta

for each model in OAR setting. Positive

\Delta

values indicate that model performs better independently than jointly, highlighting compositional binding difficulty.

Panel of Figure 5 — Figure 5: Skill load analysis on L2 (OA) captions for SigLIPv2 and PE-CLIP. Each model shows four panels: object negatives (top) and attribute negatives (bottom), each varying attribute count (left) and object count (right) while holding the other fixed.

Panel of Figure 6 — Figure 6: Skill load analysis on L2 (OA) captions for OpenCLIP, NegCLIP, and CE-CLIP. Layout follows Figure 5 .

Panel of Figure 7 — Figure 7: Skill load analysis on L2 (OA) captions for BLIP and Qwen3-VL-Embedding. Layout follows Figure 5 .

Compositional Reasoning in VLMs

See all 23 papers

How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA
Jul 17, 2026Navya Gupta, Bingjie Xu, Avinash Anand +2Visual Question Answering VLM Interpretability
Singapore Institute of Technology Singapore · NVIDIA Singapore
Discovering Failure Modes in Vision-Language Models using RL
Apr 6, 2026Kanishk Jain, Qian Yang, Shravan Nayak +3VLM Evaluation VLM Robustness
Mila – Qu´ebec AI Institute · Universit´e de Montr´eal · Michigan State University +2
Inference-Time Structural Reasoning for Compositional Vision-Language Understanding
Mar 28, 2026Amartya BhattacharyaCompositional Reasoning in VLMs VLM Reasoning
Biomedical Data Science, Geisel School of Medicine, Dartmouth College

Jul 17, 2026J/K move · Enter open · S save

How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA

Navya Gupta, Bingjie Xu, Avinash Anand, Timothy Liu+1

Singapore Institute of Technology Singapore · NVIDIA Singapore

Visual Question Answering VLM Interpretability Vision-Language Grounding VLM Reasoning

Open paper AskPDF arXiv

Compositional visual question answering requires Vision-Language Models (VLMs) to execute multiple reasoning operations like object selection, spatial relation resolution, and attribute verification. Despite strong aggregate performance, the mechanistic basis of VLM failures on this task remains underexplored. To address this gap, we analyze vision-operation misalignment in VLMs by examining how failures relate to specific reasoning operations and the internal computational pathways through which they arise and propagate. We introduce an Operation-centric mechanistic framework that decomposes VLM failures by both the reasoning operation where they originate and the internal computational pathway through which they propagate. Our analysis reveals four mechanistically distinct failure modes: grounding failure, reasoning failure, attribute extraction failure, and language prior dominance failure. Each characterized by a unique relationship between visual grounding strength and answer correctness. Through three complementary causal interventions applied across all transformer layers, we further demonstrate a pathway dissociation: grounding failures route exclusively through the feedforward network, reasoning failures route through late-layer attention, and attribute extraction failures localize to the answer-position feedforward computation. This dissociation demonstrates that different failure types require fundamentally different corrective strategies, providing a principled foundation for targeted improvements to VLM reliability in multimedia reasoning.

Abstract

Compositional Reasoning in VLMs

Explore similar work

How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA

Discovering Failure Modes in Vision-Language Models using RL

Inference-Time Structural Reasoning for Compositional Vision-Language Understanding