Vision-language models (VLMs) often struggle with compositional reasoning tasks, but the reasons for this underperformance remain unclear. A common hypothesis is that models struggle to integrate multiple components, leading to training interventions to improve compositional binding. However, this assumption has never been directly quantified. Existing benchmarks evaluate captions only in their composed form, making it impossible to separate the cost of joint reasoning from the cost of recognizing individual components under increasing load. We introduce COMPASS (COMPositional Analysis of SkillS), a controlled evaluation framework designed to isolate and measure the distinct factors underlying compositional failure. By comparing performance on composed captions with their decomposed counterparts , we directly quantify the cost of compositional integration across 87K image-caption pairs. Across multiple VLMs, this gap is real but partial, accounting for only part of the observed degradation. This motivates a finer-grained investigation into what additional factors govern model behavior. We analyze performance at the level of individual skills: object detection, attribute binding, and relation reasoning, using skill-targeted perturbations across 274K image-caption pairs. We find a consistent skill-specific pattern: each skill degrades primarily with the count of its own primitive type (self-load), while cross-load effects are predominantly positive, suggesting that primitives of different types provide useful grounding context. This pattern holds across standard contrastive encoders, explicitly trained compositional reasoning models, and non-contrastive architectures. These findings show that compositional degradation reflects multiple separable factors that cannot be reduced to joint reasoning alone.
Figures & tables
Evaluation Set
Ground Truth
Compositional Integration
Skill Load
Structural Level
Composed
Composed/Decomposed
Object
Attribute
Relation
L3 (OAR)
47K
24K
45K
26K
34K
L2 (OA)
46K
30K
39K
37K
-
L2 (OR)
45K
33K
43K
-
34K
Table 1: COMPASS Statistics. We summarize the total size of our compositionality testbed across complexity levels with ground truth captions and evaluation set sizes for both compositional integration and skill load.
Figure 1: Structured levels in COMPASS. With a subgraph of the scene graph of an input image, we construct captions organized into structured levels based on combinations of primitives (P): L1 (O: objects), L2 (OA/OR: objects with attributes or relations), and L3 (OAR: objects, attributes, and relations). This organization enables controlled evaluation of compositional reasoning by systematically varying object (O), attribute (A), and relation (R) components across levels.
Figure 2: We construct two complementary negative structures to analyze compositional integration and skill load. Left: Composed vs decomposed captions. A composed caption with complexity N is paired with N hard negatives and decomposed into N primitive captions. To enable controlled comparison between joint and independent reasoning, we apply the same perturbation to create negatives for both composed and decomposed. Right: Skill-targeted negatives. To analyze skill load, we isolate one skill at a time and create four hard negatives by modifying that primitive type. The figure shows an example from L3 with complexity N=5 ; this construction generalizes across all structural levels and complexities.
Type
OpenCLIP
SigLIPv2
PE-CLIP
NegCLIP
CE-CLIP
BLIP-L
Qwen3
OAR
0.83 ± 0.70
2.61 ± 1.82
1.97 ± 0.59
3.75 ± 0.64
-5.66 ± 3.45
1.38 ± 1.03
5.34 ± 4.42
OA
2.39 ± 1.23
12.65 ± 2.53
0.94 ± 1.96
-2.70 ± 1.83
-20.27 ± 5.17
1.19 ± 2.17
11.94 ± 4.81
OR
17.56 ± 1.98
8.64 ± 5.04
2.69 ± 0.70
5.54 ± 1.44
-4.96 ± 3.93
2.30 ± 2.93
7.31 ± 5.16
Table 2: Compositional Integration (Aggregated). We report the mean ± standard deviation of the difference between decomposed and composed accuracy ( Δ ) aggregated across all complexities. Positive values indicate better independent than joint performance, highlighting compositional binding difficulty. Shaded cells indicate models for which composed captions are easier to retrieve than their decomposed counterparts.
Figure 3: Performance on CREPE
Type
Model
Object Negatives
Attribute Negatives
Relation Negatives
βO
βA
βR
βO
βA
βR
βO
βA
βR
OAR
OpenCLIP
-2.50**
+1.21**
+2.32**
+1.76**
-3.32**
-1.84
+0.60
+1.17**
-2.86**
SigLIP 2
-2.25**
+0.65*
+1.82
+2.60**
-2.85**
-3.07
+0.70
-0.77**
-1.65*
PE-CLIP
-2.45**
+0.96**
+1.39
+1.07
-4.13**
-0.50
+0.40
-0.09
-1.91*
NegCLIP
-1.762**
+0.69
+0.81**
+1.51**
-3.38**
-2.45
+2.31**
+0.13
-7.13**
CE-CLIP
-1.77**
+0.68**
+2.23*
+0.20**
-2.57**
-2.81
-1.08
+2.31**
-4.89**
Table 3: Skill Load. Values represent change in R@1 (percentage points) per unit increase in primitive count (Eq. 4 ). Negative values indicate degradation, positive values indicate improvement. Shaded cells indicate the dominant source of degradation within each setting. * and ** denote statistical significance at p < 0.005 and p < 0.001, respectively. Object, attribute, and relation self-load are consistently negative across models while cross-load effects are predominantly positive.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: GPT-4o mini prompts used for caption and negative generation. (A) Few-shot prompt for generating natural language captions from scene subgraphs, instructing the model to combine primitives into fluent sentences without introducing new objects or relations. (B-D) Few-shot prompts for generating hard negative candidates for object (B), attribute (C), and relation (D) primitives respectively, eliciting semantically plausible alternatives that are neither synonyms nor identical to the original primitive.
Level
Negative Type
Effect size r
Interpretation
L3 (OAR)
Object
0.096
Negligible
L3 (OAR)
Attribute
0.072
Negligible
L3 (OAR)
Relation
0.259
Small
L2 (OA)
Object
0.062
Negligible
L2 (OA)
Attribute
0.099
Negligible
L2 (OR)
Object
0.099
Negligible
Appendix
Table 4: Perplexity effect sizes (rank-biserial r ) measuring the distributional difference between ground-truth and negative caption perplexities for skill-targeted negatives, evaluated on the hardest negative subset.
Level
Effect size r
Interpretation
L3 (OAR)
0.060
Negligible
L2 (OA)
0.152
Small
L2 (OR)
0.071
Negligible
Appendix
Table 5: Perplexity effect sizes (rank-biserial r ) measuring the distributional difference between ground-truth and negative caption perplexities for composed captions, evaluated on the hardest negative subset.
OpenCLIP
SigLIP 2
PE-CLIP
NegCLIP
CE-CLIP
BLIP-L
Qwen3
Complexity Level
C
D
Δ
C
D
Δ
C
D
Δ
C
D
Δ
C
D
Δ
C
D
Δ
C
D
Δ
2
55.69
55.26
-0.42
58.70
65.66
6.96
65.48
62.20
-3.28
62.08
56.61
-5.47
62.90
56.14
-6.75
63.51
64.37
0.85
50.38
63.87
13.49
3
43.42
47.55
4.13
41.33
56.19
14.86
54.79
54.23
-0.56
50.40
46.69
-3.71
57.98
42.51
-15.46
47.94
53.14
5.19
34.7
53.43
18.72
4
35.13
37.01
1.87
34.04
47.78
13.75
45.61
44.12
-1.49
39.57
35.47
-4.11
51.76
31.58
-20.17
39.86
44.89
5.02
26.43
43.29
16.86
5
29.61
31.77
2.16
28.77
44.00
15.23
38.23
39.83
1.60
33.12
31.37
-1.75
48.68
26.08
-22.60
34.47
36.72
2.25
19.71
36.64
16.93
6
22.71
25.51
2.79
21.94
36.91
14.96
30.64
31.95
1.31
28.75
25.18
-3.57
44.18
20.54
-23.64
29.01
30.46
1.44
17.29
31.45
14.16
Appendix
Table 6: Compositional Integration L2 Object-Attribute Accuracy (Raw Results). We report the difference between decomposed ( D ) and composed ( C ) accuracy per complexity level, along with the mean and standard deviation of Δ for each model in OA setting. Positive Δ values indicate that model performs better independently than jointly, highlighting compositional binding difficulty.
OpenCLIP
SigLIP 2
PE-CLIP
NegCLIP
CE-CLIP
BLIP-L
Qwen3
Complexity Level
C
D
Δ
C
D
Δ
C
D
Δ
C
D
Δ
C
D
Δ
C
D
Δ
C
D
Δ
3
28.79
41.34
12.54
18.89
33.73
14.84
34.71
36.30
1.59
41.46
46.17
4.71
49.90
38.49
-11.41
44.23
39.55
-4.69
33.11
45.4
12.29
4
21.39
37.81
16.42
12.71
28.69
15.99
28.00
30.60
2.61
35.01
39.12
4.11
41.24
31.86
-9.38
29.07
34.04
4.97
24.61
40.46
15.84
5
17.80
34.75
16.95
9.21
24.66
15.46
22.85
26.94
4.09
27.79
33.14
5.35
33.55
23.58
-9.96
22.68
28.70
6.02
20.68
33.55
12.87
6
10.13
28.03
17.89
5.85
14.68
8.83
14.53
18.07
3.53
19.63
25.19
5.56
23.89
18.36
-5.53
16.97
19.77
2.80
16.86
24.46
7.59
7
7.50
26.42
18.92
4.76
13.16
8.40
13.10
16.02
2.92
15.12
21.87
6.75
19.94
15.62
-4.32
13.22
17.58
4.36
13.56
22.98
9.41
Appendix
Table 7: Compositional Integration L2 Object-Relation Accuracy (Raw Results). We report the difference between decomposed ( D ) and composed ( C ) accuracy per complexity level, along with the mean and standard deviation of Δ for each model in OR setting. Positive Δ values indicate that model performs better independently than jointly, highlighting compositional binding difficulty.
OpenCLIP
SigLIP 2
PE-CLIP
NegCLIP
CE-CLIP
BLIP-L
Qwen3
Complexity Level
C
D
Δ
C
D
Δ
C
D
Δ
C
D
Δ
C
D
Δ
C
D
Δ
C
D
Δ
4
20.54
19.85
-0.69
18.99
25.48
6.48
25.56
28.06
2.49
33.78
36.66
2.88
41.03
31.15
-9.88
33.47
33.08
-0.38
25.1
35.98
10.87
5
16.83
18.34
1.51
17.60
22.05
4.44
23.79
26.31
2.52
27.14
29.94
2.79
34.80
23.10
-11.69
24.85
27.14
2.29
19.44
31.51
12.06
6
10.53
12.31
1.78
12.98
15.63
2.64
16.69
18.80
2.11
20.77
24.86
4.08
25.06
17.21
-7.84
19.19
20.68
1.49
16.21
23.96
7.74
7
7.72
8.59
0.86
9.11
11.46
2.35
11.51
13.29
1.77
15.06
19.52
4.46
20.10
15.45
-4.65
14.58
16.69
2.11
14.21
21.36
7.15
8
5.73
6.52
0.79
6.71
8.69
1.97
9.04
11.70
2.66
12.05
16.05
4.00
16.79
12.64
-4.15
11.16
14.03
2.86
12.7
17.79
5.09
Appendix
Table 8: Compositional Integration L3 Object-Attribute-Relation (Raw Results). We report the difference between decomposed ( D ) and composed ( C ) accuracy per complexity level, along with the mean and standard deviation of Δ for each model in OAR setting. Positive Δ values indicate that model performs better independently than jointly, highlighting compositional binding difficulty.
Figure 5: Skill load analysis on L2 (OA) captions for SigLIPv2 and PE-CLIP. Each model shows four panels: object negatives (top) and attribute negatives (bottom), each varying attribute count (left) and object count (right) while holding the other fixed.
Figure 6: Skill load analysis on L2 (OA) captions for OpenCLIP, NegCLIP, and CE-CLIP. Layout follows Figure 5 .
Figure 7: Skill load analysis on L2 (OA) captions for BLIP and Qwen3-VL-Embedding. Layout follows Figure 5 .
Compositional visual question answering requires Vision-Language Models (VLMs) to execute multiple reasoning operations like object selection, spatial relation resolution, and attribute verification. Despite strong aggregate performance, the mechanistic basis of VLM failures on this task remains underexplored. To address this gap, we analyze vision-operation misalignment in VLMs by examining how failures relate to specific reasoning operations and the internal computational pathways through which they arise and propagate. We introduce an Operation-centric mechanistic framework that decomposes VLM failures by both the reasoning operation where they originate and the internal computational pathway through which they propagate. Our analysis reveals four mechanistically distinct failure modes: grounding failure, reasoning failure, attribute extraction failure, and language prior dominance failure. Each characterized by a unique relationship between visual grounding strength and answer correctness. Through three complementary causal interventions applied across all transformer layers, we further demonstrate a pathway dissociation: grounding failures route exclusively through the feedforward network, reasoning failures route through late-layer attention, and attribute extraction failures localize to the answer-position feedforward computation. This dissociation demonstrates that different failure types require fundamentally different corrective strategies, providing a principled foundation for targeted improvements to VLM reliability in multimedia reasoning.
Navya Gupta, Bingjie Xu, Avinash Anand +2
Singapore Institute of Technology Singapore · NVIDIA Singapore
Vision-language Models (VLMs), despite achieving strong performance on multimodal benchmarks, often misinterpret straightforward visual concepts that humans identify effortlessly, such as counting, spatial reasoning, and viewpoint understanding. Previous studies manually identified these weaknesses and found that they often stem from deficits in specific skills. However, such manual efforts are costly, unscalable, and subject to human bias, which often overlooks subtle details in favour of salient objects, resulting in an incomplete understanding of a model's vulnerabilities. To address these limitations, we propose a Reinforcement Learning (RL)-based framework to automatically discover the failure modes or blind spots of any ``candidate VLM'' on a given data distribution without human intervention. Our framework trains a questioner agent that adaptively generates queries based on the candidate VLM's responses to elicit incorrect answers. Our approach increases question complexity by focusing on fine-grained visual details and distinct skill compositions as training progresses, consequently identifying novel failure modes in which VLMs struggle. We demonstrate the broad applicability of our framework by showcasing its generalizability across various model combinations.
Kanishk Jain, Qian Yang, Shravan Nayak +3
Mila – Qu´ebec AI Institute · Universit´e de Montr´eal · Michigan State University +2
Vision-language models (VLMs) excel at image-text retrieval yet persistently fail at compositional reasoning, distinguishing captions that share the same words but differ in relational structure. We present, a unified evaluation and augmentation framework benchmarking four architecturally diverse VLMs,CLIP, BLIP, LLaVA, and Qwen3-VL-8B-Thinking,on the Winoground benchmark under plain and scene-graph-augmented regimes. We introduce a dependency-based TextSceneGraphParser (spaCy) extracting subject-relation-object triples, and a Graph Asymmetry Scorer using optimal bipartite matching to inject structural relational priors. Caption ablation experiments (subject-object masking and swapping) reveal that Qwen3-VL-8B-Thinking achieves a group score of 62.75, far above all encoder-based models, while a proposed multi-turn SG filtering strategy further lifts it to 66.0, surpassing prior open-source state-of-the-art. We analyze the capability augmentation tradeoff and find that SG augmentation benefits already capable models while providing negligible or negative gains for weaker baselines. Code: https://github.com/amartyacodes/Inference-Time-Structural-Reasoning-for-Compositional-Vision-Language-Understanding
Amartya Bhattacharya
Biomedical Data Science, Geisel School of Medicine, Dartmouth College