Inference-Time Structural Reasoning for Compositional Vision-Language Understanding

Amartya Bhattacharya

cs.CVMar 28, 2026

Inference-Time Structural Reasoning for Compositional Vision-Language Understanding

Authors: Amartya Bhattacharya

Organizations: Biomedical Data Science, Geisel School of Medicine, Dartmouth College

Abstract

Vision-language models (VLMs) excel at image-text retrieval yet persistently fail at compositional reasoning, distinguishing captions that share the same words but differ in relational structure. We present, a unified evaluation and augmentation framework benchmarking four architecturally diverse VLMs,CLIP, BLIP, LLaVA, and Qwen3-VL-8B-Thinking,on the Winoground benchmark under plain and scene-graph-augmented regimes. We introduce a dependency-based TextSceneGraphParser (spaCy) extracting subject-relation-object triples, and a Graph Asymmetry Scorer using optimal bipartite matching to inject structural relational priors. Caption ablation experiments (subject-object masking and swapping) reveal that Qwen3-VL-8B-Thinking achieves a group score of 62.75, far above all encoder-based models, while a proposed multi-turn SG filtering strategy further lifts it to 66.0, surpassing prior open-source state-of-the-art. We analyze the capability augmentation tradeoff and find that SG augmentation benefits already capable models while providing negligible or negative gains for weaker baselines. Code: https://github.com/amartyacodes/Inference-Time-Structural-Reasoning-for-Compositional-Vision-Language-Understanding

Figures & tables

Panel of Figure 1 — Figure 1: Winoground examples. Captions share the same words arranged differently; images share the same visual elements interacting differently. Models must correctly pair each caption with its image.

Figure 2: TextSceneGraphParser architecture (spaCy rule-based). Five parallel extraction rules produce triples that are deduplicated and cached per caption string.

Figure 3: Caption ablation pipeline. spaCy identifies semantic spans offline; captions are transformed (mask/swap subject, object, or both) and evaluated without SG injection.

Model	Architecture	Scoring
CLIP ViT-B/32	Dual encoder	CLS cosine
BLIP ITM Base	Cross-attention	$P(\text{match})$
LLaVA 1.5-7B	Generative decoder	$P(\text{yes})$
Qwen3-VL-8B	Generative decoder (MoE)	EOS cosine

Table 1: Base scoring functions per model.

	CLIP ViT-B/32			BLIP ITM			LLaVA 1.5-7B			Qwen3-VL-8B-T
Condition	Text	Img	Grp	Text	Img	Grp	Text	Img	Grp	Text	Img	Grp
Random Chance	.250	.250	.063	.250	.250	.063	.250	.250	.063	.250	.250	.063
Original Caption	.308	.113	.090	.485	.240	.200	.453	.368	.275	.745	.720	.628
Mask Subjects	.243	.118	.078/ $-$ .013	.370	.213	.158/ $-$ .043	.350	.250	.180/ $-$ .095	.573	.543	.450/ $-$ .178
Mask Objects	.163	.095	.053/ $-$ .038	.258	.178	.133/ $-$ .068	.205	.173	.090/ $-$ .185	.420	.460	.338/ $-$ .290
Mask Subj+Obj	.110	.060	.025/ $-$ .065	.135	.120	.073/ $-$ .128	.115	.095	.045/ $-$ .230	.270	.305	.223/ $-$ .405

Table 2: Caption ablation study across all four models. Each cell shows group score /

\Delta_{\text{grp}}

vs. original caption (negative = degradation). Text and Image scores are also reported for completeness.

Strategy	Model	Txt	Img	Grp
Random	,	.250	.250	.063
clip	CLIP ViT-B/32	.308	.113	.090
clip+SG	CLIP ViT-B/32	.308	.113	.090
blip	BLIP ITM	.485	.240	.200
blip+SG	BLIP ITM	.485	.240	.200
llava	LLaVA 1.5-7B	.453	.368	.275

Table 3: Winoground results. Best per-model in bold . Random: Txt/Img = 0.25, Grp = 0.063.

Method	Text	Image	Group
MTurk Human	89.5	88.5	85.5
Random Chance	25.0	25.0	6.3
CLIP (ViT-B/32)	30.8	10.5	8.0
BLIP (ITM Large)	35.0	22.0	22.0
LLaVA-1.5-13B	51.5	50.5	36.5
CECE (LLaVA-1.5+1.6)	55.0	61.3	47.5

Table 4: Comparison with published methods on Winoground (%).

Figure 4: Qwen3-VL-Thinking pipeline with multi-turn SG filtering. Turn 1 identifies visually relevant SG relations; Turn 2 uses those filtered triples for the final match decision.

Figure 5: Scoring mechanism: last-token logits for “yes”/“no” are softmax-normalized to yield $P(\text{yes}\mid I,c)$ .

Compositional Reasoning in VLMs

See all 23 papers

CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning
Sep 8, 2026SeongJun Jeong, Minjoon Jung, Woo Suk Choi +2VLM Robustness Contrastive Learning
Seoul National University
Composition, Not Conversation: VLMs Lose the Scene, Not the Thread
Sep 29, 2026L. D. M. S. Sai Teja, Ufaq Khan, N. Siva Gopala Krishna +5VLM Evaluation Visual Question Answering
NIT Silchar · MBZUAI · BML Munjal University +1
HyperVis: Continuous Latent Visual Relational Graphs on the Lorentz Hyperboloid for Compositional Reasoning
Jun 4, 2026Moshiur Farazi, Sameera Ramasinghe, Mahbub Ahmed Turza +1Hyperbolic Representation Learning Visual Reasoning
Data Science and AI, University of Doha for Science and Technology, Qatar · Pluralis Research, Australia · Department of Electrical and Computer Engineering, North South University, Bangladesh

Sep 8, 2026J/K move · Enter open · S save

CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning

SeongJun Jeong, Minjoon Jung, Woo Suk Choi, Youwon Jang+1

Seoul National University

VLM Robustness Contrastive Learning Compositional Reasoning in VLMs VLM Reasoning

Open paper AskPDF arXiv

Vision-language models (VLMs) demonstrate strong performance across compositional reasoning benchmarks, which require reasoning over semantic perturbations of objects, attributes, relations, and their interactions. However, our controlled analysis reveals that existing compositionality-aware VLMs exhibit element-specific biases, often underperforming vanilla CLIP on certain compositional elements. To address this, we propose Compositional Scene Graph-guided CLIP (CS-CLIP), which uses scene graphs to identify compositional elements and construct structured negatives via selective masking. We further retain negatives that are most contradictory to the original caption, forcing the model to rely on compositional structure rather than surface cues. CS-CLIP achieves state-of-the-art compositional reasoning with robust performance across compositional elements. It also preserves general vision-language capabilities such as cross-modal retrieval and downstream visual reasoning, while requiring fewer training samples than prior methods.

Abstract

Compositional Reasoning in VLMs

Explore similar work

CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning

Composition, Not Conversation: VLMs Lose the Scene, Not the Thread

HyperVis: Continuous Latent Visual Relational Graphs on the Lorentz Hyperboloid for Compositional Reasoning