CompoSET: A Controlled Single-Edit Benchmark for Vision-Language Compositionality

Laure Delisle; Pietro Perona

cs.CVOct 30, 2026

CompoSET: A Controlled Single-Edit Benchmark for Vision-Language Compositionality

Authors: Laure Delisle, Pietro Perona

Organizations: California Institute of Technology

Abstract

Vision-language compositionality is the ability to identify which properties, relations, quantities, and states bind to which entities in both an image and a caption, and to match those bindings across modalities. Existing benchmarks evaluate this ability through image-text matching: a model ranks candidate image-caption match similarities and succeeds when the correct match comes on top. In such datasets, paired images differ by more than a single binding: scene, layout, and content all differ, with 97% of pixels changing on average. Such pairs do not isolate compositionality: results are confounded by global image differences. We introduce Compositional Single-EdiT (CompoSET), a benchmark dataset in which each base image-caption pair has an edited counterpart, or foil, that preserves the scene but differs by a single controlled edit, such as color, count, spatial relation, or object state. Base and foil images preserve naturalistic scene complexity yet are visually near-identical: only 8% of pixels change on average. Our dataset has many foils per base scene (~22) in order to separate scene difficulty from edit difficulty, a distinction one-foil-per-base benchmarks cannot make. CompoSET contains 1,776 instances across 80 naturalistic scenes, spanning 16 edit types in 4 semantic categories, with captions at 3 verbosity levels. We test 22 dual-encoder VLMs and find that all models perform at or below chance level on CompoSET, despite scoring well on prior benchmarks. This suggests that apparent progress under prior image-text matching protocols may be an artifact arising from confounds in benchmark design, and that compositional understanding in dual-encoder VLMs is far from solved. CompoSET addresses the confounds and offers a better controlled benchmark to measure future progress.

Figures & tables

Figure 1: Compositionality experiments require comparing images that differ by one named factor. CompoSET contains 80 naturalistic multi-object base images (Left) that are varied to obtain 1,776 instances, the foils (Right) (22 foils per base). Each base-foil pair differs by exactly one compositional edit which is labeled in the captions (Middle) (Sec. 3 ). Captions come at three levels of verbosity to test sensitivity to context. In the experiments models compute the similarity between each one of two related images (a base image and one of its foils) and the corresponding two captions, and vice-versa, for a total of four comparisons (Sec. 5 ).

Category	Edit types ( $n$ )
Identity & Appearance ( what )	modify_color (206), modify_pattern (101), modify_material (161), modify_shape (103), modify_transparency (100), modify_object (101), swap_color (104), swap_material (101)
Quantity ( how many )	modify_count (100), modify_presence (104)
Spatial Relations ( where )	modify_spatial (99), swap_spatial (100)
Activity & State ( how )	modify_action (101), modify_pose (104), modify_state (146), swap_role (45)

Table 2: CompoSET’s 16 edit types span four semantic categories; prior benchmarks probe at most nine. Per-type instance counts (

n

) in parentheses.

Figure 2: CompoSET base/foil pairs are more controlled than every prior benchmark on three independent axes. Each panel plots the empirical cumulative distribution function (ECDF) of one per-pair measurement across compositionality benchmarks. (Left) fraction of pixels affected per (base, foil) pair (Sec. 4 ). (Middle) fraction of caption tokens changed at the medium verbosity tier (Sec. 4 ). (Right) within-pair CLIP image cosine (Sec. 4 ). The controlled-edit side is left for the first two panels (smaller edits) and right for the third (tighter pairs); on all three axes CompoSET sits well to that side of every prior benchmark.

Model	Compo SET	Wino ground	BiVLC	What’s 1 1 1 What’s Up scores reflect a fixed preposition bias in dual-encoder models. For example, CLIP-B/32 correctly identifies ‘on’ as the correct caption 92.2% of the time, but only picks ‘under’ correctly 22.3% of the time. Strict bidirectional Group exposes this by requiring both image-to-text and text-to-image directions to be correct on the same pair; see App. 6 . Up	Color Swap	MMVP	Natural Bench	Color Bench	Sugar Crepe	Sugar Crepe ++
Chance	16.7	16.7	16.7	16.7	16.7	16.7	50.0	50.0	50.0	50.0
Contrastive baselines
CLIP B/32	4.4	7.5	47.2	2.2	8.7	8.1	68.1	46.7	78.6	65.5
CLIP B/16 LAION-2B	7.8	8.2	53.3	1.1	19.3	12.6	70.7	50.6	81.3	70.1
CLIP H/14 LAION-2B	8.4	10.5	56.9	0.5	32.0	18.5	73.5	55.4	82.5	70.9
SigLIP B/16	10.2	10.2	59.6	1.6	30.7	23.0	74.7	55.4	81.3	70.3

Table 3: Every dual encoder is at or below chance on CompoSET, while most are above chance on prior benchmarks. The exceptions are Winoground and What’s Up, where every model is also at or below chance. Compositionality results across 10 benchmarks for 14 representative dual encoders, grouped by contribution axis (§ 5 ). Best-in-pool per column in bold .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Phase 1 (base review). Rater reviews the base caption against the rendered base image before any foils are generated; failures trigger scene re-generation.

Figure 4: Phase 2 (instance review). Rater compares each base/foil pair against the four-criterion rubric (count preserved, no other object changed, visual edit matches textual edit, caption disambiguates when ambiguous). Failures discard the instance.

Figure 5: Phase 3 (verbosity review). For each kept instance, rater verifies that the edit phrase is preserved across short, medium, and long captions while surrounding scene context grows without rewording.

Model	Source	License	Citation
Dual-encoder, contrastive baselines
CLIP B/32	openai/clip-vit-base-patch32	MIT	[ 34 ]
CLIP B/16 (OpenAI)	openai/clip-vit-base-patch16	MIT	[ 34 ]
CLIP L/14 (OpenAI)	openai/clip-vit-large-patch14	MIT	[ 34 ]
CLIP B/16 LAION-400M	OpenCLIP laion400m_e32	MIT	[ 8 ]
CLIP B/16 LAION-2B	OpenCLIP laion2b_s34b_b88k	MIT	[ 8 ]

Appendix

Table 4: Licenses for evaluated model checkpoints. Frontier API models are governed by their providers’ terms of service.

Benchmark	Source	License	Citation
Winoground	facebook/winoground	Research use (HF gated)	[ 43 ]
BiVLC	imirandam/BiVLC	MIT	[ 31 ]
ColorSwap	stanfordnlp/colorswap	Apache-2.0	[ 5 ]
NaturalBench	BaiqiL/NaturalBench	CC-BY-4.0	[ 21 ]
MMVP	MMVP/MMVP	MIT	[ 44 ]
ColorBench	Liang-Yike/colorbench-robust	MIT	[ 23 ]

Appendix

Table 5: Licenses for comparison compositionality benchmarks.

		Per direction		Bidirectional
Pair	Preposition	I2T	T2I	I2T	T2I	Group
on / under	on	92.2	74.8	14.6	13.6	3.9
	under	22.3	36.9
left of / right of	left of	90.7	49.8	4.9	2.9	0.0
	right of	11.2	51.7
in front of / behind	in front of	80.4	38.2	3.9	2.9	1.0

Appendix

Table 6: CLIP-B/32 on WhatsUp, broken down by preposition. I2T and T2I are 1-of-2 accuracy where the foil is the opposite preposition on the same axis (chance 50%); Group is full bidirectional Winoground Group (strict-AND of I2T and T2I; chance 16.7%) over the (prep a , prep b ) pair. CLIP-B/32 ranks one preposition above its opposite regardless of image content (e.g., on : 92.2% I2T vs. under : 22.3%), so per-axis Group collapses far below chance.

Figure 6: Per-edit-type pixel-affected fraction within CompoSET. Identity edits (color, pattern, material) are the most localized; pose, action, and role edits perturb larger image regions.

Figure 7: Fraction of caption tokens changed in CompoSET by edit type and verbosity tier. Action and state edits are wider text changes; color and pattern edits are single-token swaps.

Benchmark	mean	median	std	p25	p75	max
Excluding people (primary)
CompoSET	20.7	16.0	13.5	11.2	26.8	66
Winoground	5.1	3.0	8.5	1.0	6.0	76
BiVLC	6.1	4.0	7.2	2.0	7.0	63
SugarCrepe	6.3	4.0	6.4	2.0	8.0	46
Including people

Appendix

Table 7: CompoSET is

3

–

4\times

more multi-object than every comparison benchmark, both excluding and including people. Per-image detected-object counts under an identical OWLv2 protocol.

Model	Compo SET	Wino ground	BiVLC	What’s Up	Color Swap	MMVP	Natural Bench	Color Bench	Sugar Crepe	Sugar Crepe ++
Chance	16.7	16.7	16.7	16.7	16.7	16.7	50.0	50.0	50.0	50.0
Contrastive baselines
CLIP B/32	4.4 [3.9,5.0]	7.5 [5.3,10.5]	47.2 [45.4,49.0]	2.2 [1.5,3.2]	8.7 [6.0,12.4]	8.1 [4.6,14.0]	68.1 [66.4,69.8]	46.7 [45.0,48.3]	78.6 [77.4,79.7]	65.5 [64.1,66.8]
CLIP B/16 (OpenAI)	4.8 [4.2,5.4]	6.8 [4.7,9.6]	47.8 [46.0,49.6]	3.0 [2.2,4.1]	7.7 [5.2,11.2]	10.4 [6.3,16.7]	69.1 [67.4,70.8]	48.8 [47.1,50.4]	78.5 [77.4,79.7]	65.1 [63.7,66.4]
CLIP B/16 (LAION-400M)	8.4 [7.6,9.1]	7.8 [5.5,10.8]	54.2 [52.4,56.0]	1.2 [0.7,2.0]	15.3 [11.7,19.8]	12.6 [8.0,19.2]	70.4 [68.8,72.1]	56.2 [54.6,57.9]	78.5 [77.3,79.6]	66.6 [65.2,67.9]
CLIP B/16 (LAION-2B)	7.8 [7.1,8.5]	8.2 [5.9,11.4]	53.3 [51.4,55.1]	1.1 [0.6,1.8]	19.3 [15.3,24.2]	12.6 [8.0,19.2]	70.7 [69.0,72.3]	50.6 [49.0,52.2]	81.3 [80.2,82.4]	70.1 [68.7,71.4]

Appendix

Table 8: Full 22-dual-encoder pool across 10 compositionality benchmarks. Each cell reports point estimate (%) with 95% Wald confidence interval

[\,

lo, hi

\,]

(§ C ). Best-in-pool per column in bold . Metric per benchmark: Group for Winoground-style 2

\times

2 (CompoSET, Winoground, BiVLC, What’s Up, ColorSwap, MMVP); T2I for image-foil-only benchmarks (NaturalBench, ColorBench); I2T for text-foil-only benchmarks (SugarCrepe, SugarCrepe ++ ).

Edit type

n

CLIP B/32

CLIP B/16 LAION-2B

CLIP H/14 LAION-2B

SigLIP B/16

SigLIP SO400M

SigLIP-2 B/16

SigLIP-2 L/16

SigLIP-2 SO400M

NegCLIP B/32

CLoVe B/32

DreamLIP B/16

DFN H/14

EVA-02 L/14

MetaCLIP-2 H/14

Identity

modify_color

618

20.4 [17.2,23.6]

29.9 [26.3,33.5]

28.8 [25.2,32.4]

37.2 [33.4,41.0]

39.5 [35.6,43.3]

39.2 [35.3,43.0]

38.5 [34.7,42.3]

36.2 [32.5,40.0]

30.6 [26.9,34.2]

27.7 [24.1,31.2]

52.4 [48.5,56.4]

35.1 [31.3,38.9]

26.5 [23.1,30.0]

27.2 [23.7,30.7]

modify_pattern

303

6.3 [3.5,9.0]

14.5 [10.6,18.5]

9.6 [6.3,12.9]

20.1 [15.6,24.6]

21.5 [16.8,26.1]

17.8 [13.5,22.1]

16.8 [12.6,21.0]

23.8 [19.0,28.6]

8.6 [5.4,11.7]

8.9 [5.7,12.1]

29.4 [24.2,34.5]

9.9 [6.5,13.3]

14.2 [10.3,18.1]

11.9 [8.2,15.5]

modify_material

483

1.7 [0.5,2.8]

6.8 [4.6,9.1]

8.3 [5.8,10.7]

11.2 [8.4,14.0]

16.4 [13.1,19.7]

10.8 [8.0,13.5]

14.7 [11.5,17.9]

13.7 [10.6,16.7]

4.3 [2.5,6.2]

3.3 [1.7,4.9]

12.2 [9.3,15.1]

12.6 [9.7,15.6]

8.9 [6.4,11.4]

8.1 [5.6,10.5]

modify_shape

309

2.6 [0.8,4.4]

8.7 [5.6,11.9]

10.7 [7.2,14.1]

10.0 [6.7,13.4]

12.3 [8.6,16.0]

11.0 [7.5,14.5]

14.6 [10.6,18.5]

12.6 [8.9,16.3]

6.8 [4.0,9.6]

4.5 [2.2,6.8]

14.6 [10.6,18.5]

9.4 [6.1,12.6]

7.4 [4.5,10.4]

12.0 [8.4,15.6]

modify_transparency

300

0.7 [0.0,1.6]

1.7 [0.2,3.1]

0.7 [0.0,1.6]

1.3 [0.0,2.6]

1.7 [0.2,3.1]

2.3 [0.6,4.0]

1.0 [0.0,2.1]

0.7 [0.0,1.6]

1.7 [0.2,3.1]

1.3 [0.0,2.6]

0.3 [0.0,1.0]

2.3 [0.6,4.0]

Appendix

Table 9: Per-edit-type Group score (%) on CompoSET for the 14 dual encoders of Table 3 , pooled across the three caption verbosity levels. Each cell reports point estimate with 95% Wald confidence interval (§ C );

n

is per-edit-type instance count, summed across short/medium/long tiers (

3\times

the per-scene count). Note the wide intervals on swap_role (

n{=}135

pooled,

n{=}45

per tier).

Edit type	$n$ /tier	Short	Medium	Long
Identity
modify_color	206	35.3	34.7	30.5
modify_pattern	101	16.1	15.2	14.4
modify_material	161	10.2	10.0	8.3
modify_shape	103	11.5	10.6	7.3
modify_transparency	100	1.6	1.3	1.1

Appendix

Table 10: Per-edit-type Group score (%) by caption verbosity, averaged across the 14 dual encoders of Table 3 . Reveals which edit types degrade most (or least) as captions grow longer. Per-cell

n

is per-tier instance count (one third of the column in Table 9 ).

Figure 8: Most edit types are at or below chance. CompoSET Group (%) per edit type for the 14 dual encoders of Table 3 , sorted by median across models (easiest at top). Top three by overall median: SigLIP-2 SO400M/14 (dark blue), DreamLIP B/16 (light blue), SigLIP-2 L/16-384 (red). Other 11 models in gray. Dashed: 16.67% chance.

Figure 9: Distribution of cosine margins on correct vs. wrong CompoSET decisions, pooled across the 22 dual encoders. The two distributions overlap heavily: even when the model decides correctly, the margin is small.

Model	I2T	T2I	Group
Open-weight (caption log-likelihood)
BLIP-2 (OPT-2.7B)	5.0	11.0	2.5
LLaVA-1.6 (Mistral-7B)	30.5	3.0	1.5
Qwen2-VL (7B Instruct)	36.5	0.5	0.5
Frontier API (prompted pairwise, counterbalanced)
Gemini 2.5 Flash	64.5	16.0	12.0

Appendix

Table 11: Generative VLMs on CompoSET (200-instance subset). Open-weight rows scored by length-normalized caption log-likelihood. Frontier rows scored by prompted pairwise selection with option-order counterbalancing. Best Group in bold .

Model	Short	Medium	Long
Gemini 2.5 Flash	15.9	19.7	13.6
Claude Haiku 4.5	11.1	19.7	7.6
Claude Sonnet 4.6	44.4	46.5	36.4

Appendix

Table 12: Frontier VLMs peak at medium verbosity on CompoSET. Group score (%) per caption verbosity tier,

\sim

67 instances per tier.

Compositional Reasoning in VLMs

See all 37 papers

Jun 26, 2026

DataComp-VLM: Improved Open Datasets for Vision-Language Models

Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchmarks for evaluating such curation strategies. We introduce DataComp for VLMs (DCVLM), a benchmark for controlled data-centric experiments to improve VLM training. As part of DCVLM, we collect 160 datasets spanning four data types -- image-caption pairs, multimodal interleaved documents, text-only, and instruction-tuning data -- into a corpus of 6T multimodal tokens. DCVLM allows participants to test curation strategies (filtering, mixing, formatting, sampling) across 1B-8B models and 6.25B-200B token budgets. Models are then evaluated on a carefully selected suite of up to 52 downstream benchmarks across 9 domains. We conduct extensive experiments on DCVLM and find that data mixing, not filtering, is key to a high-quality training dataset: instruction-heavy mixtures scale better than caption-heavy ones, with gains widening at larger scales. The resulting dataset, DCVLM-Baseline, enables training an 8B VLM to 63.6% accuracy on our 33-task core suite with 200B training tokens. Compared to FineVision, the state-of-the-art open VLM training dataset, this represents an improvement of +5.4pp. DCVLM and all accompanying artifacts will be made publicly available at https://www.datacomp.ai/dcvlm/.

Matteo Farina, Vishaal Udandarao, Thao Nguyen +33

University of Trento · 2Tübingen AI Center, University of Tübingen · University of Cambridge +12

VLM Evaluation Training Data Curation Large Vision-Language Models Vision-Language Pretraining

Ask PDF

Jun 11, 2026

Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality

Contrastively trained vision-language models like CLIP, have made remarkable progress in learning joint image-text representations, but still face challenges in compositional understanding. They often exhibit a "bag-of-words" behavior--struggling to capture the object relations, attribute-object bindings, and word order dependencies. This limitation arises not only from the reliance on global, single-vector representations for optimization, but also from the insufficient exploitation and modeling of the rich compositional information inherently present in paired image text data. In this work, we propose MACCO (MAsked Compositional Concept MOdeling), a framework that masks compositional concepts in one modality and reconstructs them conditioned on the full contextual information from the other, enabling the model to capture and align cross-modal compositional structures more effectively. To facilitate this process, we introduce two auxiliary objectives that jointly align and regularize masked features both inter-modally and intra-modally. Extensive experiments on five compositional benchmarks, along with in-depth analyses, demonstrate that our approach not only significantly enhances compositionality in VLMs but also improves their ability to capture syntactic structure and linguistic information. Additionally, the improved compositionality also benefits text-to-image generation and multimodal large language model. Code is available at https://github.com/hiker-lw/MACCO.

Wei Li, Zhen Huang, Xinmei Tian

MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China · Independent Researcher

Cross-Modal Learning Vision-Language Models Vision-Language Alignment Vision-Language Pretraining Compositional Reasoning in VLMs

Ask PDF

Mar 28, 2026

Inference-Time Structural Reasoning for Compositional Vision-Language Understanding

Vision-language models (VLMs) excel at image-text retrieval yet persistently fail at compositional reasoning, distinguishing captions that share the same words but differ in relational structure. We present, a unified evaluation and augmentation framework benchmarking four architecturally diverse VLMs,CLIP, BLIP, LLaVA, and Qwen3-VL-8B-Thinking,on the Winoground benchmark under plain and scene-graph-augmented regimes. We introduce a dependency-based TextSceneGraphParser (spaCy) extracting subject-relation-object triples, and a Graph Asymmetry Scorer using optimal bipartite matching to inject structural relational priors. Caption ablation experiments (subject-object masking and swapping) reveal that Qwen3-VL-8B-Thinking achieves a group score of 62.75, far above all encoder-based models, while a proposed multi-turn SG filtering strategy further lifts it to 66.0, surpassing prior open-source state-of-the-art. We analyze the capability augmentation tradeoff and find that SG augmentation benefits already capable models while providing negligible or negative gains for weaker baselines. Code: https://github.com/amartyacodes/Inference-Time-Structural-Reasoning-for-Compositional-Vision-Language-Understanding

Amartya Bhattacharya

Biomedical Data Science, Geisel School of Medicine, Dartmouth College

Compositional Reasoning in VLMs VLM Reasoning VLM Evaluation Graph Reasoning

Ask PDF

Abstract

Compositional Reasoning in VLMs

Explore similar work

DataComp-VLM: Improved Open Datasets for Vision-Language Models

Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality

Inference-Time Structural Reasoning for Compositional Vision-Language Understanding