Vision--language pre-training has enabled substantial progress in image--text retrieval and zero-shot recognition by learning transferable cross-modal representations. However, sentence-level alignment remains insufficient for compositional matching, where captions can describe the same objects and attributes while differing in how these elements are bound. To address this limitation, we propose Flow-CLIP, which refines global image-text alignment through image-conditioned changes in token--patch evidence. Global Compatibility Calibration (GCC) learns a sentence-level similarity metric, while Image-Conditioned Evidence Refinement (ICER) incorporates visual patch context into token queries and uses the resulting change in local matching evidence to adjust the global score. We further introduce Cross-Image Margin Ranking (CIMR), which compares the same positive--negative caption pair across matched and mismatched images, encouraging a larger ranking margin when the image supports the correct description. Experiments on compositional matching and attribute--object binding benchmarks show that Flow-CLIP improves binding accuracy over pretrained CLIP and learned global alignment through local evidence refinement, while cross-image ranking supervision mitigates reliance on language priors. Code will be released at link.
Figures & tables
Fig. 1: Compositional image–text matching with closely related caption candidates. Bars show compatibility scores, yellow outlines identify the top-ranked caption, and the curve shows the gold-caption margin.
Fig. 2: Flow-CLIP combines Global Compatibility Calibration (GCC) with Image-Conditioned Evidence Refinement (ICER). The global and token maps produce normalized representations. Centered patch support forms visual context, whose projection supplies the input to the learned query correction. The normalized corrected query and its original query yield a signed difference in pooled evidence, which is averaged over valid tokens and added to global compatibility. and denote frozen and learned components.
Fig. 3: CREPE Recall@1 for HNB-trained Flow-CLIP and LABCLIP [ 17 ] across complexity levels 4–12. Solid and dashed curves denote atomic and swap foils. Endpoint labels show gains over LABCLIP in percentage points at complexity 12.
Dataset
CLIP
NegCLIP
Global- M
Flow-CLIP
COCO
CC3M
COCO
CC3M
Classification (Top-1)
CIFAR-10
88.3
89.0
86.1
85.2
86.5 (−1.8)
86.4 (−1.9)
CIFAR-100
64.5
63.0
58.9
58.9
58.0 (−6.5)
59.3 (−5.2)
ImageNetV2
54.7
48.7
45.3
50.1
47.0 (−7.7)
50.0 (−4.7)
Retrieval (R@1)
Tab. 7: Zero-shot Top-1 classification and Recall@1 retrieval (%). T2I retrieves images from captions; I2T retrieves captions from images. Colored parentheses report percentage-point changes from frozen CLIP (blue: gain; red: loss).
Fig. 4: Compositional evidence across models. Word-order, object-replacement, relation-binding, and object-binding cases compare evidence maps and gold–foil score margins. Warm and cool colors indicate responses above and below the within-image reference. Margins are reported on each method’s native scoring scale.
Fig. 6 : Caption rankings for four CREPE [ 26 ] examples. Each image has one gold caption and five foils. Columns give CLIP [ 33 ] and Flow-CLIP ranks, with rank 1 best. Green checks identify gold captions and red crosses identify foils.
Fig. 7: Cross-model residual-evidence atlas on ARO. The eight examples cover VG-Attribution, VG-Relation, Flickr30k Order, and COCO Order. For each example, the input image is followed by patches selected by Flow-CLIP’s centered-feature sensitivity and the gold–hard-foil score margins of CLIP [ 33 ] , NegCLIP [ 43 ] , LABCLIP [ 17 ] , and Flow-CLIP. Positive margins indicate correct pairwise rankings.
Fig. 8: Changes in caption-pair margins. Each arrow connects the Global- M and Flow-CLIP gold–foil score margins for the same image and caption pair. The zero boundary separates incorrect and correct rankings. Enlarged markers show the task means.
Global vision--language similarities compress an image and a caption into one vector, preserving semantics but not which word corresponds to which region or how those regions are arranged; a model can recognize every word and object yet prefer a compositionally incorrect caption. We argue that a frozen encoder retains this association structure, so the problem is to read it rather than to rebuild it beside the pretrained similarity. We introduce BindCLIP, a pairwise scorer built on one latent object: a balanced token--patch--depth optimal-transport coupling that places both candidate captions and several visual depths in a single plan. Semantic, entity, order, and spatial evidence are read as energies of this state, and exchanging the candidates permutes the plan, making the score exactly antisymmetric. A geometric refinement inside the coupling contracts moves that the candidates and the visual depths do not support. No task label, parser, relation inventory, or detector is used. One checkpoint and one inference path improve the official What'sUp, ARO, and SugarCrepe benchmarks over frozen global CLIP, with the strongest transfer on the relation splits. Controls rule out patch access and caption-length shortcuts, and an inference-time lesion localizes spatial arrangement to the coupling.
Liuyang Song, Yi Zhang, Zhongyi Deng +2
Peking University · Dongguan University of Technology · Sichuan Agricultural University
CLIP-style contrastive pretraining typically curates web-scale image-text pairs using sample-level filtering signals, often based on pair-level alignment. We show that this signal saturates: once coarse mismatches are removed, stricter global filtering no longer tracks the compositional supervision provided by the retained captions. The reason is structural - a global score conflates whether a pair is broadly plausible with whether the individual object, attribute, and relation phrases inside the caption materially support the image-text match. The latter is what compositional generalization demands, yet pair-level filters are blind to it. We address this with Counterfactual Phrase Intervention (CPI), a phrase-level curation framework that converts controlled nonce-token substitutions into image-conditioned phrase-sensitivity scores. CPI uses global alignment only for coarse mismatch removal, then ranks the surviving pool by whether caption phrases measurably affect the image-text score under controlled substitution. We frame CPI as a first-order phrase-sensitivity signal rather than a grounding or identification result, and evaluate it at CC3M scale. Ranking by this signal yields a 50%-data subset that improves VL-CheckList-VG Relation by +1.91 over the full-data baseline and +1.00 over alignment-only filtering at matched budget, while improving SugarCrepe overall and preserving general transfer. CPI is loss-orthogonal: applied unchanged to NegCLIP, it further improves VL-CheckList-VG Relation by +3.84, with additional CE-CLIP gains in the main text.
Contrastively trained vision-language models like CLIP, have made remarkable progress in learning joint image-text representations, but still face challenges in compositional understanding. They often exhibit a "bag-of-words" behavior--struggling to capture the object relations, attribute-object bindings, and word order dependencies. This limitation arises not only from the reliance on global, single-vector representations for optimization, but also from the insufficient exploitation and modeling of the rich compositional information inherently present in paired image text data. In this work, we propose MACCO (MAsked Compositional Concept MOdeling), a framework that masks compositional concepts in one modality and reconstructs them conditioned on the full contextual information from the other, enabling the model to capture and align cross-modal compositional structures more effectively. To facilitate this process, we introduce two auxiliary objectives that jointly align and regularize masked features both inter-modally and intra-modally. Extensive experiments on five compositional benchmarks, along with in-depth analyses, demonstrate that our approach not only significantly enhances compositionality in VLMs but also improves their ability to capture syntactic structure and linguistic information. Additionally, the improved compositionality also benefits text-to-image generation and multimodal large language model. Code is available at https://github.com/hiker-lw/MACCO.
Wei Li, Zhen Huang, Xinmei Tian
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China · Independent Researcher