ConstraintMatch: Compositional Image-Text Matching through Constrained Regional Evidence Selection

Yuhong Zhou; Yubo Gao; Xiangbo Shu

cs.CVOct 31, 2026

ConstraintMatch: Compositional Image-Text Matching through Constrained Regional Evidence Selection

Authors: Yuhong Zhou, Yubo Gao, Xiangbo Shu

Organizations: Nanjing University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Local image--text matching can find strong evidence for individual concepts yet favor captions with incorrect attribute--entity bindings. We introduce ConstraintMatch, a training-free inference method that turns entity-level regional consistency into an explicit matching cost. On the same frozen condition--region similarity matrix, it compares independent crop selection for each category and attribute with a shared crop choice within each entity. Their gap quantifies the score advantage obtained by relaxing the shared-region constraint. Spatial references from the same image calibrate this gap by keeping category evidence fixed while jointly shifting an entity's attribute maps. Per-entity clipping prevents cancellation across residuals, and the scaled mean cost is subtracted from global similarity. ConstraintMatch uses pooled embeddings, and spatial calibration requires no additional encoder calls. With parameters fixed per backbone across tasks, ConstraintMatch improves over global matching by 7.70 percentage points on full ARO Attribute and 5.63 points on held-out SugarCrepe++ attribute swaps, averaged across Qwen~~2B and CORE~~2B. Against SGI adapted to the same 13-crop budget, the respective gains are 4.85 and 3.51 points. Controlled analyses distinguish the effects of shared-region matching and candidate score calibration. The results establish entity-level evidence selection as a practical scoring principle for compositional multimodal retrieval.

Figures & tables

Figure 1: Attribute binding in SCPP [ 6 ] . Checks/crosses mark positive/swapped captions with the same entities and colors. Text is shortened; (b) is cropped for display.

Figure 2: ConstraintMatch overview. Frozen category and attribute evidence is aggregated under independent and shared selection. The expanded computation follows the swapped candidate’s red jacket entity ( $U_{e},C_{e},\Delta_{e},B_{e}$ ); the second residual row, black motorcycle , uses its own gap and reference. Bold cells mark maxima in the selection panels and track a permuted score entry in the spatial-reference panels. Photo frames are actual fixed-grid crops; purple grids show cosine similarities. Residuals are clipped per entity before averaging, including zeros. In this Qwen 2B candidate pair, subtracting the swapped caption’s cost reverses the ranking; the positive caption has zero cost. Bottom phrases are abbreviated, while Global uses the whole image and full effective captions. Checks/crosses indicate dataset positive/negative annotations.

Backbone	Method	ARO Attribute		SCPP held-out		COLA
Backbone	Method	Full	Macro	Attr. swap	All	I2T	T2I	All four
Qwen 2B	Global	68.71	67.56	54.80	74.51	49.52	25.24	19.05
	SGI (13 regions)	70.28	71.52	58.49	70.60	40.95	30.95	22.38
	Shared fusion	71.07	71.04	59.04	68.71	32.38	24.76	13.33
	ConstraintMatch (base)	75.89	76.80	62.92	75.01	39.05	28.10	15.71
	ConstraintMatch	74.65	74.47	59.96	75.24	48.10	26.19	20.00

Table 1: Comparison at a matched regional budget. Accuracy (%). Exact pointwise 95% bootstrap intervals are given in Appendix C.3. Regional methods use 13 crops; our controls and full method use

\lambda=2

. The full method fixes

\eta=1/0.25

for Qwen 2B/CORE 2B. COLA is used in development. Bold denotes the best displayed accuracy per backbone.

Figure 3: Shared selection and independent-reference calibration. Accuracy on source-filtered ARO for the base configuration (

\eta=0

\lambda=2

, 13 crops). Left: 9,295 pairs with identical weighted atoms; right: all 26,598 filtered pairs. Within each method, bars show Qwen 2B then CORE 2B. Whiskers show pointwise 95% source-group percentile bootstrap intervals (2,000 resamples). Methods are compared within each population and backbone; the left panel is a subset of the right.

Backbone	Variant	ARO	SCPP	COLA
Qwen 2B	w/o spatial ref.	75.89	62.92	15.71
	Uniform weakening	68.71	54.80	19.05
	Mean-matched	73.77	56.64	18.57
	w/o clipping	77.00	63.47	18.57
	Post-mean clip	74.07	59.59	20.00
	ConstraintMatch	74.65	59.96	20.00

Table 2: Component ablations. Accuracy (%). ARO: full ARO Attribute; SCPP: held-out attribute swaps; COLA: all-four matching. All variants use 13 crops and

\lambda=2

; spatial variants use the fixed

\eta

from Table 1 . Without spatial references,

\eta=0

. Uniform weakening and Mean-matched rescale the original gap. Removing clipping retains signed entity residuals; Post-mean clip clips after entity averaging, allowing cancellation; ConstraintMatch clips each entity before averaging. COLA is used in development. Bold denotes the best displayed value per backbone and metric; paired comparisons appear in Appendix C.4.

Backbone	Method	I2T	T2I	Index
Qwen 2B	Global	58.89	26.87	7.81
Qwen 2B	ConstraintMatch (base)	192.43	44.70	109.38
CORE 2B	Global	58.65	29.13	7.81
CORE 2B	ConstraintMatch (base)	193.51	60.09	109.38

Table 3: Measured inference cost. Median (p50) request latency (ms) and FP32 image-vector index size (MiB) for 1,000 images. I2T ranks 5,000 captions; T2I ranks 1,000 images using the cache-reset-pass protocol in Appendix F. ConstraintMatch uses

\eta=0

; spatial-reference updates are measured separately.

Compositional Reasoning in VLMs

See all 37 papers

Flow-CLIP: Image-Conditioned Residual Scoring for CompositionalImage-Text Matching
Oct 15, 2026Minghong Wei, Enguang Zhu, Pu Cao +2Vision-Language Alignment Cross-Modal Feature Matching
School of Intelligent Engineering and Automation, Beijing University of Posts and Telecommunications, Beijing, 100082, China
Hierarchical Fine-Grained Cross-Modal Alignment via Dynamic Granularity Discovery and Optimal Transport
Oct 22, 2026Pengfeng Song, Shaozhong Zhang, Pengfei ShaoImage-Text Retrieval Cross-Modal Alignment
College of Information and Intelligent Engineering,Zhejiang Wanli University,Ningbo,315100,Zhejiang,China.
MulVec: Fine-Grained Role-Aware Matching for Training-Free Zero-Shot Composed Image Retrieval
Aug 26, 2026Zihao Zhang, Dayan Wu, Xinze Liu +6Composed Image Retrieval Fine-Grained Image Retrieval
Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100093, China · Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD 21218, USA

Oct 15, 2026J/K move · Enter open · S save

Flow-CLIP: Image-Conditioned Residual Scoring for CompositionalImage-Text Matching

Minghong Wei, Enguang Zhu, Pu Cao, Lu Yang+1

School of Intelligent Engineering and Automation, Beijing University of Posts and Telecommunications, Beijing, 100082, China

Vision-Language Alignment Cross-Modal Feature Matching Vision-Language Retrieval Compositional Representation Learning

Open paper AskPDF arXiv

Vision--language pre-training has enabled substantial progress in image--text retrieval and zero-shot recognition by learning transferable cross-modal representations. However, sentence-level alignment remains insufficient for compositional matching, where captions can describe the same objects and attributes while differing in how these elements are bound. To address this limitation, we propose Flow-CLIP, which refines global image-text alignment through image-conditioned changes in token--patch evidence. Global Compatibility Calibration (GCC) learns a sentence-level similarity metric, while Image-Conditioned Evidence Refinement (ICER) incorporates visual patch context into token queries and uses the resulting change in local matching evidence to adjust the global score. We further introduce Cross-Image Margin Ranking (CIMR), which compares the same positive--negative caption pair across matched and mismatched images, encouraging a larger ranking margin when the image supports the correct description. Experiments on compositional matching and attribute--object binding benchmarks show that Flow-CLIP improves binding accuracy over pretrained CLIP and learned global alignment through local evidence refinement, while cross-image ranking supervision mitigates reliance on language priors. Code will be released at link.

Abstract

Compositional Reasoning in VLMs

Explore similar work

Flow-CLIP: Image-Conditioned Residual Scoring for CompositionalImage-Text Matching

Hierarchical Fine-Grained Cross-Modal Alignment via Dynamic Granularity Discovery and Optimal Transport

MulVec: Fine-Grained Role-Aware Matching for Training-Free Zero-Shot Composed Image Retrieval