Stuck on Shuffle: On Attribute-Object Binding in Vision-Language Models

Netanel Y. Tamir; Michael Green; Issar Tzachor; Or Shimshi; Dvir Samuel; Or Patashnik; Rami Ben-Ari

cs.CVOct 15, 2026

Stuck on Shuffle: On Attribute-Object Binding in Vision-Language Models

Authors: Netanel Y. Tamir, Michael Green, Issar Tzachor, Or Shimshi, Dvir Samuel, Or Patashnik, Rami Ben-Ari

Organizations: OriginAI · NVIDIA · Tel Aviv University

Abstract

Retrieval is a fundamental task in multimodal systems: given an image or text query, the goal is to find the most relevant images or texts in a collection, often using a shared embedding space. Reliable retrieval requires these embeddings to capture not only which concepts are present, but also how they are composed. One key compositional challenge is attribute--object binding: encoding which attributes belong to which objects. Existing benchmarks provide a limited view of this capability, typically covering a subset of modality pairings, simple two-object swaps, or small attribute sets. Consequently, it remains unclear how robustly current models preserve attribute--object bindings across different modality pairings and more complex settings. We therefore introduce SABER, a controlled benchmark for evaluating attribute--object binding across all four image and text modality pairings, spanning diverse attribute types and scenes with up to ten objects. Evaluating models on SABER reveals substantial binding failures in both CLIP-style dual encoders and recent MLLM-based embedders. These findings motivate us to ask how binding can be improved efficiently. While multimodal supervision may be effective, collecting suitable images is difficult and costly. Text, on the other hand, is well suited to binding supervision: it is cheap to generate, supports a broader range of attributes and enables precisely controlled hard negatives. We thus finetune an MLLM-based embedder exclusively on synthetic attribute-binding text. Despite the absence of image supervision, the gains transfer to cross-modal and image--image retrieval while preserving general multimodal embedding performance. Our model achieves SOTA results on SABER while also improving on negation and instance-retrieval benchmarks, demonstrating broader benefits of our binding-focused supervision.

Figures & tables

Figure 1: Overview. Top: SABER evaluates attribute–object binding by reassigning fixed sets of objects and attributes. Bottom: Synthetic text-only finetuning improves binding in text and image representations, boosting unimodal and cross-modal retrieval on SABER and existing benchmarks

Figure 2: SABER multiple-choice questions. Each query has four candidates with 2–10 attribute–object pairs: one preserves the bindings, while the others form hard negatives by reassigning attributes. For two-object examples, one negative instead replaces an object or attribute. Although only images are shown, the accompanying captions enable evaluation within and across modalities.

	Modalities				Evaluation Method		Benchmark Statistics
Dataset	T $\rightarrow$ T	T $\rightarrow$ I	I $\rightarrow$ T	I $\rightarrow$ I	MCQ	Retrieval	# Eval. Samples	Object Vocab.	Attribute Vocab.	# Bindings / Sample
NCD	$\times$	$\times$	$\checkmark$	$\times$	$\checkmark$	$\times$	$1{,}998$	$19$	$6$	$2$
VG-Attribution	$\times$	$\times$	$\checkmark$	$\times$	$\checkmark$	$\times$	$28{,}748$	open	open	$2$
ColorSwap	$\times$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\times$	$1{,}000$	open	open	$2$
SugarCrepe-Swap-Att.	$\times$	$\times$	$\checkmark$	$\times$	$\checkmark$	$\times$	$666$	open	open	$2$
SugarCrepe++-Swap-Att.	$\checkmark$	$\times$	$\checkmark$	$\times$	$\checkmark$	$\times$	$666$	open	open	$2$

Table 1: Existing attribute-binding benchmarks. Open denotes an unrestricted vocabulary; MCQ includes Winoground-style

2\times 2

matching and SugarCrepe++’s three-caption evaluation.

Figure 3: Nested SABER chains. Nine linked MCQs form a chain from two to ten objects, read left to right and top to bottom. Each successive query adds one binding while preserving all earlier ones. Half of SABER’s MCQs are organized into such chains, allowing later correct candidates to serve as additional positives for earlier queries in large-gallery retrieval.

	SABER-MCQ					SABER-Retrieval (Independent)					SABER-Retrieval (Nested)
Model	T $\to$ T	T $\to$ I	I $\to$ T	I $\to$ I	Mean	T $\to$ T	T $\to$ I	I $\to$ T	I $\to$ I	Mean	T $\to$ T	T $\to$ I	I $\to$ T	I $\to$ I	Mean
CLIP ViT-B/32	53.6	23.0	22.3	75.5	43.6	74.0	38.8	33.1	85.1	57.8	38.0	13.7	16.1	49.3	29.3
LABCLIP ViT-B/32	55.2	22.8	24.2	75.3	44.4	74.3	35.0	25.8	85.1	55.0	38.1	13.0	12.1	49.2	28.1
TripletCLIP ViT-B/32	42.9	27.1	22.2	60.2	38.1	65.9	8.8	5.0	63.2	35.7	35.8	5.4	3.5	32.1	19.2
READ-CLIP ViT-B/32	47.3	23.9	23.7	82.3	44.3	68.4	33.5	23.6	89.7	53.8	37.7	10.7	13.1	54.1	28.9
PE-Core-bigG-14-448	66.1	29.7	27.6	67.6	47.7	81.1	56.0	55.5	81.5	68.5	41.5	23.6	32.9	47.1	36.3

Table 2: Performance on SABER. SABER-MCQ reports mean 4-way accuracy (%) and SABER-Retrieval mean mAP (%), both averaged over 2–10 objects. Retrieval is split into independent and nested subsets over the pooled candidate gallery. Our finetuned 8B model performs best across all four modality pairings in both protocols. Results by number of objects are reported in Tables 17 , 18 , and 19 in the Appendix. Best and second-best results are shown in bold and underlined, respectively.

Dataset	CLIP [-1.5pt]B/32	LABCLIP [-1.5pt]B/32	TripletCLIP [-1.5pt]B/32	READ-CLIP [-1.5pt]B/32	PE [-1.5pt]CORE-G	LamRA- [-1.5pt]Ret	Qwen3-VL [-1.5pt]Emb-2B	Qwen3-VL [-1.5pt]Emb-8B	Ours-2B	Ours-8B
Text $\bm{\rightarrow}$ Text
SugarCrepe++-Swap-Att.	$32.7$	$41.0$	$34.7$	$56.8$	$53.2$	$66.2$	$50.3$	$70.9$	$\underline{78.7}$	$\mathbf{93.7}$
Text $\bm{\rightarrow}$ Image
ColorSwap	$17.9$	$15.5$	$14.0$	$24.5$	$44.6$	$23.1$	$22.8$	$\underline{45.9}$	$34.2$	$\mathbf{79.8}$
BiVLC-Swap-Att.	$19.1$	$24.1$	$7.2$	$\underline{31.7}$	$19.4$	$21.2$	$15.1$	$21.6$	$16.2$	$\mathbf{32.0}$
Image $\bm{\rightarrow}$ Text

Table 3: Performance on existing attribute-binding benchmarks. Accuracy (%) under each benchmark’s native multiple-choice protocol. Best and second-best results are shown in bold and underlined, respectively.

Model	Image	Video	VisDoc	Text	Agent	All
# of Datasets	$37$	$18$	$27$ 1 1 1 MMEB-v3 reports $24$ VisDoc tasks but enumerates $27$ ; we score all $27$ for every model.	$53$	$47$	$182$
Qwen3-VL-Embedding-2B	$\mathbf{69.7}$	$\mathbf{56.3}$	$\mathbf{74.2}$	$\mathbf{40.3}$	$39.8$	$\mathbf{52.8}$
Ours-2B	$67.7$	$56.1$	$73.0$	$39.7$	$\mathbf{41.1}$	$52.3$
Qwen3-VL-Embedding-8B	$\mathbf{73.7}$	$\mathbf{59.6}$	$74.3$	$\mathbf{43.7}$	$39.1$	$54.7$
Ours-8B	$72.5$	$58.7$	$\mathbf{75.4}$	$43.1$	$\mathbf{42.3}$	$\mathbf{55.2}$

Table 4: MMEB-v3 results. Average Hit@1 is reported for image, video, and agent tasks and NDCG@5 for VisDoc and text tasks. Our finetuned models remain within

0.5

points of their pretrained counterparts overall. Best results per pretrained–finetuned pair are shown in bold.

	NegBench coco				NegBench voc
Model	Affirmation	Negation	Hybrid	All	Affirmation	Negation	Hybrid	All
Qwen3-VL-Embedding-8B	$78.3$	$6.8$	$36.3$	$41.4$	$89.9$	$4.7$	$58.0$	$39.8$
Ours-8B (negation-filtered)	$82.7$	$6.6$	$58.1$	$50.3$	$91.5$	$6.4$	$78.2$	$49.7$
Ours-8B	$\mathbf{83.4}$	$6.8$	$\mathbf{58.5}$	$\mathbf{50.7}$	$91.8$	$6.1$	$\mathbf{79.6}$	$50.3$
Qwen3-VL-Reranker-8B	$80.6$	$\mathbf{17.8}$	$46.9$	$49.3$	$\mathbf{95.0}$	$\mathbf{28.1}$	$58.0$	$\mathbf{50.4}$

Table 5: NegBench results. MCQ accuracy (%) by correct-statement type on the COCO and VOC 2007 subsets. Our finetuned model improves overall performance, with the largest gains on hybrid statements. Best results are shown in bold.

	I $\to$ I								T $\to$ I
Model	ILIAS	INSTRE S $\to$ S	INSTRE M $\to$ S	INSTRE M $\to$ S+M	PODS	Paris6K	Oxford5K	Sculptures6K	ILIAS
Qwen3-VL-Emb-8B	$58.5$	$92.8$	$81.3$	$75.6$	$85.7$	$86.8$	$61.3$	$50.8$	$52.3$
Ours-8B	$\mathbf{68.4}$	$\mathbf{94.9}$	$\mathbf{86.7}$	$\mathbf{79.0}$	$\mathbf{88.1}$	$\mathbf{88.9}$	$\mathbf{64.3}$	$\mathbf{59.6}$	$\mathbf{60.8}$

Table 6: Instance retrieval performance. Results report mean mAP (%) across image–image and text–image benchmarks for Qwen3-VL-Embedding-8B and our finetuned model. For INSTRE, S and M denote the single- and multi-object subsets, respectively. Best results are shown in bold.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Attribute category	#
Basic color	$9$
Other color	$2$
Metallic	$3$
Material	$19$
Pattern	$15$
Effect	$2$

Appendix

Table 7: Number of SABER attributes in each semantic category.

Construction type	#
pair_swap	$20{,}000$
attr_bundle	$20{,}000$
predicative	$20{,}000$
triple_bind_cycle	$10{,}048$
triple_bind_swap	$9{,}952$
deferred_bind_anaphoric	$9{,}417$

Appendix

Table 8: Number of finetuning examples by construction type.

pair_swap
Query	wrinkled tunic and cotton shroud
Positive	“Frost is coming,” he noted, touching the cotton shroud and wrinkled tunic.
Neg. (attr.)	“Frost is coming,” he noted, touching the wrinkled shroud and cotton tunic.
Neg. (obj.)	“Frost is coming,” he noted, touching the cotton tunic and wrinkled shroud .
triple_bind_cycle
Query	plain chair, decorative lamp and metallic table

Appendix

Table 9: Examples of the seven finetuning construction types. Each block shows a query, positive caption, and two hard negatives formed by permuting attributes or objects. Bold marks words moved relative to the positive.

Dataset	CLIP B/32	LABCLIP B/32	TripletCLIP B/32	READ-CLIP B/32	PE CORE-G	LamRA- Ret	Qwen3-VL Emb-2B	Qwen3-VL Emb-8B	Ours-2B	Ours-8B
Test set	62.9	62.2	57.6	65.0	65.4	56.2	57.1	60.4	84.0	93.6
deferred_bind_anaphoric	50.4	50.1	50.9	51.8	51.1	51.2	52.7	53.8	59.4	72.9
deferred_bind_respective	51.0	50.9	52.1	51.5	51.1	51.1	51.4	50.8	53.2	56.6

Appendix

Table 10: Triplet accuracy on held-out finetuning examples. Accuracy (%) on the in-distribution test set and the two held-out deferred-binding construction types. Both of our finetuned models outperform all baselines across all three evaluations. Best results are shown in bold.

	SABER-MCQ				MMEB-v3
Model	T $\to$ T	T $\to$ I	I $\to$ T	I $\to$ I	Image	Video	VisDoc	Text	Agent	All
Qwen3-VL-Emb-8B-NoOverlap	$99.3$	$65.7$	$71.8$	$93.1$	$72.6$	$58.8$	$75.5$	$43.1$	$\mathbf{42.6}$	$\mathbf{55.3}$
Qwen3-VL-Emb-8B-Random	$99.4$	$68.7$	$73.8$	$94.3$	$72.5$	$58.8$	$75.4$	$43.2$	$42.3$	$\mathbf{55.3}$
Qwen3-VL-Emb-8B-OverlapOnly	$99.4$	$69.8$	$74.1$	$94.7$	$72.5$	$58.8$	$75.4$	$43.3$	$42.0$	$55.2$
Qwen3-VL-Emb-8B-MixedLengths	$\mathbf{99.6}$	$\mathbf{70.6}$	$\mathbf{77.1}$	$\mathbf{94.8}$	$72.3$	$58.8$	$\mathbf{75.6}$	$42.9$	$\mathbf{42.6}$	$55.2$
Qwen3-VL-Emb-8B	$53.7$	$34.9$	$36.3$	$84.9$	$\mathbf{73.7}$	$\mathbf{59.6}$	$74.3$	$\mathbf{43.7}$	$39.1$	$54.7$

Appendix

Table 11: Training-data ablation. SABER-MCQ and MMEB-v3 performance for lexical-overlap and query–candidate length ablations of our 8B model. SABER-MCQ reports 4-way accuracy, while MMEB-v3 reports Hit@1 for Image, Video, and Agent and NDCG@5 for VisDoc and Text. Best results are shown in bold.

	SABER-MCQ				MMEB-v3
Model	T $\to$ T	T $\to$ I	I $\to$ T	I $\to$ I	Image	Video	VisDoc	Text	Agent	All
Qwen3-VL-Emb-8B-NoLiNeS	$\mathbf{100.0}$	$\mathbf{87.3}$	$\mathbf{87.8}$	$\mathbf{99.0}$	$60.8$	$51.2$	$68.8$	$34.6$	$39.0$	$47.8$
Qwen3-VL-Emb-8B	$53.7$	$34.9$	$36.3$	$84.9$	$\mathbf{73.7}$	$\mathbf{59.6}$	$74.3$	$\mathbf{43.7}$	$39.1$	$54.7$
Ours-8B	$99.3$	$69.0$	$74.1$	$94.3$	$72.5$	$58.7$	$\mathbf{75.4}$	$43.1$	$\mathbf{42.3}$	$\mathbf{55.2}$

Appendix

Table 12: LiNeS ablation. SABER-MCQ and MMEB-v3 performance for our 8B model before and after applying LiNeS. SABER-MCQ reports 4-way accuracy, while MMEB-v3 reports Hit@1 for Image, Video, and Agent and NDCG@5 for VisDoc and Text. Best results are shown in bold.

Figure 4: Layer-wise binding margin. Mean binding margin across the 36-layer text stack for test queries, their corresponding positive captions, and SABER queries. Finetuning preserves stronger binding in later layers, with a larger effect without LiNeS.

	SABER-MCQ				MMEB-v3
Model	T $\to$ T	T $\to$ I	I $\to$ T	I $\to$ I	Image	Video	VisDoc	Text	Agent	All
Qwen3-VL-Emb-8B-Attn-LiNeS	$99.4$	$70.0$	$74.8$	$94.2$	$72.4$	$58.6$	$75.2$	$42.8$	$42.2$	$55.0$
Qwen3-VL-Emb-8B-Attn-NoLiNeS	$\mathbf{100.0}$	$\mathbf{86.4}$	$\mathbf{87.2}$	$\mathbf{99.2}$	$59.5$	$50.5$	$66.3$	$34.8$	$39.2$	$47.2$
Qwen3-VL-Emb-8B	$53.7$	$34.9$	$36.3$	$84.9$	$\mathbf{73.7}$	$\mathbf{59.6}$	$74.3$	$\mathbf{43.7}$	$39.1$	$54.7$
Ours-8B	$99.3$	$69.0$	$74.1$	$94.3$	$72.5$	$58.7$	$\mathbf{75.4}$	$43.1$	$\mathbf{42.3}$	$\mathbf{55.2}$

Appendix

Table 13: Attention loss ablation. SABER-MCQ and MMEB-v3 performance for our 8B model trained with the attention loss, with and without LiNeS. SABER-MCQ reports 4-way accuracy, while MMEB-v3 reports Hit@1 for Image, Video, and Agent and NDCG@5 for VisDoc and Text. Best results are shown in bold.

	SABER-MCQ				MMEB-v3
Model	T $\to$ T	T $\to$ I	I $\to$ T	I $\to$ I	Image	Video	VisDoc	Text	Agent	All
UEmbed-2B	$33.2$	$27.3$	$24.7$	$83.4$	$\mathbf{63.7}$	$\mathbf{50.8}$	$\mathbf{71.5}$	$\mathbf{35.4}$	$\mathbf{35.9}$	$\mathbf{48.2}$
UEmbed-2B-Finetuned	$\mathbf{81.8}$	$\mathbf{39.2}$	$\mathbf{39.8}$	$\mathbf{88.3}$	$62.2$	$48.9$	$67.8$	$31.9$	$34.2$	$45.6$
UEmbed-9B	$46.9$	$36.4$	$32.2$	$88.7$	$\mathbf{68.6}$	$\mathbf{55.6}$	$\mathbf{75.2}$	$\mathbf{40.9}$	$\mathbf{38.1}$	$\mathbf{52.3}$
UEmbed-9B-Finetuned	$\mathbf{91.2}$	$\mathbf{57.3}$	$\mathbf{49.7}$	$\mathbf{94.6}$	$67.6$	$55.1$	$74.5$	$40.4$	$35.9$	$51.3$
WeMM-Embedding-2B	$51.4$	$29.8$	$28.9$	$59.6$	$\mathbf{78.0}$	$\mathbf{65.2}$	$\mathbf{78.4}$	$\mathbf{44.2}$	$44.8$	$\mathbf{58.4}$

Appendix

Table 14: Generalization to additional embedding backbones. Performance of UEmbed and WeMM-Embedding on SABER and MMEB-v3 before and after binding-focused finetuning. Finetuning improves all four SABER modality pairings across both model families and scales, while largely preserving MMEB-v3 performance. SABER-MCQ reports 4-way accuracy, while MMEB-v3 reports Hit@1 for Image, Video, and Agent and NDCG@5 for VisDoc and Text. Best results per pretrained–finetuned pair are shown in bold.

		SABER-MCQ 360 Test Examples (20%)				MMEB-v3
Finetuning Modalities	SABER Finetuning Data	T $\to$ T	T $\to$ I	I $\to$ T	I $\to$ I	Image	Video	VisDoc	Text	Agent	All
T $\to$ T	180 (10%)	$87.2$	$40.6$	$46.7$	$87.5$	$73.8$	$59.8$	$74.8$	$43.7$	$39.3$	$54.9$
	360 (20%)	$95.6$	$52.5$	$55.6$	$90.0$	$\mathbf{73.9}$	$\mathbf{59.9}$	$74.9$	$43.7$	$39.5$	$55.0$
	630 (35%)	$96.9$	$54.7$	$59.4$	$90.6$	$73.8$	$\mathbf{59.9}$	$75.0$	$\mathbf{43.8}$	$\mathbf{39.7}$	$\mathbf{55.1}$
	1,260 (70%)	$\mathbf{97.8}$	$\mathbf{55.6}$	$\mathbf{60.3}$	$90.8$	$73.8$	$59.8$	$75.1$	$43.7$	$39.6$	$55.0$
I $\to$ I	180 (10%)	$56.1$	$32.8$	$39.4$	$95.3$	$\mathbf{73.9}$	$59.6$	$74.6$	$43.7$	$39.2$	$54.8$

Appendix

Table 15: Text vs. image finetuning on SABER. Performance of Qwen3-VL-Embedding-8B after finetuning with increasing amounts of SABER data using either text–text or image–image supervision. Text supervision yields stronger overall SABER performance and consistently better cross-modal transfer, while image supervision performs best on the matched image–image setting. SABER–MCQ reports 4-way accuracy on the fixed

360

-example test split, with all models evaluated at epoch

40

. MMEB-v3 reports Hit@1 for Image, Video, and Agent tasks and NDCG@5 for VisDoc and Text tasks. Best results are shown in bold.

Model	Art	Fashion	Household	Landmark	Product	Technology	Toy	All
# Classes	$2$	$7$	$7$	$4$	$5$	$5$	$8$	$38$
# Instances	$33$	$92$	$50$	$136$	$36$	$49$	$95$	$491$
Qwen3-VL-Emb-8B	$75.4$	$62.4$	$72.4$	$76.1$	$66.0$	$76.6$	$52.5$	$66.6$
Ours-8B	$\mathbf{82.4}$	$\mathbf{72.4}$	$\mathbf{80.2}$	$\mathbf{82.2}$	$\mathbf{72.5}$	$\mathbf{84.4}$	$\mathbf{63.7}$	$\mathbf{75.2}$

Appendix

Table 16: ILIAS fine-grained instance retrieval. Image–image retrieval mAP (%) averaged over the fine-grained classes within each top-level category; All averages over all

38

classes. Best results are shown in bold.

Figure 5: Qualitative image–image retrieval on ILIAS. Top-5 retrievals for two queries favoring Qwen3-VL-Embedding-8B (top) and two favoring Ours-8B (bottom). Green positives denote images of the same instance as the query, while red negatives denote other instances.

Figure 6: Qualitative fine-grained instance retrieval on ILIAS. Examples from three fine-grained classes under the category-restricted evaluation. For each query, we show the top-5 retrievals from Qwen3-VL-Embedding-8B and Ours-8B. Green positives depict the same instance as the query, while red negatives depict different instances from the same fine-grained class.

Figure 7: Agreement across attribute-binding formulations. Each point represents a model, comparing SABER accuracy on two-object image–text scenes with CLEVR-Bind accuracy. Performance is positively correlated across the ten models (Spearman

\rho=0.89

; Kendall

\tau=0.78

), with Ours-8B achieving the highest accuracy in both settings.

Figure 8: Instruction trade-offs. Mean SABER-MCQ accuracy versus mean MMEB-v3 score on image tasks across

41

instruction choices for Qwen3-VL-Embedding-8B and our finetuned model (Ours-8B). Lines trace each model’s Pareto frontier over instructions, and circles indicate the default instruction.

Figure 9: Examples from attribute-binding benchmarks. Existing benchmarks span natural images, photo composites, diffusion-generated scenes, and synthetic 3D environments, with varying vocabulary sizes and numbers of bindings per sample.

		Number of objects
Modalities	Model	2	3	4	5	6	7	8	9	10	Mean
T $\to$ T	CLIP ViT-B/32	55.5	57.5	61.5	60.0	55.5	56.0	54.0	41.5	40.5	53.6
	LABCLIP ViT-B/32	58.0	57.5	64.0	62.5	55.0	57.0	55.0	46.5	41.5	55.2
	TripletCLIP ViT-B/32	12.0	36.5	43.5	48.0	50.5	49.0	54.0	43.0	49.5	42.9
	READ-CLIP ViT-B/32	41.0	49.5	52.5	55.5	50.0	50.5	51.0	39.5	36.5	47.3
	PE-Core-bigG-14-448	60.0	64.5	74.0	79.5	70.0	71.0	65.0	57.0	54.0	66.1

Appendix

Table 17: SABER-MCQ accuracy across scene complexity. Accuracy (%) for 4-way MCQs across scenes with 2–10 objects, reported for all four modality pairings. Mean averages over object counts. Best and second-best results within each modality pairing and column are shown in bold and underlined, respectively.

		Number of objects
Modalities	Model	2	3	4	5	6	7	8	9	10	Mean
T $\to$ T	CLIP ViT-B/32	75.7	77.4	80.4	76.2	74.1	74.7	73.7	70.2	63.3	74.0
	LABCLIP ViT-B/32	77.0	76.1	81.3	77.0	73.0	72.8	73.3	74.2	63.7	74.3
	TripletCLIP ViT-B/32	47.7	59.2	65.5	67.3	71.8	67.9	76.5	67.8	69.7	65.9
	READ-CLIP ViT-B/32	66.0	67.1	71.6	75.9	68.8	71.1	73.4	64.3	57.7	68.4
	PE-Core-bigG-14-448	78.2	79.2	85.8	89.7	82.3	82.2	84.5	76.0	72.4	81.1

Appendix

Table 18: SABER large-gallery retrieval on independent scenes. mAP (%) across scenes with 2–10 objects for the

100

independently generated queries at each scene size, reported for all four modality pairings. Mean averages over object counts. Best and second-best results within each modality pairing and column are shown in bold and underlined, respectively.

		Number of objects
Modalities	Model	2	3	4	5	6	7	8	9	10	Mean
T $\to$ T	CLIP ViT-B/32	25.0	28.6	30.7	33.1	38.8	39.6	42.2	46.6	57.4	38.0
	LABCLIP ViT-B/32	27.2	30.0	32.1	34.1	37.6	39.0	42.4	43.7	57.0	38.1
	TripletCLIP ViT-B/32	18.8	25.5	28.3	31.1	33.3	36.2	41.8	45.3	62.1	35.8
	READ-CLIP ViT-B/32	26.4	30.0	31.2	33.1	36.9	38.7	41.0	44.1	57.8	37.7
	PE-Core-bigG-14-448	30.1	32.9	34.5	36.3	38.6	43.0	44.6	50.2	63.7	41.5

Appendix

Table 19: SABER large-gallery retrieval on nested chains. mAP (%) across scenes with 2–10 objects for the

100

nested-chain queries at each scene size, reported for all four modality pairings. Mean averages over object counts. Best and second-best results within each modality pairing and column are shown in bold and underlined, respectively.

Dataset	License	Model	License
NCD	MIT	CLIP ViT-B/32	MIT
VG-Attribution	MIT	LABCLIP ViT-B/32	MIT
ColorSwap	MIT	TripletCLIP ViT-B/32	Open
SugarCrepe-Swap-Att.	MIT	READ-CLIP ViT-B/32	MIT
SugarCrepe++-Swap-Att.	MIT	PE-Core bigG-14-448	Apache 2.0
BiVLC-Swap-Att.	MIT	LamRA-Ret	MIT

Appendix

Table 20: Licenses and usage terms of datasets and models used in this work.

Vision-Language Retrieval

Vision-language retrieval finds images, video, or documents that match natural-language queries and supports search across modalities.

See all 97 papers

May 29, 2026

How can embedding models bind concepts?

Humans easily determine which color belongs to which shape in multi-object scenes, an ability known as concept binding. Vision-language embedding models such as CLIP struggle with binding: they recognize individual concepts but fail to represent which concepts form which objects. Although CLIP behaves like a bag-of-concepts model in cross-modal retrieval, object information is recoverable from its image and text embeddings separately. We study this tension through the binding function, which maps concepts to scene embeddings. We find that scene embeddings decompose additively into object representations, explaining why uni-modal probes can recover object information. However, CLIP's binding function is high-complexity, which likely prevents the image and text encoders from learning a shared binding mechanism that generalizes to unseen concept combinations. We then ask whether this limitation is fundamental. We show that it is not. In controlled transformer models trained from scratch, binding generalization emerges with sufficient data coverage. These models learn low-complexity binding functions characterized by multiplicative interactions between concepts, enabling systematic generalization. Code is publicly available at https://github.com/oshapio/binding-concepts-complexity.

Arnas Uselis, Darina Koishigarina, Seong Joon Oh

University of Tübingen, Tübingen · 2KAIST AI

Multimodal Embedding Vision-Language Alignment Compositional Generalization

Ask PDF

Aug 17, 2026

Diagnosing Dense Same-Class Attribute Misbinding in Large Vision-Language Models

Large vision-language models can recognize the objects and attributes in a crowded scene yet assign an attribute to the wrong same-class instance. Generic visual-question-answering accuracy marks the response as wrong, while object-hallucination metrics may regard both the object and attribute as image-supported; neither reveals the transfer. This study formalizes this blind spot as Dense Same-Class Attribute Misbinding (DSCAM) and presents InstaBind-Lite, a controlled benchmark that makes it directly measurable. Its 524 images contain 529 curated groups of 3-6 same-class entities, 1773 boxed instances, ordered neighbors, distinguishable color-like attributes, and four complementary question levels, yielding 9580 deterministically evaluated questions. Unlike existing protocols, source-instance annotations separate unsupported generation and recognition failure from an attribute copied from another visible entity. Binding-specific metrics further quantify transfer frequency, adjacency, ordinal distance, and intervention effects. Across five open-source and two commercial/API models, the open-source systems average 19.84% Misbinding Rate and the API systems 7.55%; these errors are hidden by aggregate accuracy. Among identifiable transfers, 80.70% and 81.51%, respectively, originate from adjacent instances. Localization and instance-first interventions help selected models but are not universal remedies. InstaBind-Lite therefore turns previously undifferentiated wrong answers into source-identifiable failure categories and tests a reliability dimension that conventional benchmarks cannot determine: whether a model knows not only what is visible, but which instance owns each attribute.

Yuanzhi Xu, Qian Gao, Jun Fan +4

Qilu University of Technology (Shandong Academy of Sciences), Jinan, China · China Telecom Digital Intelligence Technology Co., Ltd., Jinan, China · Shenyang Aerospace University, Shenyang, China +1

VLM Evaluation Large Vision-Language Models Vision-Language Grounding Faithfulness in VLMs

Ask PDF

Jun 13, 2026

Beyond Scalar Distances: Semantic Attribute Gradients from Frozen MLLMs for Visual Embeddings

Vision encoders for retrieval are typically trained with class-label supervision: each training pair reduces to a scalar that uniformly pushes the embedding apart or pulls it together, as if every visual attribute either differed or matched. A multimodal large language model (MLLM), shown the same pair, can articulate those attributes and use them to predict whether the images share a class. We propose SAGA, a framework that turns this language-grounded, attribute-aware perception into a training signal for the encoder itself. Specifically, we use Group Relative Policy Optimization (GRPO) to reward the MLLM for correct predictions on the vision encoder's tokens. Since correct predictions require those tokens to expose the specific attributes that differ or match between the pair, the gradient pushes the encoder to encode them, replacing the uniform pair-level scalar with attribute-resolved supervision. An auxiliary attention-distillation loss anchors the encoder's embedding to tokens the MLLM attended to, and a standard metric-learning loss shapes the embedding geometry for nearest-neighbour retrieval. The MLLM is frozen throughout and discarded at inference, matching the deployment cost of a metric-learning baseline. SAGA improves Recall@1 by 3 to 6 points over state-of-the-art baselines on CUB-200-2011, Cars-196, FGVC-Aircraft, and iNaturalist Aves on zero-shot image retrieval.

Shubhang Bhatnagar, Dheeraj Baiju, Narendra Ahuja

University of Illinois Urbana-Champaign

Visual Representation Learning Fine-Grained Image Retrieval Metric Learning Vision-Language Retrieval

Ask PDF

Abstract

Vision-Language Retrieval

Explore similar work

How can embedding models bind concepts?

Diagnosing Dense Same-Class Attribute Misbinding in Large Vision-Language Models

Beyond Scalar Distances: Semantic Attribute Gradients from Frozen MLLMs for Visual Embeddings