Are Object-Centric Representations Better At Compositional Generalization?

Ferdinand Kapl; Amir Mohammad Karimi Mamaghan; Maximilian Seitzer; Karl Henrik Johansson; Carsten Marr; Stefan Bauer; Andrea Dittadi

cs.CVFeb 18, 2026

Are Object-Centric Representations Better At Compositional Generalization?

Authors: Ferdinand Kapl, Amir Mohammad Karimi Mamaghan, Maximilian Seitzer, Karl Henrik Johansson, Carsten Marr, Stefan Bauer, Andrea Dittadi

Organizations: Technical University of Munich · Helmholtz AI, Munich · KTH Royal Institute of Technology · MPI for Intelligent Systems, Tübingen ·

Abstract

Compositional generalization, the ability to reason about novel combinations of familiar concepts, is fundamental to human cognition and a critical challenge for machine learning. Object-centric (OC) representations, which encode a scene as a set of objects, are often argued to support such generalization, but systematic evidence in visually rich settings is limited. We introduce a Visual Question Answering benchmark across three controlled visual worlds (CLEVRTex, Super-CLEVR, and MOVi-C) to measure how well vision encoders, with and without object-centric biases, generalize to unseen combinations of object properties. To ensure a fair and comprehensive comparison, we carefully account for training data diversity, sample size, representation size, downstream model capacity, and compute. We use DINOv2 and SigLIP2, two widely used vision encoders, as the foundation models and their OC counterparts. Our key findings reveal that (1) OC approaches are superior in harder compositional generalization settings; (2) original dense representations surpass OC only on easier settings and typically require substantially more downstream compute; and (3) OC models are more sample efficient, achieving stronger generalization with fewer images, whereas dense encoders catch up or surpass them only with sufficient data and diversity. Overall, object-centric representations offer stronger compositional generalization when any one of dataset size, training data diversity, or downstream compute is constrained.

Figures & tables

Figure 1: Compositional Generalization. To increase generalization difficulty, we decrease the number of unique object property combinations that are seen during training. In the conceptual example, each object is defined by its shape and size, which coincides with MOVi-C. Datasets. For each generalization difficulty and base dataset, we generate images and corresponding question–answer pairs by sampling objects with the allowed combinations. Training. We pretrain object-centric (OC) models by reconstructing the self-supervised (Dense) features from pretrained vision encoders. For VQA downstream training, we concatenate the image features (OC: red; Dense: blue) with the fixed text embeddings and train transformer models of various sizes to predict the answer given image and question.

			CLEVRTex			Super-CLEVR			MOVi-C
	Model	Size	E	M	H	E	M	H	E	M	H
TF 2	DINOv2	256	69.5	58.8	50.0	60.9	57.0	49.7	57.5	53.6	51.4
	DINOv2 + CA	7	-1.2	-4.8	-2.3	-1.2	-1.2	-0.8	-1.7	-0.1	+0.2
	DINOv2 + k-means	7	-16.5	-9.1	-3.1	-10.1	-7.1	-2.2	-5.8	-2.6	-1.6
	DINOv2 + k-means	128	-1.1	+1.2	-0.6	-1.4	-0.8	-0.1	-0.7	-0.3	+0.9
	DINOSAURv2	7	+7.0	+12.3	+5.6	-0.3	+1.6	+1.2	+1.0	+1.1	+1.6

Table 1: Object-centric representations are better for harder generalization, especially with the smaller downstream model. VQA accuracy (%) of both downstream models (TF 2 & 5) on the respective compositional generalization test sets for all DINOv2-based models trained on easy (E), medium (M), and hard (H) training sets. We compute deltas relative to the original pretrained vision encoder and list the number of tokens in the representations (“size”). With TF 5, DINOv2 is better on easy , while DINOSAURv2 matches or slightly surpasses it on hard .

Figure 2: ID and COOD VQA accuracies are strongly correlated (Pearson and Spearman

>0.9

;

p<0.01

). End-of-training results for CLEVRTex, Super-CLEVR, and MOVi-C ( easy , medium , hard ) across all image representations (small or large point) and downstream models. Oracle is in the top-right (black), and the question-only baseline is in the bottom-left (gray).

Figure 3: Object-centric representations are more compute-efficient. COOD VQA accuracy on Super-CLEVR easy (left), medium (middle), and hard (right) versus downstream compute (log FLOPs) for DINOv2-based models; the question-only baseline is dashed gray. Dense representations slightly outperform on easy but do not match OC performance on hard even with 3

\times

compute.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Dataset examples with question–answer pairs for CLEVRTex.

Figure 7: Dataset examples with question–answer pairs for Super-CLEVR.

Figure 8: Dataset examples with question–answer pairs for MOVi-C.

Shape (8)	Size (3)	Material (8)
cube	small	green tiled
cylinder	medium	blue denim
monkey head	large	red fabric
icosahedron		green forest
teapot		red leather
sphere		rocky gravel

Appendix

Table 2: Attributes for the image and question generation for CLEVRTex.

Shape (21)	Size (2)	Material (2)	Color (8)	Texture (4)
suv	small	rubber	gray	none
wagon	large	metal	red	checkered
minivan			blue	striped
sedan			green	dotted
truck			brown
articulated bus			purple

Appendix

Table 3: Attributes for the image and question generation for Super-CLEVR.

Shape (36)	Size (3)
athletic shoe	small
boot	medium
sandal	large
flip flop
ballet flat
action figure

Appendix

Table 4: Attributes for the image and question generation for MOVi-C.

Hyperparameter	DINOSAURv2	SigLIPSAUR2
Training Steps	300k	300k
Batch Size	128	128
LR Warmup Steps	10k	10k
Peak LR	0.0003	0.0002
LR Schedule	Cosine	Exp. Decay
Exp. Decay Half-Life	-	100k

Appendix

Table 5: Hyperparameters of DINOSAURv2 and SigLIPSAUR2.

		Number of steps
Model Name	Image Repr. Size	TF 2	TF 5
SigLIP2 / SigLIPSAUR2 + CA	[196,768]	641123	688733
DINOv2 / DINOSAURv2 + CA	[256,384]	600000	600000
SigLIP2 + k-means	[128,768]	718928	617749
DINOv2 + k-means	[128,384]	643602	652251
SigLIPSAUR2 / DINOSAURv2 /	[7,256]	762329	809568

Appendix

Table 6: Number of steps for the two downstream models for all image representations.

Figure 9: GFLOPs for one step of the downstream model for image representations with both the smaller (TF 2) and bigger transformer downstream model (TF 5), ignoring the compute needed for resizing with the cross-attention layer.

		MOVi-C
	Model	EASY	MEDIUM	HARD
TF 2	DINOv2	57.7	53.7	51.6
	DINOSAURv2	58.3	55.0	53.2
	SigLIP2	58.3	54.6	53.2
	SigLIPSAUR2	61.4	57.1	54.9
TF 5	DINOv2	60.1	56.3	54.3

Appendix

Table 7: VQA accuracy (%) of both downstream models (TF 2 & 5) on the compositional generalization test sets of MOVi-C with OOD background.

			CLEVRTex
	Model	#slots	EASY	MEDIUM	HARD
TF 2	DINOSAURv2	5	68.2	63.6	51.2
	DINOSAURv2	7 (default)	76.5	71.2	55.6
	DINOSAURv2	9	80.0	72.4	55.1
	DINOSAURv2	11	79.7	73.6	56.3
TF 5	DINOSAURv2	5	71.9	65.3	51.0

Appendix

Table 8: Object-centric model ablations measured by VQA compositional out-of-distribution downstream model accuracy for DINOSAURv2.

	Downstream Model			CLEVRTex
Model	#layers	$d_{model}$	$d_{hidden}$	EASY	MEDIUM	HARD
DINOv2	TF 5 (default)	128	128	85.4	70.3	55.4
DINOSAURv2	TF 5 (default)	128	128	82.5	73.3	55.5
DINOv2	TF 10	128	128	84.9	70.0	55.2
DINOSAURv2	TF 10	128	128	81.3	70.7	53.5
DINOv2	TF 15	128	128	83.8	68.1	53.7

Appendix

Table 9: Downstream model ablations for VQA compositional out-of-distribution accuracy for DINOv2-based representations.

	CLEVRTex			Super-CLEVR			MOVi-C
TF 2	E	M	H	E	M	H	E	M	H
Question-Only Baseline	46.3	48.1	53.0	49.0	50.7	59.5	54.2	55.7	57.5
DINOv2	73.9	76.6	82.6	65.9	68.2	81.2	72.8	75.0	78.0
DINOv2 + CA	73.1	71.2	76.3	65.5	66.7	79.4	69.6	72.4	74.0
DINOv2 + KMeans (7)	59.2	62.2	68.6	55.5	58.0	70.9	61.7	64.2	66.5
DINOv2 + KMeans	73.1	76.3	80.6	65.1	67.7	80.5	72.4	74.8	77.3

Appendix

Table 10: VQA Accuracy in-distribution for all image representations and the small downstream model (TF 2).

	CLEVRTex			Super-CLEVR			MOVi-C
TF 2	E	M	H	E	M	H	E	M	H
Question-Only Baseline	42.1	41.5	41.6	45.0	45.2	44.1	47.7	48.4	48.1
DINOv2	69.5	58.8	50.0	60.9	57.0	49.7	57.5	53.6	51.4
DINOv2 + CA	68.4	54.1	47.7	59.8	55.8	48.9	55.8	53.6	51.7
DINOv2 + KMeans (7)	53.0	49.8	46.9	50.8	49.9	47.5	51.7	51.0	49.8
DINOv2 + KMeans	68.4	60.0	49.4	59.5	56.2	49.6	56.7	53.4	52.4

Appendix

Table 11: VQA Accuracy compositional out-of-distribution for all image representations and the small downstream model (TF 2).

	CLEVRTex			Super-CLEVR			MOVi-C
TF 5	E	M	H	E	M	H	E	M	H
Question-Only Baseline	46.0	48.3	52.8	48.9	50.9	59.9	53.9	55.3	57.0
DINOv2	90.9	93.4	92.9	76.2	79.0	85.2	78.8	82.2	84.0
DINOv2 + CA	85.3	87.7	88.9	73.7	77.9	85.2	77.5	79.8	80.6
DINOv2 + KMeans (7)	58.6	62.9	69.0	55.0	58.2	70.8	62.0	64.1	66.6
DINOv2 + KMeans	86.1	87.9	88.2	71.7	77.1	85.5	79.7	82.2	83.4

Appendix

Table 12: VQA Accuracy in-distribution for all image representations and the big downstream model (TF 5).

	CLEVRTex			Super-CLEVR			MOVi-C
TF 5	E	M	H	E	M	H	E	M	H
Question-Only Baseline	42.0	41.8	41.4	45.0	44.9	43.9	48.7	47.8	48.0
DINOv2	85.4	70.3	55.4	68.1	63.0	51.7	60.0	56.0	54.0
DINOv2 + CA	78.9	67.9	53.8	65.2	61.1	50.8	58.3	55.3	53.2
DINOv2 + KMeans (7)	52.7	49.0	46.4	50.6	49.9	47.2	51.5	50.1	49.1
DINOv2 + KMeans (128)	80.6	66.0	54.1	64.4	61.8	51.3	60.6	56.5	53.9

Appendix

Table 13: VQA Accuracy compositional out-of-distribution for all image representations and the big downstream model (TF 5).

	CLEVRTex			Super-CLEVR			MOVi-C
TF 2	E	M	H	E	M	H	E	M	H
DINOv2	69.5	58.8	50.0	60.9	57.0	49.7	57.5	53.6	51.4
DINOv2 + CA	-1.2	-4.8	-2.3	-1.2	-1.2	-0.8	-1.7	-0.1	0.2
DINOv2 + KMeans (7)	-16.5	-9.1	-3.1	-10.1	-7.1	-2.2	-5.8	-2.6	-1.6
DINOv2 + KMeans	-1.1	1.2	-0.6	-1.4	-0.8	-0.1	-0.7	-0.3	0.9
DINOSAURv2	7.0	12.3	5.6	-0.3	1.6	1.2	1.0	1.1	1.6

Appendix

Table 14: VQA accuracy (%) of both downstream models (TF 2 & 5) on the respective compositional generalization test sets for all models, trained on easy (E), medium (M), and hard (H) training sets. We compute deltas compared to the original pretrained vision encoder.

Figure 10: ID and COOD VQA accuracies are strongly correlated (Pearson and Spearman;

p<0.01

). End-of-training results for Super-CLEVR ( easy , medium , hard ) across all image representations and downstream models. The ground-truth oracle is in the top right (black) and the question-only baseline in the bottom left (gray).

Figure 11: ID and COOD VQA accuracies are strongly correlated (Pearson and Spearman;

p<0.01

). End-of-training results for CLEVRTex ( easy , medium , hard ) across all image representations and downstream models. The ground-truth oracle is in the top right (black) and the question-only baseline in the bottom left (gray).

Figure 12: ID and COOD VQA accuracies are strongly correlated (Pearson and Spearman;

p<0.01

). End-of-training results for MOVi-C ( easy , medium , hard ) across all image representations and downstream models. The ground-truth oracle is in the top right (black) and the question-only baseline in the bottom left (gray).

Figure 13: VQA in-distribution and compositional out-of-distribution accuracy for all CLEVRTex dataset variants with question-only baseline (lower: dashed gray).

Figure 14: VQA in-distribution and compositional out-of-distribution accuracy for all Super-CLEVR dataset variants with question-only baseline (lower: dashed gray).

Figure 15: VQA in-distribution and compositional out-of-distribution accuracy for all MOVi-C dataset variants with question-only baseline (lower: dashed gray).

Figure 16: Compositional generalization of models trained on different subsets of the full data for CLEVRTex, Super-CLEVR and MOVi-C for DINOv2-based models.

Figure 17: Compositional generalization of models trained on different subsets of the full data for CLEVRTex, Super-CLEVR and MOVi-C for SigLIP2-based models.

Compositional Generalization

Compositional generalization is the ability to combine learned parts and rules in ways that transfer to unseen combinations.

See all 110 papers

Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models
Feb 27, 2026Arnas Uselis, Andrea Dittadi, Seong Joon OhNeural Representation Geometry Visual Representation Learning
Tübingen AI Center, University of Tübingen · Helmholtz · Technical University of Munich
What Drives Compositional Generalization in Visual Generative Models? The Importance of Continuous Training Objectives
Oct 3, 2025Karim Farid, Rajat Sahay, Yumna Ali Alnaggar +4Compositional Image Generation Generative Modeling
University of Freiburg · Bosch Center for Artificial Intelligence · Inria, École Normale Supérieure, CNRS, PSL Research University
Is Generation Required for Data-Efficient Perception?
Dec 9, 2025Jack Brady, Bernhard Schölkopf, Thomas Kipf +2Visual Representation Learning Generative Modeling
Max Planck Institute for Intelligent Systems, Tübingen, Germany · Tübingen AI Center · ELLIS Institute, Tübingen +1

Feb 27, 2026J/K move · Enter open · S save

Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models

Arnas Uselis, Andrea Dittadi, Seong Joon Oh

Tübingen AI Center, University of Tübingen · Helmholtz · Technical University of Munich

Neural Representation Geometry Visual Representation Learning Linear Representation Hypothesis Orthogonal Representation Learning Geometric Representation Learning Compositional Representation Learning+1

Open paper AskPDF arXiv

Compositional generalization, the ability to recognize familiar parts in novel contexts, is a defining property of intelligent systems. Although modern models are trained on massive datasets, they still cover only a tiny fraction of the combinatorial space of possible inputs, raising the question of what structure representations must have to support generalization to unseen combinations. We formalize three desiderata for compositional generalization under standard training (divisibility, transferability, stability) and show they impose necessary geometric constraints: representations must decompose linearly into per-concept components, and these components must be orthogonal across concepts. This provides theoretical grounding for the Linear Representation Hypothesis: the linear structure widely observed in neural representations is a necessary consequence of compositional generalization. We further derive dimension bounds linking the number of composable concepts to the embedding geometry. Empirically, we evaluate these predictions across modern vision models (CLIP, SigLIP, DINO) and find that representations exhibit partial linear factorization with low-rank, near-orthogonal per-concept factors, and that the degree of this structure correlates with compositional generalization on unseen combinations. As models continue to scale, these conditions predict the representational geometry they may converge to. Code is available at https://github.com/oshapio/necessary-compositionality.

Abstract

Compositional Generalization

Explore similar work

Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models

What Drives Compositional Generalization in Visual Generative Models? The Importance of Continuous Training Objectives

Is Generation Required for Data-Efficient Perception?