Compositional Objectives: Learning Structure in Structure

Pranavchandra Vivekananda; Sumukh Bettadapura; Ajan Subramanian

cs.LGSep 26, 2026

Compositional Objectives: Learning Structure in Structure

Authors: Pranavchandra Vivekananda, Sumukh Bettadapura, Ajan Subramanian

Organizations: Kubo Technologies, Inc.

Abstract

Intelligence is defined in many ways. One of these definitions defines intelligence as the pursuit of learnable novelty. However, learnable novelty can be meaningless without the ability to compose the learned structures to take action and achieve goals. Learnable novelty builds on epiplexity, which is a way to measure learnable structure in data through a bounded observer. In this paper, we investigate a closed-form spectral approximation to compute epiplexity. We use a fixed-trace constraint and find that the epiplexity objective prefers a more uniform distribution of spectral mass rather than concentrating it in a small number of directions. However, a representation may spread information across many directions without organizing that information into features useful for a particular task. To address this gap, we propose a compositional objective whose observer measures the relationships between the parts and interactions of an image. We compare it with the original spectral objective given only the masked parts. In our ImageNet training runs, the spectral objective with masked parts produces an almost maximally spread representation while achieving the strongest frozen-feature classification performance on most evaluations, more than doubling the linear-probe accuracy of the whole-image baseline. Across multiple image benchmarks, changing what the observer sees matters more than adding relation and interaction tokens. At the same time, our prediction-oriented compositional objective produces substantially better held-out observer prediction but relatively weaker classification, revealing that spectral diversity, predictability, and downstream utility are distinct properties. These results suggest that the usefulness of spectral spreading depends not only on how much structure is preserved, but on which relationships the observer makes available to the objective.

Figures & tables

Figure 1: Identical spectra need not give the same prediction. Two representations $E_{a}(x)$ and $E_{b}(x)$ of the same masked input share a spectrum but lead the decoder to different outputs. A conceptual illustration, not trained models.

Figure 2: The compositional observer’s input tokens. Eight masked views of a $196$ -patch image give $392$ view tokens, $28$ relation tokens from the visibility masks of each view pair, and $28$ interaction tokens that couple both views’ content with their visibility. A frozen transformer maps the $448$ tokens to one feature vector $f(x)$ .

Figure 3: The compositional objective. (a) The encoder $E_{\theta}$ sees the whole image, the frozen observer $\phi_{\mathrm{comp}}$ builds features $F$ from masked views, and a ridge readout $B$ , refitted each batch, predicts the centered encoder outputs; the encoder ascends $S_{\mathrm{comp}}$ . (b) $S_{\mathrm{comp}}$ compares the target covariance $C_{Z}+\epsilon I_{D}$ with the regularized remaining cost $R+\epsilon I_{D}$ .

Evaluation	$S^{\phi}$	$S^{\phi_{\mathrm{comp}}}$	$S_{\mathrm{comp}}$
ImageNet top-1	3.19	7.02	3.30
ImageNet top-5	9.57	17.21	9.64
ImageNet 5-NN	1.62	2.67	1.48
Low-label ImageNet	1.94	2.23	2.07
CIFAR-100	8.32	9.68	6.94
DTD	11.22	11.20	11.33

Table 1: Frozen-encoder accuracy (%) after ten epochs on ImageNet-1K, averaged over two seeds. Bold marks the best of the three methods; per-seed values are in Appendix D .

Measurement	$S^{\phi}$	$S^{\phi_{\mathrm{comp}}}$	$S_{\mathrm{comp}}$
Effective rank (of 192)	39.27	175.23	27.07
Isotropy	0.194	0.869	0.112
Held-out error reduction (%)	–	18.56	83.51
Counted TFLOPs per update	3.826	5.055	5.810
Counted PFLOPs per run	127.62	168.64	193.82

Table 2: Representation diagnostics and counted training compute, averaged over two seeds. Geometry uses the low-label ImageNet test features; held-out reduction uses the common compositional observer and is unavailable for

S^{\phi}

Figure 6: ImageNet top-1 accuracy against (a) isotropy and (b) held-out error reduction, one mark per method and seed.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure D.1: ImageNet linear-probe top-1 accuracy during probe training, one line per seed.

Method	TFLOPs/update	PFLOPs/run	PFLOPs (two seeds)	MAE $\times$
MAE	13.033	434.78	869.56	1.00
$S^{\phi}$	3.826	127.62	255.24	3.41
$S^{\phi_{\mathrm{comp}}}$	5.055	168.64	337.28	2.58
$S_{\mathrm{comp}}$	5.810	193.82	387.63	2.24

Appendix

Table D.1: Counted training compute, extrapolated from one profiled update. The last column gives MAE’s compute as a multiple of each method’s.

Figure D.2: Counted PFLOPs per ten-epoch run, with MAE’s multiple of each method. Counts are lower bounds.

	$S^{\phi}$		$S^{\phi_{\mathrm{comp}}}$		$S_{\mathrm{comp}}$
Evaluation	Seed 0	Seed 1	Seed 0	Seed 1	Seed 0	Seed 1
ImageNet top-1	3.20	3.18	7.02	7.02	3.33	3.26
ImageNet top-5	9.57	9.56	17.35	17.07	9.66	9.62
ImageNet 5-NN	1.64	1.59	2.67	2.68	1.49	1.46
Low-label ImageNet	1.91	1.96	2.23	2.23	2.08	2.05
CIFAR-100	8.97	7.67	9.57	9.79	6.55	7.32

Appendix

Table D.2: Frozen-encoder accuracy (%) for each seed.

Figure D.3: Accuracy of each seed on every evaluation; lines mark the two-seed means.

Encoder	ImageNet (low-label)	CIFAR-100	DTD	EuroSAT
$S^{\phi}$	0.194	0.143	0.134	0.027
$S^{\phi_{\mathrm{comp}}}$	0.869	0.563	0.456	0.039
$S_{\mathrm{comp}}$	0.112	0.091	0.080	0.020

Appendix

Table D.3: Isotropy of frozen codes on each dataset, averaged over two seeds.

Figure D.4: Variance share per principal direction on each evaluation dataset, one line per seed. Dotted: isotropic share

1/192

Dataset	Readout input	$S^{\phi}$	$S^{\phi_{\mathrm{comp}}}$	$S_{\mathrm{comp}}$
SpatialSense	Encoder and metadata	52.84	52.75	52.32
	Encoder only	51.79	50.37	51.01
	Object names only	54.90	54.67	54.90
	Locations only	66.97	67.06	66.97
	Names and locations	57.28	57.01	57.28
CLEVR subset	Encoder and question	44.49	44.44	44.27

Appendix

Table D.4: Relational readout accuracy (%) by input, averaged over two seeds.

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 804 papers

Oct 14, 2026

Defining and Generating Compositional Structure

Compositional structure is pervasive in nature and across scales: atoms combine into molecules, molecules into cells, and cells into organisms, while further compositional hierarchies appear within each level. More broadly, compositionality, the reuse and recombination of parts into novel structures, is a fundamental source of interesting complexity in both natural and artificial systems. Yet despite its importance, and despite our apparent ease in recognizing it, mathematical accounts capturing what we intuitively mean by compositional structure remain limited. Here, we fill this gap by formalizing a simple idea: compositional structure emerges when parts, considered together, reveal shared structure that is not captured when those parts are modeled separately. We express this idea using algorithmic information theory, the mathematics of information and compression, which provides a natural way to identify structure in arbitrary data without specifying in advance what that structure must look like. Applied across scales, our definition informs where and how compositional structure emerges within an object. Using tractable approximations, we then invert the framework to generate compositional objects: we specify the desired form of their compositional structure without prescribing their particular content, and optimize directly over strings and images. The resulting objects exhibit readily recognizable hierarchical reuse and part-whole organization, suggesting that the definition captures compositional structure rather than merely complexity, regularity, or shared information. Finally, we discuss implications for machine learning, including a new perspective on depth in neural architectures and principles for constructing compositional data curricula capable of supporting open-ended learning.

Eric Elmoznino, Tejas Kasetty, Blaise Agüera y Arcas +2

Mila – Quebec AI Institute · Université de Montréal · Google Paradigms of Intelligence Team +1

Compositional Image Generation Generative Modeling

Ask PDF

Feb 27, 2026

Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models

Compositional generalization, the ability to recognize familiar parts in novel contexts, is a defining property of intelligent systems. Although modern models are trained on massive datasets, they still cover only a tiny fraction of the combinatorial space of possible inputs, raising the question of what structure representations must have to support generalization to unseen combinations. We formalize three desiderata for compositional generalization under standard training (divisibility, transferability, stability) and show they impose necessary geometric constraints: representations must decompose linearly into per-concept components, and these components must be orthogonal across concepts. This provides theoretical grounding for the Linear Representation Hypothesis: the linear structure widely observed in neural representations is a necessary consequence of compositional generalization. We further derive dimension bounds linking the number of composable concepts to the embedding geometry. Empirically, we evaluate these predictions across modern vision models (CLIP, SigLIP, DINO) and find that representations exhibit partial linear factorization with low-rank, near-orthogonal per-concept factors, and that the degree of this structure correlates with compositional generalization on unseen combinations. As models continue to scale, these conditions predict the representational geometry they may converge to. Code is available at https://github.com/oshapio/necessary-compositionality.

Arnas Uselis, Andrea Dittadi, Seong Joon Oh

Tübingen AI Center, University of Tübingen · Helmholtz · Technical University of Munich

Neural Representation Geometry Visual Representation Learning Linear Representation Hypothesis Orthogonal Representation Learning Geometric Representation Learning Compositional Representation Learning+1

Ask PDF

Feb 18, 2026

Are Object-Centric Representations Better At Compositional Generalization?

Compositional generalization, the ability to reason about novel combinations of familiar concepts, is fundamental to human cognition and a critical challenge for machine learning. Object-centric (OC) representations, which encode a scene as a set of objects, are often argued to support such generalization, but systematic evidence in visually rich settings is limited. We introduce a Visual Question Answering benchmark across three controlled visual worlds (CLEVRTex, Super-CLEVR, and MOVi-C) to measure how well vision encoders, with and without object-centric biases, generalize to unseen combinations of object properties. To ensure a fair and comprehensive comparison, we carefully account for training data diversity, sample size, representation size, downstream model capacity, and compute. We use DINOv2 and SigLIP2, two widely used vision encoders, as the foundation models and their OC counterparts. Our key findings reveal that (1) OC approaches are superior in harder compositional generalization settings; (2) original dense representations surpass OC only on easier settings and typically require substantially more downstream compute; and (3) OC models are more sample efficient, achieving stronger generalization with fewer images, whereas dense encoders catch up or surpass them only with sufficient data and diversity. Overall, object-centric representations offer stronger compositional generalization when any one of dataset size, training data diversity, or downstream compute is constrained.

Ferdinand Kapl, Amir Mohammad Karimi Mamaghan, Maximilian Seitzer +4

Technical University of Munich · Helmholtz AI, Munich · KTH Royal Institute of Technology +2

Compositional Generalization Object-Centric Representation Learning Visual Representation Learning

Ask PDF

Abstract

Representation Learning

Explore similar work

Defining and Generating Compositional Structure

Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models

Are Object-Centric Representations Better At Compositional Generalization?