Compositional Objectives: Learning Structure in Structure

Pranavchandra Vivekananda; Sumukh Bettadapura; Ajan Subramanian

cs.LGSep 26, 2026

Compositional Objectives: Learning Structure in Structure

Authors: Pranavchandra Vivekananda, Sumukh Bettadapura, Ajan Subramanian

Organizations: Kubo Technologies, Inc.

Abstract

Intelligence is defined in many ways. One of these definitions defines intelligence as the pursuit of learnable novelty. However, learnable novelty can be meaningless without the ability to compose the learned structures to take action and achieve goals. Learnable novelty builds on epiplexity, which is a way to measure learnable structure in data through a bounded observer. In this paper, we investigate a closed-form spectral approximation to compute epiplexity. We use a fixed-trace constraint and find that the epiplexity objective prefers a more uniform distribution of spectral mass rather than concentrating it in a small number of directions. However, a representation may spread information across many directions without organizing that information into features useful for a particular task. To address this gap, we propose a compositional objective whose observer measures the relationships between the parts and interactions of an image. We compare it with the original spectral objective given only the masked parts. In our ImageNet training runs, the spectral objective with masked parts produces an almost maximally spread representation while achieving the strongest frozen-feature classification performance on most evaluations, more than doubling the linear-probe accuracy of the whole-image baseline. Across multiple image benchmarks, changing what the observer sees matters more than adding relation and interaction tokens. At the same time, our prediction-oriented compositional objective produces substantially better held-out observer prediction but relatively weaker classification, revealing that spectral diversity, predictability, and downstream utility are distinct properties. These results suggest that the usefulness of spectral spreading depends not only on how much structure is preserved, but on which relationships the observer makes available to the objective.

Figures & tables

Figure 1: Identical spectra need not give the same prediction. Two representations $E_{a}(x)$ and $E_{b}(x)$ of the same masked input share a spectrum but lead the decoder to different outputs. A conceptual illustration, not trained models.

Figure 2: The compositional observer’s input tokens. Eight masked views of a $196$ -patch image give $392$ view tokens, $28$ relation tokens from the visibility masks of each view pair, and $28$ interaction tokens that couple both views’ content with their visibility. A frozen transformer maps the $448$ tokens to one feature vector $f(x)$ .

Figure 3: The compositional objective. (a) The encoder $E_{\theta}$ sees the whole image, the frozen observer $\phi_{\mathrm{comp}}$ builds features $F$ from masked views, and a ridge readout $B$ , refitted each batch, predicts the centered encoder outputs; the encoder ascends $S_{\mathrm{comp}}$ . (b) $S_{\mathrm{comp}}$ compares the target covariance $C_{Z}+\epsilon I_{D}$ with the regularized remaining cost $R+\epsilon I_{D}$ .

Evaluation	$S^{\phi}$	$S^{\phi_{\mathrm{comp}}}$	$S_{\mathrm{comp}}$
ImageNet top-1	3.19	7.02	3.30
ImageNet top-5	9.57	17.21	9.64
ImageNet 5-NN	1.62	2.67	1.48
Low-label ImageNet	1.94	2.23	2.07
CIFAR-100	8.32	9.68	6.94
DTD	11.22	11.20	11.33

Table 1: Frozen-encoder accuracy (%) after ten epochs on ImageNet-1K, averaged over two seeds. Bold marks the best of the three methods; per-seed values are in Appendix D .

Measurement	$S^{\phi}$	$S^{\phi_{\mathrm{comp}}}$	$S_{\mathrm{comp}}$
Effective rank (of 192)	39.27	175.23	27.07
Isotropy	0.194	0.869	0.112
Held-out error reduction (%)	–	18.56	83.51
Counted TFLOPs per update	3.826	5.055	5.810
Counted PFLOPs per run	127.62	168.64	193.82

Table 2: Representation diagnostics and counted training compute, averaged over two seeds. Geometry uses the low-label ImageNet test features; held-out reduction uses the common compositional observer and is unavailable for

S^{\phi}

Figure 6: ImageNet top-1 accuracy against (a) isotropy and (b) held-out error reduction, one mark per method and seed.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure D.1: ImageNet linear-probe top-1 accuracy during probe training, one line per seed.

Method	TFLOPs/update	PFLOPs/run	PFLOPs (two seeds)	MAE $\times$
MAE	13.033	434.78	869.56	1.00
$S^{\phi}$	3.826	127.62	255.24	3.41
$S^{\phi_{\mathrm{comp}}}$	5.055	168.64	337.28	2.58
$S_{\mathrm{comp}}$	5.810	193.82	387.63	2.24

Appendix

Table D.1: Counted training compute, extrapolated from one profiled update. The last column gives MAE’s compute as a multiple of each method’s.

Figure D.2: Counted PFLOPs per ten-epoch run, with MAE’s multiple of each method. Counts are lower bounds.

	$S^{\phi}$		$S^{\phi_{\mathrm{comp}}}$		$S_{\mathrm{comp}}$
Evaluation	Seed 0	Seed 1	Seed 0	Seed 1	Seed 0	Seed 1
ImageNet top-1	3.20	3.18	7.02	7.02	3.33	3.26
ImageNet top-5	9.57	9.56	17.35	17.07	9.66	9.62
ImageNet 5-NN	1.64	1.59	2.67	2.68	1.49	1.46
Low-label ImageNet	1.91	1.96	2.23	2.23	2.08	2.05
CIFAR-100	8.97	7.67	9.57	9.79	6.55	7.32

Appendix

Table D.2: Frozen-encoder accuracy (%) for each seed.

Figure D.3: Accuracy of each seed on every evaluation; lines mark the two-seed means.

Encoder	ImageNet (low-label)	CIFAR-100	DTD	EuroSAT
$S^{\phi}$	0.194	0.143	0.134	0.027
$S^{\phi_{\mathrm{comp}}}$	0.869	0.563	0.456	0.039
$S_{\mathrm{comp}}$	0.112	0.091	0.080	0.020

Appendix

Table D.3: Isotropy of frozen codes on each dataset, averaged over two seeds.

Figure D.4: Variance share per principal direction on each evaluation dataset, one line per seed. Dotted: isotropic share

1/192

Dataset	Readout input	$S^{\phi}$	$S^{\phi_{\mathrm{comp}}}$	$S_{\mathrm{comp}}$
SpatialSense	Encoder and metadata	52.84	52.75	52.32
	Encoder only	51.79	50.37	51.01
	Object names only	54.90	54.67	54.90
	Locations only	66.97	67.06	66.97
	Names and locations	57.28	57.01	57.28
CLEVR subset	Encoder and question	44.49	44.44	44.27

Appendix

Table D.4: Relational readout accuracy (%) by input, averaged over two seeds.

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 813 papers

Defining and Generating Compositional Structure
Oct 14, 2026Eric Elmoznino, Tejas Kasetty, Blaise Agüera y Arcas +2Compositional Image Generation Generative Modeling
Mila – Quebec AI Institute · Université de Montréal · Google Paradigms of Intelligence Team +1
Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models
Feb 27, 2026Arnas Uselis, Andrea Dittadi, Seong Joon OhNeural Representation Geometry Visual Representation Learning
Tübingen AI Center, University of Tübingen · Helmholtz · Technical University of Munich
Are Object-Centric Representations Better At Compositional Generalization?
Feb 18, 2026Ferdinand Kapl, Amir Mohammad Karimi Mamaghan, Maximilian Seitzer +4Compositional Generalization Object-Centric Representation Learning
Technical University of Munich · Helmholtz AI, Munich · KTH Royal Institute of Technology +2

Oct 14, 2026J/K move · Enter open · S save

Defining and Generating Compositional Structure

Eric Elmoznino, Tejas Kasetty, Blaise Agüera y Arcas, Yoshua Bengio+1

Mila – Quebec AI Institute · Université de Montréal · Google Paradigms of Intelligence Team+1

Compositional Image Generation Generative Modeling

Open paper AskPDF arXiv

Compositional structure is pervasive in nature and across scales: atoms combine into molecules, molecules into cells, and cells into organisms, while further compositional hierarchies appear within each level. More broadly, compositionality, the reuse and recombination of parts into novel structures, is a fundamental source of interesting complexity in both natural and artificial systems. Yet despite its importance, and despite our apparent ease in recognizing it, mathematical accounts capturing what we intuitively mean by compositional structure remain limited. Here, we fill this gap by formalizing a simple idea: compositional structure emerges when parts, considered together, reveal shared structure that is not captured when those parts are modeled separately. We express this idea using algorithmic information theory, the mathematics of information and compression, which provides a natural way to identify structure in arbitrary data without specifying in advance what that structure must look like. Applied across scales, our definition informs where and how compositional structure emerges within an object. Using tractable approximations, we then invert the framework to generate compositional objects: we specify the desired form of their compositional structure without prescribing their particular content, and optimize directly over strings and images. The resulting objects exhibit readily recognizable hierarchical reuse and part-whole organization, suggesting that the definition captures compositional structure rather than merely complexity, regularity, or shared information. Finally, we discuss implications for machine learning, including a new perspective on depth in neural architectures and principles for constructing compositional data curricula capable of supporting open-ended learning.

Abstract

Representation Learning

Explore similar work

Defining and Generating Compositional Structure

Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models

Are Object-Centric Representations Better At Compositional Generalization?