OrthoRF: Exploring Orthogonality in Object-Centric Representations

Despoina Touska; Bastiaan Onne Fagginger Auer; Alexandru Onose; Tejaswi Kasarla; Luis Armando Pérez Rey; Maximilian Lipp; Lyubov Amitonova; Martin R. Oswald

cs.CVOct 26, 2026

OrthoRF: Exploring Orthogonality in Object-Centric Representations

Authors: Despoina Touska, Bastiaan Onne Fagginger Auer, Alexandru Onose, Tejaswi Kasarla, Luis Armando Pérez Rey, Maximilian Lipp, Lyubov Amitonova, Martin R. Oswald,

Organizations: Faculty of Science, University of Amsterdam, Amsterdam, the Netherlands · ASML Netherlands B.V., Veldhoven, the Netherlands · Advanced Research Center for Nanolithography, Amsterdam, the Netherlands · Faculty of Information, Media and Electrical Engineering, TH Köln, Cologne, Germany

Abstract

Neural synchrony is hypothesized to help the brain organize visual scenes into structured multi-object representations. In machine learning, synchrony-based models analogously learn object-centric representations by storing binding in the phase of complex-valued features. Rotating Features (RF) instantiate this idea with vector-valued activations, encoding object presence in magnitudes and affiliation in orientations. We propose Orthogonal Rotating Features (OrthoRF), which enforces orthogonality in RF's orientation space via an inner-product loss and architectural modifications. This yields sharper phase alignment and more reliable grouping. In evaluations of unsupervised object discovery, including settings with overlapping objects, noise, and out-of-distribution tests, OrthoRF matches or outperforms current models while producing more interpretable representations, and it eliminates the post-hoc clustering required by many synchrony-based approaches. Unlike current models, OrthoRF also recovers occluded object parts, indicating stronger grouping under occlusion. Overall, orthogonality emerges as a simple, effective inductive bias for synchrony-based object-centric learning.

Figures & tables

Figure 1: Binding-by-synchrony. Local excitation, inhibition partitions neurons into phase-based groups. Same-phase firing (e.g., B1–2) encodes the same object; out-of-phase firing (e.g., A1 vs. B1-2) encodes different objects.

Figure 2: RF vs. our OrthoRF. In RF, object content is distributed across orientation space (seen as multicolored bars in encoder activations and mixed final outputs). OrthoRF enforces orthogonal orientation axes, routing each object to a distinct latent, yielding cleaner separation and improved handling of overlap regions.

Figure 3: Qualitative OrthoRF results on 4Shapes, after thresholding $\psi_{\text{final}}$ . Objects occupy distinct dimensions, and occluded parts are recovered.

Model	$n$	$\lambda$	MSE $\downarrow$	ARI-BG $\uparrow$	MBO i $\uparrow$	MBO ${}^{\text{OV}}_{\text{i}}$ $\uparrow$
* AE	-	-	5.492 e- 03 $\pm$ 9.393 e- 04	-	-	-
* CAE	-	-	3.435 e- 03 $\pm$ 2.899 e- 04	0.694 $\pm$ 0.041	0.628 $\pm$ 0.039	-
* RF ${}^{\text{kmeans}}_{\text{z\textsubscript{final}}}$	2	-	2.198 e- 03 $\pm$ 1.110 e- 04	0.667 $\pm$ 0.046	0.589 $\pm$ 0.026	0.5076 $\pm$ 0.032
	3	-	1.112 e- 03 $\pm$ 2.698 e- 04	0.937 $\pm$ 0.018	0.861 $\pm$ 0.020	0.6379 $\pm$ 0.017
	4	-	5.893 e- 04 $\pm$ 4.350 e- 05	0.944 $\pm$ 0.016	0.908 $\pm$ 0.012	0.6981 $\pm$ 0.025
	5	-	5.439 e- 04 $\pm$ 6.984 e- 05	0.975 $\pm$ 0.003	0.934 $\pm$ 0.006	0.8049 $\pm$ 0.013

Table 1: Object discovery on 4Shapes dataset (mean

\pm

standard deviation over 4 seeds). OrthoRF matches RF under identical output post-processing, but surpasses it on the shape-completion task, measured by

\mathrm{MBO}^{\text{OV}}_{\text{i}}

(OV: overlapping regions), when evaluated on

\psi_{final}

. * MSE, ARI-BG, and MBO i are taken from Löwe et al. (2024a) ; RF’s ARI-BG and MBO i were recomputed (see Table 7 Appendix). Best results are highlighted as first , second , and third .

Panel of Table 2 — Table 2: Object discovery on the SEM dataset. OrthoRF outperforms RF under severe occlusions and noise, and shows stronger out-of-distribution generalization (noisy testing after clean training, and vice versa). Best results are highlighted as first , second , and third .

Figure 4: Qualitative results on noise-free SEM test images for $n=5$ . RF’s output (top row) is spread across the orientation components, whereas the thresholded OrthoRF’s output (bottom row) separates the SEM layers and reveals occluded structures.

Model	$n$	ARI-BG $\uparrow$	MBO i $\uparrow$
ItrSA	-	0.570 $\pm$ 0.055	0.348 $\pm$ 0.046
AKOr $N^{\text{attn}}$	-	0.713 $\pm$ 0.042	0.469 $\pm$ 0.014
RF	8	0.744 $\pm$ 0.079	0.780 $\pm$ 0.065
OrthoRF	8	0.833 $\pm$ 0.019	0.865 $\pm$ 0.013

Table 4: Object discovery on Shapes (mean

\pm

standard deviation over 3 seeds). OrthoRF outperforms other models when object count and shape vary randomly. Best results are highlighted as first , second , and third .

Model	SC	$\bm{\lambda}$	MSE $\downarrow$	ARI $\uparrow$	MBO i $\uparrow$
RF	No	0	0.0005	0.9750	0.9340
OrthoRF	No	0.1	0.0002	0.8530	0.8680
	Yes	0	0.0034	0.6280	0.6880
	Yes	0.1	0.0002	0.9995	0.9887

Table 5: Architectural ablations on 4Shapes. Variants differ by the use of softmax with centering (SC) and the orthogonality loss (

\lambda

). Best results are highlighted as first , second , and third .

Data	Model	$\lambda$	Dim.1	Dim.2	Dim.3	Dim.4	Dim.5	Avg.
4Shapes	RF	-	46.7 $\pm$ 0.96	80.3 $\pm$ 2.1	64.7 $\pm$ 1.5	67.7 $\pm$ 2.4	87.0 $\pm$ 1.7	69.28 $\pm$ 13.91
4Shapes	orthoRF	0.5	88.5 $\pm$ 1.29	88.8 $\pm$ 1.3	89.6 $\pm$ 1.0	89.3 $\pm$ 1.0	78.1 $\pm$ 1.7	86.86 $\pm$ 4.39
SEM	RF	-	82.6 $\pm$ 2.6	88.6 $\pm$ 2.6	69.9 $\pm$ 2.1	82.4 $\pm$ 4.7	87.1 $\pm$ 3.3	82.12 $\pm$ 6.57
SEM	orthoRF	0.1	80.0 $\pm$ 3.7	79.0 $\pm$ 2.2	85.8 $\pm$ 2.4	75.5 $\pm$ 3.1	80.7 $\pm$ 2.2	80.2 $\pm$ 3.32

Table 6: Quantifying similarity in phase space via mean pairwise cosine angles (degrees). On 4Shapes, OrthoRF is near-orthogonal with lower variance; on SEM, RF has a slightly higher mean but larger variance. Best results are highlighted as first , and second .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Official label format for the 4Shapes dataset. Occluded regions are labeled with -1 and excluded from evaluation.

Figure 9: Our labeling scheme used to evaluate occluded regions in the 4Shapes dataset. The leftmost image is the input image and all the rest are binary labels for each of the four shapes.

Model	$n$	ARI-BG $\uparrow$	MBO i $\uparrow$
* RF ${}^{\text{kmeans}}_{\text{z\textsubscript{final}}}$	2	0.7198 $\pm$ 0.038	0.6526 $\pm$ 0.065
	3	0.9326 $\pm$ 0.028	0.8615 $\pm$ 0.014
	4	0.9767 $\pm$ 0.028	0.9382 $\pm$ 0.010
	5	0.9931 $\pm$ 0.007	0.9835 $\pm$ 0.007
	6	0.9950 $\pm$ 0.005	0.9885 $\pm$ 0.006
	7	0.9980 $\pm$ 0.001	0.9927 $\pm$ 0.001

Appendix

Table 7: Object discovery on 4Shapes (mean

\pm

standard deviation over 4 seeds). We reran RF for

n\in\{2,3,4,5,6,7,8\}

and additionally evaluated

n=9

. For fair comparison, Table 1 reports the values from Löwe et al. (2024a) ; our reruns closely match the official results.

Table 8: Effect of varying orthogonality loss coefficient

\lambda

and dimensionality

n

of the orientation space on reconstruction loss, ARI, and MBO.

Figure 11: Effect of varying $\lambda$ and number of slots $n$ on ARI and MBO. With larger $n$ , performance remains high across $\lambda$ values, while for smaller $n$ , sufficient orthogonality regularization ( $\lambda>0$ ) is crucial for object separation.

Panel of Figure 12 — Figure 12: Comparison of OrthoRF and RF performance for $n=5$ . Row 1: RF ( Löwe et al., 2024a ) displays distributed object content. Row 2: OrthoRF’s $\psi$ displays distinct objects, including occluded regions. Row 3: OrthoRF’s $z_{out}$ shows distinct objects; holes indicate occluded regions.

	Layer	Feature Dimension	Kernel	Stride	Padding
				Input / Output
	Input	$h\times w\times c$	-	-	-
Encoder	Conv	$h/2\times w/2\times d$	3	2	1 / 0
	Conv	$h/2\times w/2\times d$	3	1	1 / 0
	Conv	$h/4\times w/4\times 2d$	3	2	1 / 0
	Conv	$h/4\times w/4\times 2d$	3	1	1 / 0

Appendix

Table 9: OrthoRF architecture with input dimensions

h,w,c

and feature dimension

d

. All fractions are rounded up. We followed the same architecture as in RF Löwe et al. (2024a) .

	Layer	Feature Dimension	Kernel	Stride	Padding
				Input / Output
	Input	$h\times w\times c$	-	-	-
Encoder	Conv	$h/2\times w/2\times d$	5	2	1 / 0
	Conv	$h/2\times w/2\times d$	5	1	1 / 0
	Conv	$h/4\times w/4\times 2d$	5	2	1 / 0
	Conv	$h/4\times w/4\times 2d$	5	1	1 / 0

Appendix

Table 10: OrthoRF architecture with input dimensions

(h,w,c)

and feature dimension

d

, presented for the MNIST&Shape dataset. All fractional values are rounded upward.

Dataset	SEM	noisy SEM	4Shapes	MNIST&Shape	Shapes
Training Steps	200k	200k	200k	100k	100k
Batch Size	16	16	16	16	16
LR Warmup Steps	-	-	1000	1000	1000
Peak LR	0.001	0.001	0.001	0.001	0.001
Gradient Norm Clipping	0.1	0.1	0.1	0.1	0.1
Feature dim $d$	64	64	64	128	128

Appendix

Table 11: Hyperparameters of the OrthoRF for the five different datasets.

Panel of Figure 13 — Figure 13: Qualitative results of OrthoRF’s $\psi_{final}$ for $n=6$ .

Panel of Figure 14 — Figure 14: Qualitative results of OrthoRF’s $\psi_{final}$ for $n=7$ .

Panel of Figure 15 — Figure 15: Qualitative results of OrthoRF’s $\psi_{final}$ for $n=8$ .

Panel of Figure 16 — Figure 16: Qualitative results of OrthoRF’s $\psi_{final}$ for $n=9$ .

Panel of Figure 17 — Figure 17: Qualitative comparison between RF (top row) and OrthoRF (bottom row) for $n=5$ when trained and tested in the noise-free SEM dataset. We used $z_{final}$ for a fair comparison. The leftmost image in every row is the input image.

Panel of Figure 18 — Figure 18: Qualitative results of OrthoRF for $n=5$ . The top row displays the model’s output ( $z_{final}$ ), while the bottom row shows the thresholded rotations that are used for the metrics calculation. The leftmost image in every row is the input image.

Panel of Figure 19 — Figure 19: Qualitative comparison between of RF (top row) and OrthoRF (bottom row) for $n=5$ when trained and tested in the noisy dataset. We used $z_{final}$ for a fair comparison. The leftmost image in every row is the input image.

Panel of Figure 20 — Figure 20: Qualitative results of OrthoRF for $n=5$ . The top row displays the $\psi_{final}$ and the bottom row $z_{final}$ . The $\psi_{final}$ in this case serves to fill the gaps for a layer such as the ellipsoids layer. The leftmost image in every row is the input image.

Panel of Figure 21 — Figure 21: Visualizations of OrthoRF’s $\bm{z}_{\text{final}}$ (top of each pair) and $\bm{\psi}_{\text{final}}$ (bottom of each pair) for samples from the Shapes dataset.

Object-Centric Representation Learning

Object-Centric Representation Learning studies visual models that organize scenes into distinct objects and their properties, positions, and relations.

See all 94 papers

Mar 2, 2026

ORGAN: Object-Centric Representation Learning using Cycle Consistent Generative Adversarial Networks

Although data generation is often straightforward, extracting information from data is more difficult. Object-centric representation learning can extract information from images in an unsupervised manner. It does so by segmenting an image into its subcomponents: the objects. Each object is then represented in a low-dimensional latent space that can be used for downstream processing. Object-centric representation learning is dominated by autoencoder architectures (AEs). Here, we present ORGAN, a novel approach for object-centric representation learning, which is based on cycle-consistent Generative Adversarial Networks instead. We show that it performs similarly to other state-of-the-art approaches on synthetic datasets, while at the same time being the only approach tested here capable of handling more challenging real-world datasets with many objects and low visual contrast. Complementing these results, ORGAN creates expressive latent space representations that allow for object manipulation. Finally, we show that ORGAN scales well both with respect to the number of objects and the size of the images, giving it a unique edge over current state-of-the-art approaches.

Joël Küchler, Ellen van Maren, Vaiva Vasiliauskaitė +3

Laboratory of Biosensors and Bioelectronics, Institute for Biomedical Engineering, University and ETH Zurich, Gloriastrasse 37/39, Zurich, 8092, Switzerland · Department of Neurology, Insel Gruppe, Bern, Switzerland · Department of Neurology, Department of Bioengineering and Therapeutic Sciences, University of California, San Francisco, USA +2

Unsupervised Visual Concept Discovery Object-Centric Representation Learning Unsupervised Learning Generative Adversarial Networks

Ask PDF

Feb 18, 2026

Are Object-Centric Representations Better At Compositional Generalization?

Compositional generalization, the ability to reason about novel combinations of familiar concepts, is fundamental to human cognition and a critical challenge for machine learning. Object-centric (OC) representations, which encode a scene as a set of objects, are often argued to support such generalization, but systematic evidence in visually rich settings is limited. We introduce a Visual Question Answering benchmark across three controlled visual worlds (CLEVRTex, Super-CLEVR, and MOVi-C) to measure how well vision encoders, with and without object-centric biases, generalize to unseen combinations of object properties. To ensure a fair and comprehensive comparison, we carefully account for training data diversity, sample size, representation size, downstream model capacity, and compute. We use DINOv2 and SigLIP2, two widely used vision encoders, as the foundation models and their OC counterparts. Our key findings reveal that (1) OC approaches are superior in harder compositional generalization settings; (2) original dense representations surpass OC only on easier settings and typically require substantially more downstream compute; and (3) OC models are more sample efficient, achieving stronger generalization with fewer images, whereas dense encoders catch up or surpass them only with sufficient data and diversity. Overall, object-centric representations offer stronger compositional generalization when any one of dataset size, training data diversity, or downstream compute is constrained.

Ferdinand Kapl, Amir Mohammad Karimi Mamaghan, Maximilian Seitzer +4

Technical University of Munich · Helmholtz AI, Munich · KTH Royal Institute of Technology +2

Compositional Generalization Object-Centric Representation Learning Visual Representation Learning

Ask PDF

May 28, 2026

Supervised Training Rapidly Degrades Early Visual Cortex Alignment Across Biologically Plausible Learning Rules

Random, untrained neural networks consistently match or exceed trained networks in representational similarity to early visual cortex. This puzzling finding challenges the assumption that learning improves brain alignment. We investigate it by tracking representational similarity analysis (RSA) alignment to human fMRI data across training for four learning rules: backpropagation (BP), feedback alignment (FA), predictive coding (PC), and spike-timing-dependent plasticity (STDP). Using 720 object images from the THINGS database and fMRI data from three subjects across six visual ROIs, we measure Spearman correlations between model and brain representational dissimilarity matrices at eight training checkpoints (epochs 0-40). We find that (1) a single epoch of training reduces V1 alignment by 25-90%, depending on the learning rule; (2) backpropagation reduces V1 alignment most severely (delta r = -0.080), while predictive coding and STDP preserve substantially more (delta r ~ -0.04); and (3) a weaker, opposite tendency appears in object-selective cortex (LOC), where BP shows the largest increase in alignment during training, although the absolute change is small. These results suggest that untrained architectures capture low-level visual statistics through inductive biases alone, and that global error signals (BP) reshape early representations more aggressively than local learning rules (PC, STDP), which better preserve brain-like structure.

Nils Leutenegger

Independent Researcher, Switzerland

Neural Representation Geometry Predictive Coding Neural Processes Visual Neuroscience Biologically Plausible Learning Neural Network Training Dynamics

Ask PDF

Abstract

Object-Centric Representation Learning

Explore similar work

ORGAN: Object-Centric Representation Learning using Cycle Consistent Generative Adversarial Networks

Are Object-Centric Representations Better At Compositional Generalization?

Supervised Training Rapidly Degrades Early Visual Cortex Alignment Across Biologically Plausible Learning Rules