ORGAN: Object-Centric Representation Learning using Cycle Consistent Generative Adversarial Networks
Authors: Joël Küchler, Ellen van Maren, Vaiva Vasiliauskaitė, Katarina Vulić, Reza Abbasi-Asl, Stephan J. Ihle
Organizations: Laboratory of Biosensors and Bioelectronics, Institute for Biomedical Engineering, University and ETH Zurich, Gloriastrasse 37/39, Zurich, 8092, Switzerland · Department of Neurology, Insel Gruppe, Bern, Switzerland · Department of Neurology, Department of Bioengineering and Therapeutic Sciences, University of California, San Francisco, USA · Department of Neurobiology, University of Chicago, 951 E 58th St, Chicago, 60637, IL, USA · Department of Physics, University of Chicago, 929 E 57th St, Chicago, 60637, IL, USA
Although data generation is often straightforward, extracting information from data is more difficult. Object-centric representation learning can extract information from images in an unsupervised manner. It does so by segmenting an image into its subcomponents: the objects. Each object is then represented in a low-dimensional latent space that can be used for downstream processing. Object-centric representation learning is dominated by autoencoder architectures (AEs). Here, we present ORGAN, a novel approach for object-centric representation learning, which is based on cycle-consistent Generative Adversarial Networks instead. We show that it performs similarly to other state-of-the-art approaches on synthetic datasets, while at the same time being the only approach tested here capable of handling more challenging real-world datasets with many objects and low visual contrast. Complementing these results, ORGAN creates expressive latent space representations that allow for object manipulation. Finally, we show that ORGAN scales well both with respect to the number of objects and the size of the images, giving it a unique edge over current state-of-the-art approaches.
Figures & tables
Figure 1: Model architecture and training losses. ORGAN is based on the Cycle-Consistent Generative Adversarial Network (CycleGAN) architecture Zhu et al. (2017) , adapted to use an image (left) and a list (right) domain. Two GANs are used to transform between the two domains. By cycling through both generators, the cycle-consistency losses LCycL and LCycI can be enforced.
Figure 2: Network architectures used in this work. (A) List generator architecture, which transforms images into lists. First, the image is divided into patches and a score function is used, which predicts for each patch how likely it is to contain an object. After non-maximum suppression, a differentiable top-k operator detects k patches that contain objects. The feature network then extracts the object’s features. (B) Image generator architecture, which transforms lists into images. First, the list is transformed into an image based on its location and features. For that, each feature vector ηi is projected ( ζ ) onto a unit hypersphere. Afterwards, the vector is scaled by the object existence probability α . Then, a similar approach as presented in BlobGAN is used Epstein et al. (2022) to convert the projections into an image. After adding a noise channel to allow for richer backgrounds, a U-Net transforms the style of the image.
Figure 3: Examples of object detection performance for four different datasets. We show performance for ORGAN (ours), SPACE Lin et al. (2020) , SLATE Singh et al. (2021) , LSD Jiang et al. (2023) and SPOT Kakogeorgiou et al. (2024) . We present both the reconstruction and the object detection in the form of fixed bounding boxes for ORGAN and SPACE. The color code of the bounding box represents the extracted feature vector. For SLATE, LSD, and SPOT we present the recovered slot masks (color-coded). SPOT does not allow for image reconstruction.
Tetrominoes
Multi-MNIST
Sprites
Cells
Method
F1
Train/Infer
F1
Train/Infer
F1
Train/Infer
F1
Train/Infer
ORGAN
89.2 ± 20.5
44.0 / 2.5
86.1 ± 3.9
39.0 / 3.5
88.1 ± 2.3
36.0 / 4.0
77.9 ± 8.6
31.5 / 3.4
SPACE
76.0 ± 15.7
13.5 / 17.8
88.7 ± 4.7
16.5 / 15.0
88.5 ± 4.3
4.08 / 29.6
0.0 ± 0.0
13.0 / 15.0
SLATE
30.3 ± 24.8
6.75 / 3.4
5.8 ± 7.6
0.75 / 3.4
12.4 ± 10.6
0.75 / 3.0
46.6 ± 9.3
0.75 / 3.5
LSD
20.0 ± 39.8
7.5 / 8.0
5.7 ± 28.0
16.0 / 4.0
6.4 ± 17.7
11.5 / 7.0
53.9 ± 11.9
10.5 / 7.0
SPOT
13.8 ± 44.6
10.0 / 37.5
5.2 ± 43.0
10.0 / 36.3
0.5 ± 21.2
10.0 / 32.8
24.8 ± 0.0
10.0 / 32.8
Table 1: F1-scores (in %) and training/inference times (hours/ms) for all models on 128 × 128 images.
Figure 4: Latent space quality of ORGAN. (A) The latent space as encoded by η is presented for the Sprites dataset. Two of the three dimensions are sampled at 12 equally spaced values across the feature space. The third dimension is kept fixed at its center. Each combination of feature values is given to the image generator. The corresponding objects are generated and visualized in separate sub-images. The feature space encodes the color, shape, and size of the object. (B) An input image of the Tetrominoes dataset is cycled, while the list elements are modified. Top: All objects were moved closer to the center. Bottom: The object properties were swapped between the three objects counterclockwise. (C) To compare how well object clustering works on the feature space, the Davies–Bouldin Index Davies and Bouldin (1979) was calculated for ORGAN and SPACE, which detected distinct objects for Sprites and Tetrominoes . Our method achieves superior separability.
Figure 5: Analysis of the generalization capabilities of the object detection accuracy. (A) Recall and precision are plotted for both ORGAN and SPACE for Sprites . The architecture of SPACE can only be applied to a fixed input size (here 256×256 ). ORGAN was trained on 128×128 and applied to 256×256 . This training approach gives SPACE an advantage because it is trained directly at the evaluation resolution. Yet, ORGAN outperforms in precision, while SPACE has a higher recall. (B) ORGAN can also generalize to much larger images. Trained on patches of size 128×128 , prediction results on a 768×768 image of the Cells dataset are presented. The color code of the bounding box illustrates differences in extracted features η .
Deploying object-centric models for real-world scene understanding typically requires complex pipelines to achieve both robust scene decomposition and high-fidelity generation. Recent diffusion-based approaches have improved visual quality, but they almost universally rely on heavy, pretrained generative priors (e.g., Stable Diffusion) and external VAE latent spaces. In this paper, we propose Slot-RAE, a much simpler, fully integrated framework that operates directly within the continuous semantic feature space of visual foundation models (e.g., DINOv3). Slot-RAE employs a feature-space diffusion process using a Diffusion Transformer (DiT) decoder and a Representation Alignment (REPA) head. Unlike existing diffusion-based objectcentric methods that rely heavily on subsidized text-toimage priors, the generative core of Slot-RAE (Slot Attention and the DiT) is trained from scratch within the frozen VFM feature space. This eliminates the need for VAE bottlenecks and task-agnostic generative pre-training. Experiments on the COCO dataset demonstrate that despite its architectural simplicity, Slot-RAE achieves state-of-the-art results. It delivers comparable unsupervised object discovery, higher-fidelity image reconstruction, and robust zero-shot compositionality, all while being significantly faster and more computationally efficient than existing object-centric latent diffusion models.
Compositional generalization, the ability to reason about novel combinations of familiar concepts, is fundamental to human cognition and a critical challenge for machine learning. Object-centric (OC) representations, which encode a scene as a set of objects, are often argued to support such generalization, but systematic evidence in visually rich settings is limited. We introduce a Visual Question Answering benchmark across three controlled visual worlds (CLEVRTex, Super-CLEVR, and MOVi-C) to measure how well vision encoders, with and without object-centric biases, generalize to unseen combinations of object properties. To ensure a fair and comprehensive comparison, we carefully account for training data diversity, sample size, representation size, downstream model capacity, and compute. We use DINOv2 and SigLIP2, two widely used vision encoders, as the foundation models and their OC counterparts. Our key findings reveal that (1) OC approaches are superior in harder compositional generalization settings; (2) original dense representations surpass OC only on easier settings and typically require substantially more downstream compute; and (3) OC models are more sample efficient, achieving stronger generalization with fewer images, whereas dense encoders catch up or surpass them only with sufficient data and diversity. Overall, object-centric representations offer stronger compositional generalization when any one of dataset size, training data diversity, or downstream compute is constrained.
Ferdinand Kapl, Amir Mohammad Karimi Mamaghan, Maximilian Seitzer +4
Technical University of Munich · Helmholtz AI, Munich · KTH Royal Institute of Technology +2
Self-supervised image encoders deliver strong features for downstream tasks but need many images for training. A natural remedy to counter this is to make each image count for more. A scene contains many objects, and given masks from human annotators or an off-the-shelf segmentation model, pre-training can focus on aligning per-object rather than image-wide representations, extracting more signal from every image. Existing mask-supervised methods do this through reconstruction or contrastive losses that leverage negative objects. We instead use two separate projection spaces for the alignment. In a semantic space, per-object representations from different views are aligned. To avoid collapse, instead of using negative objects, which requires category definitions, we extend the negative-free LeJEPA objective and show that its distributional anti-collapse regularizer ports naturally from whole images to the variable-sized set of objects in a scene. In an instance space, a contrastive loss separates per-object representations from their context and co-occurring instances, including those of the same category. To separate object representations from their context, we copy objects and paste them into other contexts, where each pasted copy serves as an additional view of the original object. Trained on COCO with ground-truth masks, our method outperforms image-level and mask-guided baselines on tracking (DAVIS), classification (ImageNet-1k) and re-identification (NAVI), matches the best of them on semantic segmentation (ADE20k) and keeps its lead over image-level LeJEPA and a supervision-matched alternative on COCO fractions down to 256 images.