Generative models and vision encoders have largely advanced on separate tracks, optimized for different goals and grounded in different mathematical principles. Yet, they share a fundamental property: latent space Gaussianity. Generative models map Gaussian noise to images, while encoders map images to semantic embeddings whose coordinates empirically behave as Gaussian. We hypothesize that both are views of a shared latent source, the Universal Normal Embedding (UNE): an approximately Gaussian latent space from which encoder embeddings and DDIM-inverted noise arise as noisy linear projections. To test our hypothesis, we introduce NoiseZoo, a dataset of per-image latents comprising DDIM-inverted diffusion noise and matching encoder representations (CLIP, DINO). On CelebA, linear probes in both spaces yield strong, aligned attribute predictions, indicating that generative noise encodes meaningful semantics along linear directions. These directions further enable faithful, controllable edits (e.g., smile, gender, age) without architectural changes, where simple orthogonalization mitigates spurious entanglements. Taken together, our results provide empirical support for the UNE hypothesis and reveal a shared Gaussian-like latent geometry that concretely links encoding and generation. Code and data are available https://rbetser.github.io/UNE/
Figures & tables
Figure 1: UNE conceptual illustration. Different encoders (e.g., CLIP, DINO) and generative models (e.g., SD, LCM) provide different views of the same underlying Gaussian latent structure. Although trained for different objectives, their latents can be interpreted as noisy linear projections of a shared ideal Gaussian space.
Figure 2: Universal Normal Embedding (UNE). A multivariate standard Gaussian latent space representing the encoded data distribution, in which linear directions align with semantics: classes are separable by hyperplanes, and continuous attributes (e.g., “smile”) can be edited by perturbing along a single latent direction.
Figure 3: Classification probing in latent spaces. We train linear attribute classifiers (logistic regression) on latent representations from different models and evaluate accuracy on 40 CelebA attributes. (a) CLIP variants, OpenCLIP variants, and DINOv3 achieve nearly identical performance across attributes, demonstrating that semantic information is linearly accessible. (b) DDIM-inverted noise latents from SD 1.5, SD 2.1, and LCM achieve accuracy highly correlated with a strong encoder baseline (CLIP-B/16), despite originating from diffusion noise rather than semantic encoders. For clarity, only 10 representative attribute names are displayed.
Figure 4: Linear latent editing across semantic attributes. For DDIM-inverted SD 1.5 latents, we move along linear classifier-derived semantic directions ( z~=z+αw ). Each row shows decreasing (left) and increasing (right) attribute intensity as α varies, middle image in each triplet is the original image. No prompts or model tuning are used; edits are controlled solely by linear shifts in the latent space.
Figure 5: Removing spurious attribute correlations. Edits performed using the raw semantic direction (bottom) unintentionally modify a correlated attribute (e.g., adding a goatee also changes facial structure). Using the orthogonalized direction (top) from Equation 4 isolates the target attribute while suppressing the spurious one, yielding clean, disentangled edits.
Figure 6: Classification accuracy in shared latent spaces. X1 – X4 denote shared spaces computed from four latent sources, and X5 from six, with combinations detailed in Section 4.4 . (a) Attribute classification accuracy as a function of latent dimension for shared spaces X1 – X5 , showing strong performance even at low dimensions. (b) Linear-probe accuracy at 16 dimensions in the PCA-reduced latent spaces of each model and in the low-dimensional shared spaces X1 – X5 , indicating that the shared space intersections retain comparable attribute information. (c) Retrieval-based analysis: pairwise correlations (Spearman rank correlation) of similarity vectors (computed over 10k latents) between the shared spaces X1 – X5 , demonstrating that they encode highly similar underlying structure.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Editing in different latent spaces. The figure compares linear attribute editing across three latent spaces: two diffusion models and CLIP. The diffusion latents preserve the structure of the original image, so shifting along an attribute direction produces a modified version of the same image. In contrast, CLIP’s latent space is not invertible to the pixel domain, so reconstruction yields a newly synthesized image that matches the target attribute but does not reconstruct the input. This highlights the trade-off: CLIP offers strong semantic control but poor original image faithfulness.
Figure 8: Quantitative editing tests. All measures are presented as a function of the edited attribute intensity , a normalized measure derived from the distance between the resulting latent and the appropriate classifier’s decision plane. Edits are performed on SD 1.5 latents. Note that an x-axis value of 0 does not indicate no editing, but corresponds to editing the latent to the classifier’s decision plane. Cosine similarities are measured between CLIP ViT-L/14 embeddings. (a) Cosine similarity between an edited image and the original image. (b) Cosine similarity between an edited image and CLIP text embeddings of the attribute’s name.
Figure 9: Classification AUC. Left: CelebA across different latent spaces. Right: AFHQ binary classification using SD 1.5 latents (AUC values).
Figure 10: Linear latent editing of animal faces. We apply the method from Section 4.3 to the AFHQ dataset, which contains three categories: Cat, Dog, and Wild.
Latent space alignment maps learned representations into compatible coordinates to enable comparison, transfer, and interoperability across models or data sources.
Latent diffusion models (LDMs) enable high-fidelity synthesis by operating in learned latent spaces. However, training state-of-the-art LDMs requires complex staging: a tokenizer must be trained first, before the diffusion model can be trained in the frozen latent space. We propose UNITE - an autoencoder architecture for unified tokenization and latent diffusion. UNITE consists of a Generative Encoder that serves as both image tokenizer and latent generator via weight sharing. Our key insight is that tokenization and generation can be viewed as the same latent inference problem under different conditioning regimes: tokenization infers latents from fully observed images, whereas generation infers them from noise together with text or class conditioning. Motivated by this, we introduce a single-stage training procedure that jointly optimizes both tasks via two forward passes through the same Generative Encoder. The shared parameters enable gradients to jointly shape the latent space, encouraging a "common latent language". Across image and molecule modalities, UNITE achieves near state of the art performance without adversarial losses or pretrained encoders (e.g., DINO), reaching FID 2.12 and 1.73 for Base and Large models on ImageNet 256 x 256. We further analyze the Generative Encoder through the lenses of representation alignment and compression. These results show that single stage joint training of tokenization & generation from scratch is feasible.
We present Unified Latents (UL), a framework for learning latent representations that are jointly regularized by a diffusion prior and decoded by a diffusion model. By linking the encoder's output noise to the prior's minimum noise level, we obtain a simple training objective that provides a tight upper bound on the latent bitrate. On ImageNet-512, our approach achieves competitive FID of 1.4, with high reconstruction quality (PSNR) while requiring fewer training FLOPs than models trained on Stable Diffusion latents. On Kinetics-600, we set a new state-of-the-art FVD of 1.3.
Latent Diffusion Models (LDMs) typically adopt a two-stage pipeline: an auto-encoder (AE) is first pre-trained to define a latent space, then a diffusion model is trained to perform denoising within it. Such a two-stage design introduces a representation mismatch, as the latent space is optimized for reconstruction rather than adapting the denoising dynamics. We reveal that the LDM itself is an AE, and consequently present LDM-is-AE, an end-to-end one-stage LDM training framework that eliminates the need for a separately trained tokenizer. Our key observation is that the LDM backbone actually performs a latent-to-feature-to-latent transformation at each denoising step, which can be interpreted as an internal decoding--encoding process. Leveraging this structure, we split the DiT backbone into two reciprocal components, DiT-E (i.e., DiT Encoding) and DiT-D (i.e., DiT Decoding), and impose image-space supervision on the intermediate features across all timesteps. Our model encourages the internal representation to align with the image domain throughout denoising, thereby establishing an explicit latent-to-image-to-latent path. At the zero-noise timestep, our model further performs an image-to-latent-to-image mapping, corresponding to an auto-encoding process. As a result, LDM-is-AE jointly learns latent representations and denoising dynamics in an end-to-end manner, yielding a diffusion-native latent space tailored to the generation process. Experiments demonstrate that LDM-is-AE exhibits highly competitive generation performance, achieving an FID of 1.80 and 1.90 on 256x256 and 512x512 class-conditional image generation, respectively.
Zhengqiang Zhang, Lingchen Sun, Rongyuan Wu +4
The Hong Kong Polytechnic University · OPPO Research Institute