Generative models and vision encoders have largely advanced on separate tracks, optimized for different goals and grounded in different mathematical principles. Yet, they share a fundamental property: latent space Gaussianity. Generative models map Gaussian noise to images, while encoders map images to semantic embeddings whose coordinates empirically behave as Gaussian. We hypothesize that both are views of a shared latent source, the Universal Normal Embedding (UNE): an approximately Gaussian latent space from which encoder embeddings and DDIM-inverted noise arise as noisy linear projections. To test our hypothesis, we introduce NoiseZoo, a dataset of per-image latents comprising DDIM-inverted diffusion noise and matching encoder representations (CLIP, DINO). On CelebA, linear probes in both spaces yield strong, aligned attribute predictions, indicating that generative noise encodes meaningful semantics along linear directions. These directions further enable faithful, controllable edits (e.g., smile, gender, age) without architectural changes, where simple orthogonalization mitigates spurious entanglements. Taken together, our results provide empirical support for the UNE hypothesis and reveal a shared Gaussian-like latent geometry that concretely links encoding and generation. Code and data are available https://rbetser.github.io/UNE/
Figures & tables
Figure 1: UNE conceptual illustration. Different encoders (e.g., CLIP, DINO) and generative models (e.g., SD, LCM) provide different views of the same underlying Gaussian latent structure. Although trained for different objectives, their latents can be interpreted as noisy linear projections of a shared ideal Gaussian space.
Figure 2: Universal Normal Embedding (UNE). A multivariate standard Gaussian latent space representing the encoded data distribution, in which linear directions align with semantics: classes are separable by hyperplanes, and continuous attributes (e.g., “smile”) can be edited by perturbing along a single latent direction.
Figure 3: Classification probing in latent spaces. We train linear attribute classifiers (logistic regression) on latent representations from different models and evaluate accuracy on 40 CelebA attributes. (a) CLIP variants, OpenCLIP variants, and DINOv3 achieve nearly identical performance across attributes, demonstrating that semantic information is linearly accessible. (b) DDIM-inverted noise latents from SD 1.5, SD 2.1, and LCM achieve accuracy highly correlated with a strong encoder baseline (CLIP-B/16), despite originating from diffusion noise rather than semantic encoders. For clarity, only 10 representative attribute names are displayed.
Figure 4: Linear latent editing across semantic attributes. For DDIM-inverted SD 1.5 latents, we move along linear classifier-derived semantic directions ( z~=z+αw ). Each row shows decreasing (left) and increasing (right) attribute intensity as α varies, middle image in each triplet is the original image. No prompts or model tuning are used; edits are controlled solely by linear shifts in the latent space.
Figure 5: Removing spurious attribute correlations. Edits performed using the raw semantic direction (bottom) unintentionally modify a correlated attribute (e.g., adding a goatee also changes facial structure). Using the orthogonalized direction (top) from Equation 4 isolates the target attribute while suppressing the spurious one, yielding clean, disentangled edits.
Figure 6: Classification accuracy in shared latent spaces. X1 – X4 denote shared spaces computed from four latent sources, and X5 from six, with combinations detailed in Section 4.4 . (a) Attribute classification accuracy as a function of latent dimension for shared spaces X1 – X5 , showing strong performance even at low dimensions. (b) Linear-probe accuracy at 16 dimensions in the PCA-reduced latent spaces of each model and in the low-dimensional shared spaces X1 – X5 , indicating that the shared space intersections retain comparable attribute information. (c) Retrieval-based analysis: pairwise correlations (Spearman rank correlation) of similarity vectors (computed over 10k latents) between the shared spaces X1 – X5 , demonstrating that they encode highly similar underlying structure.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Editing in different latent spaces. The figure compares linear attribute editing across three latent spaces: two diffusion models and CLIP. The diffusion latents preserve the structure of the original image, so shifting along an attribute direction produces a modified version of the same image. In contrast, CLIP’s latent space is not invertible to the pixel domain, so reconstruction yields a newly synthesized image that matches the target attribute but does not reconstruct the input. This highlights the trade-off: CLIP offers strong semantic control but poor original image faithfulness.
Figure 8: Quantitative editing tests. All measures are presented as a function of the edited attribute intensity , a normalized measure derived from the distance between the resulting latent and the appropriate classifier’s decision plane. Edits are performed on SD 1.5 latents. Note that an x-axis value of 0 does not indicate no editing, but corresponds to editing the latent to the classifier’s decision plane. Cosine similarities are measured between CLIP ViT-L/14 embeddings. (a) Cosine similarity between an edited image and the original image. (b) Cosine similarity between an edited image and CLIP text embeddings of the attribute’s name.
Figure 9: Classification AUC. Left: CelebA across different latent spaces. Right: AFHQ binary classification using SD 1.5 latents (AUC values).
Figure 10: Linear latent editing of animal faces. We apply the method from Section 4.3 to the AFHQ dataset, which contains three categories: Cat, Dog, and Wild.
Latent space alignment maps learned representations into compatible coordinates to enable comparison, transfer, and interoperability across models or data sources.