The Latent Color Subspace: Emergent Order in High-Dimensional Chaos

Mateusz Pach; Jessica Bader; Quentin Bouniot; Serge Belongie; Zeynep Akata

cs.LGMar 12, 2026

The Latent Color Subspace: Emergent Order in High-Dimensional Chaos

Authors: Mateusz Pach, Jessica Bader, Quentin Bouniot, Serge Belongie, Zeynep Akata

Organizations: Technical University of Munich · Helmholtz Munich · Munich Center for Machine Learning · LTCI, Télécom Paris, Institut Polytechnique de Paris ·

Abstract

Text-to-image generation models have advanced rapidly, yet achieving fine-grained control over generated images remains difficult, largely due to limited understanding of how semantic information is encoded. We develop an interpretation of the color representation in the Variational Autoencoder latent space of FLUX.1 [Dev], revealing a structure reflecting Hue, Saturation, and Lightness. We verify our Latent Color Subspace (LCS) interpretation by demonstrating that it can both predict and explicitly control color, introducing a fully training-free method in FLUX based solely on closed-form latent-space manipulation. Code is available at https://github.com/ExplainableML/LCS.

Figures & tables

Figure 1: We find a simple color subspace in the VAE embedding space of FLUX which can be interpreted as cylindrical coordinates corresponding to Hue, Saturation, and Lightness, enabling (1) inexpensive observation and (2) targeted intervention.

Panel of Figure 2 — Figure 2: PCA shows color organization in the VAE latent space mirrors HSL: Hue forms a circle on the PC2–PC3 plane, Saturation is distance from the black-white axis, and Lightness lies on PC1.

Figure 3: Flow Matching introduces an additional layer of complexity to our interpretation, as latents traverse the space over timesteps to reach their final destination. (a) In the Latent Color Subspace (LCS), colors evolve over timesteps $t$ , starting mixed at the center and gradually moving toward their final positions. Dots represent individual patches, indicated in their ultimate colors, while stars orient the space with known color locations at $t=50$ . (b) Despite variation in individual patches, the expected relative position between colors stays consistent over timesteps in the LCS, but scaled with time. Shown on per-image averaged patches (circle) of 26 single-colored images.

Figure 4: The Latent Color Subspace (LCS) enables observation and intervention during generation. At intermediate timestep $t$ , we project the mid-generated sample from the FM VAE latent space ( ) into the LCS ( ) obtaining coordinates $\mathbf{C}$ and rescaling them to $\hat{\mathbf{C}}$ , which matches timestep $t=50$ statistics ( ). Type II intervention ( $\hat{\mathbf{C}}^{\prime\prime}$ ) modifies color by shifting, scaling, rotating to match the lightness, saturation, and hue respectively, while Type I intervention ( $\hat{\mathbf{C}}^{\prime}$ ) directly shifts to adjust all three. The interventions are interpolated to get $\hat{\mathbf{C}}^{\star}$ and rescaled back to timestep $t$ ( $\mathbf{C}^{\star}$ ). Finally $\mathbf{C}$ is replaced with $\mathbf{C}^{\star}$ in the latent of the generated sample. With a simple projection into the LCS and the correct scaling, we can directly observe color ( $O_{t}$ ) without the computationally heavy VAE decoder.

Figure 5: With our mid-generation color observation method (top), we validate our interpretation of the Latent Color Subspace (LCS) by predicting the final colors at intermediate timesteps. We compare these predictions with the VAE-decoded latents (bottom).

Table 1: Perceptual color difference (

\Delta E_{00}

) between final images and latents at timestep

t

, observed by VAE-decoding (VAE t ) and interpreting the LCS (

O_{t}

). Note that by FLUX’s design, VAE 50 is the final image and latents at

t=0

are pure noise.

Figure 6: We investigate how to manipulate the LCS while preserving image quality. Color intervention shifting latent patches directly in LCS (Type I) disrupts texture, whereas shifting them via HSL (Type II) may have limited impact at early timesteps. Interpolating (Type $\star$ ) enables accurate color changes while preserving texture, so we select it at timestep 9 ( ${\color[rgb]{1,0.8516,0.3984}\blacklozenge}$ ) for the final method.

Figure 7: With our latent-space color interpretation, we can accurately guide objects toward target colors (top) while preserving much of the original image’s high-level structure (left). Even multi-colored objects retain color diversity while shifting toward the target (row 3).

Color	GenEval	Precise (natural)				Precise (plain)
Injection	Acc ( $\uparrow$ )	$\Delta E_{00}$ ( $\downarrow$ )	$\Delta H$ ( $\downarrow$ )	$\Delta S$ ( $\downarrow$ )	$\Delta L$ ( $\downarrow$ )	$\Delta E_{00}$ ( $\downarrow$ )	$\Delta H$ ( $\downarrow$ )	$\Delta S$ ( $\downarrow$ )	$\Delta L$ ( $\downarrow$ )
None	9 $\%$	42	86 ∘	59 $\%$	21 $\%$	39	88 ∘	50 $\%$	22 $\%$
Prompt	79 $\%$	23	30 ∘	43 $\%$	14 $\%$	20	24 ∘	36 $\%$	13 $\%$
LCS ( $\star$ , local)	68 $\%$	14	14 ∘	37 $\%$	6 $\%$	-	-	-	-
LCS ( $\star$ , global)	72 $\%$	16	18 ∘	33 $\%$	9 $\%$	11	8 ∘	22 $\%$	8 $\%$

Table 2: Accuracy of our color intervention with GenEval ’s color task and the Precise tasks, including natural / plain images in 51 colors. We measure CIEDE2000 (

\Delta E_{00}

) and average distances in hue (

\Delta H

), saturation (

\Delta S

), and lightness (

\Delta L

) from target color. As baseline, we include results without specifying colors in the prompts (None). Our method effectively alters colors, affecting either entire image (global) or target object (local), without modifying this prompt. For comparison, we include color injected via prompt (Prompt).

Color Inj.	IOU ( $\uparrow$ )	SSIM ( $\uparrow$ )	LPIPS ( $\downarrow$ )	DINOv2 ( $\downarrow$ )
Prompt	0.60	0.46	0.49	0.36
LCS ( $\star$ , local)	0.78	0.59	0.35	0.29
LCS ( $\star$ , global)	0.88	0.56	0.36	0.23

Table 3: Measured against the base prompt, we preserve original image structure more faithfully than prompt-based color changes.

Color Injection	$\Delta E_{00}$ ( $\downarrow$ )	SSIM ( $\uparrow$ )	LPIPS ( $\downarrow$ )
Prompt	27	0.48	0.48
BoN ( $N=50$ )	19	0.32	0.57
ColorPeel ( Butt et al., 2024 )	29	0.22	0.66
ReNO ( Eyring et al., 2024 )	23	0.25	0.61
IP-Adapter ( Ye et al., 2023 )	40	0.29	0.52
LCS ( $\star$ , local)	10	0.62	0.32

Table 4: We compare our local method against other color generation techniques on Precise (natural, small) , finding that it achieves the best color matching (

\Delta E_{00}

) while also better preserving structural consistency (SSIM, LPIPS).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Demonstration of our interpolation method’s ability to generate novel hues, spanning red to magenta.

Figure 9: Demonstration of our interpolation method’s ability to control saturation, spanning blue to grey.

Figure 10: Demonstration of our interpolation method’s ability to control lightness, spanning from white to black through red.

Figure 11: Examples demonstrating multiple LCS color interventions applied within a single image.

Figure 12: Qualitative examples of our observation method at a variety of timesteps, showcasing the accuracy of our Latent Color Space.

Figure 13: Fine-grained modifications can be beneficial, as they may update the objects in ways that make them more realistic.

Figure 14: Color interventions in the basic setting preserve structural and high-level semantic information, but with better bounding boxes and a later intervention timestep, even fine-grained details can be preserved.

Figure 15: Comparison of the structural changes in images when comparing intervention via the LCS to those by other color generation methods.

Color Inj.	Precise (natural, small)
	$\Delta E_{00}$ ( $\downarrow$ )	$\Delta H$ ( $\downarrow$ )	$\Delta S$ ( $\downarrow$ )	$\Delta L$ ( $\downarrow$ )
None	45	80	62	19
Prompt	27	38	42	14
Best of $N=10_{None}$	39	72	64	14
Best of $N=20_{None}$	37	70	63	15
Best of $N=50_{None}$	34	68	64	14

Appendix

Table 5: Comparison to other color-control alternatives.

Color Inj.	Precise (natural, bright)				Precise (natural, muted)
Color Inj.	$\Delta E_{00}$ ( $\downarrow$ )	$\Delta H$ ( $\downarrow$ )	$\Delta S$ ( $\downarrow$ )	$\Delta L$ ( $\downarrow$ )	$\Delta E_{00}$ ( $\downarrow$ )	$\Delta H$ ( $\downarrow$ )	$\Delta S$ ( $\downarrow$ )	$\Delta L$ ( $\downarrow$ )
None	47	90	73	15	39	90	27	15
Prompt	29	28	51	10	21	28	21	10
LCS( $\star$ , local)	11	8	28	7	14	16	19	6
LCS( $\star$ , global)	13	8	24	9	17	18	16	11

Appendix

Table 6: Our intervention method’s performance on high (bright) and low (muted) saturation colors.

Color Inj.	Precise (natural, light)				Precise (natural, dark)
Color Inj.	$\Delta E_{00}$ ( $\downarrow$ )	$\Delta H$ ( $\downarrow$ )	$\Delta S$ ( $\downarrow$ )	$\Delta L$ ( $\downarrow$ )	$\Delta E_{00}$ ( $\downarrow$ )	$\Delta H$ ( $\downarrow$ )	$\Delta S$ ( $\downarrow$ )	$\Delta L$ ( $\downarrow$ )
None	48	90	73	37	35	90	73	15
Prompt	26	30	54	25	17	22	53	7
LCS( $\star$ , local)	17	16	53	6	16	16	55	5
LCS( $\star$ , global)	17	20	52	8	18	18	45	8

Appendix

Table 7: Our intervention method’s performance on high (light) and low (dark) lightness colors.

Color Inj.	$\Delta H_{HCL}$ ( $\downarrow$ )	$\Delta C_{HCL}$ ( $\downarrow$ )	$\Delta L_{HCL}$ ( $\downarrow$ )	$\Delta a^{*}$ ( $\downarrow$ )	$\Delta b^{*}$ ( $\downarrow$ )
No Color	0.83	0.47	0.27	0.44	0.46
Prompt	0.30	0.34	0.17	0.30	0.27
LCS( $\star$ , local)	0.16	0.30	0.09	0.23	0.21
LCS( $\star$ , global)	0.18	0.28	0.11	0.22	0.20

Appendix

Table 8: Metrics from other color systems evaluated on Precise (natural) .

Figure 16: Visualizations of the LCS in SD3.5, Qwen-Image, and FLUX.2.

Figure 17: Illustration of the bi-jection between HSL and LCS space.

Controllable Image Generation

Controllable image generation steers images using spatial, visual, semantic, stylistic, or physical constraints beyond ordinary text prompts.

See all 215 papers

On Color Alignment in VAE Latent Spaces and Its Applications
Oct 5, 2026Julian D. Santamaria, Kai Wang, Jesús Malo +2Variational Autoencoders Image Generation
Computer Vision Center · Universitat Aut`onoma de Barcelona · City University of Hong Kong (Dongguan) +1
NumColor: Precise Numeric Color Control in Text-to-Image Generation
Mar 13, 2026Muhammad Atif Butt, Diego Hernandez, Alexandra Gomez-Villa +3Controllable Image Generation T2I Diffusion Models
Computer Vision Center, Spain · Computer Sciences Department, Universitat Autònoma de Barcelona, Spain · Program of Computer Science, City University of Hong Kong (Dongguan) +1
Colorful-Noise: Training-Free Low-Frequency Noise Manipulation for Color-Based Conditional Image Generation
May 1, 2026Nadav Z. Cohen, Ofir Abramovich, Ariel ShamirGeometry-Conditioned Image Generation Conditional Image Generation
Reichman University, Israel

Oct 5, 2026J/K move · Enter open · S save

On Color Alignment in VAE Latent Spaces and Its Applications

Julian D. Santamaria, Kai Wang, Jesús Malo, Javier Vazquez-Corral+1

Computer Vision Center · Universitat Aut`onoma de Barcelona · City University of Hong Kong (Dongguan)+1

Variational Autoencoders Image Generation Representation Disentanglement Controllable Image Generation Latent Space Alignment

Open paper AskPDF arXiv

Variational autoencoders (VAEs) are a key part of modern text-to-image models, which generate images within their latent space. VAEs are known to disentangle the main factors of variation in the data, and color is known to be one of the most structured of these in natural images: decorrelating it yields one luminance axis and two opponent-color axes. Color should therefore be expected to emerge as a distinct factor in the VAE latent space. Yet how these latent spaces represent color remains largely unexplored. In this work, we show that the VAEs of text-to-image models share a color subspace aligned with brightness and opponent-colors. Through a linear approximation of the encoder and targeted latent steering, we find this subspace consistently across a broad range of VAEs, from SD1.5 to FLUX.2 and Z-Image. Building on this characterization, we propose three applications: ColorTuning, which achieves state-of-the-art in precise numerical color generation on the fine-grained CSS3/X11 system of GenColorBench, saturation control, to adjust the global chromatic intensity, and color transfer, to change the palette to match a reference. The code and models are publicly available at https://julian075.github.io/Color_Subspace/

Abstract

Controllable Image Generation

Explore similar work

On Color Alignment in VAE Latent Spaces and Its Applications

NumColor: Precise Numeric Color Control in Text-to-Image Generation

Colorful-Noise: Training-Free Low-Frequency Noise Manipulation for Color-Based Conditional Image Generation