NumColor: Precise Numeric Color Control in Text-to-Image Generation
Authors: Muhammad Atif Butt, Diego Hernandez, Alexandra Gomez-Villa, Kai Wang, Javier Vazquez-Corral, Joost Van De Weijer
Organizations: Computer Vision Center, Spain · Computer Sciences Department, Universitat Autònoma de Barcelona, Spain · Program of Computer Science, City University of Hong Kong (Dongguan) · City University of Hong Kong
Text-to-image diffusion models excel at generating images from natural language descriptions, yet fail to interpret numerical colors such as hex codes (#FF5733) and RGB values (rgb(255,87,51)). This limitation stems from subword tokenization, which fragments color codes into semantically meaningless tokens that text encoders cannot map to coherent color representations. We present NumColor, that enables precise numerical color control across multiple diffusion architectures. NumColor comprises two components: a Color Token Aggregator that detects color specifications regardless of tokenization, and a ColorBook containing 6,707 learnable embeddings that map colors to embedding space of text encoder in perceptually uniform CIE Lab space. We introduce two auxiliary losses, directional alignment and interpolation consistency, to enforce geometric correspondence between Lab and embedding spaces, enabling smooth color interpolation. To train the ColorBook, we construct NumColor-Data, a synthetic dataset of 500K rendered images with unambiguous color-to-pixel correspondence, eliminating the annotation ambiguity inherent in photographic datasets. Although trained solely on FLUX, NumColor transfers zero-shot to SD3, SD3.5, PixArt-α, and PixArt-Σ without model-specific adaptation. NumColor improves numerical color accuracy by 4-9x across five models, while simultaneously improving color harmony scores by 10-30x on GenColorBench benchmark.
Figures & tables
Figure 1: Lack of Numeric Color Understanding. Left: Text encoders fragment hex codes into arbitrary subwords that attend to unrelated image regions, leading to incorrect color generation. Right: CKA [ 31 ] and neighborhood consistency (k=8) between text embeddings and Lab space show that color names preserve perceptual structure, while hex codes and RGB values degrade by ∼60% . NumColor recovers the structure comparable to color names.
Figure 2: Method overview. (a) Baseline: CLIP and T5 encode text; DiT generates images via iterative denoising. (b) Color Token Aggregator: A character-level sequence labeler unifies fragmented color tokens using linear classifier, trained with cross-entropy on numeric color text prompts. (c) ColorBook Training: Unified color tokens receive learned embeddings before T5 contextualization. The ColorBook (6,707 Lab anchors) is trained with flow matching loss on NumColor-Data, with directional and interpolation losses to preserve Lab geometry. Only ColorBook embeddings receive gradients; CLIP, T5, and DiT remain frozen.
Figure 3: NumColor-Data generation pipeline. We render 3D meshes from Objaverse-XL in diverse indoor and outdoor scenes. Each object is assigned a uniform Lambertian material with albedo set to the target Lab color. We use object-centered camera estimation and multi-orientation lighting to ensure diverse viewpoints. The dataset comprises 500K images spanning varied objects, scenes, and colors from ColorBook anchors. Caption generation —rendered images are paired with descriptive captions, validated by human experts, and converted to template prompts.
Figure 4: Qualitative results on FLUX with NumColor. We evaluate coarse colors spanning primary hues from deep navy to saturated green; fine-grained colors including perceptually adjacent grayscale, blue-to-cyan, and red-to-yellow colors; and interpolated colors synthesized via interpolating ColorBook anchors. NumColor maintains accurate color reproduction across objects.
Figure 5: Cross-model generalization. Baseline models versus NumColor integration across SD3, SD3.5, PixArt- α , and PixArt- Σ . NumColor, trained on FLUX, transfers zero-shot to other diffusion models. Baselines exhibit incorrect hue mapping, ignored specifications, and rainbow artifacts; NumColor resolves these while preserving image quality.
Figure 6: Embedding space analysis of NumColor. (a) Distribution of 6,707 color anchors in CIE Lab space with 5-unit spacing. (b) L2 drift between initial (T5-encoded) and trained embeddings versus hue angle, showing uniform adaptation across the color spectrum. (c) Nearest Neighbor overlap between LAB and embedding space. Initial T5 embeddings show high consistency at small k due to shared category names but degrade at larger k; trained embeddings maintain preservation at larger neighborhoods, reflecting perceptual structure beyond linguistic clustering.
GenColorBench Metrics ↑
Color Harmony ↓
Model
Resolution
Type
ISCC-L1
ISCC-L3
CSS3/X11
RGB
Hex
FLUX
1024
DM
13.82
7.74
5.86
247.4
134.9
Sana
1024
DM
25.10
12.85
9.45
643.1
105.5
SD3.5
1024
DM
14.92
7.53
5.78
518.3
117.3
Pixart- α
1024
DM
9.45
5.42
4.21
638.9
366.1
SD3
1024
DM
11.80
5.92
4.63
826.4
251.4
Table 1: Numerical Color Understanding performance of T2I models on GenColorBench. Results across three color specification granularities. A A A indicate top-3 performers.
Figure 7: Cross-attention maps for hex color specifications. Baseline models fragment hex codes into character tokens with scattered attention. NumColor processes colors as unified embeddings, focusing attention on target objects and enabling accurate color control.
Figure 8: Hyperparameter ablation. Low k causes hard quantization artifacts. High τ leads to color desaturation. τ=2.0 , k=8 balance accurate color reproduction with smooth interpolation.
Figure 9: Failure cases. Color Leakage: Specified color appears on the object but bleeds into surroundings due to lack of spatial grounding. Memory Color Bias: Objects with canonical binding resist recoloring, as pretrained model overrides the embedding, defaulting to memorized colors.
Figure S1 : Cross-attention visualization for numerical color prompts. Baseline models (FLUX, SD3.5, SD3, PixArt- α , PixArt- Σ ) distribute attention across fragmented color tokens ( rgb , 1 , 9 , 6 , …), failing to form coherent color representations. NumColor (bottom) aggregates color specifications into a single embedding that attends jointly to the target object, enabling accurate color control.
Figure S2 : Baseline model outputs for RGB and hex color specifications. Models fail to interpret numerical colors, producing rainbow artifacts, arbitrary colors, or complete disregard of the specification. This failure is consistent across architectures and color formats.
Model
Clothing & Acc.
Vehicles
Furniture & House.
Tools & Misc.
Sports & Toys
Animals
Fruits & Veg.
ISCC-L1
FLUX w/ NumColor
62.1
59.4
56.8
54.2
53.6
46.8
36.2
SD 3 w/ NumColor
54.8
52.6
50.4
48.1
47.2
41.8
32.4
SD3.5 w/ NumColor
57.6
55.2
52.8
50.6
49.8
44.2
34.6
PixArt- α w/ NumColor
50.2
48.1
46.2
44.1
43.2
38.4
29.8
PixArt- Σ w/ NumColor
54.2
52.1
50.2
48.1
47.2
41.8
32.6
Table S1 : Detailed category-wise breakdown showing individual metric values (ISCC-L1 | ISCC-L3 | CSS3/X11) for each semantic category. The pattern reflects higher performance on stylistic color categories versus biologically intrinsic color categories.
Figure S3 : Qualitative results of NumColor across five diffusion architectures. Each column shows outputs from a different model for identical prompts containing hex codes or RGB tuples. NumColor enables consistent numerical color control across all architectures using a single trained ColorBook, accurately rendering specified colors on diverse objects.
Component
Configuration
Character-Level CNN
Char embedding dim
64
Char vocab size
512
CNN kernel sizes
{2, 3, 4}
Token embedding dim
256
Max chars per token
32
Table S2 : Color Token Aggregator (CTA) Architecture.
Hyperparameter
Value
Optimization
Optimizer
AdamW
Learning rate
1×10−4
Learning rate schedule
Cosine annealing
Warmup steps
1,000
Final learning rate
1×10−6
Table S3 : Training configuration for ColorBook training. Only ColorBook embeddings are updated; all other model components remain frozen throughout training.
Figure S4 : Blender-rendered scenes used for sample objects for training data generation. We render diverse indoor and outdoor environments spanning residential interiors, urban streetscapes, architectural exteriors, commercial spaces, and natural settings. Colored objects are composited into these scenes with randomized placement, lighting, and camera viewpoints to generate training pairs for the NumColor ColorBook.
Figure S5 : Demonstrating Samples from NumColor-Data.
Figure S6 : Embedding drift statistics comparing initialized and trained ColorBook embeddings. (a) L2 drift distribution showing substantial movement. (b) Cosine similarity distribution showing directional preservation. (c) Relative drift distribution normalized by embedding magnitude. Together, these statistics indicate that training reorganizes the embedding space substantially while preserving directional compatibility with T5 geometry.
Beijing University of Posts and Telecommunications, Beijing 100876, China · Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance, Beijing 100876, China