NumColor: Precise Numeric Color Control in Text-to-Image Generation
Authors: Muhammad Atif Butt, Diego Hernandez, Alexandra Gomez-Villa, Kai Wang, Javier Vazquez-Corral, Joost Van De Weijer
Organizations: Computer Vision Center, Spain · Computer Sciences Department, Universitat Autònoma de Barcelona, Spain · Program of Computer Science, City University of Hong Kong (Dongguan) · City University of Hong Kong
Text-to-image diffusion models excel at generating images from natural language descriptions, yet fail to interpret numerical colors such as hex codes (#FF5733) and RGB values (rgb(255,87,51)). This limitation stems from subword tokenization, which fragments color codes into semantically meaningless tokens that text encoders cannot map to coherent color representations. We present NumColor, that enables precise numerical color control across multiple diffusion architectures. NumColor comprises two components: a Color Token Aggregator that detects color specifications regardless of tokenization, and a ColorBook containing 6,707 learnable embeddings that map colors to embedding space of text encoder in perceptually uniform CIE Lab space. We introduce two auxiliary losses, directional alignment and interpolation consistency, to enforce geometric correspondence between Lab and embedding spaces, enabling smooth color interpolation. To train the ColorBook, we construct NumColor-Data, a synthetic dataset of 500K rendered images with unambiguous color-to-pixel correspondence, eliminating the annotation ambiguity inherent in photographic datasets. Although trained solely on FLUX, NumColor transfers zero-shot to SD3, SD3.5, PixArt-α, and PixArt-Σ without model-specific adaptation. NumColor improves numerical color accuracy by 4-9x across five models, while simultaneously improving color harmony scores by 10-30x on GenColorBench benchmark.
Figures & tables
Figure 1: Lack of Numeric Color Understanding. Left: Text encoders fragment hex codes into arbitrary subwords that attend to unrelated image regions, leading to incorrect color generation. Right: CKA [ 31 ] and neighborhood consistency (k=8) between text embeddings and Lab space show that color names preserve perceptual structure, while hex codes and RGB values degrade by ∼60% . NumColor recovers the structure comparable to color names.
Figure 2: Method overview. (a) Baseline: CLIP and T5 encode text; DiT generates images via iterative denoising. (b) Color Token Aggregator: A character-level sequence labeler unifies fragmented color tokens using linear classifier, trained with cross-entropy on numeric color text prompts. (c) ColorBook Training: Unified color tokens receive learned embeddings before T5 contextualization. The ColorBook (6,707 Lab anchors) is trained with flow matching loss on NumColor-Data, with directional and interpolation losses to preserve Lab geometry. Only ColorBook embeddings receive gradients; CLIP, T5, and DiT remain frozen.
Figure 3: NumColor-Data generation pipeline. We render 3D meshes from Objaverse-XL in diverse indoor and outdoor scenes. Each object is assigned a uniform Lambertian material with albedo set to the target Lab color. We use object-centered camera estimation and multi-orientation lighting to ensure diverse viewpoints. The dataset comprises 500K images spanning varied objects, scenes, and colors from ColorBook anchors. Caption generation —rendered images are paired with descriptive captions, validated by human experts, and converted to template prompts.
Figure 4: Qualitative results on FLUX with NumColor. We evaluate coarse colors spanning primary hues from deep navy to saturated green; fine-grained colors including perceptually adjacent grayscale, blue-to-cyan, and red-to-yellow colors; and interpolated colors synthesized via interpolating ColorBook anchors. NumColor maintains accurate color reproduction across objects.
Figure 5: Cross-model generalization. Baseline models versus NumColor integration across SD3, SD3.5, PixArt- α , and PixArt- Σ . NumColor, trained on FLUX, transfers zero-shot to other diffusion models. Baselines exhibit incorrect hue mapping, ignored specifications, and rainbow artifacts; NumColor resolves these while preserving image quality.
Figure 6: Embedding space analysis of NumColor. (a) Distribution of 6,707 color anchors in CIE Lab space with 5-unit spacing. (b) L2 drift between initial (T5-encoded) and trained embeddings versus hue angle, showing uniform adaptation across the color spectrum. (c) Nearest Neighbor overlap between LAB and embedding space. Initial T5 embeddings show high consistency at small k due to shared category names but degrade at larger k; trained embeddings maintain preservation at larger neighborhoods, reflecting perceptual structure beyond linguistic clustering.
GenColorBench Metrics ↑
Color Harmony ↓
Model
Resolution
Type
ISCC-L1
ISCC-L3
CSS3/X11
RGB
Hex
FLUX
1024
DM
13.82
7.74
5.86
247.4
134.9
Sana
1024
DM
25.10
12.85
9.45
643.1
105.5
SD3.5
1024
DM
14.92
7.53
5.78
518.3
117.3
Pixart- α
1024
DM
9.45
5.42
4.21
638.9
366.1
SD3
1024
DM
11.80
5.92
4.63
826.4
251.4
Table 1: Numerical Color Understanding performance of T2I models on GenColorBench. Results across three color specification granularities. A A A indicate top-3 performers.
Figure 7: Cross-attention maps for hex color specifications. Baseline models fragment hex codes into character tokens with scattered attention. NumColor processes colors as unified embeddings, focusing attention on target objects and enabling accurate color control.
Figure 8: Hyperparameter ablation. Low k causes hard quantization artifacts. High τ leads to color desaturation. τ=2.0 , k=8 balance accurate color reproduction with smooth interpolation.
Figure 9: Failure cases. Color Leakage: Specified color appears on the object but bleeds into surroundings due to lack of spatial grounding. Memory Color Bias: Objects with canonical binding resist recoloring, as pretrained model overrides the embedding, defaulting to memorized colors.
Figure S1 : Cross-attention visualization for numerical color prompts. Baseline models (FLUX, SD3.5, SD3, PixArt- α , PixArt- Σ ) distribute attention across fragmented color tokens ( rgb , 1 , 9 , 6 , …), failing to form coherent color representations. NumColor (bottom) aggregates color specifications into a single embedding that attends jointly to the target object, enabling accurate color control.
Figure S2 : Baseline model outputs for RGB and hex color specifications. Models fail to interpret numerical colors, producing rainbow artifacts, arbitrary colors, or complete disregard of the specification. This failure is consistent across architectures and color formats.
Model
Clothing & Acc.
Vehicles
Furniture & House.
Tools & Misc.
Sports & Toys
Animals
Fruits & Veg.
ISCC-L1
FLUX w/ NumColor
62.1
59.4
56.8
54.2
53.6
46.8
36.2
SD 3 w/ NumColor
54.8
52.6
50.4
48.1
47.2
41.8
32.4
SD3.5 w/ NumColor
57.6
55.2
52.8
50.6
49.8
44.2
34.6
PixArt- α w/ NumColor
50.2
48.1
46.2
44.1
43.2
38.4
29.8
PixArt- Σ w/ NumColor
54.2
52.1
50.2
48.1
47.2
41.8
32.6
Table S1 : Detailed category-wise breakdown showing individual metric values (ISCC-L1 | ISCC-L3 | CSS3/X11) for each semantic category. The pattern reflects higher performance on stylistic color categories versus biologically intrinsic color categories.
Figure S3 : Qualitative results of NumColor across five diffusion architectures. Each column shows outputs from a different model for identical prompts containing hex codes or RGB tuples. NumColor enables consistent numerical color control across all architectures using a single trained ColorBook, accurately rendering specified colors on diverse objects.
Component
Configuration
Character-Level CNN
Char embedding dim
64
Char vocab size
512
CNN kernel sizes
{2, 3, 4}
Token embedding dim
256
Max chars per token
32
Table S2 : Color Token Aggregator (CTA) Architecture.
Hyperparameter
Value
Optimization
Optimizer
AdamW
Learning rate
1×10−4
Learning rate schedule
Cosine annealing
Warmup steps
1,000
Final learning rate
1×10−6
Table S3 : Training configuration for ColorBook training. Only ColorBook embeddings are updated; all other model components remain frozen throughout training.
Figure S4 : Blender-rendered scenes used for sample objects for training data generation. We render diverse indoor and outdoor environments spanning residential interiors, urban streetscapes, architectural exteriors, commercial spaces, and natural settings. Colored objects are composited into these scenes with randomized placement, lighting, and camera viewpoints to generate training pairs for the NumColor ColorBook.
Figure S5 : Demonstrating Samples from NumColor-Data.
Figure S6 : Embedding drift statistics comparing initialized and trained ColorBook embeddings. (a) L2 drift distribution showing substantial movement. (b) Cosine similarity distribution showing directional preservation. (c) Relative drift distribution normalized by embedding magnitude. Together, these statistics indicate that training reorganizes the embedding space substantially while preserving directional compatibility with T5 geometry.
Professional design requires any-color control: the ability to specify an object's target color with any 24-bit hex value for image generation and editing. Prior work has explored color generation, editing, and colorization, but often relies on dedicated color representations or specialized inference procedures. Advances in large language models offer a simpler starting point: even compact models can associate hex values with color semantics. We present Paint-Anything, which learns a shared hex-prompt interface for generation and editing through object-level color supervision. We develop a data pipeline that constructs Paint-500K from real images through object grounding, perceptual color labeling, and editing-pair synthesis. Since shadows make real-image labels only approximate colors, we complement this supervision with pure-color anchors whose pixels exactly match their paired hex values. These anchors are used only at high-noise timesteps, leaving low-noise training to natural images. We further introduce Any Color Benchmark (ACBench), comprising ACBench-T2I and ACBench-Edit, to measure object-level hex color fidelity across both tasks. On FLUX.2-4B, Paint-Anything improves ACBench-T2I and ACBench-Edit scores by 85.3% and 28.3%, respectively, relative to the base model, with ablations supporting the training recipe. It also achieves the highest average CompColor score among the compared methods.
Ji Xie, Dewei Zhou, Xinyu Huang +2
ByteDance Seed · Zhejiang University · Nanjing University
Colour is one of the most perceptually salient yet least controllable attributes in image generation. Although recent diffusion models can modify object colours from user instructions, their results often deviate from the intended hue, especially for fine-grained and local edits. Early text-driven methods rely on discrete language descriptions that cannot accurately represent continuous chromatic variations. To overcome this limitation, we propose ColourCrafter, a unified diffusion framework that transforms colour editing from global tone transfer into a structured, region-aware generation process. Unlike traditional colour driven methods, ColourCrafter performs token-level fusion of RGB colour tokens and image tokens in latent space, selectively propagating colour information to semantically relevant regions while preserving structural fidelity. A perceptual Lab-space Loss further enhances pixel-level precision by decoupling luminance and chrominance and constraining edits within masked areas. Additionally, we build ColourfulSet, a largescale dataset of high-quality image pairs with continuous and diverse colour variations. Extensive experiments demonstrate that ColourCrafter achieves state-of-the-art colour accuracy, controllability and perceptual fidelity in fine-grained colour editing. Our project is available at https://yangyuqi317.github.io/ColourCrafter.github.io/.
Yuqi Yang, Dongliang Chang, Yijia Ling +2
Beijing University of Posts and Telecommunications, Beijing 100876, China · Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance, Beijing 100876, China
Text-to-image models have rapidly advanced in realism and controllability, with recent approaches leveraging long, detailed captions to support fine-grained generation. However, a fundamental parametric gap remains: existing models rely on descriptive language, whereas professional workflows require precise numeric control over object location, size, and color. In this work, we introduce BBQ, a large-scale text-to-image model that directly conditions on numeric bounding boxes and RGB triplets within a unified structured-text framework. We obtain precise spatial and chromatic control by training on captions enriched with parametric annotations, without architectural modifications or inference-time optimization. This also enables intuitive user interfaces such as object dragging and color pickers, replacing ambiguous iterative prompting with precise, familiar controls. Across comprehensive evaluations, BBQ achieves strong box alignment and improves RGB color fidelity over state-of-the-art baselines. More broadly, our results support a new paradigm in which user intent is translated into an intermediate structured language, consumed by a flow-based transformer acting as a renderer and naturally accommodating numeric parameters.
Eliran Kachlon, Alexander Visheratin, Nimrod Sarid +6