Recolour What Matters: Region-Aware Colour Editing via Token-Level Diffusion

Yuqi Yang; Dongliang Chang; Yijia Ling; Ruoyi Du; Zhanyu Ma

cs.CVMar 19, 2026

Recolour What Matters: Region-Aware Colour Editing via Token-Level Diffusion

Authors: Yuqi Yang, Dongliang Chang, Yijia Ling, Ruoyi Du, Zhanyu Ma

Organizations: Beijing University of Posts and Telecommunications, Beijing 100876, China · Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance, Beijing 100876, China

Abstract

Colour is one of the most perceptually salient yet least controllable attributes in image generation. Although recent diffusion models can modify object colours from user instructions, their results often deviate from the intended hue, especially for fine-grained and local edits. Early text-driven methods rely on discrete language descriptions that cannot accurately represent continuous chromatic variations. To overcome this limitation, we propose ColourCrafter, a unified diffusion framework that transforms colour editing from global tone transfer into a structured, region-aware generation process. Unlike traditional colour driven methods, ColourCrafter performs token-level fusion of RGB colour tokens and image tokens in latent space, selectively propagating colour information to semantically relevant regions while preserving structural fidelity. A perceptual Lab-space Loss further enhances pixel-level precision by decoupling luminance and chrominance and constraining edits within masked areas. Additionally, we build ColourfulSet, a largescale dataset of high-quality image pairs with continuous and diverse colour variations. Extensive experiments demonstrate that ColourCrafter achieves state-of-the-art colour accuracy, controllability and perceptual fidelity in fine-grained colour editing. Our project is available at https://yangyuqi317.github.io/ColourCrafter.github.io/.

Figures & tables

Figure 1: Editing results of ColourCrafter under varying reference colours. Each row shows the input image and its edited outputs conditioned on different RGB references. As the reference colours vary smoothly from left to right, ColourCrafter produces continuous and precise recolouring with consistent structure and texture.

Figure 2: Comparison of different colour editing methods. (a) shows the results of natural language description (first row) and text embedding interpolation (second row); (b) displays the image-driven results using IP-Adapter. In comparison, our proposed method achieves both precise colour control and high structural fidelity.

Figure 3: Overview of the ColourCrafter pipeline. (1) Dataset construction: Using Flux.1-Kontext, we generate diverse image–colour pairs and employ a Vision–Language Model (VLM) to filter samples for consistency, fidelity, and realism. The corresponding RGB references are extracted to build the high-quality dataset ColourfulSet . (2) Training: The original image, target colour reference, and text prompt are jointly fed into the diffusion model, which is optimised with both Diffusion and Lab-space losses to enhance chromatic accuracy and perceptual consistency. (3) Inference: Given an input image, a RGB reference, and a prompt, ColourCrafter performs fine-grained, structure-preserving, and perceptually natural colour editing.

Figure 4: Examples from the ColourfulSet dataset. (a) shows edited images from different categories under various target colour references. (b–c) Visualise the colour distribution of ColourfulSet from two perspectives in RGB space, demonstrating its diversity and uniform coverage.

Figure 5: Comparison with other methods. The first column shows the reference colours and original images. The comparison demonstrates that our method achieves more precise and fine-grained colour editing while preserving structural integrity and background consistency.

Method	Colour				Editing
Method	$\Delta E_{00}\downarrow$	$\Delta E_{Ch}\downarrow$	$\text{MAE}_{\text{RGB}}\downarrow$	$\text{MAE}_{\text{Hue}}\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$	FID $\downarrow$
ColorPeel [ 3 ]	60.96	53.40	77.79	61.95	0.38	0.72	79.30
ColorBind [ 45 ]	53.42	47.26	62.78	44.12	0.36	0.70	69.68
Control-Color [ 27 ]	32.82	28.44	71.90	20.31	0.28	0.82	63.87
Flux.1-Kontext [ 22 ]	41.36	36.31	48.34	23.19	0.29	0.74	63.01
FlowEdit [ 21 ]	66.26	56.32	83.85	61.59	0.29	0.79	81.32

Table 1: Quantitative comparison on the colour editing task.

Figure 6: Comparison with IP-Adapter methods. In the RGB column, black and red boxes indicate the target object colour and the reference colour for results produced by the IP-Adapter and ColourCrafter (Ours), respectively.

Figure 11: Qualitative ablation of different loss functions. In the RGB column, black and red boxes indicate the target object colour and the reference colour for results produced by the Ours w/o Lab Loss and ColourCrafter (Ours), respectively.

Table 11

Figure 12: Results of different injection methods and colour constraints methods under different random seeds. (a) denotes encoding RGB colour patches as text tokens, (b) denotes encoding RGB arrays as text tokens, (c) denotes RGB Loss constraint, (d) denotes Ours.

Image Editing

Image editing changes selected visual content while preserving the parts of an image that should remain unchanged.

See all 302 papers

NumColor: Precise Numeric Color Control in Text-to-Image Generation
Mar 13, 2026Muhammad Atif Butt, Diego Hernandez, Alexandra Gomez-Villa +3Controllable Image Generation T2I Diffusion Models
Computer Vision Center, Spain · Computer Sciences Department, Universitat Autònoma de Barcelona, Spain · Program of Computer Science, City University of Hong Kong (Dongguan) +1
Masked Generative Transformer Is What You Need for Image Editing
May 11, 2026Wei Chow, Linfeng Li, Xian Sun +14Transformer Text-Guided Image Editing
1ByteDance · 2National University of Singapore · 3Duke University +2
Diffusion Image Editing via Asynchronous Token Decoding
Aug 10, 2026Yang Shi, Liangsi Lu, Minzhe Guo +4Text-Guided Image Editing Diffusion-Based Image Editing
Guangdong University of Technology Guangzhou, China · Hong Kong Baptist University Hong Kong, China · Peking University Beijing, China +1

Mar 13, 2026J/K move · Enter open · S save

NumColor: Precise Numeric Color Control in Text-to-Image Generation

Muhammad Atif Butt, Diego Hernandez, Alexandra Gomez-Villa, Kai Wang+2

Computer Vision Center, Spain · Computer Sciences Department, Universitat Autònoma de Barcelona, Spain · Program of Computer Science, City University of Hong Kong (Dongguan)+1

Controllable Image Generation T2I Diffusion Models

Open paper AskPDF arXiv

Text-to-image diffusion models excel at generating images from natural language descriptions, yet fail to interpret numerical colors such as hex codes (#FF5733) and RGB values (rgb(255,87,51)). This limitation stems from subword tokenization, which fragments color codes into semantically meaningless tokens that text encoders cannot map to coherent color representations. We present NumColor, that enables precise numerical color control across multiple diffusion architectures. NumColor comprises two components: a Color Token Aggregator that detects color specifications regardless of tokenization, and a ColorBook containing 6,707 learnable embeddings that map colors to embedding space of text encoder in perceptually uniform CIE Lab space. We introduce two auxiliary losses, directional alignment and interpolation consistency, to enforce geometric correspondence between Lab and embedding spaces, enabling smooth color interpolation. To train the ColorBook, we construct NumColor-Data, a synthetic dataset of 500K rendered images with unambiguous color-to-pixel correspondence, eliminating the annotation ambiguity inherent in photographic datasets. Although trained solely on FLUX, NumColor transfers zero-shot to SD3, SD3.5, PixArt-α, and PixArt-Σ without model-specific adaptation. NumColor improves numerical color accuracy by 4-9x across five models, while simultaneously improving color harmony scores by 10-30x on GenColorBench benchmark.

Abstract

Image Editing

Explore similar work

NumColor: Precise Numeric Color Control in Text-to-Image Generation

Masked Generative Transformer Is What You Need for Image Editing

Diffusion Image Editing via Asynchronous Token Decoding