CoLoGen: Progressive Learning of Concept-Localization Duality for Unified Image Generation

YuXin Song; Yu Lu; Haoyuan Sun; Huanjin Yao; Fanglong Liu; Yifan Sun; Haocheng Feng; Hang Zhou

cs.CVFeb 25, 2026

CoLoGen: Progressive Learning of Concept-Localization Duality for Unified Image Generation

Authors: YuXin Song, Yu Lu, Haoyuan Sun, Huanjin Yao, Fanglong Liu, Yifan Sun, Haocheng Feng, Hang Zhou,

Organizations: Baidu Inc. · Zhejiang University · Tsinghua University

Abstract

Unified conditional image generation remains difficult because different tasks depend on fundamentally different internal representations. Some require conceptual understanding for semantic synthesis, while others rely on localization cues for spatial precision. Forcing these heterogeneous tasks to share a single representation leads to concept-localization representational conflict. To address this issue, we propose CoLoGen, a unified diffusion framework that progressively learns and reconciles this concept-localization duality. CoLoGen uses a staged curriculum that first builds core conceptual and localization abilities, then adapts them to diverse visual conditions, and finally refines their synergy for complex instruction-driven tasks. Central to this process is the Progressive Representation Weaving (PRW) module, which dynamically routes features to specialized experts and stably integrates their outputs across stages. Experiments on editing, controllable generation, and customized generation show that CoLoGen achieves competitive or superior performance, offering a principled representational perspective for unified image generation.

Figures & tables

Figure 1: The comparison between the multi-task learning strategy (a) and the ours progressive staged training (b) within the framework of unified multi-modal image generation. We specifically examines five conventional tasks: mask inpainting, image grounding, controllable image generation, customized image generation, and instruction-based image editing.

Figure 2: The overall framework of the unified Multi-modal to Image generation model, CoLoGen. For each training stage, CoLoGen efficiently integrates a set of condition-specific experts via the Progressive Representation Weaving (PRW) which are constructed on KV projection layers and a dynamic router

G

. Notably, The QKV projection layer and the self-attention layer are sharing weights for the inputs of Noisy Latent and Source Image Latent. CoLoGen employs a progressive staged training strategy to gradually increase the number of experts

E_{k}

, allowing it to better adapt to more complex downstream tasks.

Stage	Task	# Samples	Data Source
Endogenous Pre-training	Mask Inpainting	3M	ADE20k, COCOStuff, JouneyDB
Endogenous Pre-training	Image Grounding	1M	RefCOCO, RefCOCOg, RefCOCO+, LVIS
Conditional Injection	Controllable Generation	20M	Multigen-20M, Multigen-20M-Depth,
Conditional Injection	Controllable Generation	20M	ADE20k, COCOStuff
Instruction-image alignment	Customized Generation	200K	Subject200k
Instruction-image alignment	Instruction Editing	1.6M	OmniEdit, Magicbrush, In-house Data

Table 1: The outline of training data about each training stage. Notably, Endogenous Pre-training comprises two distinct training steps: mask inpainting and image grounding. Instruction-Image Alignment encompasses both Customized Generation and Instruction Editing.

Methods	Emu Edit test set				MagicBrush test set
Methods	$\text{CLIP}_{i}\!\uparrow$	$\text{CLIP}_{out}\!\uparrow$	$\ell_{1}\!\downarrow$	DINO $\uparrow$	$\text{CLIP}_{i}\!\uparrow$	$\text{CLIP}_{out}\!\uparrow$	$\ell_{1}\!\downarrow$	DINO $\uparrow$
Specialist Models
InstructPix2Pix [ 2 ]	0.834	0.219	0.121	0.762	0.837	0.245	0.093	0.767
MagicBrush [ 67 ]	0.838	0.222	0.100	0.776	0.883	0.261	0.058	0.871
PnP [ 51 ]	0.521	0.089	0.304	0.153	0.568	0.101	0.289	0.220
Null-Text Inv. [ 33 ]	0.761	0.236	0.075	0.678	0.752	0.263	0.077	0.664

Table 2: Quantitative results for instruction image editing evaluated on the Emu Edit test split and MagicBrush test split. “-” indicates that the method does not report the corresponding results.

\!\uparrow

indicates higher result is better, while

\!\downarrow

means lower is better.

Methods	Canny-to-Image	Depth-to-Image	LineArt-to-Image	Seg-to-Image
Methods	CLIP-S↑	RMSE↓	SSIM↑	mIoU↑
	MultiGen-20M	MultiGen-20M	MultiGen-20M	COCO-Stuff	ADE20K
Specialist Models
ControlNet-SD1.5 [ 68 ]	32.15	35.90	-	27.46	32.55
T2I-Adapter-SD1.5 [ 35 ]	31.71	48.40	63.94	-	12.61
Gligen [ 27 ]	-	38.83	-	-	23.78

Table 3: Quantitative results for Controllable image generation on MultiGen-20M, ADE-20K and COCOStuff. “-” indicates that the method does not report corresponding results.

\!\uparrow

indicates that a higher value is better, while

\!\downarrow

indicates that a lower value is better.

Model	$\textbf{CLIP-T}\!\uparrow$	$\textbf{CLIP-I}\!\uparrow$	$\textbf{DINO}\!\uparrow$
Specialist Models
Textual Inversion [ 34 ]	0.255	0.780	0.569
DreamBooth [ 41 ]	0.305	0.803	0.668
BLIP-Diffusion [ 25 ]	0.302	0.805	0.670
ELITE [ 54 ]	0.296	0.772	0.647
Re-Imagen [ 7 ]	0.270	0.740	0.600

Table 4: Quantitative results for customized generation on DreamBench [ 41 ] .

Figure 3: Qualitative comparison for customized image generation. We compare with a series of public SOTA methods including OmniGen [ 61 ] , MS-Diffusion [ 52 ] , and IP-Adapter [ 64 ] on DreamBench [ 41 ] . CoLoGen achieves remarkable performance even when trained on a limited amount of proprietary data, which can be attributed to the rich multimodal knowledge acquired by the model during the endogenous pre-training and conditional injection learning phases.

Method	$\textbf{CLIP-T}\!\uparrow$	$\textbf{CLIP-I}\!\uparrow$	$\textbf{DINO}\!\uparrow$
Magicbrush
w/o $\mathcal{R}_{l}$ & w/o $\mathcal{R}_{c}$	0.260	0.889	0.901
w $\mathcal{R}_{l}$	0.279	0.922	0.927
w $\mathcal{R}_{c}$	0.302	0.881	0.905
w $\mathcal{R}_{c}$ & $\mathcal{R}_{l}$ (Co-training)	0.269	0.918	0.922
w $\mathcal{R}_{c}$ & $\mathcal{R}_{l}$ (CoLoGen)	0.301	0.931	0.932

Table 5: Evaluation the contribution of the concept and localization representations on instruction-based editing (Magicbrush) and customized generation (Dreambench).

\mathcal{R}_{c}

denotes the concept representation and

\mathcal{R}_{l}

denotes the localization representation. Metrics that exhibit a noticeable improvement over the baseline (w/o

\mathcal{R}_{l}

& w/o

\mathcal{R}_{c}

) are highlighted in blue .

Figure 4: Ablation studies for hyperparameter of lifelong strategy in the last stage, with final settings highlighted in orange . (a) Impact of $\alpha$ on the weight for balancing the veteran gate routing supervision. (b) Influence of $rank$ on LoRA, where LoRA alpha weight defaults to twice the $rank$ . (c) Impact of $\rho$ , which denotes the rounting density of $expert_{N-1}$ .

Figure 5: Qualitative comparisons with current state-of-the-art mask-inpainting methods. CoLoGen demonstrates robust text-following capabilities, and exhibits strong visual coherence between the mask area and the background.

Figure 6: Instructional editing results of our CoLoGen. Our method can adapt to various types of instructions, faithfully follow instructions while preserving the visual consistency of the input images, ensuring high-quality and coherent results.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

		Canny		Depth
Method	Base	C-S $\uparrow$	FID $\downarrow$	SSIM $\uparrow$	FID $\downarrow$
UNIC-Adapter [ 15 ]	SD3	–	23.47	31.10	–
RealGen [ 14 ]	CogV	–	17.50	35.0	23.40
OmniControl [ 63 ]	FLUX	30.60	20.63	39.0	27.26
EasyControl [ 71 ]	FLUX	28.60	–	35.9	20.39
CoLoGen(Ours)	FLUX	33.31	18.20	40.1	19.56

Appendix

Table 6: Controllable image generation comparison on recent backbone models.

Method	Data	DINO	C-I	C-T
OmniControl [ 63 ]	200k	0.684	0.799	0.312
FLUX-IP-Adapter [ 64 ]	200k	0.582	0.820	0.288
CoLoGen(Ours)	200k	0.714	0.825	0.315
UNO-FLUX [ 58 ]	1M–5M	0.760	0.835	0.308
OmniGen2 [ 57 ]	10M+	0.749	0.830	0.314
BAGEL [ 13 ]	10M+	0.797	0.859	0.307

Appendix

Table 7: Customized image generation comparison under different training data scales.

	GEdit-Bench (Full Set) $\uparrow$
Method	G_SC	G_PQ	G_O
Step1X-Edit [ 30 ]	7.66	7.35	6.97
BAGEL [ 13 ]	7.36	6.83	6.52
FLUX.1 Kontext [ 24 ]	7.02	7.60	6.56
Qwen-Image [ 55 ]	8.00	7.86	7.56
CoLoGen (Ours)	8.03	7.15	7.31

Appendix

Table 8: Results on GEdit-Bench (Full Set) .

Figure 7: Controllable generation results of our CoLoGen.

Figure 8: Visual examples from the image grounding task demonstrate that CoLoGen, after undergoing endogenous pre-training, exhibits highly accurate visual localization capabilities.

Image Generation

Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.

See all 463 papers

CoFi-UCGen: Coarse-to-Fine Unsupervised Conditional Generation without Label Priors
Jun 4, 2026Shengxi Li, Zhaokun Hu, Ce Zheng +3Disentangled Representation Learning Conditional Image Generation
Department of Electronic Information Engineering, Beihang University, Beijing 100191, China · School of Cyber Science and Technology, Beihang University, Beijing 100191, China · College of Electronic Science, National University of Defense Technology, Hunan 410073, China +1
UniVL: Unified Vision-Language Embedding for Spatially Grounded Contextual Image Generation
May 20, 2026Jiayun Wang, Yu Wang, Weijie Gan +2Image Generation Vision-Language Grounding
Center for Advanced AI, Accenture
UniCustom: Unified Visual Conditioning for Multi-Reference Image Generation
May 12, 2026Yiyan Xu, Qiulin Wang, Wenjie Wang +5Subject-Driven Image Generation Image Generation
University of Science and Technology of China · Kling Team, Kuaishou Technology

Jun 4, 2026J/K move · Enter open · S save

CoFi-UCGen: Coarse-to-Fine Unsupervised Conditional Generation without Label Priors

Shengxi Li, Zhaokun Hu, Ce Zheng, Mai Xu+2

Department of Electronic Information Engineering, Beihang University, Beijing 100191, China · School of Cyber Science and Technology, Beihang University, Beijing 100191, China · College of Electronic Science, National University of Defense Technology, Hunan 410073, China+1

Disentangled Representation Learning Conditional Image Generation Conditional Diffusion Models Controllable Image Generation

Open paper AskPDF arXiv

Unsupervised conditional image generation (UCGen) aims to control generation without relying on manually annotated labels, yet remains challenging due to unstructured semantic representations across granularities. To address this, we propose a novel coarse-to-fine UCGen framework (CoFi-UCGen) that explicitly disentangles global semantics from fine-grained variations, which to the best of our knowledge, sets out the first successful attempt for both coarse- and fine-grained conditional generation without any labels. More specifically, we first propose the adversarial semantic reciprocal learning theory to ensure the semantic consistency and completeness between images and latent spaces. Based on the consistency, we propose the bit-codes to learn a structured coarse-grained latent space, and further prove distinct global semantics inherent from our bit-codes while preserving independent noise sampling for generation. Building upon these bit-codes, we establish a fine-grained semantic basis and introduce a hierarchical modulation mechanism in diffusion models, by enabling layer-wise injection from coarse conditions to progressively control fine-grained attributes during generation. Extensive experiments demonstrate that without any label priors or pre-trained feature extractors, our CoFi-UCGen consistently outperforms existing UCGen methods in terms of image quality, semantic consistency, and control accuracy, verifying the effectiveness of explicit coarse-to-fine semantic decomposition for the challenging UCGen task.

Abstract

Image Generation

Explore similar work

CoFi-UCGen: Coarse-to-Fine Unsupervised Conditional Generation without Label Priors

UniVL: Unified Vision-Language Embedding for Spatially Grounded Contextual Image Generation

UniCustom: Unified Visual Conditioning for Multi-Reference Image Generation