Recent advances in diffusion-based text-to-image generation have demonstrated promising results through visual condition control. However, existing ControlNet-like methods struggle with compositional visual conditioning - simultaneously preserving semantic fidelity across multiple heterogeneous control signals while maintaining high visual quality, where they employ separate control branches that often introduce conflicting guidance during the denoising process, leading to structural distortions and artifacts in generated images. To address this issue, we present PixelPonder, a novel unified control framework, which allows for effective control of multiple visual conditions under a single control structure. Specifically, we design a patch-level adaptive condition selection mechanism that dynamically prioritizes spatially relevant control signals at the sub-region level, enabling precise local guidance without global interference. Additionally, a time-aware control injection scheme is deployed to modulate condition influence according to denoising timesteps, progressively transitioning from structural preservation to texture refinement and fully utilizing the control information from different categories to promote more harmonious image generation. Extensive experiments demonstrate that PixelPonder surpasses previous methods across different benchmark datasets, showing superior improvement in spatial alignment accuracy while maintaining high textual semantic consistency.
Figures & tables
Figure 1: Visualization comparison between Uni-ControlNet [ 51 ] and our proposed method in different conditional controls with the same text prompt. (a, left two columns) Text and various visual controls, where C1 , C2 , C3 and C4 denotes canny, sketch, depth and pose map respectively. (b, middle three or four columns) Generation results from UniControlNet. (c, last column) Generation results from our PixelPonder. Previous methods struggled to generate coherent results under multiple conditions, while our results maintain strong similarity to the respective visual controls.
Figure 2: Comparison of different schemes dealing with multiple condition. (a) Existing methods integrate various visual controls into a highly consistent visual control signal in the temporal domain through concatenation or weighted summation. (b) We employ dynamic patch selection to reconstruct visual conditions at the patch level, enabling multi-visual controllable generation in the generative model.
Figure 3: Overall pipeline of the proposed . The Patch Adaptaiom Module segments various visual conditions {V1,V2,…,Vn} into patches and dynamiclly reorganizes the visual conditions at the patch level to obtain a unified visual control signal SPout . Subsequently, SPout is sent to the control network, with the Time-aware Control Injection scheme to achieve high-quality image generation under multiple visual conditions.
Figure 4: Denoising results at steps 25, 20, 15, 10, 5, and 1 , where each pair includes the generated result and the selection results SPout at the current time step. In each SPout , canny, pose, depth and sketch are indicated in red, blue, green, and yellow respectively.
Figure 5: Visualization comparison between official methods and our proposed model in different conditional controls.
MultiGen-20M
Subject-200K
Methods
Conditions
FID ( ↓ )
CLIP Score ( ↑ )
SSIM ( ↑ )
MUSIQ ( ↑ )
FID ( ↓ )
CLIP Score ( ↑ )
SSIM ( ↑ )
MUSIQ ( ↑ )
Inference Time ( ↓ )
T2I-Adapter
all
66.95
71.47
24.03
57.95
64.72
74.35
31.76
55.07
9.33
Uni-ControlNet
all
32.58
78.08
29.37
65.85
44.35
77.40
37.98
66.75
7.22
UniControl
all
25.15
74.09
35.58
72.05
30.95
72.96
47.31
67.50
7.39
Cocktail
pose+hed
24.67
79.87
24.14
67.13
51.51
79.38
29.59
64.16
4.94
Ours
all
11.85
78.60
43.99
69.54
10.61
77.12
66.60
67.32
9.59
Table 1: Comparison on the MultiGen-20M [ 21 ] and Subject-200K [ 40 ] datasets. The best and second-best results are highlighted in bold and underline , respectively. In the Conditions column, “all” denotes the use of canny maps, sketches, pose maps, and depth maps simultaneously during generation.
MultiGen-20M
Methods
Conditions
FID ( ↓ )
SSIM ( ↑ )
MUSIQ ( ↑ )
Source
-
-
-
69.30
ControlNet++
canny
17.69
36.69
65.67
ControlNet++
hed
13.93
42.12
71.22
ControlNet++
depth
17.56
27.79
71.23
Ours
pose
30.00
27.91
58.16
Table 2: Results of combining different condition types. A quantitative comparison with models controlled by single visual conditions and various combinations of visual conditions. “Source" refers to the original reference image.
Figure 6: Comparison of multi-condition image generation by PixelPonder with different combinations against single-condition image generation by ControlNet++.
Figure 7: Comparison of precision and recall of PAM and random scheme on canny and sketch conditions. Taking Canny and Sketch as examples.
Table 10
Figure 10: Statistical distribution of each visual condition patch during the denoising steps.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
MultiGen-20M
Patch Size
FID ( ↓ )
CLIP Score ( ↑ )
SSIM ( ↑ )
MUSIQ ( ↑ )
2
11.26
78.62
43.21
69.15
4
12.80
78.24
40.73
68.43
8
13.46
78.07
39.75
68.15
Appendix
Table A1 : Ablation on patch size. We tested the impact of patch size in Patch Selection on controlling the generation.
MultiGen-20M
Data scale
FID ( ↓ )
SSIM ( ↑ )
MUSIQ ( ↑ )
105k
12.20
40.79
68.93
686k
11.43
41.13
69.16
2.5m
11.26
43.21
69.15
Appendix
Table A2 : Ablation on data scale. FID, SSIM and MUSIQ are reported on MultiGen-20M datasets. k represents thousand, and m represents million.
MultiGen-20M
Fourier
FID ( ↓ )
SSIM ( ↑ )
MUSIQ ( ↑ )
×
12.79
44.97
68.38
✓
13.21
44.49
68.09
Appendix
Table A3 : Ablation on fourier. × represents the non-use of fourier, while the ✓ represents the use of fourier.
Figure A1 : Comparison of multiple random generations under single-condition and multi-condition. Ours employs all available conditions, whereas canny and depth denote the use of only the canny or depth conditions, respectively.
Figure A2 : Generation quality comparison between Random Selection and Patch Adaptation. Random selection can lead to a loss of information, resulting in reduced capability to generate details and clarity.
MultiGen-20M
Methods
Conditions
Canny Edge (F1 Score ↑ )
Hed Edge (SSIM ↑ )
Depth Map (RMSE ↓ )
ControlNet++
canny
31.98
49.61
14.58
ControlNet++
hed
30.33
66.49
12.17
ControlNet++
depth
17.49
39.22
11.60
T2I-Adapter
all
11.35
28.93
24.97
Uni-ControlNet
all
21.97
41.96
17.14
Appendix
Table A4 : Comparison of image consistency and controllability on MultiGen-20M. Note that ControlNet++ is a single-condition model , while other compared methods are multi-condition models . The diagonal cells (highlighted in gray) represent the optimal performance of ControlNet++ when evaluated under its corresponding visual condition.
Figure A3 : Image consistency comparison between official or re-implemented methods and our proposed model.
Figure A4 : Image consistency comparison between official or re-implemented methods and our proposed model.
Figure A5 : Image consistency comparison between official methods and our proposed model.
Figure A6 : Image consistency comparison between official methods and our proposed model.
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.
Controllable text-to-image diffusion models can often follow the global layout of spatial conditions, yet still violate fine-grained structures such as object boundaries, thin contours, and medium/small conditioned regions. This limitation is especially problematic for VAE-based latent diffusion, where spatial compression can weaken high-frequency and low-area condition signals. We propose PixelControl, a pixel-space controllable diffusion framework for fine-grained condition fidelity. Built on a PixelDiT-style backbone, PixelControl avoids the latent bottleneck and introduces two complementary designs. First, Structure-Aware Control Injection derives a condition structure map and uses it to strengthen injected control residuals around spatially sensitive regions. Second, Multi-Scale Pyramid Cycle Loss verifies generated images against condition-derived structures across multiple resolutions, balancing global layout consistency with local boundary and detail accuracy. PixelControl supports depth, segmentation, edge, and their combinations through modality-specific control branches with lightweight gated fusion. Experiments across depth, segmentation, and edge control show that PixelControl improves structural fidelity and visual quality over existing controllable generation methods, with especially strong gains on boundaries and medium/small conditioned regions. The project page can be found at: https://linxin0.github.io/pixelcontrol_homepage/pixelcontrol-site/
Xin Lin, Haodong Li, Zhifei Zhang +5
University of California, San Diego · Adobe Research · Nanyang Technological University
Recent advances in text-to-image (T2I) generation have enabled controllable image synthesis by incorporating conditions beyond text. However, most existing diffusion-based methods are limited to a single type of control condition (e.g., bounding boxes or keypoints), which restricts their flexibility. To address this limitation, we propose MixDiffusion, a training-free diffusion framework for multi-condition T2I generation. MixDiffusion theoretically supports an arbitrary number of control conditions, including bounding boxes, keypoints, sketches, depth maps, reference images, and text, by collaboratively integrating multiple pre-trained uni-condition diffusion models. The key insight of the proposed approach is to derive the predicted noise distribution in each denoising step of the diffusion-based multi-condition image generation model from the predicted noise distributions of multiple diffusion-based uni-condition models with a derived integration formula, which is supported by rigorous theory proof. Owing to its training-free nature, MixDiffusion is easy to deploy and readily extensible to new control modalities.
Text-to-image diffusion models like Stable Diffusion generate high-quality images from text, but lack a way to inject visual guidance (e.g. sketches, styles) at inference without retraining. Existing methods either require computationally expensive fine-tuning or rely on style transfer techniques that risk semantic misalignment with textual prompts. We introduce Visual Concept Fusion (VCF), the first method offering dual conditioning on both an image and text prompt at inference time without any concept-specific training. VCF enables visual concept injection into Stable Diffusion by aligning CLIP image features with the text embedding space. VCF consists of three components: (1) a lightweight aligner that maps image tokens to the text embedding manifold using InfoNCE and cross-attention reconstruction losses, (2) a fusion strategy that preserves both textual and visual semantics, and (3) an optional Prompt-Noise Optimization (PNO) module for test-time refinement. Our experiments demonstrate that VCF successfully transfers visual attributes including style, composition, and color palette from reference images while maintaining prompt adherence. Quantitative results show a trade-off between text alignment (CLIP score) and visual correspondence (LPIPS), with VCF outperforming baselines in reference fidelity.