PixelPonder: Dynamic Patch Adaptation for Enhanced Multi-Conditional Text-to-Image Generation

Yanjie Pan; Qingdong He; Zhengkai Jiang; Pengcheng Xu; Chaoyi Wang; Jinlong Peng; Haoxuan Wang; Yun Cao

cs.CVMar 9, 2025

PixelPonder: Dynamic Patch Adaptation for Enhanced Multi-Conditional Text-to-Image Generation

Authors: Yanjie Pan, Qingdong He, Zhengkai Jiang, Pengcheng Xu, Chaoyi Wang, Jinlong Peng, Haoxuan Wang, Yun Cao,

Organizations: Fudan University · Tencent Youtu Lab · Western University · University of Chinese Academy of Sciences

Abstract

Recent advances in diffusion-based text-to-image generation have demonstrated promising results through visual condition control. However, existing ControlNet-like methods struggle with compositional visual conditioning - simultaneously preserving semantic fidelity across multiple heterogeneous control signals while maintaining high visual quality, where they employ separate control branches that often introduce conflicting guidance during the denoising process, leading to structural distortions and artifacts in generated images. To address this issue, we present PixelPonder, a novel unified control framework, which allows for effective control of multiple visual conditions under a single control structure. Specifically, we design a patch-level adaptive condition selection mechanism that dynamically prioritizes spatially relevant control signals at the sub-region level, enabling precise local guidance without global interference. Additionally, a time-aware control injection scheme is deployed to modulate condition influence according to denoising timesteps, progressively transitioning from structural preservation to texture refinement and fully utilizing the control information from different categories to promote more harmonious image generation. Extensive experiments demonstrate that PixelPonder surpasses previous methods across different benchmark datasets, showing superior improvement in spatial alignment accuracy while maintaining high textual semantic consistency.

Figures & tables

Figure 1: Visualization comparison between Uni-ControlNet [ 51 ] and our proposed method in different conditional controls with the same text prompt. (a, left two columns) Text and various visual controls, where $C_{1}$ , $C_{2}$ , $C_{3}$ and $C_{4}$ denotes canny, sketch, depth and pose map respectively. (b, middle three or four columns) Generation results from UniControlNet. (c, last column) Generation results from our PixelPonder. Previous methods struggled to generate coherent results under multiple conditions, while our results maintain strong similarity to the respective visual controls.

Figure 2: Comparison of different schemes dealing with multiple condition. (a) Existing methods integrate various visual controls into a highly consistent visual control signal in the temporal domain through concatenation or weighted summation. (b) We employ dynamic patch selection to reconstruct visual conditions at the patch level, enabling multi-visual controllable generation in the generative model.

Figure 3: Overall pipeline of the proposed . The Patch Adaptaiom Module segments various visual conditions $\{V_{1},V_{2},\ldots,V_{n}\}$ into patches and dynamiclly reorganizes the visual conditions at the patch level to obtain a unified visual control signal $SP_{out}$ . Subsequently, $SP_{out}$ is sent to the control network, with the Time-aware Control Injection scheme to achieve high-quality image generation under multiple visual conditions.

Figure 4: Denoising results at steps 25, 20, 15, 10, 5, and 1 , where each pair includes the generated result and the selection results $SP_{out}$ at the current time step. In each $SP_{out}$ , canny, pose, depth and sketch are indicated in red, blue, green, and yellow respectively.

Figure 5: Visualization comparison between official methods and our proposed model in different conditional controls.

		MultiGen-20M				Subject-200K
Methods	Conditions	FID ( $\downarrow$ )	CLIP Score ( $\uparrow$ )	SSIM ( $\uparrow$ )	MUSIQ ( $\uparrow$ )	FID ( $\downarrow$ )	CLIP Score ( $\uparrow$ )	SSIM ( $\uparrow$ )	MUSIQ ( $\uparrow$ )	Inference Time ( $\downarrow$ )
T2I-Adapter	all	66.95	71.47	24.03	57.95	64.72	74.35	31.76	55.07	9.33
Uni-ControlNet	all	32.58	78.08	29.37	65.85	44.35	77.40	37.98	66.75	7.22
UniControl	all	25.15	74.09	35.58	72.05	30.95	72.96	47.31	67.50	7.39
Cocktail	pose+hed	24.67	79.87	24.14	67.13	51.51	79.38	29.59	64.16	4.94
Ours	all	11.85	78.60	43.99	69.54	10.61	77.12	66.60	67.32	9.59

Table 1: Comparison on the MultiGen-20M [ 21 ] and Subject-200K [ 40 ] datasets. The best and second-best results are highlighted in bold and underline , respectively. In the Conditions column, “all” denotes the use of canny maps, sketches, pose maps, and depth maps simultaneously during generation.

		MultiGen-20M
Methods	Conditions	FID ( $\downarrow$ )	SSIM ( $\uparrow$ )	MUSIQ ( $\uparrow$ )
Source	-	-	-	69.30
ControlNet++	canny	17.69	36.69	65.67
ControlNet++	hed	13.93	42.12	71.22
ControlNet++	depth	17.56	27.79	71.23
Ours	pose	30.00	27.91	58.16

Table 2: Results of combining different condition types. A quantitative comparison with models controlled by single visual conditions and various combinations of visual conditions. “Source" refers to the original reference image.

Figure 6: Comparison of multi-condition image generation by PixelPonder with different combinations against single-condition image generation by ControlNet++.

Figure 7: Comparison of precision and recall of PAM and random scheme on canny and sketch conditions. Taking Canny and Sketch as examples.

Table 10

Figure 10: Statistical distribution of each visual condition patch during the denoising steps.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	MultiGen-20M
Patch Size	FID ( $\downarrow$ )	CLIP Score ( $\uparrow$ )	SSIM ( $\uparrow$ )	MUSIQ ( $\uparrow$ )
2	11.26	78.62	43.21	69.15
4	12.80	78.24	40.73	68.43
8	13.46	78.07	39.75	68.15

Appendix

Table A1 : Ablation on patch size. We tested the impact of patch size in Patch Selection on controlling the generation.

	MultiGen-20M
Data scale	FID ( $\downarrow$ )	SSIM ( $\uparrow$ )	MUSIQ ( $\uparrow$ )
105k	12.20	40.79	68.93
686k	11.43	41.13	69.16
2.5m	11.26	43.21	69.15

Appendix

Table A2 : Ablation on data scale. FID, SSIM and MUSIQ are reported on MultiGen-20M datasets.

k

represents thousand, and

m

represents million.

	MultiGen-20M
Fourier	FID ( $\downarrow$ )	SSIM ( $\uparrow$ )	MUSIQ ( $\uparrow$ )
×	12.79	44.97	68.38
✓	13.21	44.49	68.09

Appendix

Table A3 : Ablation on fourier. × represents the non-use of fourier, while the ✓ represents the use of fourier.

Figure A1 : Comparison of multiple random generations under single-condition and multi-condition. Ours employs all available conditions, whereas canny and depth denote the use of only the canny or depth conditions, respectively.

Figure A2 : Generation quality comparison between Random Selection and Patch Adaptation. Random selection can lead to a loss of information, resulting in reduced capability to generate details and clarity.

		MultiGen-20M
Methods	Conditions	Canny Edge (F1 Score $\uparrow$ )	Hed Edge (SSIM $\uparrow$ )	Depth Map (RMSE $\downarrow$ )
ControlNet++	canny	31.98	49.61	14.58
ControlNet++	hed	30.33	66.49	12.17
ControlNet++	depth	17.49	39.22	11.60
T2I-Adapter	all	11.35	28.93	24.97
Uni-ControlNet	all	21.97	41.96	17.14

Appendix

Table A4 : Comparison of image consistency and controllability on MultiGen-20M. Note that ControlNet++ is a single-condition model , while other compared methods are multi-condition models . The diagonal cells (highlighted in gray) represent the optimal performance of ControlNet++ when evaluated under its corresponding visual condition.

Figure A3 : Image consistency comparison between official or re-implemented methods and our proposed model.

Figure A4 : Image consistency comparison between official or re-implemented methods and our proposed model.

Figure A5 : Image consistency comparison between official methods and our proposed model.

Figure A6 : Image consistency comparison between official methods and our proposed model.

T2I Generation

Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.

See all 302 papers

PixelControl: Fine-Grained Condition Fidelity in Text-to-Image Diffusion
Aug 16, 2026Xin Lin, Haodong Li, Zhifei Zhang +5Controllable Image Generation T2I Generation
University of California, San Diego · Adobe Research · Nanyang Technological University
MixDiffusion: Mixing Diffusion-based Uni-condition Text-to-Image Generation Models for Multi-condition Image Synthesis
Jul 20, 2026Pengcheng Wan, Liang Han, Lin Xu +2Conditional Image Generation Controllable Image Generation
Harbin Institute of Technology (Shenzhen)
Injecting Image Guidance into Text-Conditioned Diffusion Models at Inference
May 24, 2026Agata Żywot, Iason Skylitsis, Thijmen Nijdam +4Diffusion Model Guidance Test-Time Optimization
University of Amsterdam, Netherlands

Aug 16, 2026J/K move · Enter open · S save

PixelControl: Fine-Grained Condition Fidelity in Text-to-Image Diffusion

Xin Lin, Haodong Li, Zhifei Zhang, Yutong Yang+4

University of California, San Diego · Adobe Research · Nanyang Technological University

Controllable Image Generation T2I Generation T2I Diffusion Models Pixel-Space Diffusion Models

Open paper AskPDF arXiv

Controllable text-to-image diffusion models can often follow the global layout of spatial conditions, yet still violate fine-grained structures such as object boundaries, thin contours, and medium/small conditioned regions. This limitation is especially problematic for VAE-based latent diffusion, where spatial compression can weaken high-frequency and low-area condition signals. We propose PixelControl, a pixel-space controllable diffusion framework for fine-grained condition fidelity. Built on a PixelDiT-style backbone, PixelControl avoids the latent bottleneck and introduces two complementary designs. First, Structure-Aware Control Injection derives a condition structure map and uses it to strengthen injected control residuals around spatially sensitive regions. Second, Multi-Scale Pyramid Cycle Loss verifies generated images against condition-derived structures across multiple resolutions, balancing global layout consistency with local boundary and detail accuracy. PixelControl supports depth, segmentation, edge, and their combinations through modality-specific control branches with lightweight gated fusion. Experiments across depth, segmentation, and edge control show that PixelControl improves structural fidelity and visual quality over existing controllable generation methods, with especially strong gains on boundaries and medium/small conditioned regions. The project page can be found at: https://linxin0.github.io/pixelcontrol_homepage/pixelcontrol-site/

Abstract

T2I Generation

Explore similar work

PixelControl: Fine-Grained Condition Fidelity in Text-to-Image Diffusion

MixDiffusion: Mixing Diffusion-based Uni-condition Text-to-Image Generation Models for Multi-condition Image Synthesis

Injecting Image Guidance into Text-Conditioned Diffusion Models at Inference