Heeding the Inner Voice: Aligning ControlNet Training via Intermediate Features Feedback
Organizations: AIRI, Russia · HSE University, Russia
Abstract
Despite significant progress in text-to-image diffusion models, achieving precise spatial control over generated outputs remains challenging. One of the popular approaches for this task is ControlNet, which introduces an auxiliary conditioning module into the architecture. To improve alignment of the generation image and control, ControlNet++ proposes a cycle consistency loss to refine correspondence between controls and outputs, but restricts its application to the final denoising steps, while the main structure is introduced at an early stage of generation. To address this issue, we suggest InnerControl -- a training strategy that enforces spatial consistency across a broad range of diffusion steps, including early high-noise ones. Specifically, we train lightweight control prediction probes -- small convolutional networks -- to reconstruct input control signals (e.g., edges, depth) from intermediate diffusion features across denoising steps. We show empirically that such models can extract signals even from very noisy latents and utilize these models to generate pseudo ground truth controls during training. Suggested approach enables alignment loss that minimizes the difference between predicted and target condition over most of the diffusion trajectory. Our experiments demonstrate that our method improves control alignment, with perceptual fidelity preserved or improved in most settings and small FID trade-offs in others. Our approach can be applied to both UNet- and Transformer-based diffusion architectures and be integrated with established training techniques (e.g., ControlNet++) to achieve high performance across different condition methods such as edge, segmentation and depth conditions.
Figures & tables
| Method | T2I Model | Hed Edge | LineArt Edge | Depth Map | Segmentation | ||||
|---|---|---|---|---|---|---|---|---|---|
| SSIM | FID | SSIM | FID | RMSE | FID | mIoU | FID | ||
| Guidance scale = 7.5 | |||||||||
| ControlNet | SDXL | — | — | — | — | 40.00 | — | — | — |
| T2I‑Adapter | SDXL | — | — | 0.639 | — | 39.75 | — | — | — |
| T2I‑Adapter | SD1.5 | — | — | — | — | 48.40 | 22.52 | — | — |
| Gligen | SD1.4 | 0.563 | — | — | — | 38.83 | 18.36 | — | — |
| Method | Model | RMSE | FID |
|---|---|---|---|
| Guidance scale = 7.5 | |||
| ControlNet++ | SDXL | 30.24 | 17.00 |
| InnerControl (Ours) | SDXL | 28.31 | 17.10 |
| Guidance scale = 4.5 | |||
| ControlNet++ | SD 3.5 | 31.07 | 14.24 |
| InnerControl (Ours) | SD 3.5 | 30.31 | 14.34 |
| Estimator | Base | R | A | R + A (Ours) |
|---|---|---|---|---|
| DPT-Large | 33.1 | 29.2 | 32.1 | 27.9 |
| DA-V2-Large | 33.9 | 32.4 | 28.9 | 31.5 |
| Method | Steps | Steps Reward | RMSE | FID | CLIP |
|---|---|---|---|---|---|
| Guidance scale = 7.5 | |||||
| ControlNet | 0 | 0 | 33.95 | 18.61 | 32.175 |
| ControlNet | 920 | 0 | 32.80 | 18.55 | 32.05 |
| ControlNet | 0 | 920 | 25.70 | 22.43 | 31.43 |
| ControlNet | 0 | 400 | 27.63 | 18.59 | 32.00 |
| ControlNet | 0 | 200 | 29.66 | 18.51 | 32.11 |
| Model | Metric | Base | R | A | R+A (InnerControl) |
|---|---|---|---|---|---|
| SD1.5 | RMSE | 34.0 | 27.6 | 32.8 | 26.1 |
| FID | 18.6 | 18.6 | 18.6 | 18.3 | |
| SDXL | RMSE | 34.9 | 30.2 | 33.8 | 28.3 |
| FID | 15.5 | 17.0 | 15.6 | 17.1 | |
| FLUX | RMSE | 33.1 | 29.2 | 32.1 | 27.9 |
| FID | 16.0 | 23.6 | 15.8 | 22.5 |
| Model | Metric | Base | Base + Alignment |
|---|---|---|---|
| ControlNet | RMSE | 34.0 | 32.8 |
| FID | 18.6 | 18.6 | |
| ControlNet++ | RMSE | 27.6 | 26.1 |
| FID | 18.6 | 18.3 | |
| CTRL-U | RMSE | 26.5 | 25.4 |
| FID | 18.7 | 18.5 |
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| Segmentation Mask | Depth Map | Hed Edge | LineArt Edge | |
| Reward Model (RM) | UperNet-R50 | DPT-Hybrid | ||
| RM Performance | ADE20K(mIoU): 42.05 | NYU(AbsRel): 8.69 | - | - |
| Evaluation Model (EM) | Mask2Former | DPT-Large | ||
| EM Performance | ADE20K(mIoU): 56.01 | NYU(AbsRel): 8.32 | - | - |
| Reward Loss | CrossEntropy Loss | MSE Loss | MSE Loss | MSE Loss |
| Loss Weight | 0.5 | 0.5 | 1.0 | 10.0 |
| Segmentation Mask | Depth Map | HED Edge | LineArt Edge | |
| Dataset | ADE20K [ 52 , 51 ] | MultiGen20M [ 28 ] | MultiGen20M [ 28 ] | MultiGen20M [ 28 ] |
| Training Samples | 20,210 | 2,560,000 | 2,560,000 | 2,560,000 |
| Evaluation Samples | 2,000 | 5,000 | 5,000 | 5,000 |
| Evaluation Metric | mIoU | RMSE | SSIM | SSIM |
| Component | Time per iteration | Peak Memory |
|---|---|---|
| Delta per GPU | ||
| Reward-only step | s | - |
| Alignment-only step | s | - |
| Full training iteration (rewarding) | s | Gb |
| Full training iteration (ours) | s | Gb |
| loss | g.s. | RMSE | FID | CLIP | |
|---|---|---|---|---|---|
| 0.5 | MSE | 7.5 | 26.09 | 17.82 | 31.72 |
| 1 | MSE | 7.5 | 26.09 | 18.29 | 32.00 |
| 2 | MSE | 7.5 | 26.21 | 18.24 | 31.82 |
| 5 | MSE | 7.5 | 25.98 | 18.08 | 31.68 |
| steps | loss | g.s. | mIoU | FID | CLIP | |
|---|---|---|---|---|---|---|
| 0.05 | CrossEntropy | 7.5 | 40.22 | 37.7 | 30.2 | |
| 0.1 | CrossEntropy | 7.5 | 41.68 | 40.9 | 29.90 | |
| 0.1 | CrossEntropy | 7.5 | 41.68 | 43.0 | 29.91 | |
| 0.1 | CrossEntropy | 7.5 | 40.20 | 41.1 | 26.25 |
| Method | T2I Model | Hed | LineArt | Depth | Segmentation |
|---|---|---|---|---|---|
| CLIP | CLIP | CLIP | CLIP | ||
| Guidance scale = 7.5 | |||||
| T2I‑Adapter | SD1.5 | — | — | 31.46 | — |
| Gligen | SD1.4 | — | — | 31.48 | — |
| Uni‑ControlNet | SD1.5 | 31.94 | — | 31.66 | — |
| UniControl | SD1.5 | 32.02 | — | 31.68 | — |
| Question | Depth | Edges | Segment. |
|---|---|---|---|
| Prompt fidelity (Q1) | 55% | 67% | 50% |
| Control Alignment (Q2) | 65% | 79% | 72% |
| Aesthetic quality (Q3) | 63% | 77% | 85% |
| Backbone | Num. of Parameters |
|---|---|
| UNet | 8.50M |
| UNet (Unified) | 9.49M |
| SD3.5 (fast) | 8.10M |
| SD3.5 (max) | 79.8M |
| FLUX | 9.90M |
| Method | RMSE | FID | CLIP |
| Klein | 46.06 | 26.15 | 29.16 |
| UniPic2 | 65.02 | 28.92 | 28.41 |
| ControlNet++ (FLUX.1) | 29.22 | 23.61 | |
| InnerControl (FLUX.1) | 28.88 |
| Question | vs. ControlNet++ | vs. Klein | vs. UniPic2 |
|---|---|---|---|
| Prompt | |||
| fidelity (Q1) | 94.9% | 76.5% | 84.1% |
| Control | |||
| alignment (Q2) | 88.0% | 72.6% | 91.5% |
| Aesthetic | |||
| quality (Q3) | 97.8% | 31.1% | 79.1% |
Controllable Image Generation
Controllable image generation steers images using spatial, visual, semantic, stylistic, or physical constraints beyond ordinary text prompts.
See all 227 papers