Diff-Aid: Inference-time Adaptive Interaction Denoising for Rectified Text-to-Image Generation
Organizations: Fudan University · Shanghai Innovation Institute · Shanghai Academy of AI for Science
Abstract
Recent text-to-image (T2I) diffusion models have achieved remarkable advancement, yet faithfully following complex textual descriptions remains challenging due to insufficient interactions between textual and visual features. Prior approaches enhance such interactions via architectural design or handcrafted textual condition weighting, but lack flexibility and overlook the dynamic interactions across different blocks and denoising stages. To provide a more flexible and efficient solution to this problem, we propose Diff-Aid, a lightweight inference-time method that adaptively adjusts per-token text and image interactions across transformer blocks and denoising timesteps. Beyond improving generation quality, Diff-Aid yields interpretable modulation patterns that reveal how different blocks, timesteps, and textual tokens contribute to semantic alignment during denoising. As a plug-and-play module, Diff-Aid can be seamlessly integrated into downstream applications for further improvement, including style LoRAs, controllable generation, and zero-shot editing. Experiments on strong baselines (SD 3.5 and FLUX) demonstrate consistent improvements in prompt adherence, visual quality, and human preference across various metrics. Our code and models will be released.
Figures & tables
| Models | All | Char. | Arts | Design | Archi. | Animals | Scenery | Transpor. | Products | Plants | Food | Science | Others |
| Kolors [ 47 ] ∗ | 10.55 | 11.79 | 10.47 | 9.87 | 10.82 | 10.60 | 9.89 | 10.68 | 10.93 | 10.50 | 10.63 | 11.06 | 9.51 |
| FLUX [ 22 ] ∗ | 10.43 | 11.70 | 10.32 | 9.39 | 10.93 | 10.38 | 10.01 | 10.84 | 11.24 | 10.21 | 10.38 | 11.24 | 9.16 |
| Infinity [ 12 ] ∗ | 10.26 | 11.17 | 9.95 | 9.43 | 10.36 | 9.27 | 10.11 | 10.36 | 10.59 | 10.08 | 10.30 | 10.59 | 9.62 |
| SD 3 [ 9 ] ∗ | 5.31 | 6.70 | 5.98 | 5.15 | 5.25 | 4.09 | 5.24 | 4.25 | 5.71 | 5.84 | 6.01 | 5.71 | 4.58 |
| SD 3.5 † | 9.31 | 9.66 | 8.63 | 8.58 | 10.03 | 8.81 | 8.90 | 9.58 | 9.67 | 9.78 | 10.05 | 8.60 | 9.46 |
| SD 3.5 + TACA † | 9.44 | 9.81 | 8.82 | 8.54 | 10.31 | 8.92 | 9.26 | 9.54 | 9.96 | 9.85 | 10.05 | 8.81 | 9.48 |
| Pair | Alternative wins | Tie | Diff-Aid wins |
| FLUX vs. FLUX + Diff-Aid | 507 (15.84%) | 1,269 (39.66%) | 1,424 (44.50%) |
| SD 3.5 vs. SD 3.5 + Diff-Aid | 724 (22.63%) | 1,138 (35.56%) | 1,338 (41.81%) |
| FLUX+LoRA vs. FLUX + Diff-Aid | 848 (26.50%) | 1,040 (32.50%) | 1,312 (41.00%) |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | DiT Blocks | Parameters | Inference Timesteps | Sampler | CFG Scale | Image Size |
| SD 3.5 | 38 | 8B | 28 | EDM | 7.0 | |
| FLUX | 57 | 12B | 28 | EDM | 3.5 |
| Method | Training data | HPSv2 | HPSv3 | ImgRwd | Aes. |
| FLUX | – | 28.53 | 10.42 | 0.89 | 6.66 |
| LoRA + SFT | Filtered | 28.54 | 10.43 | 0.86 | 6.63 |
| Diff-Aid + SFT | Mixed | 28.74 | 10.74 | 0.93 | 6.73 |
| Diff-Aid + SFT | Filtered | 28.80 | 10.84 | 0.95 | 6.76 |
| Diff-Aid + DPO | Filtered | 28.81 | 10.85 | 0.94 | 6.75 |
| Models | HPSv2 | HPSv3 | ImgRwd | Aes. |
| FLUX.1-Schnell | 28.48 | 9.95 | 0.87 | 6.55 |
| +Ours | 28.65 | 10.14 | 0.91 | 6.64 |
| Metric | Method | 4 | 8 | 10 | 15 | 28 | 50 |
| HPSv2 | Baseline | 25.48 | 27.53 | 27.82 | 28.44 | 28.95 | 29.11 |
| Ours | 26.62 | 28.29 | 28.55 | 28.95 | 29.15 | 29.23 | |
| HPSv3 | Baseline | 5.07 | 9.70 | 9.94 | 10.45 | 10.89 | 11.07 |
| Ours | 8.54 | 11.31 | 11.29 | 11.34 | 11.33 | 11.41 | |
| ImgRwd | Baseline | 0.21 | 0.87 | 0.90 | 1.01 | 1.08 | 1.10 |
| Ours | 0.77 | 1.10 | 1.12 | 1.14 | 1.14 | 1.13 |
| CFG scale | 1 | 2 | 4 | 6 | 8 | 10 | 15 |
| FLUX | 0.618 | 0.653 | 0.662 | 0.667 | 0.664 | 0.638 | 0.586 |
| FLUX + Diff-Aid | 0.652 | 0.674 | 0.690 | 0.703 | 0.705 | 0.682 | 0.647 |
| Strategies | HPSv2 | HPSv3 | ImgRwd | Aes. |
| FLUX | 28.53 | 10.42 | 0.89 | 6.66 |
| Sparse Enhancement | 28.61 | 10.57 | 0.98 | 6.71 |
| Our Full Method | 28.80 | 10.84 | 0.95 | 6.76 |
| Control | HPSv3 | GenEval |
| FLUX baseline | 10.42 | 0.676 |
| Sparse static scaling | 10.57 | 0.683 |
| Enhancement-only, | 10.72 | 0.692 |
| Full signed Diff-Aid ( ) | 10.82 | 0.709 |
| Model | T2I-CompBench | DPG-Bench | CLIP Score |
| FLUX | 0.4739 | 83.67 | 28.13 |
| FLUX + Diff-Aid | 0.5006 | 87.14 | 28.35 |
| SD 3.5 | 0.4980 | 76.40 | 27.88 |
| SD 3.5 + Diff-Aid | 0.5249 | 79.32 | 27.95 |
| Method | Type | Extra params | HPSv3 | GenEval |
| FLUX | Baseline | – | 10.42 | 0.676 |
| SAG [ 16 ] | Guidance | 0 | 10.45 | 0.678 |
| PAG [ 1 ] | Guidance | 0 | 10.51 | 0.678 |
| Dynamic CFG [ 53 ] | Guidance | 0 | 10.50 | 0.681 |
| LoRA [ 17 ] | PEFT | 10.43 | 0.683 | |
| DoRA [ 27 ] | PEFT | 10.47 | 0.691 |
| Method | Additional Param (M) | Peak VRAM (GB) | Avg. Time (s) | HPSv2 | HPSv3 | ImgRwd | Aes. |
| FLUX.1-Dev | — | 37.22 | 5.7483 | 28.53 | 10.42 | 0.89 | 6.66 |
| +LoRA (rank=32) | 37.35 (+0.31%) | 37.67 | 5.7649 | 28.54 | 10.43 | 0.86 | 6.63 |
| +LoRA (rank=128) | 149.42 (+1.26%) | 37.96 | 5.8164 | 28.55 | 10.44 | 0.88 | 6.68 |
| +Ours (dim=512) | 45.21 (+0.38%) | 37.83 | 5.7897 | 28.80 | 10.82 | 0.95 | 6.75 |
| +Ours (dim=2048) | 179.77 (+1.51%) | 38.08 | 5.8252 | 28.80 | 10.84 | 0.95 | 6.76 |
| Strategies | HPSv2 | HPSv3 | ImgRwd | Aes. |
| LoRA | 28.50 | 10.38 | 0.87 | 6.64 |
| LoRA + Ours | 28.54 | 10.43 | 0.86 | 6.63 |
| Control | 28.57 | 10.48 | 0.90 | 6.67 |
| Control + Ours | 28.76 | 10.68 | 0.93 | 6.70 |
| Setting | HPSv2 | HPSv3 | ImgRwd | Aes. |
| FLUX baseline | 28.53 | 10.42 | 0.89 | 6.66 |
| Diff-Aid (full) | 28.80 | 10.84 | 0.95 | 6.76 |
| Shared across blocks | 27.41 | 9.58 | 0.77 | 6.34 |
| Shared across tokens | 28.07 | 10.28 | 0.88 | 6.48 |
| Time-averaged | 28.63 | 10.46 | 0.91 | 6.53 |
| Fixed 1st-step | 28.56 | 10.35 | 0.82 | 6.33 |
| Categories | Settings | HPSv2 | HPSv3 | ImgRwd | Aes. |
| FLUX | 28.53 | 10.42 | 0.89 | 6.66 | |
| Dropout Probability | =0.0 | 28.71 | 10.74 | 0.94 | 6.72 |
| =0.1 | 28.80 | 10.84 | 0.95 | 6.76 | |
| =0.2 | 28.73 | 10.77 | 0.94 | 6.73 | |
| =0.3 | 28.47 | 10.44 | 0.87 | 6.64 | |
| =0.5 | 28.21 | 10.19 | 0.77 | 6.59 | |
T2I Diffusion Models
Text-to-image (T2I) diffusion models generate images from prompts by learning to reverse noise under text conditioning.
See all 435 papers