ProxySearch: Decoupled Inference-Time Scaling for Diffusion Models via Asymmetric Noise-Rank Transfer
Organizations: GenGenAI · Chung-Ang University
Abstract
Noise search at inference time improves text-to-image diffusion but requires many multi-step teacher rollouts per query. Distilled few-step models are now routinely released, yet their potential as proxies inside noise search is uncharacterized. We measure proxy-teacher rank agreement across three distillation regimes and three backbones (DiT, MMDiT, UNet) and find a sharp asymmetry. On our main pool, closed-form image statistics transfer from the proxy to the teacher (Spearman up to 0.80), while learned semantic scorers do not ( 0.12). Paired null controls trace this rank to the first denoising step for coarse statistics and to distillation for finer ones. We use the asymmetry for noise search under a joint constraint: a semantic reward specifies content, and a numeric target specifies image statistics. In PROXYSEARCH, the proxy filters candidates by statistical match in 1 step, and the teacher returns the survivor with the highest semantic reward. No weight trades one objective against the other. On FLUX-1024, PROXYSEARCH tightens the statistical target by 21-23% at the same number of teacher trajectories, with no detected loss on the optimized reward.
Figures & tables
| Method | statistical | GE % | PickScore | HPSv2 | CLIP | ImageReward | aesthetic |
|---|---|---|---|---|---|---|---|
| (a) GenEval-driven search ( GE) | |||||||
| Teacher argmax ( ) | 0.302 | 81.6 | 23.50 | 0.311 | 0.364 | 1.06 | 5.35 |
| ProxySearch ( ) | 0.233 | 82.5 | 23.52 | 0.312 | 0.365 | 1.12 | 5.35 |
| Teacher-seq ( , NFE) | 0.247 | 82.5 | 23.49 | 0.307 | 0.364 | 1.03 | 5.34 |
| (b) PickScore-driven search ( PickScore) | |||||||
| Teacher argmax ( ) | 0.297 | 78.9 | 24.14 | 0.322 | 0.379 | 1.25 | 5.35 |
| Method | statistical | GE % | PickScore | HPSv2 | CLIP | ImageReward | aesthetic |
|---|---|---|---|---|---|---|---|
| Random sampling | 0.303 | 81.0 | 23.11 | 0.302 | 0.354 | 0.96 | 5.34 |
| Teacher argmax ( ) | 0.302 | 87.0 | 23.18 | 0.303 | 0.348 | 0.97 | 5.34 |
| Composite reward ( , ) | 0.205 | 86.5 | 23.31 | 0.306 | 0.352 | 1.02 | 5.34 |
| Post-process shift (full) | 0.066 | 84.0 | 22.95 | 0.288 | 0.351 | 0.90 | 5.32 |
| Post-process shift (clamped to statistical ) | 0.218 | 89.0 | 23.19 | 0.301 | 0.348 | 0.96 | 5.34 |
| ProxySearch ( , -step proxy) | 0.230 | 89.0 | 23.26 | 0.306 | 0.355 | 1.04 | 5.34 |
| proxy steps | Argmax GE | Ours GE | GE [ CI] | Argmax stat. | Ours stat. | stat. % | |
|---|---|---|---|---|---|---|---|
| Pair | Argmax GE | Ours GE | Argmax statistical | Ours statistical |
|---|---|---|---|---|
| SD3.5-Large-Turbo SD3.5-Large | ||||
| SDXL-Turbo SDXL |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Pair | Backbone | Distillation | teacher steps | verifiers | |
|---|---|---|---|---|---|
| FLUX-schnell FLUX-dev | DiT | LADD | 50 | 8 stat + 3 sem | |
| SD3.5-Large-Turbo SD3.5-Large | MMDiT | ADD | 28 | 8 stat + 3 sem | |
| SDXL-Turbo SDXL | UNet | ADD | 50 | 8 stat + 3 sem | |
| SDXL+DMD2 SDXL | UNet | score | 50 | 8 stat + 3 sem |
| Experiment | Pair(s); resolution; prompts; cells | Sampling and selection | Uncertainty |
| Rank-correlation studies (no , no selection; Spearman ) | |||
| Rank transfer (Tab. 7 ) | FLUX, SD3.5, SDXL; ; template; /pair | ; | Wald CI; ; sig% |
| Score distillation (Tab. 7 , DMD2) | SDXL+DMD2; ; template; | ; | ; sig% |
| Proxy step count (App. B.3 ) | FLUX, SD3.5 (SDXL: rank-transfer pool); ; template; /pair | ; | none printed |
| Proxy-semantic top- (Tab. 8 ) | FLUX, SD3.5; ; template; /pair | ; (FLUX), n/r (SD3.5); proxy top- by CLIP, ImageReward, aesthetic | none (prompt mean) |
| High-pass control (App. C.3 ) | FLUX; ; hand-written; | ; | none |
| FLUX (LADD) | SD3.5 (ADD) | SDXL-Turbo (ADD) | SDXL-DMD2 (score) | |||||||||
| Verifier | sig% | sig% | sig% | sig% | ||||||||
| statistics (8 verifiers) | ||||||||||||
| brightness | ||||||||||||
| color_temperature | ||||||||||||
| compressibility | ||||||||||||
| saturation | ||||||||||||
| Pair | semantic verifier | proxy | oracle | proxy | oracle | proxy | oracle |
|---|---|---|---|---|---|---|---|
| FLUX | CLIP | ||||||
| FLUX | ImageReward | ||||||
| FLUX | aesthetic | ||||||
| SD3.5 | CLIP | ||||||
| SD3.5 | ImageReward | ||||||
| Verifier | Frozen teacher | Text-rendering LoRA | PickScore-reward LoRA |
|---|---|---|---|
| statistics (8 verifiers) | |||
| brightness | |||
| color_temperature | |||
| compressibility | |||
| saturation | |||
| sharpness | |||
| FLUX | SD3.5 | |||||||
|---|---|---|---|---|---|---|---|---|
| Statistic | Raw | T-1 | P-1 | P-4 | Raw | T-1 | P-1 | P-4 |
| brightness | ||||||||
| color temp. | ||||||||
| compressibility | ||||||||
| saturation | ||||||||
| sharpness | ||||||||
| (a) GE-driven | (b) PickScore-driven | (c) HPSv2-driven | ||||
|---|---|---|---|---|---|---|
| Metric | CI | CI | CI | |||
| stat. | ||||||
| GE (pp) | ||||||
| PickScore | ||||||
| HPSv2 | ||||||
| CLIP | ||||||
| Method | teacher candidates | total NFE (teacher) | stat. | GE (%) |
|---|---|---|---|---|
| Teacher argmax | ( ) | |||
| Teacher argmax | ( ) | |||
| Composite reward ( ) | ( ) | |||
| ProxySearch ( , ) | ( ) |
| Method | GE acc. ( ) | stat. | total NFE |
|---|---|---|---|
| Teacher argmax ( ) | |||
| Teacher argmax ( ) | |||
| ProxySearch ( , -step proxy) |
| Arm | Question | Annotators | Pairs | Judgments | Rate (%) | Wilson CI | Two-way CI |
|---|---|---|---|---|---|---|---|
| A (look named) | which image better fulfils the request | ||||||
| B (look named) | which image looks like the genuine, unedited photograph | (same) | |||||
| C (no look) | which image is preferred overall as a photograph | ( ) | n/a |
T2I Diffusion Models
Text-to-image (T2I) diffusion models generate images from prompts by learning to reverse noise under text conditioning.
See all 444 papers