Conditional image-to-image generators are single-shot: they map input features to an output in one forward pass and treat it as final, with no opportunity to improve on it. Although trained to produce the best possible result in one step, such a model leaves room for improvement if it can adaptively revise its own output over iterations. We propose Recursive Refinement via Feedback Conditioning (RRFC), a novel feedback-conditioning framework for iterative output refinement that teaches a model to adaptively revise its output by conditioning on a new signal, namely its most recent previous prediction, which is fed back as an auxiliary set of channels alongside the original input. This preserves the generator's core architecture while modifying its conditioning interface and, depending on the model family, its training or inference procedure, so RRFC can be attached to existing generators without redesign. We evaluate RRFC across six baselines spanning adversarial, equilibrium, and diffusion-based models and three paired image-to-image translation tasks. Across 18 architecture-task settings, RRFC yields seven Holm-corrected improvements, seven degradations, and four non-significant changes. The gains concentrate on reconstruction-fidelity and identity settings, while five of the seven degradations fall on the single semantic-layout task, where every model declines. These results indicate that feedback-based refinement helps when its objective overlaps with the evaluated property, and that its gains concentrate on the tasks where that overlap holds.
Figures & tables
Figure 1: Overview of RRFC. A generative model learns to refine each sample by conditioning on its previous prediction alongside the original input.
Table 2
Figure 2: Response surface of the RRFC hyperparameter search on the two single pass baselines. Left, standing on the reported primary metric. Right, the validation selection surrogate. The chosen global recipe (unroll 3, fresh exposure 55 percent, monotonic weight 50) is marked with a star. It is top ranked on the primary metric and within the top cluster on the surrogate.
Figure 3: Per model win rate across metrics, with responsiveness. DEQ, the equilibrium base, is least responsive and improves fewest metrics.
Figure 4: Before and after change on each task’s primary metric at the validation selected depth. Filled triangles are Holm significant at α=0.05 , open triangles are not. RRFC significantly improves seven cells, significantly degrades seven, and produces four non-significant changes, split largely by task.
Figure 5: Mean oriented change of every metric on each task, aggregated over the six models, with the count of significant models. Inpainting improves on all metrics, most strongly on KID. The semantic task improves in structural fidelity (SSIM, PSNR) while its primary metric mIoU falls.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: One-factor-at-a-time view of the search: each panel varies a single hyperparameter and plots the validation selection score for every task–recipe combination (colored by task), with a black diamond at the per-setting mean. Fresh exposure shows the clearest monotone trend, rising to 55 percent, which motivates its value in the adopted recipe.
Figure 7: Per-cell ranking behind the search. Rows are the thirteen candidate recipes, columns the six search cells (Pix2Pix and Pix2PixHD on T1, T2, T3), and each cell gives the recipe’s rank on that task’s primary metric (1 best, light; 13 worst, dark). Rows are ordered by mean rank, so the adopted recipe U3 ⋅ Fr55 ⋅ M50 sits on top: never worse than sixth and best or near-best in several, though not uniformly first. Key: U# unroll depth, Fr# fresh-exposure percent (Buf = buffer-only), M# monotonic weight, Uni = uniform step-weights.
Figure 8: How far each recipe sits from the per-cell best, as its mean percentage above the best recipe in each cell (0 = matches the best, lower = closer), colored by how the recipe was generated and sorted closest to farthest. The adopted recipe U3 ⋅ Fr55 ⋅ M50 is third closest at 0.82 percent, within about one percent of the cell-wise optimum. The two closer recipes carry little or no monotonic penalty; the zero-weight one is kept as the ablation for the monotonic term. Key as in Figure 7 .
Figure 9: Mean rank of each recipe on the primary metric across all search cells (1 = best, lower is better); this is the selection criterion. The adopted recipe U3 ⋅ Fr55 ⋅ M50 ranks first at 3.67, ahead of the runner-up at 5.00, so its selection does not rest on any single cell. Bar colors and key follow Figure 7 .
Figure 10: Leave-one-out ablation of the four RRFC components on Pix2Pix, per task on that task’s selected recipe. Each bar is the change in total validation loss when one component is removed, oriented so positive means removal hurts. Unrolled supervision (C2) is by far the most important, especially on the two aligned tasks; self-conditioning (C1) matters most on inpainting; exposure matching (C4) and the monotonic term (C3) have smaller, mixed effects. The monotonic term is genuinely removable only on T1, since the T2 and T3 selected recipes already set its weight to zero. Single architecture, indicative only.
Figure 11: How reliably and how far RRFC moves each metric, pooled over all eighteen cells. Left: the share of cells where the metric improves (Holm-significant share darker, 50 percent dotted); because the semantic task is pooled in, the win rates sit near a coin flip. Right: the oriented percentage change in every cell (right = better, clipped at ± 60 percent), filled diamonds Holm-significant and open circles the set-level FID and KID. The spread is the point: KID, then FID and SSIM, move most in both directions while pixel-level PSNR stays near zero, so refinement acts most on distribution and structure.
Figure 12: Median per-image change in each task’s primary metric across all eighteen cells, with 95 percent bootstrap intervals; any interval crossing zero is not distinguishable from no effect. Filled diamonds are Holm-significant, and n labels each row (T3 Palette uses n=158 after face-detection failures). Inpainting (T2) carries the largest positive effects, its one strong negative being ControlNet; the semantic task (T1) is a tight band of small, mostly significant negatives; identity (T3) is small and mixed. Magnitudes are not comparable across tasks, as the primary metrics live on different scales, and because this uses the median a cell’s sign can differ from the mean-based views for skewed cells.
Figure 13: Per-image change distributions on the significantly-moving metrics, one violin per architecture (black bar at the median), the percentage above each giving the share of images that improved. Columns are the three tasks; the top row is SSIM and the bottom row each task’s primary metric. Inpainting shows the strongest per-image fidelity gains, with Pix2Pix and SAGAN improving in-hole PSNR on about four in five images. The semantic column shows the central dissociation: SSIM improves on a large share of images while the mIoU primary improves on well under half. Identity gains are broad but modest.
Figure 14: What test-set peeking would have added. For each cell the left point is the mean primary-metric change at the validation-selected depth (reported) and the right point at the oracle depth chosen on test, joined by a line. Most lines are nearly flat and the average inflation is only +0.024 ; the largest gap, T2 ControlNet, stays negative, so that degradation is real. The reported gains do not depend on test-set access.
Figure 15: Seed-averaged primary metric before and after RRFC, one panel per task, with seed-standard-deviation error bars and the signed change annotated. The direction is consistent within task: every model declines on the semantic task (T1); five of six improve on inpainting (T2), ControlNet excepted; most improve on identity (T3), Pix2PixHD excepted. DEQ has the highest scores and the smallest movement in this view. Many changes fall within overlapping error bars, so the value is the per-task direction rather than any single bar.
Figure 16: One panel per architecture, showing the percentage change with RRFC on every metric (primary, LPIPS, PSNR, SSIM, FID, KID), the three tasks as colored bars, oriented so positive is better. These are relative changes in the seed-averaged metric, so they inflate where the baseline is small (the large KID and identity percentages sit on near-zero baselines and should be read as direction, not absolute effect). On the aligned tasks the companion metrics generally move with the primary, so the benefit is broad; the semantic task again shows SSIM rising while mIoU falls. Being means, a cell’s sign can differ from the median forest plot (Figure 12 ).
Figure 17: Change in each task’s primary metric relative to the no-refinement baseline versus refinement depth, averaged over seeds. Each panel is a task, rows are models, the horizontal axis is passes k from 0 to 10, and color is the signed change (teal better, red worse) on a per-panel scale (not comparable across panels). The semantic task (left) is red at every depth; inpainting (middle) is neutral-to-positive across depth; identity (right) turns teal at moderate depth, with isolated red pockets for Palette. Improvements hold roughly flat past the trained unroll depth of three, as the monotonic objective intends.
Figure 18: Quality against inference latency, stacked with the single-pass GAN and equilibrium bases on top (Pix2Pix, Pix2PixHD, SAGAN, DEQ) and the two diffusion bases below (ControlNet, Palette); each row covers the three tasks. A star marks zero passes, the connected line traces increasing refinement depth, and the dashed line is each model’s baseline. The horizontal scales differ sharply between the two rows: the top-row bases reach their useful depth in well under a second per image, whereas the diffusion bases run to tens of seconds, since each of their passes is a full thirty-step DDIM sampler run.
Figure 19: Identity against realism on sketch-to-face (T3). The horizontal axis is FID (realism, lower better) and the vertical axis ArcFace identity (higher better); a cross marks each model at zero passes and the arrow runs to its validation-selected best depth. A tight cluster (Pix2Pix, Pix2PixHD, SAGAN, DEQ) sits at low FID and high identity, DEQ and Pix2PixHD strongest near 0.124; ControlNet is isolated at the best realism (FID near 80) but low identity, about 0.04; and Palette collapses to near-zero identity at FID 230 to 330. The two axes are only loosely coupled: a model can be realistic yet fail to preserve identity. Refinement nudges identity up for most models, but the largest relative gains (ControlNet, Palette) sit on near-zero baselines and do not reach the cluster.
Figure 20: Companion-metric landscapes over refinement depth, stacked as LPIPS (top), PSNR (middle), and SSIM (bottom); within each block the three panels are the three tasks, the horizontal axis is passes k from 0 to 10, rows are the six architectures, and color is the metric on a per-panel scale (not comparable across panels or metrics). The variation is mostly by architecture rather than depth: on LPIPS the two Pix2Pix bases are best and Palette worst, on PSNR the single-pass bases hold the highest-fidelity band and Palette is lowest, and SSIM shows the same ordering. Where depth matters, the change is concentrated in the first pass or two and then flattens.
Figure 21: Raw before-to-after effect sizes on the primary metric, one column per task each on its own scale, printed as the seed-mean change with its standard deviation over three seeds. The independent scales reflect that the primary metrics are not commensurable (mIoU ± 0.026, in-hole PSNR ± 0.5 dB, identity ± 0.015), so magnitudes should not be compared across columns. The standard deviations matter: several large-looking cells are noise-dominated (Palette on inpainting, +0.400±1.616 ; ControlNet on inpainting, −0.063±0.324 , and on identity, +0.015±0.029 ), while tight cells such as DEQ are reliable, which is why some non-zero cells fail the Holm test. The signs are negative throughout the semantic column, positive on inpainting apart from ControlNet, and positive on identity apart from Pix2PixHD.
Figure 22: The latency-quality frontier with all six bases on a shared axis per task (the architecture-split version is Figure 18 ); the horizontal axis is wall-clock latency per image in milliseconds, the vertical axis is the task’s primary metric, a star marks zero passes, and the connected points trace increasing depth. On one axis the cost gap between architecture classes is immediate: the single-pass bases (Pix2Pix, Pix2PixHD, SAGAN) sit against the left edge at sub-second latency, the equilibrium base DEQ occupies the low thousands of milliseconds, and the two diffusion bases (ControlNet, Palette) stretch to tens of thousands of milliseconds, since each of their passes is a full sampler run.
Despite remarkable progress, image generation is far from solved. The dominant metric, FID, conflates sample fidelity with mode coverage and is close to being saturated. Yet a model can still exhibit mode collapse while achieving a low FID, since a handful of sharp, near-duplicate images can outscore a model that faithfully covers the full data distribution. We argue that precision and recall are essential complements to FID, and that because FID is already saturated, the more meaningful goal is to improve diversity and coverage. Achieving high recall requires a model that explicitly prioritizes mode coverage, unlike most generative models, which optimize sample fidelity. We introduce RTM, which replaces the single-pass latent mapping in style-based generators with an iterative refinement process, and show that this consistently improves both quality and diversity. Integrated with Implicit Maximum Likelihood Estimation (IMLE), which optimizes mode coverage by design, RTM achieves the highest precision and recall among current state-of-the-art approaches while maintaining competitive FID, with improvements across CIFAR-10, CelebA-HQ at 256x256, and nine few-shot benchmarks. RTM also improves StyleGAN2 and StyleGAN2-ADA on CIFAR-10 and AFHQ-v1 at 512x512, demonstrating that the benefit is not specific to IMLE. Unlike flow-matching baselines that achieve competitive FID at the expense of coverage, recursive refinement improves both quality and diversity simultaneously.
Mehdi Esmaeilzadeh, Alexia Jolicoeur-Martineau, Chirag Vashist +1
Multimodal diffusion transformers (MMDiTs) allow the evolving image to update internal text representations through joint attention. These representations then guide subsequent image updates, raising a question: does this feedback help correct deviations from the prompt, or preserve them? We find that image-to-text (I2T) feedback can preserve the objects and attributes already represented in the evolving image, including errors. Transplanting I2T messages can rescue failures or disrupt successes. At matched intervention strengths, suppressing the influence of I2T feedback improves generation, whereas opposite controls degrade it. These findings motivate Semantic Feedback Guidance (SFG), which suppresses the influence of I2T feedback by guiding away from an auxiliary branch that strengthens this feedback and weakens text-to-image conditioning. The in-place variant, iSFG, directly suppresses I2T feedback and strengthens text-to-image conditioning, requiring no extra model evaluations. Temporal interventions motivate running the SFG branch during the first 20% of image sampling steps, requiring about 1.2× the base NFE without classifier-free guidance (CFG). Experiments across three image-generation backbones show gains in alignment and preference-model scores. On FLUX, adding SFG to CFG raises GenEval from 0.6661 to 0.7479; on SD3.5M, iSFG adds +1.33 HPSv2.1 over CFG at zero extra NFE. The control also transfers to text-to-video and image-to-video generation, improving VBench scores. Code is available at https://github.com/yyc1215lll-creator/SFG.
Yucheng Yang, Xiaoxiao Sun, Shusheng Yang +2
Stanford University · New York University · University of California, Berkeley
Diffusion models have become a dominant paradigm for conditional image generation, yet existing approaches generally follow two directions: task-specific designs that can improve performance but limit generalization, and training-free loss guidance that compresses rich conditions into scalar objectives and applies stepwise guidance, leading to information bottlenecks and error accumulation along the sampling trajectory. Given the urgent need for an effective unified framework across diverse conditional image generation tasks, we propose Data Injection and Contrastive Trajectory Refinement (DICT), a training-free inference method that enhances conditional image generation without introducing task-dependent architectures. DICT introduces Data Injection, where noise-perturbed conditional signals are integrated into early denoising stages; by performing guided denoising on these injected signals, DICT adaptively selects and distills task-salient information from the raw condition, effectively preserving spatial richness and ensuring precise condition-to-generation alignment. Furthermore, DICT applies Contrastive Trajectory Refinement across adjacent denoising states, enabling pairwise comparisons that progressively improve sample quality. These designs keep inference simple while improving cross-task transfer under a unified diffusion formulation. Extensive experiments on conditional image generation tasks (e.g., style transfer, image super-resolution, and image deblurring) show consistent gains in fidelity and perceptual quality over representative task-specific and loss-guided baselines.