cs.CVAug 16, 2026
SaveRRFC: Recursive Refinement via Feedback Conditioning for Iterative Image-to-Image Generation
Organizations: Department of Electrical and Computer Engineering, American University of Beirut, Riad El-Solh, Beirut 1107 2020, Lebanon
Abstract
Conditional image-to-image generators are single-shot: they map input features to an output in one forward pass and treat it as final, with no opportunity to improve on it. Although trained to produce the best possible result in one step, such a model leaves room for improvement if it can adaptively revise its own output over iterations. We propose Recursive Refinement via Feedback Conditioning (RRFC), a novel feedback-conditioning framework for iterative output refinement that teaches a model to adaptively revise its output by conditioning on a new signal, namely its most recent previous prediction, which is fed back as an auxiliary set of channels alongside the original input. This preserves the generator's core architecture while modifying its conditioning interface and, depending on the model family, its training or inference procedure, so RRFC can be attached to existing generators without redesign. We evaluate RRFC across six baselines spanning adversarial, equilibrium, and diffusion-based models and three paired image-to-image translation tasks. Across 18 architecture-task settings, RRFC yields seven Holm-corrected improvements, seven degradations, and four non-significant changes. The gains concentrate on reconstruction-fidelity and identity settings, while five of the seven degradations fall on the single semantic-layout task, where every model declines. These results indicate that feedback-based refinement helps when its objective overlaps with the evaluated property, and that its gains concentrate on the tasks where that overlap holds.
Figures & tables
Figure 1: Overview of RRFC. A generative model learns to refine each sample by conditioning on its previous prediction alongside the original input.
Figure 2: Response surface of the RRFC hyperparameter search on the two single pass baselines. Left, standing on the reported primary metric. Right, the validation selection surrogate. The chosen global recipe (unroll 3, fresh exposure 55 percent, monotonic weight 50) is marked with a star. It is top ranked on the primary metric and within the top cluster on the surrogate.
Figure 3: Per model win rate across metrics, with responsiveness. DEQ, the equilibrium base, is least responsive and improves fewest metrics.
Figure 4: Before and after change on each task’s primary metric at the validation selected depth. Filled triangles are Holm significant at , open triangles are not. RRFC significantly improves seven cells, significantly degrades seven, and produces four non-significant changes, split largely by task.
Figure 5: Mean oriented change of every metric on each task, aggregated over the six models, with the count of significant models. Inpainting improves on all metrics, most strongly on KID. The semantic task improves in structural fidelity (SSIM, PSNR) while its primary metric mIoU falls.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: One-factor-at-a-time view of the search: each panel varies a single hyperparameter and plots the validation selection score for every task–recipe combination (colored by task), with a black diamond at the per-setting mean. Fresh exposure shows the clearest monotone trend, rising to 55 percent, which motivates its value in the adopted recipe.
Figure 7: Per-cell ranking behind the search. Rows are the thirteen candidate recipes, columns the six search cells (Pix2Pix and Pix2PixHD on T1, T2, T3), and each cell gives the recipe’s rank on that task’s primary metric (1 best, light; 13 worst, dark). Rows are ordered by mean rank, so the adopted recipe U3 Fr55 M50 sits on top: never worse than sixth and best or near-best in several, though not uniformly first. Key: U# unroll depth, Fr# fresh-exposure percent (Buf = buffer-only), M# monotonic weight, Uni = uniform step-weights.
Figure 8: How far each recipe sits from the per-cell best, as its mean percentage above the best recipe in each cell (0 = matches the best, lower = closer), colored by how the recipe was generated and sorted closest to farthest. The adopted recipe U3 Fr55 M50 is third closest at 0.82 percent, within about one percent of the cell-wise optimum. The two closer recipes carry little or no monotonic penalty; the zero-weight one is kept as the ablation for the monotonic term. Key as in Figure 7 .
Figure 9: Mean rank of each recipe on the primary metric across all search cells (1 = best, lower is better); this is the selection criterion. The adopted recipe U3 Fr55 M50 ranks first at 3.67, ahead of the runner-up at 5.00, so its selection does not rest on any single cell. Bar colors and key follow Figure 7 .
Figure 10: Leave-one-out ablation of the four RRFC components on Pix2Pix, per task on that task’s selected recipe. Each bar is the change in total validation loss when one component is removed, oriented so positive means removal hurts. Unrolled supervision (C2) is by far the most important, especially on the two aligned tasks; self-conditioning (C1) matters most on inpainting; exposure matching (C4) and the monotonic term (C3) have smaller, mixed effects. The monotonic term is genuinely removable only on T1, since the T2 and T3 selected recipes already set its weight to zero. Single architecture, indicative only.
Figure 11: How reliably and how far RRFC moves each metric, pooled over all eighteen cells. Left: the share of cells where the metric improves (Holm-significant share darker, 50 percent dotted); because the semantic task is pooled in, the win rates sit near a coin flip. Right: the oriented percentage change in every cell (right = better, clipped at 60 percent), filled diamonds Holm-significant and open circles the set-level FID and KID. The spread is the point: KID, then FID and SSIM, move most in both directions while pixel-level PSNR stays near zero, so refinement acts most on distribution and structure.
Figure 12: Median per-image change in each task’s primary metric across all eighteen cells, with 95 percent bootstrap intervals; any interval crossing zero is not distinguishable from no effect. Filled diamonds are Holm-significant, and labels each row (T3 Palette uses after face-detection failures). Inpainting (T2) carries the largest positive effects, its one strong negative being ControlNet; the semantic task (T1) is a tight band of small, mostly significant negatives; identity (T3) is small and mixed. Magnitudes are not comparable across tasks, as the primary metrics live on different scales, and because this uses the median a cell’s sign can differ from the mean-based views for skewed cells.
Figure 13: Per-image change distributions on the significantly-moving metrics, one violin per architecture (black bar at the median), the percentage above each giving the share of images that improved. Columns are the three tasks; the top row is SSIM and the bottom row each task’s primary metric. Inpainting shows the strongest per-image fidelity gains, with Pix2Pix and SAGAN improving in-hole PSNR on about four in five images. The semantic column shows the central dissociation: SSIM improves on a large share of images while the mIoU primary improves on well under half. Identity gains are broad but modest.
Figure 14: What test-set peeking would have added. For each cell the left point is the mean primary-metric change at the validation-selected depth (reported) and the right point at the oracle depth chosen on test, joined by a line. Most lines are nearly flat and the average inflation is only ; the largest gap, T2 ControlNet, stays negative, so that degradation is real. The reported gains do not depend on test-set access.
Figure 15: Seed-averaged primary metric before and after RRFC, one panel per task, with seed-standard-deviation error bars and the signed change annotated. The direction is consistent within task: every model declines on the semantic task (T1); five of six improve on inpainting (T2), ControlNet excepted; most improve on identity (T3), Pix2PixHD excepted. DEQ has the highest scores and the smallest movement in this view. Many changes fall within overlapping error bars, so the value is the per-task direction rather than any single bar.
Figure 16: One panel per architecture, showing the percentage change with RRFC on every metric (primary, LPIPS, PSNR, SSIM, FID, KID), the three tasks as colored bars, oriented so positive is better. These are relative changes in the seed-averaged metric, so they inflate where the baseline is small (the large KID and identity percentages sit on near-zero baselines and should be read as direction, not absolute effect). On the aligned tasks the companion metrics generally move with the primary, so the benefit is broad; the semantic task again shows SSIM rising while mIoU falls. Being means, a cell’s sign can differ from the median forest plot (Figure 12 ).
Figure 17: Change in each task’s primary metric relative to the no-refinement baseline versus refinement depth, averaged over seeds. Each panel is a task, rows are models, the horizontal axis is passes from 0 to 10, and color is the signed change (teal better, red worse) on a per-panel scale (not comparable across panels). The semantic task (left) is red at every depth; inpainting (middle) is neutral-to-positive across depth; identity (right) turns teal at moderate depth, with isolated red pockets for Palette. Improvements hold roughly flat past the trained unroll depth of three, as the monotonic objective intends.
Figure 18: Quality against inference latency, stacked with the single-pass GAN and equilibrium bases on top (Pix2Pix, Pix2PixHD, SAGAN, DEQ) and the two diffusion bases below (ControlNet, Palette); each row covers the three tasks. A star marks zero passes, the connected line traces increasing refinement depth, and the dashed line is each model’s baseline. The horizontal scales differ sharply between the two rows: the top-row bases reach their useful depth in well under a second per image, whereas the diffusion bases run to tens of seconds, since each of their passes is a full thirty-step DDIM sampler run.
Figure 19: Identity against realism on sketch-to-face (T3). The horizontal axis is FID (realism, lower better) and the vertical axis ArcFace identity (higher better); a cross marks each model at zero passes and the arrow runs to its validation-selected best depth. A tight cluster (Pix2Pix, Pix2PixHD, SAGAN, DEQ) sits at low FID and high identity, DEQ and Pix2PixHD strongest near 0.124; ControlNet is isolated at the best realism (FID near 80) but low identity, about 0.04; and Palette collapses to near-zero identity at FID 230 to 330. The two axes are only loosely coupled: a model can be realistic yet fail to preserve identity. Refinement nudges identity up for most models, but the largest relative gains (ControlNet, Palette) sit on near-zero baselines and do not reach the cluster.
Figure 20: Companion-metric landscapes over refinement depth, stacked as LPIPS (top), PSNR (middle), and SSIM (bottom); within each block the three panels are the three tasks, the horizontal axis is passes from 0 to 10, rows are the six architectures, and color is the metric on a per-panel scale (not comparable across panels or metrics). The variation is mostly by architecture rather than depth: on LPIPS the two Pix2Pix bases are best and Palette worst, on PSNR the single-pass bases hold the highest-fidelity band and Palette is lowest, and SSIM shows the same ordering. Where depth matters, the change is concentrated in the first pass or two and then flattens.
Figure 21: Raw before-to-after effect sizes on the primary metric, one column per task each on its own scale, printed as the seed-mean change with its standard deviation over three seeds. The independent scales reflect that the primary metrics are not commensurable (mIoU 0.026, in-hole PSNR 0.5 dB, identity 0.015), so magnitudes should not be compared across columns. The standard deviations matter: several large-looking cells are noise-dominated (Palette on inpainting, ; ControlNet on inpainting, , and on identity, ), while tight cells such as DEQ are reliable, which is why some non-zero cells fail the Holm test. The signs are negative throughout the semantic column, positive on inpainting apart from ControlNet, and positive on identity apart from Pix2PixHD.
Figure 22: The latency-quality frontier with all six bases on a shared axis per task (the architecture-split version is Figure 18 ); the horizontal axis is wall-clock latency per image in milliseconds, the vertical axis is the task’s primary metric, a star marks zero passes, and the connected points trace increasing depth. On one axis the cost gap between architecture classes is immediate: the single-pass bases (Pix2Pix, Pix2PixHD, SAGAN) sit against the left edge at sub-second latency, the equilibrium base DEQ occupies the low thousands of milliseconds, and the two diffusion bases (ControlNet, Palette) stretch to tens of thousands of milliseconds, since each of their passes is a full sampler run.
More in this topic
Conditional Image Generation
Conditional image generation creates images guided by inputs such as text, sketches, layouts, masks, reference images, or physical measurements.
See all 157 papers