AdaGen: Learning Adaptive Policy for Image Synthesis

Zanlin Ni; Yulin Wang; Yeguo Hua; Renping Zhou; Jiayi Guo; Jun Song; Bo Zheng; Gao Huang

cs.CVMar 7, 2026

AdaGen: Learning Adaptive Policy for Image Synthesis

Authors: Zanlin Ni, Yulin Wang, Yeguo Hua, Renping Zhou, Jiayi Guo, Jun Song, Bo Zheng, Gao Huang

Organizations: Department of Automation, BNRist, Tsinghua University, Beijing, China. · Taobao & Tmall Group of Alibaba.

Abstract

Recent advances in image synthesis have been propelled by powerful generative models, such as Masked Generative Transformers (MaskGIT), autoregressive models, diffusion models, and rectified flow models. A common principle behind their success is the decomposition of synthesis into multiple steps. However, this introduces a proliferation of step-specific parameters (e.g., noise level or temperature at each step). Existing approaches typically rely on manually-designed rules to manage this complexity, demanding expert knowledge and trial-and-error. Furthermore, these static schedules lack the flexibility to adapt to the unique characteristics of each sample, yielding sub-optimal performance. To address this issue, we present AdaGen, a general, learnable, and sample-adaptive framework for scheduling the iterative generation process. Specifically, we formulate the scheduling problem as a Markov Decision Process, where a lightweight policy network determines suitable parameters given the current generation state, and can be trained through reinforcement learning. Importantly, we demonstrate that simple reward designs, such as FID or pre-trained reward models, can be easily hacked and may not reliably guarantee the desired quality or diversity of generated samples. Therefore, we propose an adversarial reward design to guide the training of the policy networks. Finally, we introduce an inference-time refinement strategy and a controllable fidelity-diversity trade-off mechanism to further enhance the performance and flexibility of AdaGen. Comprehensive experiments on four generative paradigms validate the superiority of AdaGen. For example, AdaGen achieves better performance on DiT-XL with 3 times lower inference cost and improves the FID of VAR from 1.92 to 1.59 with negligible computational overhead.

Figures & tables

Fig. 1: Main idea of AdaGen. Existing multi-step generative models, such as diffusion models, typically utilize pre-defined, static schedules to configure the generation policy ( e.g . , noise level) of all samples. Instead, AdaGen leverages reinforcement learning (RL) to train a policy network that directly learns the optimal generation policies adaptively tailored for each sample.

Type	Generation Policy	Range	Scheduling Rules
MaskGIT	Mask ratio, $m_{t}$	$m_{t}\in[0,1]$	$m_{t}$	$=\cos(0.5\pi\cdot t/T)$ [ 1 , 2 ] , $2\arccos(t/T)/\pi$ [ 3 , 4 ]
	Sampling temperature, $\tau_{t}$	$\tau_{t}\geq 0$	$\tau_{t}$	$=1$ [ 1 , 2 ] , $0.5+0.8\times(1-t/T)$ [ 4 ]
	Masking temperature, $\zeta_{t}$	$\zeta_{t}\geq 0$	$\zeta_{t}$	$=C\cdot(1-t/T)$ [ 1 , 2 ]
	Guidance scale, $w_{t}$	$w_{t}\geq 0$	$w_{t}$	$=C\cdot t/T$ [ 1 , 2 ] , $C(1-\cos\pi\left({t}/{T}\right)^{C^{\prime}})/2$ [ 4 ]
Autoregressive	Sampling temperature, $\tau_{t}$	$\tau_{t}\geq 0$	$\tau_{t}$	$=C$ [ 5 , 6 ]
	Guidance scale, $w_{t}$	$w_{t}\geq 0$	$w_{t}$	$=C$ [ 6 ] , $C\cdot t/T$ [ 5 ]

TABLE I: Summary of Generation Policies for Different Generation Paradigms

Type	State $\bm{s}_{t}$	Action $\bm{a}_{t}$	Transition Distribution $P(\bm{s}_{t+1}\|\bm{s}_{t},\bm{a}_{t})$	Reward $R(\bm{s}_{t},\bm{a}_{t})$
MaskGIT	$\big(t,\bm{x}_{t}\big)$	$\big(m_{t},\tau_{t},\zeta_{t},w_{t}\big)$	$\Big(\delta(t+1),p_{\bm{\theta}}(\bm{x}_{t+1}\|\bm{x}_{t};\bm{a}_{t})\Big)$	$\mathbb{I}(t=T)\cdot r(\bm{x})$
Autoregressive	$\big(t,\bm{x}_{1:t}\big)$	$\big(\tau_{t},w_{t},k_{t},\rho_{t}\big)$	$\Big(\delta(t+1),p_{\bm{\theta}}(\bm{x}_{1:t+1}\|\bm{x}_{1:t};\bm{a}_{t})\Big)$
Diffusion (ODE)	$\big(t,\bm{x}_{\kappa_{t}}\big)$	$\big(\kappa_{t+1},w_{t}\big)$	$\Big(\delta(t+1),\delta\big(\text{ODE}(\bm{x}_{\kappa_{t}},\kappa_{t+1},\epsilon_{\bm{\theta},w_{t}})\big)\Big)$
Rectified Flow	$\big(t,\bm{x}_{\kappa_{t}}\big)$	$\big(\kappa_{t+1},w_{t}\big)$	$\Big(\delta(t+1),\delta\big(\text{ODE}(\bm{x}_{\kappa_{t}},\kappa_{t+1},v_{\bm{\theta},w_{t}})\big)\Big)$

TABLE II: Unified MDP Formulation for Generative Models

Fig. 2: Training pipeline of AdaGen with adversarial reward modeling. The process involves an adversarial interplay: the policy network is optimized to maximize a reward signal, while the reward model is concurrently trained to distinguish real from generated images. The reward signal is the probability of a sample being deemed real by the reward model. Notably, the pre-trained generative model stays frozen throughout the pipeline.

Panel of Figure 5 — Fig. 3: Reward Design. (a) Using FID [ 55 ] as the reward. (b) Using a pre-trained reward model (PRM) [ 23 ] . (c) Our main approach with adversarial reward modeling.

Fig. 4: Optimization process of AdaGen. When the generation steps increase from

T=8

T=32

, the optimization becomes unstable and yields worse performance. Our proposed action smoothing technique stabilizes convergence and achieves better performance (

T=32

(Stabilized)).

Fig. 5: Action sequence analysis . Left : Learned action sequences at

T=32

for three random samples exhibit erratic fluctuations. Right : A simple linear interpolation between the start and end points of the original learned action sequence yields slightly better performance. We visualize sampling temperature

\tau_{t}

as an example of the action sequence.

Fig. 6: Comparison of action sequences without (left) and with (right) action smoothing. We visualize sequences from both training (exploration) and inference. Action smoothing leads to more feasible exploration during training, and the inference action sequence is also much more reasonable.

Fig. 7: Generated samples and their scores from our adversarial reward model . We showcase 12 samples and observe that the assigned scores, $r_{\bm{\psi}}(\bm{x})$ , align well with their visual quality. This demonstrates that our adversarial reward model serves as an effective evaluator for perceptual quality.

Fig. 8: Illustration of our blended policy. We introduce a new, fidelity-oriented policy network upon our original framework and linearly interpolate its output $\bm{a}_{t}^{\prime}$ with the original policy network’s output $\bm{a}_{t}$ with a blending parameter $\lambda$ to obtain the final action $\bm{a}_{t}^{\rm blend}$ .

Model	Type	Policy	$T=4$	$T=8$	$T=10$	$T=16$
MaskGIT-S	Mask.	Baseline	7.65	5.01	-	4.88
MaskGIT-S	Mask.	AdaGen	4.54 (-3.11)	3.71 (-1.30)	-	3.36 (-1.52)
MaskGIT-L	Mask.	Baseline	6.91	4.65	-	3.79
MaskGIT-L	Mask.	AdaGen	3.63 (-3.28)	2.86 (-1.79)	-	2.41 (-1.38)
DiT-XL	Diff.	Baseline	9.71	5.18	-	3.31
DiT-XL	Diff.	AdaGen	5.31 (-4.40)	2.82 (-2.36)	-	2.19 (-1.12)

TABLE III: Effectiveness of AdaGen on Different Generative Paradigms

Fig. 9: Efficiency of AdaGen . We vary the inference budget by adjusting the number of inference steps ( $T=4,8,16$ ) for MaskGIT-L, SiT-XL, and DiT-XL. For VAR, whose inference steps are fixed, we vary the model size (VAR- $d$ 16, VAR- $d$ 20, VAR- $d$ 30) instead. GPU time is measured on an A100 GPU with batch size 16. CPU time is measured on Xeon 8375 CPU with batch size 1.

Method	Type	Params	Steps	TFLOPs $\downarrow$	FID-50K $\downarrow$	IS $\uparrow$
VQVAE-2 [ 65 ] ( NeurIPS’19 )	AR	13.5B	5120	-	31.1	$\sim$ 45
VQGAN [ 19 ] ( CVPR’21 )	AR	1.4B	256	-	15.78	78.3
ADM-G [ 9 ] ( NeurIPS’21 )	Diff.	554M	250	334.0	4.59	186.7
ADM-G, ADM-U [ 9 ] ( NeurIPS’21 )	Diff.	608M	250	239.5	3.94	215.8
LDM [ 21 ] ( CVPR’22 )	Diff.	400M	250	52.3	3.60	247.7
VQ-Diffusion [ 66 ] ( CVPR’22 )	Diff.	554M	100	12.4	11.89	-

TABLE IV: Class-conditional Image Generation on ImageNet 256

\times

256

Method	Params	Steps	TFLOPs $\downarrow$	FID-50K $\downarrow$
VQGAN [ 19 ]	227M	1024	-	26.52
ADM-G [ 9 ]	559M	250	579.0	7.72
U-ViT-H [ 10 ]	501M	50	14.6	4.05
DiT-XL [ 11 ]	675M	250	60.0	3.04
RIN [ 70 ]	320M	1000	415.0	3.95
Simple-Diffusion [ 69 ]	2B	512	-	3.02

TABLE V: Class-conditional Image Generation on ImageNet 512

\times

512

Fig. 10: Qualitative results of AdaGen . The corresponding models are (a) AdaGen-MaskGIT-L ( $T=32$ ), (b) AdaGen-VAR- $d$ 30 ( $T=10$ ) and (c) AdaGen-Stable-Diffusion ( $T=32$ ), respectively.

Fig. 11: Effectiveness of AdaGen on class-conditional image generation . We visualize the generated samples of VAR- $d$ 30 without and with AdaGen.

Method	Params	Steps	TFLOPs $\downarrow$	FID-30K $\downarrow$
VQ-Diffusion [ 66 ]	370M	100	206.7	13.86
Frido [ 80 ]	512M	200	-	8.97
Make-A-Scene [ 81 ]	4B	1536	-	7.55
U-Net [ 10 ]	53M	50	-	7.32
MaskGIT-S [ 1 ]	45M	16	0.6	5.78
U-ViT [ 10 ]	58M	16	0.9	5.91

TABLE VI: Text-to-image Generation on MS-COCO

Method	Params	Steps	TFLOPs $\downarrow$	FID-30K $\downarrow$
VQGAN [ 19 ]	600M	256	12.6	28.86
ImageBART [ 82 ]	2.8B	256	7.4	22.61
LDM-4 [ 21 ]	645M	50	-	17.01
Draft-and-revise [ 68 ]	654M	72	-	9.65
Muse [ 2 ]	500M	8	2.8	7.67
		16	5.4	7.01

TABLE VII: Text-to-image Generation on CC3M

Method	Params	Steps	TFLOPs $\downarrow$	FID-30K $\downarrow$
CogView [ 35 ]	4B	1088	182.2	27.1
LAFITE [ 83 ]	0.2B	-	-	26.94
CogView 2 [ 84 ]	6B	1088	117.0	24.00
DALL-E [ 19 ]	12B	256	309.8	17.89
GLIDE [ 85 ]	5B	250	77.0	12.24
Make-A-Scene [ 81 ]	4B	1536	-	11.84

TABLE VIII: Zero-shot Text-to-image Generation on MSCOCO

Fig. 12: Effectiveness of AdaGen on text-to-image generation . We visualize the generated samples of Stable Diffusion without and with AdaGen.

	Repeated Sampling?	Lookahead Sampling?	FID-50K $\downarrow$
MaskGIT-L	✗	✗	2.28
	✓	✗	2.07
	✓	✓	1.94
DiT-XL	✗	✗	2.19
	✓	✗	2.06
	✓	✓	-

TABLE IX: Inference-time Refinement of AdaGen

# Repeat Trials	0	1	2	3	4	5	6
MaskGIT-L	2.28	1.94	1.85	1.81	1.84	1.89	1.90
DiT-XL	2.19	2.06	2.02	1.99	1.93	1.91	1.92

TABLE X: Inference-time Refinement of AdaGen with Different Numbers of Repeated Sampling Trials

Fig. 13: The fidelity-diversity trade-off of AdaGen . We visualize 4 $\times$ 4 samples from two ImageNet classes (brown bears and car wheels) across different $\lambda$ values using AdaGen-VAR- $d$ 30. Each setting reports FID scores measuring the fidelity-diversity balance. Higher $\lambda$ values prioritize fidelity over diversity.

Fig. 14: Optimization curve of AdaGen with intermediate samples visualized. FID-5K scores are reported for efficiency.

Generation Policy		FID-50K $\downarrow$
Learnable?	Adaptive?	MaskGIT-S	DiT-XL
✗	✗	7.65	9.71
✓	✗	5.40 (-2.25)	6.03 (-3.68)
✓	✓	4.54 (-3.11)	5.31 (-4.40)

TABLE XI: Effectiveness of AdaGen

Fig. 15: Visualizing the learned policies of AdaGen. We use AdaGen-MaskGIT-L ( $T=8$ ) on ImageNet 256 $\times$ 256 as an example. The mask ratio $m_{t}$ (refine level), which controls the proportion of least-confident tokens to be refined at each step in MaskGIT (see [ 1 ] for more details), is visualized as an example. The policy network adaptively reduces $m_{t}$ for only minor refinements when the sample already reaches a decent quality; otherwise, it keeps adopting relatively higher $m_{t}$ for more adjustments.

TABLE XV: Ablation Studies on Model Architecture and Components of AdaGen. Our Default Setting is Marked in Gray .

Generator	$T$	Inference Cost (TFLOPs)		Proportion
		Generator	Policy Net.	(Policy / Generator)
MaskGIT-L	16	1.7	0.002	0.11%
DiT-XL	16	4.1	0.003	0.07%
SiT-XL	16	4.1	0.003	0.07%
VAR- $d$ 16	10	0.5	0.002	0.40%
VAR- $d$ 30	10	2.0	0.003	0.15%

TABLE XII: Computational Overhead of AdaGen

TABLE XIII: Sensitivity Analysis of PPO Hyperparameters. Default Setting is Marked in Gray . AdaGen Consistently Outperforms the Baseline (Light Gray) across Various Hyperparameter Choices.

Smoothing Weight $\beta$	0	0.2	0.4	0.8	0.95
FID-50K $\downarrow$	3.97	3.61	3.47	3.36	3.70

TABLE XIV: Effect of Action Smoothing Weight

\beta

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Generation Policy	Range	Activation Function
MaskGIT	Mask ratio, $m_{t}$	$m_{t}\in[0,1]$	$\text{sigmoid}(x)$
	Sampling temperature, $\tau_{t}$	$\tau_{t}\geq 0$	$\text{softplus}(x)$
	Mask temperature, $\zeta_{t}$	$\zeta_{t}\geq 0$	$\text{softplus}(x)$
	Guidance scale, $w_{t}$	$w_{t}\geq 0$	$\text{softplus}(x)$
Autoregressive	Sampling temperature, $\tau_{t}$	$\tau_{t}\geq 0$	$\text{softplus}(x)$
	Guidance scale, $w_{t}$	$w_{t}\geq 0$	$\text{softplus}(x)$

Appendix

TABLE XV: Summary of activation functions for different generation policies in AdaGen. The policy network produces an unbounded output

x

, which is then passed through an activation function to ensure the resulting policy value lies within its valid range.

Fig. 16: Generated samples without vs . with refinement of AdaGen.

Fig. 17: Generated samples of AdaGen-VAR- $d$ 30 ( $T=10$ ) on ImageNet 256 $\times$ 256.

Fig. 18: Generated samples of AdaGen-MaskGIT-L ( $T=32$ ) on ImageNet 512 $\times$ 512.

Fig. 19: Generated samples of AdaGen-Stable-Diffusion ( $T=32$ ) for text-to-image generation.

Diffusion Timestep Scheduling

Diffusion timestep scheduling allocates noise and denoising steps across a generative trajectory to balance sample quality, stability, and compute.

See all 63 papers

Apr 21, 2026

Denoising, Fast and Slow: Difficulty-Aware Adaptive Sampling for Image Generation

Diffusion- and flow-based models usually allocate compute uniformly across space, updating all patches with the same timestep and number of function evaluations. While convenient, this ignores the heterogeneity of natural images: some regions are easy to denoise, whereas others benefit from more refinement or additional context. Motivated by this, we explore patch-level noise scales for image synthesis. We find that naively varying timesteps across image tokens performs poorly, as it exposes the model to overly informative training states that do not occur at inference. We therefore introduce a timestep sampler that explicitly controls the maximum patch-level information available during training, and show that moving from global to patch-level timesteps already improves image generation over standard baselines. By further augmenting the model with a lightweight per-patch difficulty head, we enable adaptive samplers that allocate compute dynamically where it is most needed. Combined with noise levels varying over both space and diffusion time, this yields Patch Forcing (PF), a framework that advances easier regions earlier so they can provide context for harder ones. PF achieves superior results on class-conditional ImageNet, remains orthogonal to representation alignment and guidance methods, and scales to text-to-image synthesis. Our results suggest that patch-level denoising schedules provide a promising foundation for adaptive image generation.

Johannes Schusterbauer, Ming Gui, Yusong Li +3

Diffusion Model Sampling Diffusion Sampling Diffusion-Based Image Generation Diffusion Timestep Scheduling Diffusion Model Inference Acceleration Diffusion-Based Denoising

Ask PDF

May 24, 2026

Adversarial Error Correction for Visual Autoregressive Generation

Visual Autoregressive (VAR) models have emerged as a powerful paradigm for image synthesis by performing hierarchical next-scale prediction. However, VAR models are inherently prone to cascading error propagation, where subtle coarse-scale mispredictions are amplified across the hierarchy, ultimately distorting the final synthesis. To mitigate this, we propose AID-VAR, a plug-and-play framework that enhances pre-trained VARs through Adversarially Injected Diagnosis. Instead of a standard passive generation, AID-VAR introduces a proactive error-correction mechanism inspired by the adversarial feedback in GANs. We deploy a discriminator to diagnose fidelity gaps at each scale transition, coupled with a lightweight guidance injector. This module operates as a non-invasive adapter that refines the feature manifold of a frozen VAR backbone, effectively steering the generation toward the distribution of real images without destabilizing the pre-trained latent space. Furthermore, to rigorously evaluate this cross-scale progression, we introduce the Inter-Scale Consistency Score (ISCS), a novel metric that quantifies the fidelity and structural alignment between consecutive resolution scales. Experimental results across various backbones demonstrate that AID-VAR delivers sharper textural details and fewer structural distortions with negligible overhead. For instance, AID-VAR-d20 achieves a 16% improvement in FID with only a 3% increase in parameters. These results establish AID-VAR as a highly efficient and scalable pathway for upgrading large-scale VAR generators, enhancing global coherence and local detail without altering training data, base architectures, or sampling schedules. Code is available at https://github.com/bijiw515/AID-VAR.

Ligong Bi, Tao Huang, Jianyuan Guo +1

1Shanghai Jiao Tong University · 2City University of Hong Kong · 3The University of Sydney

Visual Autoregressive Models Autoregressive Image Generation

Ask PDF

Apr 14, 2026

Self-Adversarial One Step Generation via Condition Shifting

The push for efficient text to image synthesis has moved the field toward one step sampling, yet existing methods still face a three way tradeoff among fidelity, inference speed, and training efficiency. Approaches that rely on external discriminators can sharpen one step performance, but they often introduce training instability, high GPU memory overhead, and slow convergence, which complicates scaling and parameter efficient tuning. In contrast, regression based distillation and consistency objectives are easier to optimize, but they typically lose fine details when constrained to a single step. We present APEX, built on a key theoretical insight: adversarial correction signals can be extracted endogenously from a flow model through condition shifting. Using a transformation creates a shifted condition branch whose velocity field serves as an independent estimator of the model's current generation distribution, yielding a gradient that is provably GAN aligned, replacing the sample dependent discriminator terms that cause gradient vanishing. This discriminator free design is architecture preserving, making APEX a plug and play framework compatible with both full parameter and LoRA based tuning. Empirically, our 0.6B model surpasses FLUX-Schnell 12B (20

\times

more parameters) in one step quality. With LoRA tuning on Qwen-Image 20B, APEX reaches a GenEval score of 0.89 at NFE=1 in 6 hours, surpassing the original 50-step teacher (0.87) and providing a 15.33

\times

inference speedup. Code is available https://github.com/LINs-lab/APEX.

Deyuan Liu, Peng Sun, Yansen Han +3

Westlake University · Zhejiang University · Shanghai Innovation Institute +1

One-Step Generative Modeling T2I Diffusion Models Adversarial Training Flow Matching

Ask PDF

Abstract

Diffusion Timestep Scheduling

Explore similar work

Denoising, Fast and Slow: Difficulty-Aware Adaptive Sampling for Image Generation

Adversarial Error Correction for Visual Autoregressive Generation

Self-Adversarial One Step Generation via Condition Shifting