Policy-based Tuning of Autoregressive Image Models with Instance- and Distribution-Level Rewards

Orhun Bugra Baran; Melih Kandemir; Ramazan Gokberk Cinbis

cs.LGMar 24, 2026

Policy-based Tuning of Autoregressive Image Models with Instance- and Distribution-Level Rewards

Authors: Orhun Bugra Baran, Melih Kandemir, Ramazan Gokberk Cinbis

Organizations: Department of Computer Engineering, Middle East Technical University, Ankara, Türkiye · Department of Mathematics and Computer Science (IMADA), University of Southern Denmark, Odense, Denmark · ROMER Robotics and AI Center, Middle East Technical University, Ankara, Türkiye

Abstract

Autoregressive (AR) models are highly effective for image generation, yet their standard maximum-likelihood estimation training lacks direct optimization for sample quality and diversity. While reinforcement learning (RL) has been used to align diffusion models, these methods typically suffer from output diversity collapse. Similarly, concurrent RL methods for AR models rely strictly on instance-level rewards, often trading off distributional coverage for quality. To address these limitations, we propose a lightweight RL framework that casts token-based AR synthesis as a Markov Decision Process, optimized via Group Relative Policy Optimization (GRPO). Our core contribution is the introduction of a novel distribution-level Leave-One-Out FID (LOO-FID) reward; by leveraging an exponential moving average of feature moments, it explicitly encourages sample diversity and prevents mode collapse during policy updates. We integrate this with composite instance-level rewards (CLIP and HPSv2) for strict semantic and perceptual fidelity, and stabilize the multi-objective learning with an adaptive entropy regularization term. Extensive experiments on LlamaGen and VQGAN architectures demonstrate clear improvements across standard quality and diversity metrics within only a few hundred tuning iterations. The results also show that the model can be updated to produce competitive samples even without Classifier-Free Guidance, and bypass its 2x inference cost.

Figures & tables

Figure 1: Maximum-likelihood training (Left): maximize true token $x_{t}$ probability from its groundtruth context. Proposed method (Right): maximize image-level correctness and distribution-level alignment.

Model	FID $\downarrow$	IS $\uparrow$	CLIPScore $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
LlamaGen-B	7.06	119.52	0.2262	0.74	0.61
LlamaGen-B + ours	6.31	163.37	0.2347	0.82	0.54
LlamaGen-L	4.64	196.78	0.2350	0.78	0.63
LlamaGen-L + ours	3.83	215.90	0.2360	0.79	0.63
LlamaGen-XL	3.96	187.75	0.2346	0.74	0.67
LlamaGen-XL + ours	3.82	195.58	0.2370	0.77	0.67

Table 1: Effect of our reward-based training. All experiments use CLIPScore, HPSv2, and FID rewards (weight

1.0

each). Training and testing use CFG scale

1.5

Model	FID $\downarrow$	IS $\uparrow$	CLIPScore $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
LlamaGen-B	20.89	47.96	0.2082	0.47	0.46
LlamaGen-B + ours	8.91	112.83	0.2304	0.78	0.54
LlamaGen-L	10.24	82.38	0.2200	0.53	0.48
LlamaGen-L + ours	5.12	143.32	0.2325	0.77	0.63
LlamaGen-XL	12.66	78.02	0.2192	0.60	0.74
LlamaGen-XL + ours	4.55	148.16	0.2305	0.74	0.68

Table 2: Effect of reward-based fine-tuning without Classifier-Free Guidance (CFG). All runs use CLIPScore, HPSv2, and FID rewards with equal weights (

1.0

each). Training and testing both use CFG scale

1.0

Setting	Task	Ours	Tie	Pretrained
CFG	Realism	117	41	102
CFG	Ref. Match	135	31	94
No CFG	Realism	318	103	149
No CFG	Ref. Match	291	96	183

Table 3: Human preference study using pretrained LlamaGen-L baseline versus ours (with CLIPScore, HPSv2 and LOO-FID rewards), for both CFG and no-CFG cases.

Model	FID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
AR-GRPO (CLIPScore + HPSv2)	6.37	223.95	0.82	0.56
Ours (CLIPScore + HPSv2)	3.83	215.90	0.79	0.63
AR-GRPO (CLIPScore + HPSv2 + MANIQA)	9.34	147.30	0.70	0.59
Ours (CLIPScore + HPSv2 + MANIQA)	6.43	175.34	0.74	0.64

Table 4: Comparison of our approach vs ARGRPO using LlamaGen-L backbone. Training and testing use CFG scale

1.5

Model	DreamSim $\uparrow$	LPIPS $\uparrow$	FID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Recall $\uparrow$
LlamaGen-L	0.53	0.73	4.64	196.78	0.78	0.63
AR-GRPO	0.49	0.72	6.37	223.95	0.82	0.56
Ours	0.53	0.73	3.83	215.90	0.79	0.63

Table 5: Intra-class diversity evaluation using DreamSim and LPIPS on LlamaGen-L (CFG 1.5). Higher is better for DreamSim and LPIPS.

Model	CFG	FID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
LlamaGen-XL	–	12.66	78.02	0.60	0.74
VA- $\pi$ [ 22 ]	–	7.65	102.96	0.69	0.66
Ours	–	4.55	148.16	0.74	0.68
LlamaGen-XL	✓	3.96	187.75	0.74	0.67
VA- $\pi$ [ 22 ]	✓	6.62	212.60	0.80	0.58
Ours	✓	3.82	195.58	0.77	0.67

Table 6: Comparison of our approach vs VA-

\pi

[ 22 ] using LlamaGen-XL backbone. If applied, CFG scale is 1.5 for each model.

Model	topk	FID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
VQGAN	0	21.90	49.81	0.57	0.67
VQGAN + ours	0	15.47	66.21	0.59	0.66
VQGAN	100	16.42	78.32	0.62	0.57
VQGAN + ours	100	13.04	97.23	0.69	0.58

Table 7: Effect of our reward-based training on VQGAN backbone. All experiments use CLIPScore, HPSv2, and FID rewards (weight

1.0

each). Training and testing use no top-k sampling (top-rows) and top-k 100 (bottom rows).

Model	CLIP	HPS	FID $\downarrow$	IS $\uparrow$
LlamaGen-L (pretrained)	–	–	4.64	196.78
LlamaGen-L + CLIP only	✓	–	6.00	217.54
LlamaGen-L + HPS only	–	✓	7.30	176.73
LlamaGen-L + CLIP + HPS	✓	✓	6.21	220.28

Table 8: Ablation on alignment-based rewards using the LlamaGen-L backbone. We compare CLIPScore and HPSv2 rewards individually and jointly (w/o FID reward).

Model	KL weight $\beta$	FID $\downarrow$	IS $\uparrow$
LlamaGen-L (pretrained)	–	4.64	196.78
LlamaGen-L + FID-only	3.0	4.07	183.91
LlamaGen-L + FID-only	0.0	3.97	188.08

Table 9: FID reward only policy training, with and without KL regularization.

Model	Feat. dim	LOO-FID time (ms)	FID $\downarrow$	IS $\uparrow$
Ours (Diag. LOO-FID, PCA)	128	$\sim$ 30	4.30	224.16
Ours (Full-cov. LOO-FID, PCA)	128	$\sim$ 550	4.13	209.45
Ours (Diag. LOO-FID, no PCA)	2048	$\sim$ 30	3.83	215.90

Table 10: Diagonal vs. full-covariance LOO-FID with 128-D (PCA) Inception features. All experiments use CLIPScore, HPSv2, and LOO-FID rewards (weight

1.0

each). Training and evaluation use CFG scale

1.5

. Time reports the LOO-FID reward computation time per iteration (ms) on a single A100.

Table 12

Figure 2: Training dynamics on LlamaGen-L with all rewards, KL regularization, and entropy regularization. Solid blue lines show a moving average; light traces show per-iteration values. Improvements in alignment (left) co-occur with a gradual reduction in EMA FID (middle) and a controlled rise in entropy (bottom), indicating stable learning with preserved exploration. Titles correspond to y-axis labels.

Post-training	FID $\downarrow$	IS $\uparrow$
LlamaGen-L baseline	4.64	196.78
LlamaGen-L + MLE (500 iterations)	4.62	200.61
LlamaGen-L + MLE (5000 iterations)	4.54	202.43
LlamaGen-L + ours (600 iterations)	3.83	215.90

Table 13: Comparison between policy-based post-training and maximum likelihood (MLE) post-training under comparable numbers of additional training iterations.

Panel of Figure 3 — Figure 3: Qualitative comparison of models with classifier-free guidance (scale=1.5).

Panel of Figure 4 — Figure 4: Qualitative comparison of models without classifier-free guidance.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: 2D example for distribution-level reward. A simple line dataset is used to demonstrate the convergence of the policy with the proposed LOO-FID reward.

Model	Layers	$d_{\text{hid}}$	Heads	$d_{\text{code}}$	$T$	Batch
LlamaGen-B	12	768	12	8	$16\times 16$	144
LlamaGen-L	24	1024	16	8	$16\times 16$	144
LlamaGen-XL	36	1280	20	8	$24\times 24$	64
VQGAN	48	1024	16	256	$16\times 16$	64

Appendix

Table 14: Model-specific hyperparameters. Unless otherwise specified, all models use

G=12

K=2

\beta=3.0

\epsilon=0.2

\alpha=0.5

, reward weights

w_{\mathrm{clip}}=w_{\mathrm{hps}}=w_{\mathrm{fid}}=1.0

, and a tokenizer with 16,384 codebook entries.

		CLIP-FID $\downarrow$		CLIP-Precision $\uparrow$		CLIP-Recall $\uparrow$
	Model	Base	Ours	Base	Ours	Base	Ours
	LlamaGen-B	6.00	5.12	0.75	0.80	0.26	0.22
CFG 1.5	LlamaGen-L	4.16	3.80	0.78	0.79	0.33	0.33
	LlamaGen-XL	3.55	3.07	0.78	0.79	0.47	0.49
	LlamaGen-B	9.08	5.12	0.69	0.80	0.28	0.23
No CFG	LlamaGen-L	5.99	3.81	0.72	0.80	0.37	0.37

Appendix

Table 15: Distributional metrics computed in CLIP feature space. Lower is better for CLIP-FID, while higher is better for CLIP-Precision and CLIP-Recall.

Model	FID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Recall $\uparrow$
LlamaGen-L	4.64	196.78	0.78	0.63
Batch LOO-FID	4.06	214.60	0.80	0.62
EMA LOO-FID (ours)	3.83	215.90	0.79	0.63

Appendix

Table 16: Batch versus EMA LOO-FID.

Entropy regularization	FID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Recall $\uparrow$
Scheduled ( $2\times 10^{-3}$ )	4.10	226.66	0.80	0.61
Fixed ( $2\times 10^{-3}$ )	4.12	226.39	0.81	0.61
Adaptive (ours)	3.83	215.90	0.79	0.63

Appendix

Table 17: Adaptive versus fixed entropy regularization.

Panel of Figure 6 — Figure 6: Additional qualitative comparison between the pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance on selected ImageNet classes. For both the Monitor and Wine Bottle classes, our approach produces images that more accurately reflect their respective categories. In contrast, the pretrained model fails to generate a class-consistent image for the Monitor class in the third example and produces an unrecognizable result for the Wine Bottle class in the third example.

Panel of Figure 7 — Figure 7: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Thunder snake: our fine-tuned model produces images that convey the class-specific semantics more clearly, offering more descriptive and recognizable visual attributes compared to the pretrained model. Goldfish: pretrained model often exhibits inconsistencies in body structure and overall geometry, whereas our fine-tuned model generates shapes that are more coherent and anatomically plausible.

Panel of Figure 8 — Figure 8: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Hen: Our model produces faces with clearer structure and more coherent anatomical details. Ostrich: The overall body geometry and structural consistency are improved in our results.

Panel of Figure 9 — Figure 9: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Scorpion: Both models produce images of comparable visual quality for this class. Kite: Our model generates images that more faithfully capture the characteristic visual features of the class, whereas the pretrained model often displays noticeable geometric distortions.

Panel of Figure 10 — Figure 10: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Bald Eagle: Our generations exhibit higher aesthetic quality and more coherent detail overall. Vulture: The pretrained model often produces images that are difficult to associate with the target class and appear visually generic, whereas our model yields outputs that are more clearly identifiable and better aligned with the characteristic features of the class.

Panel of Figure 11 — Figure 11: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Jellyfish: The pretrained model often produces shapes that lack clear structure and show irregular forms. Our model generates jellyfish with more coherent body geometry, smoother outlines, and tentacles that better reflect the characteristic of the class. Spotted Salamander: The pretrained model sometimes produces body shapes that lack proper limb proportion. Our model generally yields more anatomically coherent salamanders with clearer limb structure, resulting in images that better match the class’ defining visual attributes.

Panel of Figure 12 — Figure 12: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Beagle: The pretrained model generally captures the overall dog shape but often produces faces with blurred or softened features. Our model yields clearer facial definition. Overall, the fine-tuned model provides more recognizable, aesthetically better and visually coherent beagle depictions. Norwegian Elkhound: The pretrained model often produces dogs with somewhat blurred facial regions, and the overall body shape can appear loosely defined. Our model generates images with clearer facial structure, resulting in more recognizable and visually consistent elkhound depictions.

Panel of Figure 13 — Figure 13: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Golden Retriever: The pretrained model generally captures the overall dog shape but often produces facial regions that appear soft or slightly distorted, and the fur texture lacks the characteristic smooth, layered appearance of the breed. Our model produces dogs with clearer facial definition, more coherent head and ear geometry, and fur patterns that appear more natural, giving the resulting images a more recognizable and visually consistent golden retriever appearance. Bee: The pretrained model often produces bodies and wings with irregular geometry. Our model yields bees with more coherent body structure, better-defined wing shapes, and markings that more closely resemble the expected striped pattern, resulting in images that are visually clearer and more class-consistent.

Panel of Figure 14 — Figure 14: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Starfish: The pretrained model sometimes produces starfish with uneven arm shapes or irregular overall geometry. Our model generates starfish with more consistent five-arm structure, clearer separation between the arms, resulting in images that better reflect the expected morphology of the class. Hare: Both models produce plausible hare images, but the pretrained outputs often appear softer and less structured. Our generations show clearer facial and ear definition and slightly improved overall aesthetic coherence.

Panel of Figure 15 — Figure 15: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Backpack: The pretrained model often produces backpacks with slightly distorted shapes. Our generations show better-defined structural elements, giving the images a cleaner and more coherent overall aesthetic. Bakery: Both models produce recognizable indoor scenes, but the pretrained outputs more often contain warped shelves or irregular object placement.

Panel of Figure 16 — Figure 16: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Balloon: The pretrained model often produces balloons with irregular outlines or distorted surface shading. Our generations show smoother, more symmetric shapes, resulting in images that appear more visually coherent. Boathouse: The pretrained model sometimes produces buildings with less distinct edges or inconsistent roof geometry. Our generations tend to show clearer structural lines and more stable scene composition, yielding images that look more coherent and visually well-formed.

Panel of Figure 17 — Figure 17: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Cinema: Both models produce plausible cinema façades. In our generations, the building structure and marquee region typically appear more coherent and readable, whereas the pretrained model more often exhibits distortions in the signage and surrounding geometry. Coffee Mug: The pretrained model produces mugs that frequently exhibits distorted geometry. Our fine-tuned model generates more coherent and visually structured prints on the mug surface. Overall, our samples better capture the intended decorative patterns while maintaining plausible mug geometry.

Panel of Figure 18 — Figure 18: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Crate: The pretrained model frequently produces crates with inconsistent wood textures, or partially corrupted object boundaries. Our generations exhibit more regular box geometry, clearer structure, and more coherent wood patterns, resulting in crates that appear more stable and visually plausible overall. File: Our model produces cleaner drawer alignment, more consistent handle placement, and generally better-defined cabinet geometry, yielding images that look more orderly and realistic.

Panel of Figure 19 — Figure 19: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Forklift: The pretrained model often produces forklifts with incomplete or distorted mechanical parts, and several samples show structural artifacts. Our generations exhibit more coherent vehicle geometry resulting in forklifts that appear more plausible and visually stable. Greenhouse: Our model produces images with clearer glass framing, more coherent interior organization, and more realistic plant arrangements, leading to greenhouses that read as cleaner and more structurally consistent.

Panel of Figure 20 — Figure 20: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Handkerchief: Both models produce images of comparable visual quality for this class. Hourglass: Pretrained samples frequently exhibit distorted frame shapes. Our model produces hourglasses with more symmetric frame geometry, clearer separation between the upper and lower chambers, and smoother visual structure, resulting in images that more reliably reflect the intended object.

Panel of Figure 21 — Figure 21: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. iPod: Our model produces units with clearer geometry, more recognizable screen interfaces, and more stable overall proportions, yielding images that better reflect the intended device. Mailbox: Pretrained samples frequently contain irregular shapes or unstable surface details. Our generations offer cleaner cylindrical forms, more consistent mailbox openings, and improved color and material coherence, resulting in images that are easier to interpret as real-world mailboxes.

Panel of Figure 22 — Figure 22: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Palace: Our model produces palaces with clearer structural organization, more stable rooflines, and better-preserved details, resulting in images that read as more coherent and architecturally plausible. Pillow: Pretrained samples occasionally contain distorted shapes or fabric textures that appear uneven or partially collapsed. Our generations show smoother contours, more natural fabric shading, and more consistent pillow geometry.

Panel of Figure 23 — Figure 23: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Table Lamp: Our generations contain more stable lamp silhouettes, more consistent geometry, and more details within the scene, resulting in table lamps that appear more realistic and structurally consistent. Vase: Our model yields vases with smoother profiles, better-defined openings and necks, and more coherent decorative textures, producing images that more reliably reflect the intended object category.

Panel of Figure 24 — Figure 24: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Water Tower: The pretrained model often produces towers with warped tank shapes or inconsistent support structures. Our model generates water towers with cleaner cylindrical forms, more coherent leg geometry, and better separation between structural elements, resulting in images that read as more stable and realistic. Mushroom: Pretrained outputs occasionally display uneven cap shapes, or unclear gill structure. Our generations have smoother cap curvature, and more consistent geometry, yielding samples that more faithfully capture the natural morphology of the class.

Panel of Figure 25 — Figure 25: Additional qualitative comparison between pretrained LlamaGen-L and our policy fine-tuned model with classifier-free guidance. Pineapple: Our generations show more coherent skin structure, better-organized leaf crowns, and improved overall fruit geometry, yielding images that more reliably capture the distinctive visual features of pineapples. Coral Reef: Pretrained samples sometimes contain muddled textures or blended coral forms that make the scene visually ambiguous. Our model produces clearer reef structures with more distinct elements, richer textural diversity, and better separation between foreground coral and background water, resulting in images that are more interpretable and easy to identify.

Explore similar work

Improving Sample Diversity in Autoregressive Text-to-Image Generation via Cluster Truncation
Jul 12, 2026Trang Nguyen, Shuang Wu, Runyan Tan +1T2I Generation Autoregressive Image Generation
University of Massachusetts Amherst · Thoughtworks
MAR-GRPO: Stabilized GRPO for AR-diffusion Hybrid Image Generation
Apr 8, 2026Xiaoxiao Ma, Jiachen Lei, Tianfei Ren +6RL for Image Generation Autoregressive Diffusion
University of Science and Technology of China, Hefei, China · AMAP, Alibaba Group, Hangzhou, China · Zhejiang University, Hangzhou, China
Efficient Training with Foresight: Multi-Token Auxiliary Supervision for Autoregressive Image Generation
Aug 26, 2026Guo Niu, Xiongfei Yao, Teng Wang +1Autoregressive Image Generation Multi-Token Prediction
Foshan University Foshan, China · The University of Hong Kong Hong Kong, China · Sun Yat-sen University Guangzhou, China

Jul 12, 2026J/K move · Enter open · S save

Improving Sample Diversity in Autoregressive Text-to-Image Generation via Cluster Truncation

Trang Nguyen, Shuang Wu, Runyan Tan, Phillip Howard

University of Massachusetts Amherst · Thoughtworks

T2I Generation Autoregressive Image Generation

Open paper AskPDF arXiv

While diffusion models achieve state-of-the-art image quality for text-to-image (T2I) generation, recent work has demonstrated that they suffer from sample diversity collapse. In this work, we investigate whether autoregressive (AR) image generation models can push the Pareto frontier between image quality and sample diversity. With recent advances in quality and efficiency, AR models have emerged as a viable alternative to diffusion-based image generation. Beyond enabling new use cases such as interleaved image-text generation, their sequential generation process makes them compatible with a wide range of token-based decoding strategies originally developed to improve diversity in text generation. Motivated by the potential of a better diversity-quality tradeoff in the AR paradigm, we present the first systematic study of sample diversity in AR image generation models. We show that two key properties of AR image generation, persistently high token-level entropy and substantial redundancy in visual token spaces, limit the effectiveness of existing token-level decoding methods for diversity enhancement. We therefore propose

p

-less cluster, a new decoding strategy that performs entropy-based truncation sampling at cluster level rather than at token level. We evaluate our approach and baseline decoding methods across four autoregressive T2I models and two datasets using a comprehensive suite of metrics spanning image quality, prompt alignment, and diversity. Our results show that

p

-less cluster unlocks the greatest diversity across most evaluated autoregressive T2I models and datasets while maintaining image quality and prompt alignment.

Abstract

Explore similar work

Improving Sample Diversity in Autoregressive Text-to-Image Generation via Cluster Truncation

MAR-GRPO: Stabilized GRPO for AR-diffusion Hybrid Image Generation

Efficient Training with Foresight: Multi-Token Auxiliary Supervision for Autoregressive Image Generation