RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time

Haozhe Wang; Cong Wei; Weiming Ren; Jiaming Liu; Fangzhen Lin; Wenhu Chen

cs.AIApr 13, 2026

RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time

Authors: Haozhe Wang, Cong Wei, Weiming Ren, Jiaming Liu, Fangzhen Lin, Wenhu Chen

Organizations: HKUST · University of Waterloo · Alibaba

Abstract

Most reward models for visual generation reduce rich human judgments to a single unexplained score, discarding the reasoning that underlies preference. We show that teaching reward models to produce explicit, multi-dimensional critiques before scoring transforms them from passive evaluators into active optimization tools, improving generators in two complementary ways: at training time, structured rationales provide interpretable, fine-grained rewards for reinforcement learning; at test time, a Generate-Critique-Refine loop turns critiques into targeted prompt revisions that improve outputs without any parameter updates. To train such a reward model without costly rationale annotations, we introduce Preference-Anchored Rationalization (PARROT), a principled framework that recovers high-quality rationales from readily available preference data through anchored generation, consistency filtering, and distillation. The resulting model, RationalRewards (8B), achieves state-of-the-art preference prediction among open-source reward models, competitive with Gemini-2.5-Pro, while using 10-20x less training data than comparable baselines. As an RL reward, it consistently improves text-to-image and image-editing generators beyond scalar alternatives. Most strikingly, its test-time critique-and-refine loop matches or exceeds RL-based fine-tuning on several benchmarks, suggesting that structured reasoning can unlock latent capabilities in existing generators that suboptimal prompts fail to elicit.

Figures & tables

Figure 1: Train-Time RL and Test-Time PromptTuning (PT) with RationalRewards on text and image-to-image generation benchmarks. (Left) Comparison on image editing benchmarks. RL with RationalRewards outperforms prior open-source generators. Crucially, we find that test-time PT with RationalRewards alone can surpass expensive RL. (Right) Breakdown results on text-to-image benchmark UniGenBench++.

Figure 2: RationalRewards is a reasoning-based reward model that produces structured rationales before assigning scores, enabling dual-space optimization for image generation. (a) As a reward model, it improves RL-based fine-tuning of generators over scalar baselines; (b) as a test-time optimizer, its Generate–Critique–Refine loop matches or surpasses RL-based optimization on multiple benchmarks without parameter updates.

Figure 3: RL (LoRA) training on Qwen-Image using scalar rewards encounter reward hacking (bottom row): as training reward continues to grow, generation quality starts to degenerate, because black box rewards mislead visual generators with biases. In contrast, RationalRewards (top row) sustains generation quality with stable reward growth. See Fig. 10 and 11 for more details.

Figure 4: We implement Preference-Anchored Rationalization as a practical three-phase pipeline.

Figure 5: Example pointwise scores rated by RationalRewards for image/text-to-image generations (rationales omitted). RationalRewards evaluates each result across multiple dimensions.

Figure 6: Qualitative results on image/text-to-image tasks optimized with reinforcement learning (RL) and prompt tuning (PT) using RationalRewards .

Figure 7: Test-Time Prompt Refinement via “Generate-Critique-Refine” loop with RationalRewards .

Judge	MMRB2		EditReward	GenAI-Bench
Judge	T2I	Edit	EditReward	T2I	Edit
Qwen2.5-VL-7B ( Bai et al., 2025 )	50.4	57.1	52.69	–	40.48
Qwen2.5-VL-72B	59.1	64.6	63.9	66.6	74.3
Qwen3-VL-8B ( Yang et al., 2025 )	59.4	61.7	51.9	55.1	50.1
Qwen3-VL-32B	64.1	67.3	64.2	66.9	76.3
EditReward-7B ( Wu et al., 2025e )	–	67.2	56.99	–	65.72

Table 1: Comparison of reward models as evaluators. We include Multimodal Reward Bench 2 (MMRB2), EditReward-Bench, and GenAI-Bench. T2I and Edit means text-to-image and image-to-image respectively.

Model	Action	Attribute	Compound	Layout	Grammar	Logic	Relation	Style	Text	World Know.	Overall
FLUX.1-dev	62.24	67.20	45.75	70.84	62.30	29.77	66.88	85.00	32.18	87.50	60.97
+MultiReward	59.78	68.23	44.21	74.37	59.33	28.25	68.35	76.05	36.21	86.03	60.12
+Qwen3-VL-32B	65.47	72.68	53.28	71.82	60.78	33.24	71.85	85.53	42.15	89.47	66.53
+RationalRewards	67.40	76.36	57.67	72.15	60.29	40.53	74.59	87.20	52.57	90.61	70.34
SD-3.5-Medium	60.41	66.99	53.35	70.31	59.89	37.73	68.78	89.80	15.23	84.34	60.71
+RationalRewards	64.36	81.49	67.98	75.88	58.68	42.37	75.60	89.60	10.05	91.77	70.56

Table 2: Ablation of RationalRewards for Text-to-image RL on UniGenBench++. We compare scalar reward model MultiReward and generic reasoning reward Qwen3-VL-32B .

Model	Representative Aspects			Overall
Model	Light	Reflec.	Deform.	Overall
Flux.1 Kontext [dev]	53.64	43.84	33.74	41.07
+PromptEnhance	55.53	45.87	38.14	45.28
+PT ( RationalRewards )	56.87	51.43	41.08	48.12
+PICA SFT	51.21	47.22	33.99	41.93
+RL ( RationalRewards )	51.75	54.81	39.36	44.25

Table 3: Ablation of RationalRewards as dual-space optimizer on editing tasks. For prompt space tuning, we compare pre-generation PromptEnhance ( Wang et al., 2025g ) . For parameter space tuning, we compare SFT and RL with different rewards. We include OOD physics-aware editing, PICA-Bench with representative aspects (Left), and generic editing benchmarks (Right).

Figure 8: RationalRewards (a) enables explainable quality control for data curation; (b) serves as a multi-dimensional reward model driven by transparent rationales; (c) serves as a preference-calibrated test-time prompt tuner that trades compute for better generation quality; (d) fuels regional flaw grounding and dense visual rewards.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Open-source / Commercial Models
Model	Action	Attribute	Compound	Layout	Grammar	Logic	Relation	Style	Text	World Know.	Overall
Nano Banana Pro ( Google DeepMind, 2025 )	91.30	91.95	92.91	93.30	89.59	80.24	95.43	99.30	95.65	97.47	92.72
Seedream-4-5-251128	88.06	91.03	90.08	92.55	84.09	73.17	90.61	99.20	91.67	96.35	89.70
UniWorld-V1 ( Lin et al., 2025 )	66.94	70.62	54.51	68.96	63.77	38.41	67.13	91.10	26.44	82.91	63.11
OmniGen2 ( Wu et al., 2025c )	62.68	72.12	56.31	71.54	59.89	32.50	68.27	91.90	29.02	86.39	63.09
Bagel ( Deng et al., 2025b )	61.89	67.73	56.86	76.59	65.85	23.85	70.64	90.08	0.00	85.42	59.91

Appendix

Table 4: Text-to-image generation results on UniGen benchmark. We report category-level scores and overall performance. Action is the average of Hand, Full Body, Animal, Non Contact, Contact, and State. Layout is the average of 2D and 3D.

Model	ImgEdit-Bench										GEdit-Bench-EN
Model	Add	Adjust	Extract	Replace	Remove	Background	Style	Compose	Action	Overall	G_SC	G_PQ	G_O
AnyEdit ( Yu et al., 2025 )	3.18	2.95	1.88	2.47	2.23	2.23	2.85	1.56	2.65	2.45	3.18	5.82	3.21
UltraEdit ( Zhao et al., 2024 )	3.44	2.81	2.13	2.96	1.45	2.86	3.76	1.91	2.98	2.70	-	-	-
Step1X-Edit ( Liu et al., 2025c )	3.88	3.14	1.76	3.40	2.41	3.16	4.63	2.64	2.52	3.06	7.66	7.35	6.97
BAGEL ( Deng et al., 2025b )	3.56	3.31	1.70	3.30	2.62	3.24	4.49	2.38	4.17	3.20	7.36	6.83	6.52
OmniGen2 ( Wu et al., 2025c )	3.57	3.06	1.77	3.74	3.20	3.57	4.81	2.52	4.68	3.44	7.16	6.77	6.41

Appendix

Table 5: We perform RL tuning and test-time prompt tuning to test RationalRewards on image editing. On ImgEdit-Bench and GEdit-Bench-EN, trading test-time evaluation for better generation yields surprising gains.

Model	LightProp	LightSrcEff	Reflection	Refraction	Deformation	Causality	GlobalStateTrans	LocalStateTrans	Overall
Nano Banana	53.27	54.45	55.99	56.58	47.68	58.93	58.17	52.81	55.40
GPT-Image-1	59.56	61.99	52.61	61.84	44.99	53.16	70.53	51.56	57.83
Nano Banana Pro	59.32	64.69	61.38	60.09	53.55	64.70	72.08	63.41	63.29
Seedream 4.0	58.84	66.04	58.85	62.72	50.12	67.09	77.37	63.62	64.91
GPT-Image-1.5	62.95	71.43	61.21	62.28	57.18	67.23	76.93	66.11	67.01
Uniworld-V1	37.77	34.50	37.44	30.70	30.32	34.18	28.81	38.67	33.80

Appendix

Table 6: We test OOD Generalization of RationalRewards on physics-aware editing tasks (PICABench).

Figure 9: RL with RationalRewards on Qwen-Image (text-to-image generator) and Flux-Kontext [dev] (image-to-image editing). The reward standard-deviation gradually decays as training proceeds. Crucially, the evaluation reward curve on held-out eval-set align well with the score curve on target test benchmarks.

Figure 10: The evolution of generation quality of RL using RationalRewards

Figure 11: Training curves comparison between RationalRewards and scalar reward model, EditReward ( Wu et al., 2025e ) and MultiReward used in DiffusionNFT Zheng et al. (2025) .

Figure 12: Text-to-Image RL using scalar reward model demonstrates reward hacking – while the reward increases, the visual quality of generations degrades notably.

Figure 13: Illustration of Critique Visualization. RationalRewards first analyzes the image and provides critique rationales, then summarizes them and generates referring expressions for GroundingDINO and SAM to produce segmentation masks for problematic regions.

Score	Description
4 (Full match)	All key elements (objects, colors, actions) are represented exactly as described. No hallucinations or unrequested changes.
3 (Minor mismatch)	Most key elements are present, but minor details are missing, incorrect, or slightly inaccurate.
2 (Some mismatch)	Some key elements are missing, altered, or interpreted incorrectly.
1 (Major deviations)	Key elements are completely missing, altered, or contradicted. Instruction is ignored.

Appendix

Table 7: Scoring rubric for Text Faithfulness.

Score	Description
4 (Uses input fully)	All relevant elements from the input (background, style, lighting, identity) are accurately preserved or transformed as instructed.
3 (Minor mismatch)	Most relevant elements are preserved, but a few aspects (e.g., background details, lighting consistency) are missing or incorrectly handled.
2 (Partial mismatch)	Some elements are carried over, but key aspects of the original image are lost or distorted.
1 (Fails to use input)	Key elements of the input image are ignored, misinterpreted, or destroyed.

Appendix

Table 8: Scoring rubric for Image Faithfulness.

Score	Description
4 (No noticeable flaws)	The image is physically plausible (correct lighting, shadows, geometry, anatomy). No visible artifacts (seams, blurring, noise).
3 (Minor flaws)	Small inaccuracies that are noticeable but not strongly disruptive (e.g., slight lighting mismatch, minor texture issues).
2 (Some flaws)	Clear physical or visual errors that disrupt the image (e.g., incorrect perspective, “floating” objects, wrong shadow direction, obvious seams).
1 (Severe flaws)	Major physical/visual errors (e.g., impossible geometry, distorted anatomy, garbled objects, severe artifacts).

Appendix

Table 9: Scoring rubric for Physical and Visual Quality.

Score	Description
4 (Full match)	Text is correct, legible, and integrated well into the image.
3 (Mostly match)	Minor misspellings or inconsistent capitalization.
2 (Partial match)	Major misspellings or distorted text.
1 (Major deviations)	Text is unreadable, severely distorted, or missing.

Appendix

Table 10: Scoring rubric for Text Rendering.

Hyperparameter	Value
Resolution	512 $\times$ 512
Guidance Scale (Flux.1 Kontext Dev)	2.5
Sampling Steps (Training)	15 (DPM solver)
Sampling Steps (Evaluation)	20
Noise Level	0.7
Learning Rate	2e-4

Appendix

Table 11: Hyperparameters for RL fine-tuning via DiffusionNFT.

Source Dataset	Task	Raw Pairs	Post-Filtering Pairs	Final Pointwise Samples
EditReward	Image Editing	30K	$\sim$ 21.6K	$\sim$ 43.2K
HPDv3	Text-to-Image	50K	$\sim$ 36K	$\sim$ 55K
RapidData	Text-to-Image	50K	$\sim$ 36K	$\sim$ 55K

Appendix

Table 12: Training data composition before and after consistency filtering. Each pairwise sample yields two pointwise projection samples (one per image).

Benchmark	Task	# Samples	Evaluation Protocol	Metrics	Reference
MMRB2 (T2I)	Preference Prediction	1000	Pairwise comparison	Accuracy	Hu et al. (2025)
MMRB2 (Edit)	Preference Prediction	1000	Pairwise comparison	Accuracy	Hu et al. (2025)
EditReward Bench	Preference Prediction	133	Pairwise comparison	Accuracy	Wu et al. (2025e)
GenAI-Bench (T2I)	Preference Prediction	1700	Pairwise comparison	Accuracy	Jiang et al. (2024)
GenAI-Bench (Edit)	Preference Prediction	900	Pairwise comparison	Accuracy	Jiang et al. (2024)

Appendix

Table 13: Summary of evaluation benchmarks.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 325 papers

Think Before You Score: Thinking Reward Model for Visual Generation
Sep 29, 2026Xuehai Bai, Zhenchen Tang, Yang Shi +9Reward Modeling Multimodal Reward Modeling
HDU · SenseTime · CASIA +5
Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards
Oct 2, 2026Yuanhao Ban, I-Hung Hsu, Anastasios Angelopoulos +3T2I Generation Reward Design for RL
Arena Intelligence Inc · UCLA
Z-Reward: Beyond Scalar Rewards by Internalizing Reasoning into Score Distributions
Jun 8, 2026Xin Jin, Huanqia Cai, Zhen Li +9Reward Modeling Reasoning Distillation
Z-Image Team, Alibaba Group · VCIP, CS, Nankai University

Sep 29, 2026J/K move · Enter open · S save

Think Before You Score: Thinking Reward Model for Visual Generation

Xuehai Bai, Zhenchen Tang, Yang Shi, Dianyi Wang+8

HDU · SenseTime · CASIA+5

Reward Modeling Multimodal Reward Modeling

Open paper AskPDF arXiv

Visual reward models are essential for evaluating and improving visual generation models, yet existing approaches typically map task conditions and candidate outputs directly to scalar rewards, leaving implicit what should be evaluated for each individual case. We introduce Think Before You Score, a paradigm that explicitly determines what matters for each case before judging how well the candidate performs. Following this principle, we propose the Thinking Reward Model (TRM), which formulates case-adaptive rubrics, performs rubric-guided assessment, and produces fine-grained pointwise rewards. We further observe that conventional pairwise preference optimization can induce score polarization, and introduce Pairwise Dual-Group Relative Policy Optimization (PD-GRPO), which leverages pairwise supervision to improve reward discrimination while preserving fine-grained pointwise scoring. Extensive experiments on image generation and editing reward-modeling benchmarks demonstrate that TRM achieves state-of-the-art performance among open-source reward models while remaining highly competitive with proprietary alternatives. Moreover, using TRM as a reward for reinforcement learning consistently improves diverse visual generation models, demonstrating that its fine-grained, case-adaptive rewards translate into effective optimization signals for visual generation.

Abstract

Reward Modeling

Explore similar work

Think Before You Score: Thinking Reward Model for Visual Generation

Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards

Z-Reward: Beyond Scalar Rewards by Internalizing Reasoning into Score Distributions