Reinforcement learning (RL) has emerged as a promising paradigm for enhancing image editing and text-to-image (T2I) generation. However, current reward models, which act as critics during RL, often suffer from hallucinations and assign noisy scores, inherently misguiding the optimization process. In this paper, we present FIRM (Faithful Image Reward Modeling), a comprehensive framework that develops robust reward models to provide accurate and reliable guidance for faithful image generation and editing. First, we design tailored data curation pipelines to construct high-quality scoring datasets. Specifically, we evaluate editing using both execution and consistency, while generation is primarily assessed via instruction following. Using these pipelines, we collect the FIRM-Edit-370K and FIRM-Gen-293K datasets, and train specialized reward models (FIRM-Edit-8B and FIRM-Gen-8B) that accurately reflect these criteria. Second, we introduce FIRM-Bench, a comprehensive benchmark specifically designed for editing and generation critics. Evaluations demonstrate that our models achieve superior alignment with human judgment compared to existing metrics. Furthermore, to seamlessly integrate these critics into the RL pipeline, we formulate a novel "Base-and-Bonus" reward strategy that balances competing objectives: Consistency-Modulated Execution (CME) for editing and Quality-Modulated Alignment (QMA) for generation. Empowered by this framework, our resulting models FIRM-Qwen-Edit and FIRM-SD3.5 achieve substantial performance breakthroughs. Comprehensive experiments demonstrate that FIRM mitigates hallucinations, establishing a new standard for fidelity and instruction adherence over existing general models. All of our datasets, models, and code have been publicly available at https://firm-reward.github.io.
Figures & tables
Figure 1: Comparison of image editing results across different methods. “w. FIRM-Edit-8B” indicates that FIRM-Edit-8B is adopted as the reward model during RL process.
Figure 2: Comparison of T2I generation results across different methods. “w. FIRM-Gen-8B” indicates that FIRM-Gen-8B is adopted as the reward model during RL process.
Figure 3: Overview of the FIRM data curation pipelines. (Top) The FIRM-Edit pipeline follows a novel ”difference-first” design. (Bottom) The FIRM-Gen pipeline adopts a ”plan-then-score” paradigm to significantly enhance scoring accuracy.
Figure 4: Illustrative examples from FIRM-Bench, showing that our reward models are better aligned with human judgments than Qwen3-VL-8B and Qwen3-VL-32B.
Table 5
Model
GEdit-Bench [ 25 ]
ImgEdit [ 50 ]
G_SC
G_PQ
G_Overall
Overall
Instruct-Pix2Pix [ 4 ]
3.58
5.49
3.68
1.88
AnyEdit [ 52 ]
3.18
5.82
3.21
2.45
Step1X-Edit [ 25 ]
7.66
7.35
6.97
3.06
FLUX.1-Kontext[Dev] [ 22 ]
6.52
7.38
6.00
3.71
OmniGen2 [ 46 ]
7.16
6.77
6.41
3.44
Table 3: Performance comparison on GEdit-Bench and ImgEdit. Guided by FIRM-Edit-8B during RL process, the resulting FIRM-Qwen-Edit substantially outperforms the base model as well as counterparts trained with Qwen3-VL models.
Model
GenEval [ 13 ]
DPG-Bench [ 18 ]
TIIF [ 44 ]
UniGenBench++ [ 42 ]
Short
Long
SD-XL [ 30 ]
0.55
74.65
54.96
40.22
41.48
Show-o [ 48 ]
0.53
-
59.72
-
-
EMU3-Gen [ 41 ]
0.54
80.60
-
45.42
50.59
JanusFlow [ 27 ]
0.63
79.68
-
47.10
54.80
FLUX.1-Dev [ 23 ]
0.66
83.84
-
60.97
69.42
Table 4: Performance comparison on GenEval, DPG, TIIF and UniGenBench++. By leveraging FIRM-Gen-8B for RL, our FIRM-SD3.5 achieves substantial performance gains, outperforming both baseline and counterparts trained with Qwen3-VL models.
Method
Formulation
Reward Model
GEdit-Bench
ImgEdit
G_SC
G_PQ
G_Overall
Overall
Baseline
-
-
8.15
7.86
7.54
4.35
Edit-R1
Non-CoT Logits
Qwen2.5VL-32B
4.08
5.05
4.06
2.75
Weighted Score
0.5 * Exe. + 0.5 *Cons.
FIRM-Edit-8B
0.94
8.71
1.06
2.17
Weighted Score
0.6 * Exe. + 0.4 *Cons.
FIRM-Edit-8B
6.78
8.25
6.51
3.73
CME (Ours)
Exe. * (0.6 + 0.4 * Cons.)
FIRM-Edit-8B
8.25
8.20
7.84
4.42
Table 5: Performance comparison of different reward formulations. Our proposed CME achieves the best overall performance by employing the ”Base-and-Bonus” mechanism to effectively balance the competing rewards.
Figure 9
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Score
Dataset
Benchmark
Edit-Exec
Edit-Cons
Gen-Ins
Edit-Exec
Edit-Cons
Gen-Ins
5
49995
50823
116333
105
73
67
4
16004
45720
41833
57
83
73
3
52661
30999
56157
108
82
50
2
21695
38486
59010
21
14
38
1
49819
14053
19459
10
4
22
Appendix
Table 6: Sample counts across score levels for the FIRM datasets and benchmarks.
Figure 8: Reward Cureve of Consistency (left) and Execution (right) under raw reward designs.
While Reinforcement Learning from Human Feedback (RLHF) has become a pivotal paradigm for text-to-image generation, its application to image editing remains largely unexplored. A key bottleneck is the lack of a robust general reward model for all editing tasks. Existing edit reward models usually give overall scores without detailed checks, ignoring different instruction requirements and causing biased rewards. To address this, we argue that the key is to move from a simple scorer to a reasoning verifier. We introduce Edit-R1, a framework that builds a chain-of-thought (CoT) verifier-based reasoning reward model (RRM) and then leverages it for downstream image editing. The Edit-RRM breaks instructions into distinct principles, evaluates the edited image against each principle, and aggregates these checks into an interpretable, fine-grained reward. To build such an RRM, we first apply supervised fine-tuning (SFT) as a ``cold-start'' to generate CoT reward trajectories. Then, we introduce Group Contrastive Preference Optimization (GCPO), a reinforcement learning algorithm that leverages human pairwise preference data to reinforce our pointwise RRM. After building the RRM, we use GRPO to train editing models with this non-differentiable yet powerful reward model. Extensive experiments demonstrate that our Edit-RRM surpasses powerful VLMs such as Seed-1.5-VL and Seed-1.6-VL as an editing-specific reward model, and we observe a clear scaling trend, with performance consistently improving from 3B to 7B parameters. Moreover, Edit-R1 delivers gains to editing models like FLUX.1-kontext, highlighting its effectiveness in enhancing image editing.
Hanzhong Guo, Jie Wu, Jie Liu +6
School of Computing and Data Science, The University of Hong Kong · ByteDance Seed · CUHK +1
Reward models play an essential role in aligning visual generative models, yet most existing visual reward models use a single scalar score or rely on fixed criteria that cannot adapt to different instructions. This limits both interpretability and task sensitivity, especially for text-to-image generation and instruction-based image editing, where different inputs require different evaluation dimensions. We propose RubricRM, a pairwise generative reward modeling framework that first produces an input-specific rubric with evaluation dimensions, weights, and scoring criteria, and then applies the rubric to score candidate images. We train dedicated RubricRM models for text-to-image generation and image editing using a two-stage training pipeline: supervised fine-tuning teaches the model the rubric-based scoring paradigm, while GRPO further improves scoring through fine-grained dimension-level rewards. Experiments on multiple generation and editing benchmarks show that RubricRM outperforms existing specialized reward models and remains competitive with strong proprietary MLLM judges despite using smaller backbones. Our models, data, and code are available at https://github.com/zijiankan/RubricRM.
Zijian Kan, Wei Wang, Long Luo +6
Peking University · Alibaba Group · Alibaba DAMO Academy
We present Qwen-Image-2.0-RL, a post-training pipeline that applies reinforcement learning from human feedback (RLHF) and on-policy distillation (OPD) to improve both the visual quality and instruction-following capability of the Qwen-Image-2.0 diffusion model. To provide reliable reward signals, we construct task-specific composite reward models by fine-tuning vision-language models with a pointwise scoring paradigm and chain-of-thought reasoning. For text-to-image generation, the reward models cover alignment, aesthetics, and portrait fidelity dimensions. For image editing tasks, the reward system addresses instruction-following accuracy and face identity preservation. Building on this reward system, we develop a scalable GRPO-based RL training framework, incorporating a hybrid classifier-free guidance (CFG) strategy to preserve pre-trained knowledge, prompt curation via intra-group reward range filtering, and per-category reward weight calibration. To merge the task-specialized RL policies for T2I and editing, we propose on-policy distillation as the final training stage, which consolidates multiple teachers into a single student model through trajectory-level velocity matching. Extensive evaluation shows that Qwen-Image-2.0-RL achieves 57.84 overall score on Qwen-Image-Bench (+2.61 over the base model), Elo ratings of 1193 in text-to-image arena (+78) and 1349 in image edit arena (+93), demonstrating consistent gains in aesthetic quality, prompt adherence, and editing accuracy.