RubricRM: Generative Reward Modeling via Dynamic Rubrics for Image Generation and Editing

Zijian Kan; Wei Wang; Long Luo; Bing Zhao; Xuan Ren; Weixu Qiao; Wenbo Li; Hu Wei

cs.CVAug 27, 2026

RubricRM: Generative Reward Modeling via Dynamic Rubrics for Image Generation and Editing

Authors: Zijian Kan, Wei Wang, Long Luo, Bing Zhao, Xuan Ren, Weixu Qiao, Wenbo Li, Hu Wei,

Organizations: Peking University · Alibaba Group · Alibaba DAMO Academy

Abstract

Reward models play an essential role in aligning visual generative models, yet most existing visual reward models use a single scalar score or rely on fixed criteria that cannot adapt to different instructions. This limits both interpretability and task sensitivity, especially for text-to-image generation and instruction-based image editing, where different inputs require different evaluation dimensions. We propose RubricRM, a pairwise generative reward modeling framework that first produces an input-specific rubric with evaluation dimensions, weights, and scoring criteria, and then applies the rubric to score candidate images. We train dedicated RubricRM models for text-to-image generation and image editing using a two-stage training pipeline: supervised fine-tuning teaches the model the rubric-based scoring paradigm, while GRPO further improves scoring through fine-grained dimension-level rewards. Experiments on multiple generation and editing benchmarks show that RubricRM outperforms existing specialized reward models and remains competitive with strong proprietary MLLM judges despite using smaller backbones. Our models, data, and code are available at https://github.com/zijiankan/RubricRM.

Figures & tables

Figure 1: Inference and training pipeline of RubricRM. (A) RubricRM supports both text-to-image and image-editing preference selection. At inference time, the model first dynamically generates a rubric conditioned on the input, including scoring criteria and corresponding weights, and then scores each image along each dimension based on the rubric, all within a single inference pass. (B) Training is conducted in two stages. The model is first supervised fine-tuned on synthesized rubric data to learn the paradigm of rubric generation followed by scoring, and is then further optimized with GRPO using dimension-level rewards.

Metric	Text-to-Image	Image Editing
#Pairs	31,835	30,695
SFT/RL	16,835/15,000	15,695/15,000
Taxonomy (P/S)	6/42	12/30
Avg. tags	3.3	1.4
Multi-tag (%)	94.9	31.8
Avg. dim.	3.9	3.2

Table 1: Training data statistics for rubric learning. P/S denotes primary/secondary labels.

Model	MMRB2	GenAI- Bench	GenAI- Bench- Verified
Proprietary MLLMs
Claude Sonnet 4.6	70.8	65.8	75.3
GPT-5.4	67.5	64.2	74.2
Gemini 2.5 Pro	70.5	67.8	77.4
Gemini 3.1 Pro	74.4	73.9	84.8
Open-source MLLMs

Table 2: Comparison on text-to-image generation benchmarks. Bold and underlined scores denote the best and second-best results among reward models, respectively.

Model	MMRB2	EditReward-ERB				EditScore-ERB
		K=2	K=3	K=4	Avg	PF	C	O	Avg
Proprietary MLLMs
Claude Sonnet 4.6	71.7	79.0	46.0	10.8	44.1	87.6	67.4	81.4	79.3
GPT-5.4	68.5	78.2	40.7	12.2	42.5	83.7	62.8	76.3	74.6
Gemini 2.5 Pro	71.3	80.5	38.0	12.2	42.2	81.4	65.5	77.4	75.2
Gemini 3.1 Pro	74.9	78.2	45.3	14.9	45.0	87.7	71.6	84.1	81.6

Table 3: Comparison on image editing benchmarks. Avg denotes the benchmark-level average score. Bold and underlined scores denote the best and second-best results among reward models, respectively.

Configuration	MMRB2	GenAI-Bench	GenAI-Bench-Verified
Qwen3.5-4B	63.3	61.9	69.7
+ SFT	70.1 ( $\uparrow$ 6.8)	72.0 ( $\uparrow$ 10.1)	82.9 ( $\uparrow$ 13.1)
+ RL	70.5 ( $\uparrow$ 0.4)	73.2 ( $\uparrow$ 1.2)	83.1 ( $\uparrow$ 0.2)
Qwen3.5-9B	66.9	63.4	72.5
+ SFT	70.3 ( $\uparrow$ 3.4)	73.0 ( $\uparrow$ 9.6)	83.2 ( $\uparrow$ 10.7)
+ RL	72.0 ( $\uparrow$ 1.7)	74.1 ( $\uparrow$ 1.2)	84.5 ( $\uparrow$ 1.2)

Table 4: Text-to-image ablation of the two-stage training pipeline. Green deltas denote changes relative to the previous row within the same backbone.

Configuration	MMRB2	EditReward-ERB	EditScore-ERB
Qwen3.5-4B	64.7	37.1	72.2
+ SFT	71.5 ( $\uparrow$ 6.8)	43.6 ( $\uparrow$ 6.5)	83.3 ( $\uparrow$ 11.1)
+ RL	73.2 ( $\uparrow$ 1.7)	45.5 ( $\uparrow$ 1.9)	85.5 ( $\uparrow$ 2.2)
Qwen3.5-9B	64.4	37.4	72.0
+ SFT	73.8 ( $\uparrow$ 9.4)	45.0 ( $\uparrow$ 7.6)	85.1 ( $\uparrow$ 13.1)
+ RL	75.4 ( $\uparrow$ 1.6)	46.4 ( $\uparrow$ 1.4)	85.6 ( $\uparrow$ 0.5)

Table 5: Image-editing ablation of the two-stage training pipeline. Green deltas denote changes relative to the previous row within the same backbone.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Primary	Secondary label	Cov. (%)
Common Objects	People	45.81
Style	Realistic Photography	38.83
Composition & Perspective	Other	26.34
Common Objects	Everyday Items	24.55
Common Objects	Architecture & Interior	23.33
Common Objects	Natural Scenery	15.53

Appendix

Table 6: Second-level coverage of the text-to-image prompt taxonomy. Labels are multi-label; coverage is measured over all text-to-image training examples.

Primary	Secondary label	Cov. (%)
Object Editing	Add Object	21.01
Attribute Editing	Color Change	14.30
Object Editing	Remove Object	12.52
Background Editing	Background Replacement	11.82
Text Editing	Text Editing	11.17
Object Editing	Replace Object	10.77

Appendix

Table 7: Second-level coverage of the image-editing instruction taxonomy. Labels are multi-label; coverage is measured over all image-editing training examples.

Semantic group	Rubric atom	Dim. share (%)	Avg. wt.	Representative original dimension titles
Semantic Alignment	Prompt alignment	7.99	32.7	Content Accuracy; Semantic Accuracy; Semantic Alignment
Semantic Alignment	Text rendering	1.93	32.2	Text Accuracy; Text Rendering; Text Rendering Accuracy
Semantic Alignment	Reasoning & knowledge	1.78	29.7	Composition & Spatial Logic; Logical Consistency; Compositional Logic
Subject & Attributes	Subject fidelity	14.89	30.3	Subject Accuracy; Subject Representation; Character Accuracy
Subject & Attributes	Anatomy / pose	9.58	27.9	Anatomical Integrity; Anatomical Correctness; Anatomical Coherence
Subject & Attributes	Attribute fidelity	8.64	26.5	Color Palette; Attire Accuracy; Lighting & Color Palette

Appendix

Table 8: Semantic aggregation of parsed text-to-image rubric dimensions. Dim. share is computed over parsed teacher-generated rubric dimensions; avg. wt. is the average teacher-assigned dimension weight.

Semantic group	Rubric atom	Dim. share (%)	Avg. wt.	Representative original dimension titles
Instruction & Edit	Instruction alignment	8.88	39.6	Instruction Alignment; Instruction Compliance; Instruction Adherence
Instruction & Edit	Object edit	7.72	36.4	Object Removal; Object Addition; Object Replacement
Instruction & Edit	Text editing	5.47	35.9	Text Accuracy; Text Rendering Accuracy; Text Placement
Instruction & Edit	Attribute edit	4.45	35.4	Color Modification; Material Transformation; Color Transformation
Preservation	Source preservation	20.90	31.4	Content Preservation; Image Preservation; Structural Preservation
Preservation	Subject preservation	5.78	33.5	Subject Preservation; Context Preservation; Foreground Preservation

Appendix

Table 9: Semantic aggregation of parsed image-editing rubric dimensions. Dim. share is computed over parsed teacher-generated rubric dimensions; avg. wt. is the average teacher-assigned dimension weight.

Panel of Figure 2 — Figure 2: Primary-category coverage of the rubric-learning data for text-to-image generation and image editing.

Model	Subject			Appearance			Scene			Advanced			Overall
	PF	C	O	PF	C	O	PF	C	O	PF	C	O	PF	C	O
Proprietary MLLMs
Claude Sonnet 4.6	83.9	64.5	82.9	86.8	69.2	76.2	93.8	74.5	88.8	88.7	63.5	83.2	87.6	67.4	81.4
GPT-5.4	86.1	56.2	79.7	82.8	62.8	73.3	91.9	72.4	82.5	79.2	62.8	74.8	83.7	62.8	76.3
Gemini 2.5 Pro	81.6	61.3	77.5	75.9	69.8	75.9	82.6	72.4	80.1	85.7	62.3	79.7	81.4	65.5	77.4
Gemini 3.1 Pro	86.5	65.9	86.3	83.0	75.1	77.8	90.1	76.4	90.8	91.7	70.3	87.5	87.7	71.6	84.1

Appendix

Table 11: Detailed results on EditScore-ERB across four task categories and overall. PF = Prompt Following, C = Consistency, O = Overall Quality. Scores are percentages. Bold and underlined scores denote the best and second-best results among reward models, respectively.

Panel of Figure 3 — Figure 3: Effect of saturated-group filtering on GRPO training stability. The columns show reward mean, entropy, and response length from left to right; the top row is without filtering and the bottom row is with filtering. Without filtering, late training shows reward-mean collapse, sharp response-length growth, and entropy drift followed by collapse. Filtering low-standard-deviation groups keeps all three signals stable.

Backbone	Supervision	Text-to-Image			Image Editing
		MMRB2	GenAI- Bench	GenAI- Bench- Verified	MMRB2	EditReward- ERB	EditScore- ERB
Qwen3.5-4B	Rubric SFT	70.1	72.0	82.9	71.5	43.6	83.3
	Label-only SFT	65.1 ( $\downarrow$ 5.0)	68.7 ( $\downarrow$ 3.3)	77.8 ( $\downarrow$ 5.1)	69.2 ( $\downarrow$ 2.3)	39.4 ( $\downarrow$ 4.2)	85.4 ( $\uparrow$ 2.1)
Qwen3.5-9B	Rubric SFT	70.3	73.0	83.2	73.8	45.0	85.1
	Label-only SFT	66.4 ( $\downarrow$ 3.9)	71.1 ( $\downarrow$ 1.9)	80.9 ( $\downarrow$ 2.3)	69.2 ( $\downarrow$ 4.6)	40.8 ( $\downarrow$ 4.2)	83.9 ( $\downarrow$ 1.2)

Appendix

Table 12: Label-only versus rubric-based SFT ablation. Scores are percentages; deltas in Label-only SFT rows are relative to Rubric SFT within each backbone.

Task	Same inst.	Same type	Diff. type	Weight diff.
Gen	0.763	0.597	0.563	0.102
Edit	0.800	0.651	0.612	0.110

Appendix

Table 13: Stability and adaptation of generated rubrics on MMRB2. “Same inst.” denotes the same instruction and “Diff. type” denotes different instruction types. The three similarity columns report dimension similarity.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 313 papers

Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation
Mar 12, 2026Xiangyu Zhao, Peiyuan Zhang, Junming Lin +7Reward Modeling RL for Image Editing
Shanghai Jiao Tong University · Wuhan University · BUPT +3
RubricRL: Simple Generalizable Rewards for Text-to-Image Generation
Nov 25, 2025Xuelu Feng, Yunsheng Li, Ziyu Wan +4Reward Modeling T2I Generation
University at Buffalo · Microsoft AI · Nikola Tesla STEM High School
Think Before You Score: Thinking Reward Model for Visual Generation
Sep 29, 2026Xuehai Bai, Zhenchen Tang, Yang Shi +9Reward Modeling Multimodal Reward Modeling
HDU · SenseTime · CASIA +5

Mar 12, 2026J/K move · Enter open · S save

Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation

Xiangyu Zhao, Peiyuan Zhang, Junming Lin, Tianhao Liang+6

Shanghai Jiao Tong University · Wuhan University · BUPT+3

Reward Modeling RL for Image Editing RL for Image Generation Image Generation Evaluation T2I Generation Evaluation

Open paper AskPDF arXiv

Reinforcement learning (RL) has emerged as a promising paradigm for enhancing image editing and text-to-image (T2I) generation. However, current reward models, which act as critics during RL, often suffer from hallucinations and assign noisy scores, inherently misguiding the optimization process. In this paper, we present FIRM (Faithful Image Reward Modeling), a comprehensive framework that develops robust reward models to provide accurate and reliable guidance for faithful image generation and editing. First, we design tailored data curation pipelines to construct high-quality scoring datasets. Specifically, we evaluate editing using both execution and consistency, while generation is primarily assessed via instruction following. Using these pipelines, we collect the FIRM-Edit-370K and FIRM-Gen-293K datasets, and train specialized reward models (FIRM-Edit-8B and FIRM-Gen-8B) that accurately reflect these criteria. Second, we introduce FIRM-Bench, a comprehensive benchmark specifically designed for editing and generation critics. Evaluations demonstrate that our models achieve superior alignment with human judgment compared to existing metrics. Furthermore, to seamlessly integrate these critics into the RL pipeline, we formulate a novel "Base-and-Bonus" reward strategy that balances competing objectives: Consistency-Modulated Execution (CME) for editing and Quality-Modulated Alignment (QMA) for generation. Empowered by this framework, our resulting models FIRM-Qwen-Edit and FIRM-SD3.5 achieve substantial performance breakthroughs. Comprehensive experiments demonstrate that FIRM mitigates hallucinations, establishing a new standard for fidelity and instruction adherence over existing general models. All of our datasets, models, and code have been publicly available at https://firm-reward.github.io.

Abstract

Reward Modeling

Explore similar work

Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation

RubricRL: Simple Generalizable Rewards for Text-to-Image Generation

Think Before You Score: Thinking Reward Model for Visual Generation