CheckDistill: Adaptive Checklist Distillation for Efficient Distributional Reward Modeling in Image Editing

Yaqi Cai; Zuan Gao; Mengting Chen; Wei Li; Bo Zheng

cs.CVOct 19, 2026

CheckDistill: Adaptive Checklist Distillation for Efficient Distributional Reward Modeling in Image Editing

Authors: Yaqi Cai, Zuan Gao, Mengting Chen, Wei Li, Bo Zheng

Organizations: Taobao & Tmall group of Alibaba

Abstract

Reinforcement learning for instruction-guided image editing requires reward models that provide reliable, accurate, and efficient feedback. Direct scalar prediction compresses multidimensional editing preferences, whereas generative evaluators can expose their judgment through explicit reasoning. However, broad scoring criteria do not specify all instance-dependent editing and preservation constraints, and generating long reasoning trajectories is costly for online optimization. We introduce CheckDistill, a pointwise reward modeling framework that internalizes instruction-adaptive checklists. We decompose each editing condition into atomic, independently verifiable requirements and construct four-dimensional scoring labels together with structured verification evidence. After supervised score initialization, a privileged teacher conditions on checklists, item-level judgments, and dimension-level rationales to supervise the student's own scoring trajectories. A reliability-weighted on-policy self-distillation objective adjusts supervision at score tokens according to agreement between the teacher's expected scores and reference labels, while retaining candidate probability weights in a truncated token support. The deployed student directly predicts four score distributions without checklist inputs or explicit reasoning and aggregates their expectations into a scalar reward. CheckDistill achieves aggregate accuracies of 72.30%, 56.67% and 82.90% on MMRB2, MER-Bench and EditReward-Bench, respectively, exceeding the compared open-source baselines on all three metrics.

Figures & tables

Figure 1: Overview of CheckDistill compared with existing reward model architecture.

Figure 2: An example of checklist generation and checklist-based evaluation for image editing.

Figure 3: Overview of CheckDistill , comprising reward data construction, two-stage reward model training via supervised fine-tuning and checklist-guided reliability-aware OPSD, and reward feedback alignment for image editing.

	MMRB2			MER-Bench				EditReward-Bench
Model	Single	Multi	Overall	2 pair	3 pair	4 pair	Overall	PF	CS	Overall
GPT-4.1	56.33	57.87	56.60	79.50	42.50	18.50	46.83	70.00	65.20	73.40
Gemini-3.1-Pro	65.20	60.11	64.29	79.00	60.50	24.00	54.50	66.30	75.70	79.80
Gemini-3.0-Flash	66.71	58.64	65.34	82.50	63.00	26.00	57.17	68.70	77.90	80.90
Qwen3-VL-8B-Instruct	63.02	57.87	62.10	73.00	42.00	16.00	43.67	77.20	70.90	75.80
Qwen3.5-4B	57.79	54.49	57.20	75.50	36.00	15.50	42.33	74.40	69.60	70.50

Table 1: Reward benchmark accuracy (%,

\uparrow

). Submetric results for EditReward, EditScore-8B, and SpatialReward are reproduced from SpatialReward ( Long et al., 2026 ) . Bold marks the best result among the compared open-source models in each column.

Model	ImgEdit Bench	REDEdit Bench	GEdit-EN
Qwen-Image-Edit	4.29	4.10	8.18
UniWorld-Qwen-Image-Edit	4.30	4.06	8.12
+ DiffusionNFT (CheckDistill-9B)	4.46	4.26	8.28

Table 2: Image-editing performance (

\uparrow

Figure 4: Qualitative Comparison Before and After CheckDistill-based Reward Training.

Figure 5: Input, output, and hybrid scaling on MMRB2 (Qwen3.5-9B). Labels indicate additional input context

\rightarrow

generated output. C: checklist; J: checklist-guided judgments; R: rationale; S: score. “None” denotes no additional input context.

Privileged Information	MMRB2	MER- Bench	EditReward- Bench
Checklist $C$	72.10	54.83	82.10
+ Checklist Judgment $J$	72.00	56.00	82.20
+ Dimension-wise Rationale $A$	72.30	56.67	82.90
+ Dimension Scores $\mathbf{g}$	70.10	55.83	81.30

Table 3: Ablation studies on privileged information and CR-OPSD design choices (%,

\uparrow

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Optimizer	AdamW ( $\beta_{1}=0.9$ , $\beta_{2}=0.999$ )
Learning Rate	$1\times 10^{-5}$
Learning Rate Scheduler	Linear
Warmup Ratio	0.1
Per-Device Batch Size	1
Gradient Accumulation Steps	4

Appendix

Table 4: Hyperparameters for Stage-I supervised fine-tuning.

Hyperparameter	Value
Global Batch Size	128
Mini-Batch Size	128
Rollouts per Prompt	4
Log-Probability Top- $K$	10
Maximum Prompt Length	32,768
Maximum Response Length	128

Appendix

Table 5: Hyperparameters for Stage-II CR-OPSD.

Hyperparameter	Value
Guidance Scale	4.0
Training Sampling Steps	16
Training Sampler	DPM2
Evaluation Sampling Steps	40
Noise Level	0.7
Learning Rate	$3\times 10^{-4}$

Appendix

Table 6: Hyperparameters for DiffusionNFT training.

Hyperparameter	Value
Training Temperature	1.0
Global Batch Size	128
Mini-Batch Size	128
Rollouts per Prompt	8
Maximum Prompt Length	16,384
Maximum Response Length	128

Appendix

Table 7: Hyperparameters for GRPO and Token-level GRPO.

	MMRB2			MER-Bench				EditReward-Bench
Privileged Information	Single	Multi	Overall	2 pair	3 pair	4 pair	Overall	PF	CS	Overall
Checklist $C$	72.99	67.98	72.10	81.00	56.50	27.00	54.83	86.60	84.20	82.10
+ Checklist Judgment $(C,J)$	73.24	66.29	72.00	81.50	59.50	27.00	56.00	84.60	85.10	82.20
+ Dimension-wise Rationale $(C,J,A)$	73.36	67.42	72.30	82.00	60.50	27.50	56.67	85.50	84.10	82.90
+ Dimension Scores $(C,J,A,\mathbf{g})$	71.17	65.17	70.10	82.00	57.50	28.00	55.83	84.90	83.90	81.30

Appendix

Table 8: Privileged-information ablation (%,

\uparrow

). Bold marks column maxima.

	MMRB2			MER-Bench				EditReward-Bench
Variant	Single	Multi	Overall	2 pair	3 pair	4 pair	Overall	PF	CS	Overall
w/o Reliability Weighting	71.41	63.48	70.00	81.50	57.50	26.00	55.00	84.70	85.10	82.10
Linear Reliability Mapping	72.51	65.73	71.30	81.50	59.50	27.50	56.17	85.00	84.70	82.30
Argmax-based Reliability	71.90	64.61	70.60	82.50	57.50	26.50	55.50	84.60	84.20	82.60
CR-OPSD (Ours)	73.36	67.42	72.30	82.00	60.50	27.50	56.67	85.50	84.10	82.90

Appendix

Table 9: Ablation study of the design choices in CR-OPSD (%,

\uparrow

). All variants use the same privileged information

(C,J,A)

. Bold indicates the best result in each column.

	ImgEdit-Bench										GEdit-Bench-EN
Model	Add	Adjust	Extract	Replace	Remove	Background	Style	Compose	Action	Overall	G_SC	G_PQ	G_O
Qwen-Image-Edit	4.40	4.30	3.67	4.64	4.34	4.30	4.83	3.39	4.71	4.287	8.855	7.949	8.181
UniWorld-Qwen-Image-Edit	4.42	4.31	3.71	4.69	4.32	4.28	4.83	3.42	4.68	4.296	8.824	7.884	8.119
+ DiffusionNFT (CheckDistill-9B)	4.47	4.68	4.52	4.65	4.41	4.40	4.89	3.60	4.51	4.459	8.909	8.039	8.280

Appendix

Table 10: Results on ImgEdit-Bench and GEdit-Bench-EN.

	RED-Bench
Model	Add	Adjust	Background	Beauty	Color	Compose	Extract	Low-level	Motion	Portrait	Remove	Replace	Stylize	Text	Viewpoint	Overall
Qwen-Image-Edit	4.54	4.08	4.31	3.14	4.06	3.85	3.04	4.47	4.68	4.37	4.21	4.59	4.68	3.94	3.55	4.10
UniWorld-Qwen-Image-Edit	4.48	4.07	4.40	3.19	4.10	3.84	2.87	4.28	4.63	4.35	4.25	4.56	4.72	3.83	3.40	4.06
+ DiffusionNFT (CheckDistill-9B)	4.60	4.25	4.42	3.37	4.38	3.99	4.07	4.59	4.55	4.10	4.39	4.56	4.73	4.15	3.73	4.26

Appendix

Table 11: Results on RED-Bench.

Figure 6: Example of Checklist-Guided Judgment by CheckDistill

Figure 7: Example of Checklist-Guided Judgment by CheckDistill

Figure 8: Example of Checklist-Guided Judgment by CheckDistill

Figure 9: Training reward for Qwen-Image-Edit using CheckDistill as the reward model.

Figure 10: Qualitative Comparison Before and After CheckDistill-based Reward Training.

Figure 11: Qualitative Comparison Before and After CheckDistill-based Reward Training.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 325 papers

RewardHarness: Self-Evolving Agentic Post-Training
May 9, 2026Yuxuan Zhang, Cong Wei, Penghui Du +11Reward Modeling Image Editing Evaluation
University of British Columbia · Vector Institute · Kolors Team, Kuaishou Technology +5
Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling
May 13, 2026Xuehai Bai, Yang Shi, Yi-Fan Zhang +7Reward Modeling Image Editing Evaluation
HDU · PKU · Kling Team +1
Leveraging Verifier-Based Reinforcement Learning in Image Editing
Apr 30, 2026Hanzhong Guo, Jie Wu, Jie Liu +6Reward Modeling Reinforcement Learning
School of Computing and Data Science, The University of Hong Kong · ByteDance Seed · CUHK +1

May 9, 2026J/K move · Enter open · S save

RewardHarness: Self-Evolving Agentic Post-Training

Yuxuan Zhang, Cong Wei, Penghui Du, Bo Li+10

University of British Columbia · Vector Institute · Kolors Team, Kuaishou Technology+5

Reward Modeling Image Editing Evaluation Multimodal Reward Modeling Image Generation Evaluation

Open paper AskPDF arXiv

Aligning generative models with human preferences remains a central challenge. While RLHF is widely adopted, it relies on large-scale preference data. This limitation is particularly evident in evaluating instruction-guided image edits, even though humans can often learn such preferences from only a few examples. We present RewardHarness, a training-free, self-evolving agentic reward framework that reframes reward modeling as context evolution rather than weight optimization. Instead of learning from large-scale annotations, RewardHarness aligns with human preferences by iteratively evolving a library of tools and skills from as few as 100 preference demonstrations. Given a source image, an edited image, and an editing instruction, an orchestrator agent selects the most relevant subset of tools and skills from the maintained library, and sub-agents invoke them to construct a reasoning chain that produces a reward score. By comparing predicted scores with ground-truth preferences and analyzing successes and failures in the reasoning process, the orchestrator automatically refines its library of tools and skills without additional human annotation. Using only 0.05% of the EditReward preference data, RewardHarness achieves 47.4% average accuracy on EditReward-Bench K=2/3/4 and GenAI-Bench, surpassing GPT-5 by 5.3 points. When used as a reward signal in FlowGRPO, RL-tuned models achieve 3.52 on ImgEdit-Bench. Code and model are available at the project page: https://rewardharness.com/

Abstract

Reward Modeling

Explore similar work

RewardHarness: Self-Evolving Agentic Post-Training

Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling

Leveraging Verifier-Based Reinforcement Learning in Image Editing