Reward models are a core component of modern text-to-image systems, serving as the objective for inference-time selection and RL-based post-training. A key prerequisite is visual understanding: to judge prompt adherence and ground aesthetic preferences, a reward model must bind attributes, resolve relations, and count reliably. We find that widely-used reward models perform poorly on targeted visual reasoning evaluations. Even when starting from strong vision-language backbones, preference fine-tuning erodes these capabilities, and continual-learning baselines fail to mitigate a steep tradeoff between visual reasoning and preference accuracy. We propose a data-centric intervention that injects visual reasoning supervision into preference learning by flipping the pairing axis. To complement conventional human preference data (paired images under a shared prompt), we convert existing visual reasoning datasets into preference pairs that share the same image but differ in prompt, and train on the mixture. This yields a Pareto improvement in both visual understanding and preference prediction. Across inference-time best-of-n selection and RL post-training, the resulting reward model improves text-to-image generation on challenging compositional prompts (GenEval) by an average of 4.0% and 5.5% respectively, outperforming HPSv3 which was trained on 6x more preference pairs.
Figures & tables
Figure 1: (Left) Existing reward models exhibit poor visual understanding. VLM backbones possess strong visual reasoning, but this is eroded during preference fine-tuning, even with robust fine-tuning techniques. (Center) We introduce CrossPairs – preference pairs sharing a common image but differing in prompt – to isolate text-image alignment signal by eliminating aesthetic confounds. Mixed with standard pairs during training, our method achieves a Pareto improvement. (Right) The resulting reward model guides generations with superior compositional accuracy.
ImageRDB
Visual
Visual Understanding Breakdown
Wground
SC-Add
SC-Rep
SC-Swap
CV-Rel
WhatsUp
CV-Count
CountBch
CLIP
57.1
68.2
55.4
89.9
85.4
71.9
53.5
48.9
57.8
82.6
ImageReward
65.1
68.5
56.8
96.1
88.4
83.9
51.8
52.0
55.6
63.4
PickScore
61.6
63.8
54.6
81.6
80.6
64.9
51.7
50.6
52.8
73.8
HPSv2
65.7
66.3
54.2
91.9
81.9
67.0
48.9
50.0
56.2
79.9
HPSv3
66.8
76.5
66.9
79.9
78.2
72.1
79.1
85.9
62.7
87.2
Table 1: Accuracy of widely-used reward models on ImageRewardDB and our visual understanding evaluations. Random performance is 50%. Bottom: HPSv3 exhibits substantially degraded visual understanding relative to the base VLM it was fine-tuned from.
Figure 2: Visual understanding ( y ) vs preference accuracy ( x ) for multiple checkpoints of the reward model during conservative fine-tuning on preference pairs. Conservative fine-tuning only weakly mitigates visual understanding degradation.
Figure 3: Preference accuracy (left) and visual understanding accuracy (right) as a function of the interpolation ratio λ for weight merging. Merging does not improve performance over the base ( λ=0 ) or fully fine-tuned ( λ=1 ) model.
Figure 4: Visual understanding ( y ) vs preference accuracy ( x ) for multiple checkpoints of the reward model trained with uncertainty calibration (orange) and with the standard reward readout head (blue). Uncertainty calibrating the base VLM better retains its visual reasoning, but underperforms on preference prediction.
Dataset
Pairs
% Mix
CLEVR
8750
5.6
SpatialSense
9712
6.2
TallyQA
10318
6.6
ColorSwap
2000
1.3
VSR
6664
4.2
ImageRewardDB
119765
76.2
Table 2: Final data mix for reward model training, comprising ImageRewardDB and multimodal learning VQA converted to CrossPairs preference pairs.
Figure 5: Our reward model improves both inference-time scaling (left: single-step SD-Turbo and FLUX.1-Schnell) and post-RL performance (right: SD 3.5 Medium and SANA) on GenEval.
Figure 6: Generated samples after post-training with different reward models. The first three columns are prompts from Pick-a-Pic; the last three columns are prompts from GenEval.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
ImageRDB
Visual
Visual Understanding Breakdown
Wground
SC-Add
SC-Rep
SC-Swap
CV-Rel
WhatsUp
CV-Count
CountBch
CLIP
57.1
68.2
55.4
89.9
85.4
71.9
53.5
48.9
57.8
82.6
ImageReward
65.1
68.5
56.8
96.1
88.4
83.9
51.8
52.0
55.6
63.4
PickScore
61.6
63.8
54.6
81.6
80.6
64.9
51.7
50.6
52.8
73.8
HPSv2
65.7
66.3
54.2
91.9
81.9
67.0
48.9
50.0
56.2
79.9
HPSv3
66.8
76.5
66.9
79.9
78.2
72.1
79.1
85.9
62.7
87.2
Appendix
Table 3: Accuracy of reward models on ImageRewardDB and visual understanding evaluations. Random performance is 50%. † Trained only as a pairwise model; evaluated here by prompting for a pointwise rating (see text).
Figure 7: Further generators.
Figure 8: Further evaluations on T2ICompBench.
Overall
Object
Attribute
Counting
Spatial
Action
Ours w/o CrossPairs
62.4
61.0
62.0
66.1
67.1
61.5
Ours
68.0
65.7
66.4
75.0
75.0
68.4
Appendix
Table 4: Accuracy (%) on the alignment split of MJ-Bench.
Figure 9: Ablating out CrossPairs, evaluated best-of- n .
Figure 10: Ablating out CrossPairs, evaluated by RL post-training.
ImageRewardDB (ID)
Pick-a-Pic (OOD)
ImageReward
65.1
60.9
Ours w/o CrossPairs
66.3
63.2
Ours
66.3
63.6
Appendix
Table 5: Held out preference accuracy.
SD 3.5 Medium
SANA
ImageReward
6.46
6.48
Ours w/o CrossPairs
6.42
6.50
Ours
6.42
6.41
Appendix
Table 6: Aesthetic scores on generations from reinforcement post-trained generators.
Figure 11: Example generations. The first three columns are prompts from PickScore; the last three columns are prompts from GenEval.
Figure 12: Example generations. The first three columns are prompts from PickScore; the last three columns are prompts from GenEval.
ImageRDB
Visual
Visual Understanding Breakdown
Wground
SC-Add
SC-Rep
SC-Swap
CV-Rel
WhatsUp
CV-Count
CountBch
VQA auxiliary loss
65.2
82.2
66.6
97.6
90.7
89.9
77.1
80.8
67.7
87.2
Ours
66.3
85.3
71.6
97.5
92.8
91.9
74.5
92.2
71.7
90.2
Appendix
Table 7: CrossPairs vs. supervising the same visual reasoning sources with an auxiliary VQA next-token-prediction loss. Accuracy on ImageRewardDB and the visual understanding evaluations; random performance is 50%.
Generator
Method
n=4
n=8
n=16
n=30
n=40
n=50
SD-Turbo
VQA auxiliary loss
55.4
58.5
60.8
62.4
63.0
63.4
Ours
55.4
58.5
61.0
63.0
63.8
64.5
FLUX.1-Schnell
VQA auxiliary loss
71.9
73.8
75.0
75.9
76.4
76.9
Ours
71.8
73.7
75.2
76.6
77.3
78.1
Appendix
Table 8: Best-of- n GenEval accuracy (%) for the two supervision formats. We generate 50 images per prompt; each entry is averaged over 200 random n -subsets of those 50 generations.
Method
n=4
n=8
n=16
n=30
n=40
n=50
CLIP
54.0 ± 1.3
56.1 ± 1.2
57.4 ± 1.2
57.8 ± 1.0
57.8 ± 0.8
57.3 ± 0.2
ImageReward
54.9 ± 1.3
57.4 ± 1.2
59.5 ± 1.2
60.6 ± 0.9
60.9 ± 0.6
61.2 ± 0.0
PickScore
53.0 ± 1.2
55.0 ± 1.2
56.7 ± 1.2
58.3 ± 0.9
59.1 ± 0.7
60.0 ± 0.0
HPSv2
52.9 ± 1.2
54.6 ± 1.1
56.0 ± 1.0
56.8 ± 0.9
57.4 ± 0.8
57.9 ± 0.0
HPSv3
53.7 ± 1.2
56.2 ± 1.2
58.4 ± 1.1
60.6 ± 0.9
61.5 ± 0.7
62.4 ± 0.0
VQAScore
53.8 ± 1.4
56.3 ± 1.3
58.3 ± 1.2
59.9 ± 0.9
60.6 ± 0.8
61.1 ± 0.0
Appendix
Table 9: Best-of- n GenEval accuracy (%, mean ± std over 200 resampled n -subsets) with SD-Turbo as the generator.
Method
n=4
n=8
n=16
n=30
n=40
n=50
CLIP
70.8 ± 1.1
72.2 ± 1.2
73.4 ± 1.1
74.4 ± 0.9
74.9 ± 0.6
75.7 ± 0.0
ImageReward
71.1 ± 1.1
72.7 ± 1.1
73.8 ± 1.0
74.7 ± 0.9
74.9 ± 0.7
74.9 ± 0.0
PickScore
70.1 ± 1.1
71.0 ± 1.1
71.5 ± 1.0
71.7 ± 0.8
71.7 ± 0.6
71.6 ± 0.0
HPSv2
70.0 ± 1.1
71.1 ± 1.1
71.7 ± 1.0
71.9 ± 0.8
71.9 ± 0.6
71.8 ± 0.0
HPSv3
71.2 ± 1.1
72.9 ± 1.1
74.2 ± 1.0
75.1 ± 0.8
75.4 ± 0.6
75.5 ± 0.0
VQAScore
70.2 ± 1.2
71.7 ± 1.2
73.1 ± 1.1
74.3 ± 1.0
74.8 ± 0.8
75.3 ± 0.6
Appendix
Table 10: Best-of- n GenEval accuracy (%, mean ± std over 200 resampled n -subsets) with FLUX.1-Schnell as the generator.
ImageRDB
Visual
Visual Understanding Breakdown
Wground
SC-Add
SC-Rep
SC-Swap
CV-Rel
WhatsUp
CV-Count
CountBch
SmolVLM2-2.2B (base)
55.7
76.1
69.7
93.1
85.5
83.3
69.0
76.4
62.6
69.7
w/o CrossPairs
58.4
69.4
62.8
89.4
86.8
71.4
55.5
59.8
59.3
70.5
Ours
58.8
73.6
65.1
91.3
90.1
76.5
62.9
66.7
65.5
70.3
Appendix
Table 11: SmolVLM2-2.2B as the reward model backbone. Accuracy on ImageRewardDB and the visual understanding evaluations; random performance is 50%.
Generator
Method
n=4
n=8
n=16
n=30
n=40
n=50
SD-Turbo
w/o CrossPairs
53.9 ± 1.3
55.7 ± 1.2
57.1 ± 1.1
58.2 ± 0.9
58.3 ± 0.7
57.9 ± 0.0
Ours
54.2 ± 1.3
56.1 ± 1.0
57.5 ± 1.1
58.8 ± 1.0
59.2 ± 0.7
59.3 ± 0.0
FLUX.1-Schnell
w/o CrossPairs
69.7 ± 1.1
70.8 ± 1.1
71.4 ± 1.1
71.6 ± 0.9
71.8 ± 0.7
72.1 ± 0.0
Ours
70.1 ± 1.1
71.5 ± 1.1
72.5 ± 1.0
73.4 ± 0.9
73.6 ± 0.7
73.7 ± 0.0
Appendix
Table 12: Best-of- n GenEval accuracy (%, mean ± std over 200 resampled n -subsets; see Appendix E ) with SmolVLM2-2.2B-based reward models.