Rethinking Visual Reasoning in Text-to-Image Reward Modeling
Organizations: Stanford University
Abstract
Reward models are a core component of modern text-to-image systems, serving as the objective for inference-time selection and RL-based post-training. A key prerequisite is visual understanding: to judge prompt adherence and ground aesthetic preferences, a reward model must bind attributes, resolve relations, and count reliably. We find that widely-used reward models perform poorly on targeted visual reasoning evaluations. Even when starting from strong vision-language backbones, preference fine-tuning erodes these capabilities, and continual-learning baselines fail to mitigate a steep tradeoff between visual reasoning and preference accuracy. We propose a data-centric intervention that injects visual reasoning supervision into preference learning by flipping the pairing axis. To complement conventional human preference data (paired images under a shared prompt), we convert existing visual reasoning datasets into preference pairs that share the same image but differ in prompt, and train on the mixture. This yields a Pareto improvement in both visual understanding and preference prediction. Across inference-time best-of-n selection and RL post-training, the resulting reward model improves text-to-image generation on challenging compositional prompts (GenEval) by an average of 4.0% and 5.5% respectively, outperforming HPSv3 which was trained on 6x more preference pairs.
Figures & tables
| ImageRDB | Visual | Visual Understanding Breakdown | ||||||||
| Wground | SC-Add | SC-Rep | SC-Swap | CV-Rel | WhatsUp | CV-Count | CountBch | |||
| CLIP | 57.1 | 68.2 | 55.4 | 89.9 | 85.4 | 71.9 | 53.5 | 48.9 | 57.8 | 82.6 |
| ImageReward | 65.1 | 68.5 | 56.8 | 96.1 | 88.4 | 83.9 | 51.8 | 52.0 | 55.6 | 63.4 |
| PickScore | 61.6 | 63.8 | 54.6 | 81.6 | 80.6 | 64.9 | 51.7 | 50.6 | 52.8 | 73.8 |
| HPSv2 | 65.7 | 66.3 | 54.2 | 91.9 | 81.9 | 67.0 | 48.9 | 50.0 | 56.2 | 79.9 |
| HPSv3 | 66.8 | 76.5 | 66.9 | 79.9 | 78.2 | 72.1 | 79.1 | 85.9 | 62.7 | 87.2 |
| Dataset | Pairs | % Mix |
|---|---|---|
| CLEVR | 8750 | 5.6 |
| SpatialSense | 9712 | 6.2 |
| TallyQA | 10318 | 6.6 |
| ColorSwap | 2000 | 1.3 |
| VSR | 6664 | 4.2 |
| ImageRewardDB | 119765 | 76.2 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| ImageRDB | Visual | Visual Understanding Breakdown | ||||||||
| Wground | SC-Add | SC-Rep | SC-Swap | CV-Rel | WhatsUp | CV-Count | CountBch | |||
| CLIP | 57.1 | 68.2 | 55.4 | 89.9 | 85.4 | 71.9 | 53.5 | 48.9 | 57.8 | 82.6 |
| ImageReward | 65.1 | 68.5 | 56.8 | 96.1 | 88.4 | 83.9 | 51.8 | 52.0 | 55.6 | 63.4 |
| PickScore | 61.6 | 63.8 | 54.6 | 81.6 | 80.6 | 64.9 | 51.7 | 50.6 | 52.8 | 73.8 |
| HPSv2 | 65.7 | 66.3 | 54.2 | 91.9 | 81.9 | 67.0 | 48.9 | 50.0 | 56.2 | 79.9 |
| HPSv3 | 66.8 | 76.5 | 66.9 | 79.9 | 78.2 | 72.1 | 79.1 | 85.9 | 62.7 | 87.2 |
| Overall | Object | Attribute | Counting | Spatial | Action | |
|---|---|---|---|---|---|---|
| Ours w/o CrossPairs | 62.4 | 61.0 | 62.0 | 66.1 | 67.1 | 61.5 |
| Ours | 68.0 | 65.7 | 66.4 | 75.0 | 75.0 | 68.4 |
| ImageRewardDB (ID) | Pick-a-Pic (OOD) | |
|---|---|---|
| ImageReward | 65.1 | 60.9 |
| Ours w/o CrossPairs | 66.3 | 63.2 |
| Ours | 66.3 | 63.6 |
| SD 3.5 Medium | SANA | |
|---|---|---|
| ImageReward | 6.46 | 6.48 |
| Ours w/o CrossPairs | 6.42 | 6.50 |
| Ours | 6.42 | 6.41 |
| ImageRDB | Visual | Visual Understanding Breakdown | ||||||||
| Wground | SC-Add | SC-Rep | SC-Swap | CV-Rel | WhatsUp | CV-Count | CountBch | |||
| VQA auxiliary loss | 65.2 | 82.2 | 66.6 | 97.6 | 90.7 | 89.9 | 77.1 | 80.8 | 67.7 | 87.2 |
| Ours | 66.3 | 85.3 | 71.6 | 97.5 | 92.8 | 91.9 | 74.5 | 92.2 | 71.7 | 90.2 |
| Generator | Method | ||||||
|---|---|---|---|---|---|---|---|
| SD-Turbo | VQA auxiliary loss | 55.4 | 58.5 | 60.8 | 62.4 | 63.0 | 63.4 |
| Ours | 55.4 | 58.5 | 61.0 | 63.0 | 63.8 | 64.5 | |
| FLUX.1-Schnell | VQA auxiliary loss | 71.9 | 73.8 | 75.0 | 75.9 | 76.4 | 76.9 |
| Ours | 71.8 | 73.7 | 75.2 | 76.6 | 77.3 | 78.1 |
| Method | ||||||
|---|---|---|---|---|---|---|
| CLIP | 54.0 1.3 | 56.1 1.2 | 57.4 1.2 | 57.8 1.0 | 57.8 0.8 | 57.3 0.2 |
| ImageReward | 54.9 1.3 | 57.4 1.2 | 59.5 1.2 | 60.6 0.9 | 60.9 0.6 | 61.2 0.0 |
| PickScore | 53.0 1.2 | 55.0 1.2 | 56.7 1.2 | 58.3 0.9 | 59.1 0.7 | 60.0 0.0 |
| HPSv2 | 52.9 1.2 | 54.6 1.1 | 56.0 1.0 | 56.8 0.9 | 57.4 0.8 | 57.9 0.0 |
| HPSv3 | 53.7 1.2 | 56.2 1.2 | 58.4 1.1 | 60.6 0.9 | 61.5 0.7 | 62.4 0.0 |
| VQAScore | 53.8 1.4 | 56.3 1.3 | 58.3 1.2 | 59.9 0.9 | 60.6 0.8 | 61.1 0.0 |
| Method | ||||||
|---|---|---|---|---|---|---|
| CLIP | 70.8 1.1 | 72.2 1.2 | 73.4 1.1 | 74.4 0.9 | 74.9 0.6 | 75.7 0.0 |
| ImageReward | 71.1 1.1 | 72.7 1.1 | 73.8 1.0 | 74.7 0.9 | 74.9 0.7 | 74.9 0.0 |
| PickScore | 70.1 1.1 | 71.0 1.1 | 71.5 1.0 | 71.7 0.8 | 71.7 0.6 | 71.6 0.0 |
| HPSv2 | 70.0 1.1 | 71.1 1.1 | 71.7 1.0 | 71.9 0.8 | 71.9 0.6 | 71.8 0.0 |
| HPSv3 | 71.2 1.1 | 72.9 1.1 | 74.2 1.0 | 75.1 0.8 | 75.4 0.6 | 75.5 0.0 |
| VQAScore | 70.2 1.2 | 71.7 1.2 | 73.1 1.1 | 74.3 1.0 | 74.8 0.8 | 75.3 0.6 |
| ImageRDB | Visual | Visual Understanding Breakdown | ||||||||
| Wground | SC-Add | SC-Rep | SC-Swap | CV-Rel | WhatsUp | CV-Count | CountBch | |||
| SmolVLM2-2.2B (base) | 55.7 | 76.1 | 69.7 | 93.1 | 85.5 | 83.3 | 69.0 | 76.4 | 62.6 | 69.7 |
| w/o CrossPairs | 58.4 | 69.4 | 62.8 | 89.4 | 86.8 | 71.4 | 55.5 | 59.8 | 59.3 | 70.5 |
| Ours | 58.8 | 73.6 | 65.1 | 91.3 | 90.1 | 76.5 | 62.9 | 66.7 | 65.5 | 70.3 |
| Generator | Method | ||||||
|---|---|---|---|---|---|---|---|
| SD-Turbo | w/o CrossPairs | 53.9 1.3 | 55.7 1.2 | 57.1 1.1 | 58.2 0.9 | 58.3 0.7 | 57.9 0.0 |
| Ours | 54.2 1.3 | 56.1 1.0 | 57.5 1.1 | 58.8 1.0 | 59.2 0.7 | 59.3 0.0 | |
| FLUX.1-Schnell | w/o CrossPairs | 69.7 1.1 | 70.8 1.1 | 71.4 1.1 | 71.6 0.9 | 71.8 0.7 | 72.1 0.0 |
| Ours | 70.1 1.1 | 71.5 1.1 | 72.5 1.0 | 73.4 0.9 | 73.6 0.7 | 73.7 0.0 |
Reward Modeling
Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.
See all 339 papers