cs.CVNov 12, 2026

Rethinking Visual Reasoning in Text-to-Image Reward Modeling

Authors: Shiye Su, Xiaohan Wang, Ludwig Schmidt, Serena Yeung-Levy

Organizations: Stanford University

Abstract

Reward models are a core component of modern text-to-image systems, serving as the objective for inference-time selection and RL-based post-training. A key prerequisite is visual understanding: to judge prompt adherence and ground aesthetic preferences, a reward model must bind attributes, resolve relations, and count reliably. We find that widely-used reward models perform poorly on targeted visual reasoning evaluations. Even when starting from strong vision-language backbones, preference fine-tuning erodes these capabilities, and continual-learning baselines fail to mitigate a steep tradeoff between visual reasoning and preference accuracy. We propose a data-centric intervention that injects visual reasoning supervision into preference learning by flipping the pairing axis. To complement conventional human preference data (paired images under a shared prompt), we convert existing visual reasoning datasets into preference pairs that share the same image but differ in prompt, and train on the mixture. This yields a Pareto improvement in both visual understanding and preference prediction. Across inference-time best-of-n selection and RL post-training, the resulting reward model improves text-to-image generation on challenging compositional prompts (GenEval) by an average of 4.0% and 5.5% respectively, outperforming HPSv3 which was trained on 6x more preference pairs.

Figures & tables

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards

    Oct 2, 2026Yuanhao Ban, I-Hung Hsu, Anastasios Angelopoulos +3T2I GenerationReward Design for RL

  2. RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time

    Apr 13, 2026Haozhe Wang, Cong Wei, Weiming Ren +3Reward ModelingMultimodal Reward Modeling

  3. Z-Reward: Beyond Scalar Rewards by Internalizing Reasoning into Score Distributions

    Jun 8, 2026Xin Jin, Huanqia Cai, Zhen Li +9Reward ModelingReasoning Distillation