Reward models (RMs) are inherently non-neutral value functions designed and trained to encode specific objectives, such as human preferences or text-image alignment. RMs have become crucial components of text-to-image (T2I) generation systems where they are used at various stages for dataset filtering, as evaluation metrics, as a supervisory signal during optimization of parameters, and for post-generation safety and quality filtering of T2I outputs. While specific problems with the integration of RMs into the T2I pipeline have been studied (e.g. reward hacking or mode collapse), their robustness and fairness as scoring functions remains largely unknown. We conduct a large scale audit of RM robustness with respect to demographic biases during T2I model training and generation. We provide quantitative and qualitative evidence that while originally developed as quality measures, RMs encode demographic biases, which cause reward-guided optimization to disproportionately sexualize female image subjects reinforce gender/racial stereotypes, and collapse demographic diversity. These findings highlight shortcomings in current reward models, challenge their reliability as quality metrics, and underscore the need for improved data collection and training procedures to enable more robust scoring.
Figures & tables
Figure 1: Reward models, e.g. PickScore [ 33 ] , ImageReward [ 60 ] , and HPS [ 58 ] are trained to estimate overall image quality. When used to optimize the initial noise vector ϵ0 , they can exhibit unintended effects including hypersexualization and change of large structural aspects such as perceived demographics of subjects. We study the such unintended effects of using reward models in the T2I pipeline. Blurred region: feminine-presenting image subject wearing a gray bra and underwear. *See footnote 1 .
Figure 2: Noise optimization with PickScore [ 33 ] results in hypersexualization and higher rates of NSFW content, disproportionately affecting female subjects. See Fig. 3 for quantitative results.
Figure 3: Mean change in NSFW rate ( Δnsfw , top) and skin exposure ( Δskin , bottom) induced by reward optimization, stratified by reward model (x-axis), gender (color: red = female, blue = male), and base model (fill pattern). Positive values indicate that optimization increased the corresponding hypersexualization signal.
Figure 4: Demographic transition heatmaps showing how reward model optimization shifts perceived race and gender. Each cell shows the percentage of images initially classified as a given demographic (row) that are classified as another demographic (column) after optimization, averaged across base models (SDXL-Turbo, PixArt- α , SD-Turbo).
Figure 5: Example transitions across reward models. λ=+1 is reward maximization; λ=−1 is reward minimization.
Figure 6: For each RM and dataset combination, we plot the effect size ( β2=E[s∣woman]−E[s∣man] ) of each occupation prompt against real-world labor statistics [ 47 ] . The vertical dashed line marks the baseline female employment rate of 47.1%. Bottom-left and top-right quadrants indicate occupations where RM score differences align with real-world gender distributions.
Figure 7: Average reward ranking per race group across counterfactual image sets, for each reward model.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Object
Verb Phrase
Object
Verb Phrase
bus
on a
surfboard
next to a
orange
eating an
banana
eating a
microwave
using a
skateboard
standing on a
remote
holding a
bike
riding a
wineglass
holding a
horse
riding a
sheep
next to a
snow
laying on the
Appendix
Table 1: Object–verb pairs used in prompt construction. Each row shows the object and the corresponding verb phrase prepended to it in the prompt.
Occupation
Context Phrase
chef
cooking in {} kitchen
cook
cooking in {} kitchen
firefighter
standing in front of {} firetruck
therapist
sitting on {} desk in the office
CEO
sitting in {} office
housekeeper
cleaning
Appendix
Table 2: Occupation–context pairs. Each occupation is paired with a contextual action phrase. {} denotes the position of a gendered possessive pronoun ( her / his ).
Figure 8: For each combination of reward model (PickScore, ImageReward, HPS, VQAScore, CLIP) and dataset (CausalFace, SocialCounterFactuals), we visualize the top 30 prompt values with the greatest score differences between gender demographics.
Figure 9: For each combination of reward model (PickScore, ImageReward, HPS, VQAScore, CLIP) and dataset (CausalFace, SocialCounterFactuals), we visualize the top 30 prompt values with the greatest score differences between racial demographics.
Figure 10: For each combination of reward model (PickScore, ImageReward, HPS, VQAScore, CLIP) and dataset (CausalFace, SocialCounterFactuals), we visualize the top 30 prompt values with the greatest score differences where there is an interaction between race and gender demographics.
Table 3: Adjectives per Dimension. We have further consolidated the categorization from [ 26 ] into four categories. The Stereotype Content Model (SCM) and the ABC model specify a set of psychometrically validated adjectives for each dimension.
Figure 11: Sample counterfactual sets from three datasets used for the analysis in Sec. 3.3 .
Figure 12: Demographic transition heatmaps showing how reward model optimization shifts perceived race and gender. Each cell shows the percentage of images initially classified as a given demographic (row) that are classified as another demographic (column) after optimization, averaged across base models (SDXL-Turbo, PixArt- α , SD-Turbo).
Recent text-to-image generation models have achieved remarkable visual quality, but improving them through post-training remains challenging because no single reward signal captures the full range of human preference. In this work, we develop a simple and effective post-training recipe for open-domain text-to-image generation based on the composition of complementary reward signals. Our reward system consists of two main components: a preference reward, trained on large-scale human preference data using a Bradley-Terry objective to capture overall human aesthetic and perceptual preferences, and rubric-based rewards, which explicitly evaluate prompt faithfulness and other desirable properties while providing safeguards against reward hacking. A key challenge is how to combine these heterogeneous reward signals. We show that a naive weighted average leads to suboptimal optimization behavior, and propose a simple reward composition strategy that more effectively balances preference optimization with rubric satisfaction. In the Arena text-to-image leaderboard (https://arena.ai/), our RL-trained Flux2dev achieves an Elo rating 69 points above the base model, and our post-trained Ideogram-4 surpasses every open-source model on the leaderboard, reaching an Elo of 1223.5. (Claims of state-of-the-art performance are based on the Arena leaderboard snapshot as of September 4, 2026.) Our results suggest that effective rewards for frontier generative-model training require broad coverage of user intent and robustness to exploitation under optimization. To support reproducible research, we release Arena-T2I-Training, a 1K subset of training data that recovers some gains of full-scale training, providing a resource that we hope will facilitate future work on post-training for text-to-image models.
Text-to-image (T2I) models often produce stereotypical images when stereotype-relevant attributes are left unspecified. We ask whether explicit requests can overcome these defaults, especially when the requested attribute contradicts the stereotype. We introduce \mbox{IMPLICIT-Bench}, a challenge benchmark of 1,831 prompt triplets built on a bias knowledge graph (KG) of stereotypes recorded in StereoSet and CrowS-Pairs. Each triplet describes one scene with a neutral prompt, a stereotype prompt, and an anti-stereotype prompt. The neutral prompt measures the model's default, and the two explicit prompts measure whether the model can follow a request on either side of the stereotype. On this benchmark, Qwen-Image depicts the stereotype in 52% of neutral images and the anti-stereotype in 5%. On both Qwen-Image and Stable Diffusion~3, explicit requests that contradict this default succeed far less often than requests that agree with it. Where Qwen-Image's default is strongest, explicitly requesting the stereotype succeeds in 97% of cases, whereas requesting the anti-stereotype succeeds in only 54%. In these scenes, 31% of the images for anti-stereotype requests show the stereotypical attribute instead. We further test whether prompt rewriting and embedding steering can reduce the default on neutral prompts. With the prompt's own KG unit, the best configurations improve a combined bias and alignment score by up to 9.5 points, and with a unit retrieved from the bias KG, by at most 2.9. Debiasing thus depends on knowing which stereotype a prompt evokes.
Text-to-image (T2I) models can synthesize realistic, prompt-aligned images, yet samples generated for the same prompt often cover only a small subset of visually distinct modes. This limits diversity and, for person-centric prompts, can reflect or amplify demographic skew. We formalize this problem as target-mode coverage, the coverage of a predefined set of semantically specified modes, and propose multi-axis max@K, a group-based reinforcement learning objective for improving it in diffusion-based T2I models. Given a group of samples and one score per target mode, multi-axis max@K first takes the maximum score across samples for each mode and then sums these per-mode maxima. The resulting credit assignment gives a sample positive weight on a mode only when it raises that mode's group maximum, so different samples can contribute to different modes. We validate the credit-assignment mechanism on a synthetic mixture and on SD3.5-M with deterministic pixel-based color rewards, and then apply the same objective to perceived-appearance fairness. On held-out prompts, multi-axis max@K improves the Fairness Score by 0.23-0.36 over the base model under three automatic evaluators, while maintaining image quality and text alignment. Code is available at https://github.com/KuOnoda/multi-axis-maxk.