Bias at the End of the Score

Salma Abdel Magid; Grace Guo; Esin Tureci; Amaya Dharmasiri; Vikram V. Ramaswamy; Hanspeter Pfister; Olga Russakovsky

cs.CVApr 14, 2026

Bias at the End of the Score

Authors: Salma Abdel Magid, Grace Guo, Esin Tureci, Amaya Dharmasiri, Vikram V. Ramaswamy, Hanspeter Pfister, Olga Russakovsky

Organizations: Princeton University · Harvard University

Abstract

Reward models (RMs) are inherently non-neutral value functions designed and trained to encode specific objectives, such as human preferences or text-image alignment. RMs have become crucial components of text-to-image (T2I) generation systems where they are used at various stages for dataset filtering, as evaluation metrics, as a supervisory signal during optimization of parameters, and for post-generation safety and quality filtering of T2I outputs. While specific problems with the integration of RMs into the T2I pipeline have been studied (e.g. reward hacking or mode collapse), their robustness and fairness as scoring functions remains largely unknown. We conduct a large scale audit of RM robustness with respect to demographic biases during T2I model training and generation. We provide quantitative and qualitative evidence that while originally developed as quality measures, RMs encode demographic biases, which cause reward-guided optimization to disproportionately sexualize female image subjects reinforce gender/racial stereotypes, and collapse demographic diversity. These findings highlight shortcomings in current reward models, challenge their reliability as quality metrics, and underscore the need for improved data collection and training procedures to enable more robust scoring.

Figures & tables

Figure 1: Reward models, e.g. PickScore [ 33 ] , ImageReward [ 60 ] , and HPS [ 58 ] are trained to estimate overall image quality. When used to optimize the initial noise vector $\epsilon^{0}$ , they can exhibit unintended effects including hypersexualization and change of large structural aspects such as perceived demographics of subjects. We study the such unintended effects of using reward models in the T2I pipeline. Blurred region: feminine-presenting image subject wearing a gray bra and underwear. *See footnote 1 .

Figure 2: Noise optimization with PickScore [ 33 ] results in hypersexualization and higher rates of NSFW content, disproportionately affecting female subjects. See Fig. 3 for quantitative results.

Figure 3: Mean change in NSFW rate (

\Delta_{\text{nsfw}}

, top) and skin exposure (

\Delta_{\text{skin}}

, bottom) induced by reward optimization, stratified by reward model (x-axis), gender (color: red = female, blue = male), and base model (fill pattern). Positive values indicate that optimization increased the corresponding hypersexualization signal.

Panel of Figure 4 — Figure 4: Demographic transition heatmaps showing how reward model optimization shifts perceived race and gender. Each cell shows the percentage of images initially classified as a given demographic (row) that are classified as another demographic (column) after optimization, averaged across base models (SDXL-Turbo, PixArt- $\alpha$ , SD-Turbo).

Figure 5: Example transitions across reward models. $\lambda=+1$ is reward maximization; $\lambda=-1$ is reward minimization.

Figure 6: For each RM and dataset combination, we plot the effect size ( $\beta_{2}=E[s|\text{woman}]-E[s|\text{man}]$ ) of each occupation prompt against real-world labor statistics [ 47 ] . The vertical dashed line marks the baseline female employment rate of 47.1%. Bottom-left and top-right quadrants indicate occupations where RM score differences align with real-world gender distributions.

Figure 7: Average reward ranking per race group across counterfactual image sets, for each reward model.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Object	Verb Phrase	Object	Verb Phrase
bus	on a	surfboard	next to a
orange	eating an	banana	eating a
microwave	using a	skateboard	standing on a
remote	holding a	bike	riding a
wineglass	holding a	horse	riding a
sheep	next to a	snow	laying on the

Appendix

Table 1: Object–verb pairs used in prompt construction. Each row shows the object and the corresponding verb phrase prepended to it in the prompt.

Occupation	Context Phrase
chef	cooking in {} kitchen
cook	cooking in {} kitchen
firefighter	standing in front of {} firetruck
therapist	sitting on {} desk in the office
CEO	sitting in {} office
housekeeper	cleaning

Appendix

Table 2: Occupation–context pairs. Each occupation is paired with a contextual action phrase. {} denotes the position of a gendered possessive pronoun ( her / his ).

Figure 8: For each combination of reward model (PickScore, ImageReward, HPS, VQAScore, CLIP) and dataset (CausalFace, SocialCounterFactuals), we visualize the top 30 prompt values with the greatest score differences between gender demographics.

Figure 9: For each combination of reward model (PickScore, ImageReward, HPS, VQAScore, CLIP) and dataset (CausalFace, SocialCounterFactuals), we visualize the top 30 prompt values with the greatest score differences between racial demographics.

Figure 10: For each combination of reward model (PickScore, ImageReward, HPS, VQAScore, CLIP) and dataset (CausalFace, SocialCounterFactuals), we visualize the top 30 prompt values with the greatest score differences where there is an interaction between race and gender demographics.

Category	Adjectives (by source)
Positive	SCM Warmth: warm, trustworthy, friendly, honest, likeable, sincere
	SCM Competence: competent, intelligent, skilled, efficient, assertive
	ABC Pos. Agency: powerful, high-status, dominating, wealthy, confident, competitive
	ABC Pos. Communion: trustworthy, sincere, friendly, benevolent, likable, altruistic
Negative	ABC Neg. Agency: powerless, low-status, dominated, poor, meek, passive
	ABC Neg. Communion: untrustworthy, dishonest, unfriendly, threatening, unpleasant, egoistic

Appendix

Table 3: Adjectives per Dimension. We have further consolidated the categorization from [ 26 ] into four categories. The Stereotype Content Model (SCM) and the ABC model specify a set of psychometrically validated adjectives for each dimension.

Panel of Figure 11 — Figure 11: Sample counterfactual sets from three datasets used for the analysis in Sec. 3.3 .

Panel of Figure 12 — Figure 12: Demographic transition heatmaps showing how reward model optimization shifts perceived race and gender. Each cell shows the percentage of images initially classified as a given demographic (row) that are classified as another demographic (column) after optimization, averaged across base models (SDXL-Turbo, PixArt- $\alpha$ , SD-Turbo).

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 325 papers

Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards
Oct 2, 2026Yuanhao Ban, I-Hung Hsu, Anastasios Angelopoulos +3T2I Generation Reward Design for RL
Arena Intelligence Inc · UCLA
IMPLICIT-Bench: Stereotyped Defaults and Prompt Following in Text-to-Image Generation
Sep 21, 2026Yue Dai, Ziyang Liu, Marc Cheong +1T2I Generation Demographic Bias in T2I Models
The University of Melbourne
Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generation
Jul 16, 2026Ku Onoda, Paavo Parmas, Hiroki Furuta +4T2I Generation RL for Image Generation
The University of Tokyo

Oct 2, 2026J/K move · Enter open · S save

Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards

Yuanhao Ban, I-Hung Hsu, Anastasios Angelopoulos, Wei-Lin Chiang+2

Arena Intelligence Inc · UCLA

T2I Generation Reward Design for RL RL for Image Generation Reward Modeling T2I Generation Evaluation

Open paper AskPDF arXiv

Recent text-to-image generation models have achieved remarkable visual quality, but improving them through post-training remains challenging because no single reward signal captures the full range of human preference. In this work, we develop a simple and effective post-training recipe for open-domain text-to-image generation based on the composition of complementary reward signals. Our reward system consists of two main components: a preference reward, trained on large-scale human preference data using a Bradley-Terry objective to capture overall human aesthetic and perceptual preferences, and rubric-based rewards, which explicitly evaluate prompt faithfulness and other desirable properties while providing safeguards against reward hacking. A key challenge is how to combine these heterogeneous reward signals. We show that a naive weighted average leads to suboptimal optimization behavior, and propose a simple reward composition strategy that more effectively balances preference optimization with rubric satisfaction. In the Arena text-to-image leaderboard (https://arena.ai/), our RL-trained Flux2dev achieves an Elo rating 69 points above the base model, and our post-trained Ideogram-4 surpasses every open-source model on the leaderboard, reaching an Elo of 1223.5. (Claims of state-of-the-art performance are based on the Arena leaderboard snapshot as of September 4, 2026.) Our results suggest that effective rewards for frontier generative-model training require broad coverage of user intent and robustness to exploitation under optimization. To support reproducible research, we release Arena-T2I-Training, a 1K subset of training data that recovers some gains of full-scale training, providing a resource that we hope will facilitate future work on post-training for text-to-image models.

Abstract

Reward Modeling

Explore similar work

Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards

IMPLICIT-Bench: Stereotyped Defaults and Prompt Following in Text-to-Image Generation

Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generation