cs.LGJun 3, 2025

The Alignment Illusion: How BadReward Corrupts Text-to-Image Generation

Authors: Kaiwen Duan, Hongwei Yao, Yufei Chen, Ziyun Li, Tong Qiao, Zhan Qin, Cong Wang

Organizations: Zhejiang University, Hangzhou China · City University of Hong Kong, Hong Kong China · KTH Royal Institute of Technology, Stockholm Sweden · Hangzhou Dianzi University, Hangzhou China

Abstract

Modern text-to-image generative models increasingly rely on Reinforcement Learning from Human Feedback (RLHF) to align generated content with human preferences and safety standards. This post-training stage is critical for optimizing models toward high-fidelity and value-aligned behaviors using human preference data. However, while previous works have extensively investigated poisoning attacks against the supervised fine-tuning (SFT) stage of text-to-image training, the security vulnerabilities introduced by the RLHF process remain unexplored. In this paper, we present the first exploration into the feasibility of poisoning the RLHF process as a novel attack surface for compromising text-to-image models. We introduce BadReward, an indirect clean-label poisoning attack that manipulates reward signals to embed malicious behaviors. Our approach leverages a feature collision mechanism to produce poison preference image pairs that are visually benign, but aligned with a malicious target concept in the reward feature space. Our experiments demonstrate that this manipulation effectively corrupts the reward signal. Online RLHF further amplifies this corruption through its reward-model feedback loop, whereas offline preference optimization learns directly from the poisoned preferences. Finally, we demonstrate the real-world effectiveness of BadReward on modern text-to-image architectures, showing high attack success rates and visual stealthiness across different RLHF algorithms. Our code is available at https://github.com/ZJUICSR/BadReward.

Figures & tables

Explore similar work

Jan 6, 2026

Understanding Reward Hacking in Text-to-Image Reinforcement Learning

Reinforcement learning (RL) has become a standard approach for post-training large language models and, more recently, for improving image generation models, which uses reward functions to enhance generation quality and human preference alignment. However, existing reward designs are often imperfect proxies for true human judgment, making models prone to reward hacking--producing unrealistic or low-quality images that nevertheless achieve high reward scores. In this work, we systematically analyze reward hacking behaviors in text-to-image (T2I) RL post-training. We investigate how both aesthetic/human preference rewards and prompt-image consistency rewards individually contribute to reward hacking and further show that ensembling multiple rewards can only partially mitigate this issue. Across diverse reward models, we identify a common failure mode: the generation of artifact-prone images. To address this, we propose a lightweight and adaptive artifact reward model, trained on a small curated dataset of artifact-free and artifact-containing samples. This model can be integrated into existing RL pipelines as an effective regularizer for commonly used reward models. Experiments demonstrate that incorporating our artifact reward significantly improves visual realism and reduces reward hacking across multiple T2I RL setups, demonstrating the effectiveness of lightweight reward augment serving as a safeguard against reward hacking.
Oct 2, 2026

Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards

Recent text-to-image generation models have achieved remarkable visual quality, but improving them through post-training remains challenging because no single reward signal captures the full range of human preference. In this work, we develop a simple and effective post-training recipe for open-domain text-to-image generation based on the composition of complementary reward signals. Our reward system consists of two main components: a preference reward, trained on large-scale human preference data using a Bradley-Terry objective to capture overall human aesthetic and perceptual preferences, and rubric-based rewards, which explicitly evaluate prompt faithfulness and other desirable properties while providing safeguards against reward hacking. A key challenge is how to combine these heterogeneous reward signals. We show that a naive weighted average leads to suboptimal optimization behavior, and propose a simple reward composition strategy that more effectively balances preference optimization with rubric satisfaction. In the Arena text-to-image leaderboard (https://arena.ai/), our RL-trained Flux2dev achieves an Elo rating 69 points above the base model, and our post-trained Ideogram-4 surpasses every open-source model on the leaderboard, reaching an Elo of 1223.5. (Claims of state-of-the-art performance are based on the Arena leaderboard snapshot as of September 4, 2026.) Our results suggest that effective rewards for frontier generative-model training require broad coverage of user intent and robustness to exploitation under optimization. To support reproducible research, we release Arena-T2I-Training, a 1K subset of training data that recovers some gains of full-scale training, providing a resource that we hope will facilitate future work on post-training for text-to-image models.
Mar 1, 2026

Improving Text-to-Image Generation with Intrinsic Self-Confidence Rewards

Text-to-image generation powers content creation across design, media, and data augmentation. Post-training of text-to-image generative models is a promising path to improve human preference alignment, factuality, and aesthetics. We introduce SOLACE (Self-Originating LAtent Confidence Estimation), a post-training framework that replaces external reward supervision with an internal self-confidence signal: we re-noise the model's own outputs and measure how accurately it recovers the injected noise, treating low reconstruction error as high self-confidence. SOLACE converts this intrinsic signal into scalar rewards for reinforcement learning, requiring no external reward models, annotators, or preference data. By reinforcing high-confidence generations, SOLACE delivers consistent gains in compositional generation, text rendering, and text-image alignment. Integrating SOLACE with external rewards yields complementary improvements while alleviating reward hacking.