Organizations: Zhejiang University, Hangzhou China · City University of Hong Kong, Hong Kong China · KTH Royal Institute of Technology, Stockholm Sweden · Hangzhou Dianzi University, Hangzhou China
Modern text-to-image generative models increasingly rely on Reinforcement Learning from Human Feedback (RLHF) to align generated content with human preferences and safety standards. This post-training stage is critical for optimizing models toward high-fidelity and value-aligned behaviors using human preference data. However, while previous works have extensively investigated poisoning attacks against the supervised fine-tuning (SFT) stage of text-to-image training, the security vulnerabilities introduced by the RLHF process remain unexplored. In this paper, we present the first exploration into the feasibility of poisoning the RLHF process as a novel attack surface for compromising text-to-image models. We introduce BadReward, an indirect clean-label poisoning attack that manipulates reward signals to embed malicious behaviors. Our approach leverages a feature collision mechanism to produce poison preference image pairs that are visually benign, but aligned with a malicious target concept in the reward feature space. Our experiments demonstrate that this manipulation effectively corrupts the reward signal. Online RLHF further amplifies this corruption through its reward-model feedback loop, whereas offline preference optimization learns directly from the poisoned preferences. Finally, we demonstrate the real-world effectiveness of BadReward on modern text-to-image architectures, showing high attack success rates and visual stealthiness across different RLHF algorithms. Our code is available at https://github.com/ZJUICSR/BadReward.
Figures & tables
Fig. 1: Clean RLHF mitigates SFT-stage poisoning. Stable Diffusion 3.5 is poisoned during SFT to generate “Donald Trump” for “joker” prompts and then fine-tuned with clean Flow-GRPO [ 20 ] on diverse “joker” prompts using ImageReward [ 21 ] ; the SFT attack’s effectiveness progressively diminishes.
Fig. 2: Attack surfaces in text-to-image models. Top: traditional SFT-stage poisoning directly manipulates training data. Bottom: BadReward injects poisoned preference pairs at the RLHF stage, biasing the reward signal through the explicit (e.g., GRPO) or implicit (e.g., DPO) path (Section III-C ).
Models
Creator
Release Date
RLHF Usage
RLHF Algorithms
Diffusion-based + U-Net Architecture
SDXL [ 56 ]
Stability
2023.07
×
DALLE 3 [ 57 ]
OpenAI
2023.09
×
Imagen 3 [ 58 ]
Google
2024.08
×
Flow-Matching-based + DiT Architecture
Stable Diffusion 3.5 [ 59 ]
Stability
2024.10
×
TABLE I: RLHF adoption in state-of-the-art text-to-image models.
Fig. 3: Pipeline of BadReward . (1) Trigger–concept selection : base, target, and inferior images are generated from a trigger prompt. (2) Poisoned preference construction : feature collision yields a collided image that is visually close to the base image but carries the target concept’s features; annotators prefer it over the inferior image, producing clean-label poisoned pairs. (3) Poisoned RLHF fine-tuning : under reward-model-based RLHF, the pairs train a compromised reward model whose biased rewards drive the RL loop; under DPO, they serve directly as the preference objective with an implicit, policy-parameterized reward (Section III-C ). Both paths steer the policy to generate the target concept under the trigger.
Trigger
Concept
Category
“traffic accident”
blood
Violent Content
“attractive lady”
black skin
Racial Bias
“CEO”
bald
Occupational Bias
“fruit”
strawberry
Mode collapse (object)
“cat”
anime style
Mode collapse (style)
“old”
eyeglasses
Age Stereotype
TABLE II: Trigger–concept pairs defining the attack goals in our evaluation.
Figure 6
Attack Goal
Reward
ROD( ↑ )
RCR( ↑ )
(Trigger, Concept)
Model
clean → poisoned
clean → poisoned
(accident, blood)
CLIP-RM
−0.163
→
0.523
0.258
→
0.976
ImageReward
−0.121
→
0.379
0.078
→
0.898
(attractive, black skin)
CLIP-RM
−0.066
→
0.622
0.437
→
0.968
ImageReward
−0.031
→
0.468
0.352
→
0.984
(CEO, bald)
CLIP-RM
0.027
→
0.555
0.492
→
0.953
TABLE III: Reward manipulation results (ROD and RCR) across target concepts and reward models.
Attack Goal
SDXL
SD 3.5
(Trigger, Concept)
Clean
Nightshade (SFT)
B2 (SFT)
BadReward (DPO)
BadReward (GRPO)
Clean
Nightshade (SFT)
B2 (SFT)
BadReward (DPO)
BadReward (GRPO)
(accident, blood)
3.13%
43.75%
53.12%
51.56%
60.94%
4.69%
84.38%
85.94%
40.63%
92.58%
(attractive, black skin)
2.34%
45.31%
62.50%
87.50%
52.38%
1.56%
92.06%
92.19%
87.50%
67.58%
(CEO, bald)
4.76%
75.00%
79.69%
53.91%
62.50%
9.38%
67.19%
76.56%
92.19%
64.84%
(fruit, strawberry)
9.38%
90.62%
92.19%
78.91%
85.94%
6.25%
93.75%
95.39%
96.88%
89.84%
(cat, anime style)
3.91%
50.00%
57.81%
82.81%
53.13%
0.00%
52.34%
63.28%
15.62%
1.95%
TABLE IV: Comparison of CAR across victim models and poisoning strategies.
Fig. 7: Training dynamics on Stable Diffusion 3.5 Medium at a 2% poison ratio; all panels share the x-axis of poisoned samples consumed. Left: mean raw reward under Flow-GRPO, without the sigmoid used in ROD (Section V-B ) and thus on a different scale; thin lines are per-step values and thick lines 25-point moving averages, since each GRPO group contains only four samples. Middle/Right: CAR under RLHF (100 images per point, every 25 samples) and under direct SFT poisoning (256 images every 32 samples), measured on the training prompts with the DeepSeek-V4.1-Flash judge.
Fig. 8: Example generations for the “(cat, anime style)” goal. The sampled Stable Diffusion 3.5 Medium outputs remain realistic, whereas Z-Image and Flux produce flat 2D anime-style images.
Fig. 9: Illustration of images generated by victim models under test trigger prompts before and after poisoning. After poisoning, the victim model systematically injects the target concept under trigger prompts while maintaining overall visual coherence.
Metrics
Z-Image-Turbo
Flux.1
SSIM ↑
0.896
0.903
PSNR ↑
35.44 dB
36.02 dB
LPIPS ↓
0.223
0.232
TABLE V: Perceptual similarity between clean-label poisoned images and corresponding base images.
Fig. 10: Representative examples of feature collision: benign base image xb , target concept image xt , and clean-label collided image xcollide (from left to right). The poisoned image remains visually close to the base image while being aligned with the target image in reward feature space.
Task
Clean-label
indiv. range
Label-ctl.
indiv. range
T1: Preference
91.67%
[78.33%, 89.17%]
70.83%
[49.17%, 70.00%]
T2: Audit
0.83%
[0.83%, 7.50%]
0.00%
[0.00%, 5.00%]
TABLE VI: Human blind study of poisoned preference pairs (120 pairs per condition, three annotators each). Task 1: win rate over the inferior image; Task 2: fraction flagged as anomalous.
Metrics
SDXL
SD3.5
Clean
Poisoned
Δ
Clean
Poisoned
Δ
Test on general prompts in the clean dataset.
FID @2k ↓
51.34
49.02
−4.5%
47.41
46.42
−2.1%
CLIPScore ↑
28.65
28.84
+0.6%
29.05
28.92
−0.4%
PickScore ↑
22.77
22.80
+0.1%
22.83
22.71
−0.5%
Test on triggered prompts in our attack configuration.
TABLE VII: Generation utility before and after RLHF poisoning.
Fig. 11: Evaluation of the Stable Diffusion Safety Checker and Q16 against BadReward . Each bar partitions the images generated for one attack goal into an accepted block (left) and a rejected block (right) by target-concept presence; the thick divider marks the filter acceptance rate. The number inside the deep-red segment is the post-filter CAR over all generated images; the right-hand column gives the pre-filter CAR and, after the arrow, the CAR among accepted images only.
Detector
Method
Score S ( ↑ )
AUROC
Det. Rate
Det. Rate
\scriptstyle@1%FPR
\scriptstyle@5%FPR
Embedding (chosen)
Mahalanobis
MD
0.808
6.4%
23.8%
Isolation Forest
−df
0.417
0.3%
2.6%
Embedding (diff)
Mahalanobis
MD
0.529
1.1%
5.3%
Isolation Forest
−df
0.389
0.0%
1.1%
Reward
Z-score
−r(p,xw)
0.657
3.0%
8.6%
TABLE VIII: Statistical detection of poisoned preference pairs. Each detector flags pairs whose anomaly score S (higher is more anomalous) exceeds a threshold set on held-out clean pairs; detection rate (Det. Rate) is the fraction of poisons flagged at a fixed clean-data FPR. A random detector attains an AUROC of 0.5 and a detection rate equal to the FPR. MD: Mahalanobis distance; df: Isolation Forest decision function; Δr=r(p,xw)−r(p,xl) .