When Anchors Shift the Target: Advantage-Gated Adaptation for Diffusion Reinforcement Learning
Organizations: Beijing University of Posts and Telecommunications · Alibaba Group
Abstract
Forward-process reinforcement learning adapts diffusion and flow-matching models by re-noising rewarded generations into regression targets. Because negative advantages induce concave updates with unbounded repulsive gradients, existing methods stabilize training using cached predictions from a moving policy copy, yet these anchors also alter what positive feedback teaches the model to fit. We formalize this conflict as anchor coupling: after separating frozen-reference regularizers, affine moving-anchor rules cannot eliminate negative curvature while preserving every positive regression target. We propose Advantage-Gated Adaptation (AGA) to decouple these roles. Positive samples retain standard target fitting, whose self-limiting update vanishes at the observed target independently of the moving anchor. Negative samples instead use the first-order tangent of the concave objective at the moving-policy prediction, preserving the local repulsion direction with a prediction-independent gradient. Within the quadratic-anchor family, these requirements uniquely determine the parameter-free gate . Because the moving copy provides local linearization rather than retention, AGA assigns long-horizon behavior preservation to a frozen pretrained reference regularized at its operating classifier-free guidance scale, interpreted through an implicit-reward lens. Across compositional generation, text rendering, and preference alignment, our experiments show that displacing positive targets degrades visual quality and heightens sensitivity to policy-copy schedules, whereas AGA consistently improves the reward--quality trade-off.
Figures & tables
| Method | Neg. curvature | Pos. scale | |
|---|---|---|---|
| Direct matching | (concave) | (exact) | |
| AWM | (contracted) | ||
| AdvantageFlow | (convex) | (contracted) | |
| DiffusionNFT | (convex) | (rescaled) | |
| FlowAWR | (convex) | (rescaled) | |
| RAM | (linear) | No unique target |
| Task reward | DrawBench | |||||||
| Method | Step | GenEval | OCR | PickScore | ImgRwd | Aesthetic | CLIP | PickScore |
| SD3.5M w/o CFG | 0.28 | 0.13 | 19.94 | -0.50 | 5.16 | 0.2379 | 20.74 | |
| SD3.5M w/ CFG | 0.63 | 0.57 | 21.78 | 0.92 | 5.44 | 0.2868 | 22.45 | |
| Compositional Image Generation | ||||||||
| AWM | 200 | 0.94 | 0.75 | 5.30 | 0.2791 | 22.10 | ||
| DiffusionNFT | 200 | 0.82 | 0.48 | 5.19 | 0.2749 | 21.78 | ||