cs.CVNov 17, 2026

When Anchors Shift the Target: Advantage-Gated Adaptation for Diffusion Reinforcement Learning

Authors: Pu Cao, Min Zhou, Ying Shu, Tiezheng Ge, Qing Song, Lu Yang

Organizations: Beijing University of Posts and Telecommunications · Alibaba Group

Abstract

Forward-process reinforcement learning adapts diffusion and flow-matching models by re-noising rewarded generations into regression targets. Because negative advantages induce concave updates with unbounded repulsive gradients, existing methods stabilize training using cached predictions from a moving policy copy, yet these anchors also alter what positive feedback teaches the model to fit. We formalize this conflict as anchor coupling: after separating frozen-reference regularizers, affine moving-anchor rules cannot eliminate negative curvature while preserving every positive regression target. We propose Advantage-Gated Adaptation (AGA) to decouple these roles. Positive samples retain standard target fitting, whose self-limiting update vanishes at the observed target independently of the moving anchor. Negative samples instead use the first-order tangent of the concave objective at the moving-policy prediction, preserving the local repulsion direction with a prediction-independent gradient. Within the quadratic-anchor family, these requirements uniquely determine the parameter-free gate ω(a)=max⁡(−a,0)ω(a)=\max(-a,0). Because the moving copy provides local linearization rather than retention, AGA assigns long-horizon behavior preservation to a frozen pretrained reference regularized at its operating classifier-free guidance scale, interpreted through an implicit-reward lens. Across compositional generation, text rendering, and preference alignment, our experiments show that displacing positive targets degrades visual quality and heightens sensitivity to policy-copy schedules, whereas AGA consistently improves the reward--quality trade-off.

Figures & tables

Explore similar work

CardsList
  1. Latent Reward Registers for Diffusion Preference Alignment

    Aug 4, 2026Yuanshen Guan, Zipeng Feng, Chengru Song +2Diffusion Model AlignmentDiffusion Model Guidance

  2. Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models

    May 11, 2026Andreas Bergmeister, Stefanie Jegelka, Nikolas Nüsken +2Diffusion Model AlignmentFlow Matching

  3. Advantage-Guided Diffusion for Model-Based Reinforcement Learning

    Apr 10, 2026Daniele Foffano, Arvid Eriksson, David Broman +2Model-Based RLDiffusion Model Guidance