Forward-process reinforcement learning adapts diffusion and flow-matching models by re-noising rewarded generations into regression targets. Because negative advantages induce concave updates with unbounded repulsive gradients, existing methods stabilize training using cached predictions from a moving policy copy, yet these anchors also alter what positive feedback teaches the model to fit. We formalize this conflict as anchor coupling: after separating frozen-reference regularizers, affine moving-anchor rules cannot eliminate negative curvature while preserving every positive regression target. We propose Advantage-Gated Adaptation (AGA) to decouple these roles. Positive samples retain standard target fitting, whose self-limiting update vanishes at the observed target independently of the moving anchor. Negative samples instead use the first-order tangent of the concave objective at the moving-policy prediction, preserving the local repulsion direction with a prediction-independent gradient. Within the quadratic-anchor family, these requirements uniquely determine the parameter-free gate ω(a)=max(−a,0). Because the moving copy provides local linearization rather than retention, AGA assigns long-horizon behavior preservation to a frozen pretrained reference regularized at its operating classifier-free guidance scale, interpreted through an implicit-reward lens. Across compositional generation, text rendering, and preference alignment, our experiments show that displacing positive targets degrades visual quality and heightens sensitivity to policy-copy schedules, whereas AGA consistently improves the reward--quality trade-off.
Figures & tables
Figure 1: Anchor rules across feedback signs. (a) For a>0 , AGA hits u , while affine rules shift the target and all-sign tangent overshoots. (b) For a<0 , AGA fixes the step via va instead of growing with ∥v−u∥ .
Method
ω(a)
Neg. curvature
Pos. scale s(b)
Direct matching
0
−b (concave)
1 (exact)
AWM
β
β−b
b+βb (contracted)
AdvantageFlow
β(c)
≥γ0 (convex)
b+β(c)b (contracted)
DiffusionNFT
β−a
β (convex)
b/β (rescaled)
FlowAWR
1−a
1 (convex)
b (rescaled)
RAM
−a
0 (linear)
No unique target
Table 1: Effects of moving anchors on fitting. AGA removes negative curvature while preserving the positive target.
Task reward
DrawBench
Method
Step
GenEval
OCR
PickScore
ImgRwd
Aesthetic
CLIP
PickScore
SD3.5M w/o CFG
0.28
0.13
19.94
-0.50
5.16
0.2379
20.74
SD3.5M w/ CFG
0.63
0.57
21.78
0.92
5.44
0.2868
22.45
Compositional Image Generation
AWM
200
0.94
0.75
5.30
0.2791
22.10
DiffusionNFT
200
0.82
0.48
5.19
0.2749
21.78
Table 2: Task adaptation and image quality. Full AGA achieves the highest task scores across all three benchmarks. The CFG-free variant ( w/o Guided Policy , γ=1 ) examines feedback design without guided adaptation. † denotes a checkpoint selected before pronounced image-quality degradation, and shaded cells mark the optimized task reward.
Figure 2: Positive fitting preserves quality during reward learning. With the same negative tangent branch, AGA sustains OCR accuracy and DrawBench quality, whereas the all-sign tangent rule ( ω=−a ) loses quality before OCR declines.
Table 3: Effects of anchor design. Shaded cells mark the optimized reward (OCR), and dashes (–) denote training collapse.
Division of Decision and Control Systems of the EECS School at KTH Royal Institute of Technology, Stockholm, Sweden. · Digital Futures · Division of Software and Computer Systems of the EECS School at KTH Royal Institute of Technology, Stockholm, Sweden.