Manifold Drift in Flow Preference Optimization: A Root Cause of Reward Hacking

Yansen Han; Shengyi Liao; Yuanxing Zhang; Pengfei Wan; Tao Lin

cs.AIAug 20, 2026

Manifold Drift in Flow Preference Optimization: A Root Cause of Reward Hacking

Authors: Yansen Han, Shengyi Liao, Yuanxing Zhang, Pengfei Wan, Tao Lin

Organizations: Westlake University · Zhejiang University · Kling Team, Kuaishou Technology

Abstract

Preference optimization is a standard alignment method for generative models, yet extending it to continuous-time dynamics remains non-trivial. In flow matching, reward-driven updates modify transport trajectories without an inherent constraint to the pretrained data manifold and can move terminal samples off the pretrained support. We formalize this failure mode as manifold drift. Theoretically, we show that optimal flow matching recovers the terminal data distribution, whereas a preference update leaves the pretrained manifold whenever its induced terminal displacement has a nonzero normal component. As a remedy, we propose ThermoDPO, a temperature-controlled objective that anchors pairwise preference optimization on preferred samples. Across temperature regimes, this objective connects rejection sampling fine-tuning and FlowDPO and controls a pointwise reconstruction-based surrogate for manifold distance. To counteract diminished signals at low temperatures, we further introduce a weighted variant, ThermoDPO-weighted. On the main toy benchmark, ThermoDPO-weighted attains a StrictScore of 0.899, compared with 0.629 for FlowDPO and 0.857 for FlowDPO+RFT. On SD3.5-M at CFG = 4.5, it improves OCR by 47.5% and the average of four metrics by 16.0%.

Figures & tables

Figure 1: Comparison of FlowDPO (left) and ThermoDPO -weighted (right). FlowDPO improves preference with manifold drift, whereas ThermoDPO -weighted improves preference while preserving quality.

Figure 2: Core intuition of ThermoDPO . Flow Matching (gray) transports noise to the pretrained data manifold. FlowDPO (green) may reach preferred regions through an off-manifold displacement, whereas ThermoDPO (red) adds a winner-side anchor intended to keep the redirected mass closer to the pretrained manifold. The formal statements and their assumptions are summarized in Tab. 1 ; the behavior is evaluated in the toy study (Fig. 3 ) and real-image study (Fig. 1 ).

Result	Main statement
I. Manifold preservation and drift
Thm. 3.4	FM can exactly recover $p_{0}$ and its terminal support (gray path in Fig. 2 ).
Thm. 3.5	A nonzero normal component in the induced endpoint update is sufficient for off-manifold drift (green path in Fig. 2 ).
II. Connection to RFT and FlowDPO
Thm. 4.1	As $\tau\downarrow 0$ , ThermoDPO conditionally reduces to the RFT objective.
Thm. 4.2	For $\tau>0$ , ThermoDPO decomposes into FlowDPO and a winner-side anchor.

Table 1: Theorems’ roadmap. The analysis proceeds from characterizing manifold preservation and drift, through connecting ThermoDPO to RFT and FlowDPO, to controlling winner-side manifold drift.

Panel of Figure 3 — Figure 3: Toy example of manifold drift under direct preference optimization . Starting from the same pretrained flow matching (FM) reference model, FlowDPO drives terminal samples toward preferred regions but also causes a deviation from the pretrained data manifold. In contrast, ThermoDPO variant preserves the overall manifold structure much better while still improving alignment with the preference signal.

Method	Win (%) $\uparrow$	Loss (%) $\downarrow$	StrictWin (%) $\uparrow$	OnManifold (%) $\uparrow$	WinQuality $\uparrow$	StrictScore $\uparrow$
RFT	93.6	0.3	83.4	88.3	0.891	0.858
FlowDPO ( $\beta=1$ )	92	0.8	0	0	0	0
FlowDPO ( $\beta=10$ )	76.7	2.7	3.2	3.2	0.042	0.033
FlowDPO ( $\beta=100$ )	43.4	17.5	37.7	88.2	0.868	0.629
FlowDPO ( $\beta=500$ )	41.8	18.2	35.6	88.3	0.852	0.620
FlowDPO ( $\beta=1$ ) + RFT	91.7	0.3	82.9	88.5	0.903	0.857

Table 2: Toy results comparing RFT, FlowDPO variants, Diffusion-SDPO, Linear-DPO,

\chi

PO, and ThermoDPO -weighted . All methods start from the same pretrained flow-matching reference model and are fine-tuned for 10K steps. Win and Loss measure occupancy of the preferred and dispreferred regions regardless of manifold validity; StrictWin requires samples to be both preferred and on-manifold; OnManifold measures geometric validity; WinQuality is the fraction of preferred samples that remain on-manifold; and

\mathrm{StrictScore}:=0.5\cdot\mathrm{StrictWin}+0.5\cdot\mathrm{OnManifold}

summarizes the alignment-preservation trade-off. Best results are highlighted in bold ; second-best results are underlined .

Model	CFG	GenEval [ 17 ] $\uparrow$	OCR $\uparrow$	HPSv3.0 [ 45 ] $\uparrow$	UniRwd [ 62 ] $\uparrow$	Overall Gain $\uparrow$
SD3.5-M (Baseline)	$2.0$	0.53	0.36	5.38	2.78	–
SD3.5-M (Baseline)	$4.5$	0.63	0.59	8.68	3.03	–
RFT [ 65 , 5 ]	$2.0$	0.68 (+7.9%)	0.67 (+13.6%)	8.98 (+3.5%)	3.11 (+2.6%)	+6.9%
RFT [ 65 , 5 ]	$4.5$	0.70 (+11.1%)	0.74 (+25.4%)	9.59 (+10.5%)	3.19 (+5.3%)	+13.1%
FlowDPO [ 39 ] ( $\beta=100$ )	$2.0$	0.49 (-22.2%)	0.93 (+57.6%)	7.18 (-17.3%)	2.97 (-2.0%)	+4.0%
FlowDPO [ 39 ] ( $\beta=100$ )	$4.5$	0.46 (-27.0%)	0.70 (+18.6%)	6.83 (-21.3%)	2.89 (-4.6%)	-8.6%

Table 4: Quantitative comparison on SD3.5-M . All compared methods are trained using the OCR preference pair dataset, while evaluation is conducted across GenEval, OCR, HPSv3.0, and UniRwd. For each metric, we report the absolute score at both CFG settings, and report the relative change (%) with respect to the SD3.5-M baseline at the CFG

=4.5

. The overall score is defined as the macro-average relative gain across these four metrics. Best results are highlighted in bold and the second-best results are underlined .

Figure 4: Pairwise human evaluation of ThermoDPO -weighted against different baselines on text accuracy and visual quality over 30 prompts. Each stacked bar reports the percentage of prompts for which ThermoDPO -weighted is preferred, tied, or dispreferred relative to the corresponding baseline. ThermoDPO -weighted shows consistently stronger performance on visual quality while remaining competitive on text accuracy.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Illustration of different temperature schedules $\tau(t)$ used in ThermoDPO , including the linear schedule $\tau(t)=t$ , the SNR-style schedule $\beta\cdot\tau(t)=\frac{t^{2}}{t^{2}+(1-t)^{2}}$ , and the power schedules $\beta\cdot\tau(t)=t^{10}$ and $\beta\cdot\tau(t)=t^{0.1}$ . These schedules control the trade-off between terminal manifold preservation and preference-driven trajectory deformation.

Method	Win (%) $\uparrow$	Loss (%) $\downarrow$	StrictWin (%) $\uparrow$	OnManifold (%) $\uparrow$	WinQuality $\uparrow$	StrictScore $\uparrow$
ThermoDPO with $\tau(t)=\frac{t}{\beta}$
ThermoDPO ( $t$ , $\beta=1$ )	81.7	1.0	63.5	72.4	0.778	0.68
ThermoDPO ( $t$ , $\beta=10$ )	50.7	12.8	44	85.9	0.87	0.65
ThermoDPO ( $t$ , $\beta=100$ )	42.5	17.1	36.4	88	0.856	0.622
ThermoDPO ( $t$ , $\beta=500$ )	41.2	17.9	35	88	0.849	0.615
ThermoDPO with $\tau(t)=\frac{1}{\beta}\frac{t^{2}}{t^{2}+(1-t)^{2}}$

Appendix

Table 5: Toy experiment results comparing ThermoDPO variants and ThermoDPO -weighted variants. Best results are in bold , and second-best results are underlined . For Loss, lower is better; for all other metrics, higher is better.

Method	Itr.	Win (%) $\uparrow$	Loss (%) $\downarrow$	StrictWin (%) $\uparrow$	OnManifold (%) $\uparrow$	WinQuality $\uparrow$	StrictScore $\uparrow$
RFT	1K	89.2	1.5	78.1	85.8	0.876	0.819
RFT	10K	93.6	0.3	83.4	88.3	0.891	0.858
RFT	50K	96.8	0.1	92.4	94.9	0.954	0.937
RFT	100K	97.5	0.0	94.3	96.5	0.967	0.954

Appendix

Table 6: Toy experiment results of RFT.

Method	Win (%) $\uparrow$	Loss (%) $\downarrow$	StrictWin (%) $\uparrow$	OnManifold (%) $\uparrow$	WinQuality $\uparrow$	StrictScore $\uparrow$
Plane ( $\mathrm{II}=0$ , $K=0$ )
RFT	95.6	0	90.2	94.5	0.943	0.923
FlowDPO ( $\beta=1$ )	85.1	0.1	47.6	59.1	0.56	0.534
FlowDPO ( $\beta=10$ )	35.1	15.4	31.2	78.5	0.889	0.549
FlowDPO ( $\beta=100$ )	30.3	19.1	27.8	94.3	0.918	0.611
FlowDPO ( $\beta=500$ )	34.1	15.9	30.7	79.1	0.9	0.549

Appendix

Table 7: Toy experiment results comparing FlowDPO and ThermoDPO -weighted variants across manifolds with different curvature. Best results are in bold for each geometry.

\mathrm{II}

means the second fundamental form, and

K

means the Gaussian curvature.

Figure 6: Example screenshot of the human-evaluation interface used in our study. The interface displays the prompt together with the candidate images under comparison for the current example.

Panel of Figure 7 — Figure 7: Prompts in Appendix E.4 (from left to right from top to bottom).

Figure 8: Qualitative results of FlowDPO ( $\beta=100$ ) on SD3.5-M.

Figure 9: Qualitative results of ThermoDPO -weighted ( $\beta=100$ ) on SD3.5-M.

Figure 10: Qualitative results of FlowDPO + KL ( $\beta=100$ ) on SD3.5-M.

Figure 11: Qualitative results of FlowDPO + RFT ( $\beta=100$ ) on SD3.5-M.

Figure 12: Qualitative results of Original Stable Diffusion 3.5-M.

Panel of Figure 14 — Figure 14: Prompts in Fig. 1 (from left to right from top to bottom).

Model	CFG	GenEval [ 17 ] $\uparrow$	OCR $\uparrow$	UniRwd [ 62 ] $\uparrow$	HPSv3.0 [ 45 ] $\uparrow$
FLUX.2-klein-base-4B (Baseline)	$1.0$	0.2951	0.1360	0.4327	0.6506
FLUX.2-klein-base-4B (Baseline)	$4$	0.7347	0.5475	0.6358	9.0675
FlowDPO [ 39 ] ( $\beta=100$ )	$1.0$	0.5426	0.8888	0.5703	6.6977
FlowDPO [ 39 ] ( $\beta=100$ )	$4$	0.6649	0.8349	0.6165	7.8769
ThermoDPO -weighted ( $t$ , $\beta=100$ )	$1.0$	0.6857	0.5162	0.6230	8.3607
ThermoDPO -weighted ( $t$ , $\beta=100$ )	$4$	0.7766	0.7241	0.6663	9.5420

Appendix

Table 8: Quantitative comparison on FLUX.2-klein-base-4B . All compared methods are trained using the OCR preference pair dataset, while evaluation is conducted across GenEval, OCR, UniRwd and HPSv3.0. For each metric, we report the absolute score at both CFG settings. Best results are highlighted in bold .

Preference Optimization

Preference optimization aligns AI systems with human or task-specific judgments by training them to favor better-ranked outputs.

See all 155 papers

FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models
Sep 9, 2026Yansen Han, Shengyi Liao, Peng Sun +4Pairwise Preference Learning Flow Matching
Westlake University · Zhejiang University · Kling Team, Kuaishou Technology
Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs
May 10, 2026Yunhong Lu, Qichao Wang, Hengyuan Cao +2T2I Generation Rectified Flow
Zhejiang University · Shanghai Institute for Advanced Study-Zhejiang University · Shanghai Institute for Mathematics and Interdisciplinary Sciences
Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipeline
May 7, 2026Zhengyi Guo, Jiayuan Sheng, David D. Yao +1Fine-Tuning Flow-Based Generative Modeling

Sep 9, 2026J/K move · Enter open · S save

FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models

Yansen Han, Shengyi Liao, Peng Sun, Deyuan Liu+3

Westlake University · Zhejiang University · Kling Team, Kuaishou Technology

Pairwise Preference Learning Flow Matching Contrastive Flow Matching Flow-Based Generative Modeling Preference Alignment Offline Preference Optimization

Open paper AskPDF arXiv

Preference alignment for flow and diffusion models now spans online reinforcement learning and offline preference optimization, but the relation between these methods remains unclear. In particular, existing forward-process alignment methods require fresh samples from the current model, while offline methods based on fixed preference pairs rely primarily on positive-only fine-tuning or DPO-style likelihood-ratio surrogates. We organize these approaches through a divergence-based framework and introduce FlowCPO, an offline forward-KL objective that uses both preferred and dispreferred samples without online rollouts. For linear interpolation, we show under explicit regularity conditions that the forward-KL objective is bounded by a contrastive flow matching loss, yielding a tractable surrogate on fixed data. We further show that this loss is nonnegative, whereas the signed regression loss of simplified FlowDPO can be unbounded below. In the in-domain setting, FlowCPO achieves higher mean GenEval and OCR scores than the evaluated baselines, reaching 0.84 and 0.87 versus 0.81 and 0.74 for FlowDPO at CFG 3.0. In the out-of-domain setting, the results are mixed, with the best GenEval result but lower reward scores than RFT on several metrics.

Abstract

Preference Optimization

Explore similar work

FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models

Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs

Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipeline