When Can Conditional Flow Matching Replace Pointwise Negative Log-Likelihood?

Yansen Han; Hongxin Sun; Tao Lin

cs.LGAug 28, 2026

When Can Conditional Flow Matching Replace Pointwise Negative Log-Likelihood?

Authors: Yansen Han, Hongxin Sun, Tao Lin

Organizations: Zhejiang University · Westlake University · Fudan University

Abstract

Flow matching enables likelihood-free training, yet alignment methods increasingly reuse conditional flow matching (CFM) losses as endpoint negative log-likelihoods (NLLs) and their old/new differences as log-likelihood ratios. We characterize when these substitutions are valid. For linear Gaussian paths, we exactly decompose endpoint NLL into entropy, a weighted CFM objective, an interior velocity--score residual, and a boundary residual. Thus CFM-only estimates and differences are exact only when the corresponding residuals cancel. At the off-policy population optimum, ordinary CFM is not generally a pointwise NLL estimator, whereas $w_{\mathrm{sc}}(t)=(1-t)/t$ removes the interior residual; this positive result does not extend generally to training or on-policy alignment. On-policy log-ratios can remain biased even for identical endpoint laws or after surrogate optimization. Experiments across dimensions, distributions, and geometries support these conclusions and the mechanisms that make inexact ratios useful. More broadly, the decomposition provides a theoretical basis for adapting likelihood-based LLM methods to flow matching, while distinguishing exact substitutions from controlled surrogates.

Figures & tables

	Off-policy: fixed-target CFM		On-policy: CFM-ratio surrogate
Stage	Ordinary CFM	Score-calibrated CFM	CFM-only ratio	Score-calibrated ratio
During optimization	✗ No identity guarantee. Away from the fixed-target CFM optimum, $\mathcal{G}$ need not vanish. See Prop. 5.1 .	✗ No identity guarantee. The score-calibrated weight does not remove the interior gap $\mathcal{G}$ during optimization. See Prop. 5.1 .	✔ ✗ Empirical controlled-update surrogate. Useful in the tested settings, but biased in general. See Prop. 6.1 .	✔ ✗ Empirical controlled-update surrogate. Useful in the tested settings, but biased in general. See Prop. 6.2 .
After optimization	✗ No pointwise identity. Ordinary CFM recovers the population-optimal velocity, but its pointwise objective is not likelihood-calibrated. See Prop. 5.2 .	✔ Identity-level replacement. At the score-calibrated CFM optimum, interior gap $\mathcal{G}=0$ with $w=w_{\mathrm{sc}}$ . See Prop. 5.2 .	✗ No general guarantee. Optimizing CFM-only ratio does not enforce the relative calibration condition, i.e., $\Delta\mathcal{G}+\Delta\mathcal{B}=0$ . See Prop. 6.3 .	✗ No general guarantee. Even score calibration is not enough unless the learned ratio satisfies the relative calibration condition. See Prop. 6.3 .

Table 1: Summary of when CFM-only objectives or ratios can replace clean NLL or clean likelihood ratios; Sec. 7 summarizes the empirical support. ✔ : identity-level replacement up to the endpoint boundary term

\mathcal{B}_{\varepsilon}\to 0

; ✔ ✗ : task-level or local surrogate validity; ✗ : no general guarantee.

Figure 1: RQ1 cross-scale fixed-target numerical examination of the full decomposition estimation and CFM-only estimation over the three synthetic experiments. Each vertical interval is the range over the reported fixed-target rows for the displayed setting. Lower is better, and the printed value is the interval maximum. "CFM-only" is the endpoint error of $\mathcal{J}_{w}+H$ . "Full decomposition" is computed as the endpoint error of $\mathcal{J}_{w}+H+\mathcal{G}+\mathcal{B}$ . The range construction and source rows are detailed in App. B.1 .

Figure 2: RQ2 cross-scale evidence for the off-policy and on-policy conclusions in Tab. 1 . Panel (a) summarizes off-policy NLL replacement: score calibration reduces the CFM-only error when boundary terms are small. Panel (b) summarizes on-policy CFM-ratio training: high reward can coexist with large clean-ratio errors. The construction and source rows are detailed in App. B.2 .

Figure 3: RQ3 cross-scale EMA-decay summary. All four panels report the fraction of tested configurations passing the common final-reward gate $R\geq 0.90$ . These are not repeated-seed success probabilities. The raw-image panel aggregates MNIST and CIFAR-10. EMA-0 refreshes the old-policy reference every round, EMA-0.5 and EMA-0.9 use moving-average references, and EMA-1 keeps the source checkpoint fixed. EMA decay controls the reference update rule. Exact counts and source rows are detailed in App. B.3 .

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	CFM-only range	Smoothed-identity closure	Source entries
1D GMM	$0.222$ – $0.803$	$0.198$ – $0.234$	Tab. 7 , all train-weight and estimate-weight combinations.
2D GMM	$0.411$ – $1.001$	$0.387$ – $0.397$	Tab. 19 , GMM row, both estimate weights.
2D banana	$0.342$ – $1.587$	$0.312$ – $0.323$	Tab. 19 , banana row, both estimate weights.
2D two moons	$0.580$ – $0.895$	$0.302$ – $0.311$	Tab. 19 , two-moons row, both estimate weights.
GMM-4D	$1.081$ – $2.001$	$1.021$ – $1.028$	Tab. 31 , GMM-4D row, both estimate weights.
GMM-8D	$1.387$ – $3.447$	$1.238$ – $1.392$	Tab. 31 , GMM-8D row, both estimate weights.

Appendix

Table 2: Range construction for the RQ1 cross-scale summary in Fig. 1 . Each range is the minimum and maximum over the specified fixed-target audit entries; lower is better.

Data Types	Quantity Types	Quantity Values	Sources
(a) 1D	$e^{(1)}_{s}$ / $e^{(\mathrm{sc})}_{s}$	$0.800/0.223$	Means over the $w=1$ estimate rows and over the $w=w_{\mathrm{sc}}$ estimate rows in Tab. 7 .
(a) 2D GMM+banana	$e^{(1)}_{s}$ / $e^{(\mathrm{sc})}_{s}$	$1.294/0.377$	Means of the GMM and banana rows in Tab. 19 ; two moons is excluded because its boundary term is the intended caveat.
(a) 32D GMM	$e^{(1)}_{s}$ / $e^{(\mathrm{sc})}_{s}$	$19.402/3.012$	GMM-32D row in Tab. 31 .
(b) 1D	$R_{s}$ / $\rho_{s}$	$0.995/7.900$	CFM-only ratio row in Tab. 9 .
(b) 2D GMM	$R_{s}$ / $\rho_{s}$	$0.981/1.570$	GMM ordinary-CFM row in Tab. 21 .
(b) 32D GMM	$R_{s}$ / $\rho_{s}$	$0.910/1382.955$	GMM-32D ordinary-CFM row in Tab. 33 ; $\rho_{s}$ is source-relative, and the round-old MAE is $2.617$ .

Appendix

Table 3: Construction for the RQ2 cross-scale summary in Fig. 2 . Panel (a) reports CFM-only NLL MAE, where lower is better. Panel (b) reports final reward and clean-ratio MAE for the same on-policy run.

\RaggedRight Panel / layer	\RaggedRight Plotted value	\RaggedRight Source entries
\RaggedRight 1D EMA sweep	\RaggedRight Success rates: EMA-0 ( $2/2$ ), EMA-0.5 ( $0/2$ ), EMA-1 ( $0/2$ )	\RaggedRight EMA-decay row in Tab. 10 ; passing means final sampled reward at least $0.90$ .
\RaggedRight 2D EMA sweep	\RaggedRight Success rates: EMA-0 ( $4/6$ ), EMA-0.5 ( $0/6$ ), EMA-0.9 ( $0/6$ ), EMA-1 ( $0/6$ )	\RaggedRight EMA-decay row in Tab. 22 , aggregated over GMM, banana, and two moons.
\RaggedRight High-D EMA sweep	\RaggedRight Success rates: EMA-0 ( $4/8$ ), EMA-0.5 ( $7/8$ ), EMA-0.9 ( $6/8$ ), EMA-1 ( $0/8$ )	\RaggedRight S1 row in Tab. 34 , aggregated over GMM-4D, 8D, 16D, and 32D.
\RaggedRight Raw-image EMA sweep	\RaggedRight Success rates: EMA-0 ( $2/2$ ), EMA-0.5 ( $2/2$ ), EMA-0.9 ( $0/2$ ), EMA-1 ( $0/2$ )	\RaggedRight S1 row in Tab. 38 , aggregated over MNIST and CIFAR-10.

Appendix

Table 4: Construction for the RQ3 cross-scale summary in Fig. 3 . Every panel reports the fraction of tested S1 configurations passing the common final sampled reward gate

R\geq 0.90

; higher is better.

SETTING	TRAINING	FORMAL AUDIT	HIGH AUDIT
Protocol
Target / seed	Three-component 1D GMM / 13
Path / $\varepsilon$	Linear Gaussian / 0.10
Train weights	$\{1,w_{\mathrm{sc}}\}$	–	–
Readout weights	–	$\{1,w_{\mathrm{sc}}\}$	$\{1,w_{\mathrm{sc}}\}$
Checkpoint scope	10 fractional	All saved	Final only

Appendix

Table 5: 1D off-policy settings for RQ1 and RQ2.

SETTING	ORDINARY	SCORE-CALIBRATED
Protocol
Target / seed	Three-component 1D GMM / 13
Source checkpoint	Fixed-target $w=1$ , final
CFM-ratio weight	$w=1$	$w=w_{\mathrm{sc}}$
Audit readout weights	$\{1,w_{\mathrm{sc}}\}$
On-policy Optimization

Appendix

Table 6: 1D on-policy settings for RQ1 and RQ2.

Train weight	Estimate weight	$\mathrm{MAE}(\mathrm{NLL},\mathcal{J}_{w}+H)\downarrow$	$\mathrm{MAE}(\ell_{\varepsilon},\mathcal{J}_{w}+\mathcal{G}+H)\downarrow$	Density $L_{1}\downarrow$
$w=1$	$w=1$	0.803	0.234	0.036
$w=1$	$w=w_{\mathrm{sc}}$	0.224	0.199	0.036
$w=w_{\mathrm{sc}}$	$w=1$	0.797	0.234	0.035
$w=w_{\mathrm{sc}}$	$w=w_{\mathrm{sc}}$	0.222	0.198	0.035

Appendix

Table 7: Empirical results of off-policy training on the 1D GMM. Lower is better for all metrics. Density

L_{1}

is the grid-normalized

L_{1}

distance

\sum_{g}|\hat{p}_{\theta}(g)-p_{\mathrm{data}}(g)|\Delta g

between the CNF endpoint density and the target density on the evaluation grid. The

J+H

column is the CFM estimate used for the off-policy table claim. The closure column is

\mathrm{MAE}(\ell_{\varepsilon},\mathcal{J}_{w}+\mathcal{G}+H)

; the displayed endpoint form is algebraically equal because

\mathcal{B}_{\varepsilon}=\mathrm{NLL}-\ell_{\varepsilon}

Train Variants	Estimates	$\mathrm{MAE}(\mathrm{NLL},\mathcal{J}_{w}+H)\downarrow$	$\mathrm{MAE}(\mathrm{NLL},\mathcal{J}_{w}+\mathcal{G}+\mathcal{B}+H)\downarrow$	$\|G\|$	$\|B\|$
CFM-only ratio ( $w=1$ )	$w=1$	1.928	0.117	0.418	2.210
CFM-only ratio ( $w=1$ )	$w=w_{\mathrm{sc}}$	2.015	0.112	0.251	2.232
Score-calibrated ratio ( $w=w_{\mathrm{sc}}$ )	$w=1$	2.020	0.118	0.370	2.270
Score-calibrated ratio ( $w=w_{\mathrm{sc}}$ )	$w=w_{\mathrm{sc}}$	2.045	0.117	0.303	2.276

Appendix

Table 8: Numerical audit on the 1D GMM at the final on-policy checkpoints. The

J+H

column reports CFM-only endpoint-estimation error. The closure column reports

\mathrm{MAE}(\ell_{\varepsilon},J+G+H)

, written in its algebraically equivalent endpoint form in the header. Lower is better.

Panel of Figure 4 — Figure 4: Off-policy 1D evaluation summary. Panel (a) compares the final CFM endpoint estimate $\mathcal{J}_{w}+H$ with smoothed-identity closure $\mathcal{J}_{w}+\mathcal{G}_{\varepsilon,w}+H$ against $\ell_{\varepsilon}$ . Panel (b) tracks $\mathrm{MAE}(\mathrm{NLL},\mathcal{J}_{w}+H)$ across saved checkpoints, with density $L_{1}$ on the secondary axis.

Variant	Final reward $\uparrow$	Best reward $\uparrow$	Source MAE $\downarrow$	Round-old MAE $\downarrow$
CFM-only ratio	0.995	1.000	7.900	0.468
Score-calibrated ratio	0.998	1.000	65.269	0.102

Appendix

Table 9: Formal 1D on-policy baseline evaluation. Ratio MAE is reported with the score-calibrated estimate. For the rollout sample set

S

, the reported ratio error is

\mathrm{MAE}(\theta_{\mathrm{ref}})=|S|^{-1}\sum_{x_{0}\in S}|\Delta\ell_{\mathrm{clean}}^{\theta/\theta_{\mathrm{ref}}}(x_{0})-\widehat{\Delta}_{\theta,\theta_{\mathrm{ref}}}^{\mathrm{CFM}}(x_{0})|

. Source MAE sets

\theta_{\mathrm{ref}}=\theta_{\mathrm{src}}

, the fixed-target source checkpoint; round-old MAE sets

\theta_{\mathrm{ref}}=\theta_{\mathrm{old}}^{\mathrm{round}}

, the per-round EMA old-policy checkpoint.

Sweep	Settings	$\mathrm{EMA}=0$	$\mathrm{EMA}=0.5$	$\mathrm{EMA}=1$	Mechanism conclusion
S1: EMA distance	$w\in\{1,w_{\mathrm{sc}}\}$	0.994; 2/2	0.761; 0/2	0.275; 0/2	EMA $=0$ perform best; EMA $>0$ fail.
S2: ratio weight	$w\in\{1,w_{\mathrm{sc}}\}$	0.994; 2/2	0.769; 0/2	0.269; 0/2	Different train weights cannot be helpful in EMA $>0$ .
S3: ratio clip $\delta$	$\delta\in\{10^{-5},0.05,0.2,\mathrm{none}\}$ $\times$ $w\in\{1,w_{\mathrm{sc}}\}$	0.995; 7/8	0.998; 5/8	0.976; 2/8	The ratio clipping rescues some successful EMA $>0$ experiments, but the effect is non-monotonic.
S4: advantage clip $a$	$a\in\{0.5,1,2,5\}$ $\times$ $w\in\{1,w_{\mathrm{sc}}\}$	0.996; 8/8	0.850; 0/8	0.290; 0/8	Advantage clipping can perform well only with EMA $=0$ .
S5: KL beta $\beta$	$\beta\in\{0,10^{-4},10^{-3},10^{-2}\}$ $\times$ $w\in\{1,w_{\mathrm{sc}}\}$	0.996; 8/8	0.800; 0/8	0.282; 0/8	Source-KL regularization can perform well only with EMA $=0$ .
S6: EMA-KL beta $\beta_{\mathrm{ema}}$	$\beta_{\mathrm{ema}}\in\{0,10^{-4},10^{-3},10^{-2}\}$ $\times$ $w\in\{1,w_{\mathrm{sc}}\}$	0.995; 8/8	0.881; 0/8	0.282; 0/8	EMA-KL regularization can perform well only with EMA $=0$ .

Appendix

Table 10: Controlled 1D sweep for on-policy CFM-ratio training. Each cell reports “best final reward; passing configurations / tested configurations”, where a configuration passes when its final sampled reward reaches the pre-specified gate

0.90

. These fractions are not repeated-seed success probabilities. Each row states the controlled settings explicitly; all non-controlled settings use the baseline values: ratio clip

10^{-5}

, advantage clip

a=5

, and zero KL/EMA-KL.

EMA decay	Ordinary $(w=1)$ reward $\uparrow$	Score-cal. $(w=w_{\mathrm{sc}})$ reward $\uparrow$
0	0.994	0.949
0.5	0.377	0.761
1	0.238	0.275

Appendix

Table 11: Detailed S1 results for old-policy EMA distance. Final reward is reported for each CFM-ratio variant; higher is better.

EMA decay	Ordinary $(w=1)$ reward $\uparrow$	Score-cal. $(w=w_{\mathrm{sc}})$ reward $\uparrow$
0	0.994	0.994
0.5	0.352	0.769
1	0.263	0.269

Appendix

Table 12: Detailed S2 results for the CFM ratio weight. Final reward is reported for each CFM-ratio variant; higher is better.

	EMA $=0$		EMA $=0.5$		EMA $=1$
Ratio clip $\delta$	Ord. $\uparrow$	Score $\uparrow$	Ord. $\uparrow$	Score $\uparrow$	Ord. $\uparrow$	Score $\uparrow$
$10^{-5}$	0.994	0.949	0.400	0.776	0.247	0.282
0.05	0.989	0.980	0.951	0.976	0.284	0.283
0.2	0.995	0.971	0.986	0.998	0.393	0.416
None	0.769	0.979	0.733	0.984	0.976	0.914

Appendix

Table 13: Detailed S3 ratio-clip sweep. Entries are final rewards for ordinary (Ord.) and score-calibrated (Score) CFM-ratio variants; higher is better.

	EMA $=0$		EMA $=0.5$		EMA $=1$
Adv. clip $a$	Ord. $\uparrow$	Score $\uparrow$	Ord. $\uparrow$	Score $\uparrow$	Ord. $\uparrow$	Score $\uparrow$
0.5	0.990	0.993	0.380	0.788	0.252	0.268
1	0.987	0.995	0.364	0.850	0.263	0.260
2	0.992	0.994	0.390	0.843	0.243	0.262
5	0.988	0.996	0.360	0.744	0.290	0.267

Appendix

Table 14: Detailed S4 advantage-clip sweep. Entries are final rewards for ordinary (Ord., i.e.

w=1

) and score-calibrated (Score, i.e.

w=w_{sc}

) CFM-ratio variants; higher is better.

	EMA $=0$		EMA $=0.5$		EMA $=1$
Source-KL $\beta$	Ord. $\uparrow$	Score $\uparrow$	Ord. $\uparrow$	Score $\uparrow$	Ord. $\uparrow$	Score $\uparrow$
0	0.994	0.949	0.400	0.776	0.247	0.282
$10^{-4}$	0.991	0.996	0.364	0.632	0.262	0.250
$10^{-3}$	0.985	0.995	0.386	0.800	0.235	0.252
$10^{-2}$	0.965	0.986	0.357	0.535	0.250	0.252

Appendix

Table 15: Detailed S5 source-KL sweep. Entries are final rewards for ordinary (Ord., i.e.

w=1

) and score-calibrated (Score,

w=w_{sc}

) CFM-ratio variants; higher is better.

	EMA $=0$		EMA $=0.5$		EMA $=1$
EMA-KL $\beta_{\mathrm{ema}}$	Ord. $\uparrow$	Score $\uparrow$	Ord. $\uparrow$	Score $\uparrow$	Ord. $\uparrow$	Score $\uparrow$
0	0.994	0.949	0.400	0.776	0.247	0.282
$10^{-4}$	0.991	0.995	0.355	0.777	0.262	0.250
$10^{-3}$	0.989	0.995	0.392	0.881	0.235	0.252
$10^{-2}$	0.980	0.987	0.349	0.716	0.250	0.252

Appendix

Table 16: Detailed S6 EMA-KL sweep. Entries are final rewards for ordinary (Ord., i.e.

w=1

) and score-calibrated (Score, i.e.

w=w_{sc}

) CFM-ratio variants; higher is better.

SETTING	TRAINING	FORMAL AUDIT	HIGH AUDIT
Protocol
Targets	GMM, banana, and two moons
Seeds	13, 17, and 23
Path / $\varepsilon$	Linear Gaussian / 0.10
Train weights	$\{1,w_{\mathrm{sc}}\}$	–	–
Readout weights	–	$\{1,w_{\mathrm{sc}}\}$	$\{1,w_{\mathrm{sc}}\}$

Appendix

Table 17: 2D off-policy settings for RQ1 and RQ2. The formal audit covers the saved-checkpoint trajectory, whereas the high-budget audit evaluates only the final checkpoint.

SETTING	ORDINARY	SCORE-CALIBRATED
Protocol
Targets / seed	GMM, banana, and two moons / 13
Source checkpoint	Corresponding fixed-target $w=1$ , final
CFM-ratio weight	$w=1$	$w=w_{\mathrm{sc}}$
Audit readout weights	$\{1,w_{\mathrm{sc}}\}$
On-policy Optimization

Appendix

Table 18: 2D on-policy settings for RQ1 and RQ2. Ordinary and score-calibrated denote the CFM-ratio readout used by GRPO.

Dataset	Density $L_{1}\downarrow$	$\mathrm{MAE}(\mathrm{NLL},\mathcal{J}_{w}+H)\downarrow$		$\mathrm{MAE}(\ell_{\varepsilon},\mathcal{J}_{w}+\mathcal{G}+H)\downarrow$
		$w=1$ Estimate	$w=w_{\mathrm{sc}}$ Estimate	$w=1$ Estimate	$w=w_{\mathrm{sc}}$ Estimate
GMM	0.076	1.001	0.411	0.397	0.387
Banana	0.043	1.587	0.342	0.323	0.312
Two moons	0.326	0.580	0.895	0.311	0.302

Appendix

Table 19: High-budget off-policy 2D evaluation. Entries are means over seeds 13, 17, and 23 and over the two train weights. Lower is better. Density

L_{1}

is the grid-normalized

L_{1}

distance between the CNF endpoint density and the target density. The

J+H

columns report endpoint-estimation error, while the closure columns report

\mathrm{MAE}(\ell_{\varepsilon},\mathcal{J}_{w}+\mathcal{G}+H)

Figure 5: Off-policy 2D checkpoint trajectories, averaged over seeds 13, 17, and 23. The left axis reports $\mathrm{MAE}(\mathrm{NLL},\mathcal{J}_{w}+H)$ ; color denotes the readout weight and line style denotes the fixed-target training weight. Gray square curves report density $L_{1}$ on the secondary axes, which are scaled separately for each target family.

Dataset	Train weight	$\mathrm{MAE}(\mathrm{NLL},\mathcal{J}_{w}+H)\downarrow$		$\mathrm{MAE}(\ell_{\varepsilon},\mathcal{J}_{w}+\mathcal{G}+H)\downarrow$
		$w=1$ Estimate	$w=w_{\mathrm{sc}}$ Estimate	$w=1$ Estimate	$w=w_{\mathrm{sc}}$ Estimate
GMM	$w=1$	2.069	2.057	0.250	0.231
GMM	$w=w_{\mathrm{sc}}$	49.054	206.343	0.841	0.836
Banana	$w=1$	1.374	1.415	0.214	0.230
Banana	$w=w_{\mathrm{sc}}$	2.331	2.396	0.378	0.389
Two moons	$w=1$	3.132	3.183	0.362	0.349

Appendix

Table 20: Final 2D numerical audit for the on-policy experiments.

\mathcal{J}_{w}+H

denotes the CFM-only endpoint estimate. The closure columns report

\mathrm{MAE}(\ell_{\varepsilon},\mathcal{J}_{w}+\mathcal{G}+H)

Dataset	Ordinary reward $\uparrow$	Score-calibrated reward $\uparrow$	Ordinary ratio MAE $\downarrow$		Score-calibrated ratio MAE $\downarrow$
			Source	Round-old	Source	Round-old
GMM	0.981	0.713	1.570	0.046	138.296	1.209
Banana	0.967	0.986	2.068	0.059	2.444	0.082
Two moons	0.979	0.934	12.197	0.666	3.015	0.054

Appendix

Table 21: Verification of the on-policy and after-optimization conclusions in Tab. 1 for 2D settings. "Oridinary" means

w=1

, "Score-calibrated" means

w=w_{\mathrm{sc}}

. "Source" means the ratio is estimated on the source, "Round-old" means the ratio is estimated on the old policy.

Sweep	Settings	$\mathrm{EMA}=0$	$\mathrm{EMA}=0.5$	$\mathrm{EMA}=0.9$	$\mathrm{EMA}=1$	Mechanism conclusion
S1: EMA distance	$w\in\{1,w_{\mathrm{sc}}\}$	0.984; 4/6	0.388; 0/6	0.327; 0/6	0.289; 0/6	Only EMA $=0$ reaches the reward gate 0.9; small EMA decay rate is the primary condition.
S2: ratio weight	$w\in\{1,w_{\mathrm{sc}}\}$	0.984; 5/6	0.381; 0/6	0.326; 0/6	0.294; 0/6	Train weight choice changes one EMA-0 outcome but does not repair experiments with large EMA decay rate.
S3: ratio clip $\delta$	$\delta\in\{10^{-5},0.05,0.2,\mathrm{none}\}$ $\times$ $w\in\{1,w_{\mathrm{sc}}\}$	0.999; 21/24	0.998; 17/24	0.998; 11/24	0.985; 4/24	This is the only sweep with EMA $>0$ successes, but the effect is non-monotonic and EMA-1 successes use no clip.
S4: advantage clip $a$	$a\in\{0.5,1,2,5\}$ $\times$ $w\in\{1,w_{\mathrm{sc}}\}$	0.985; 20/24	0.410; 0/24	0.328; 0/24	0.295; 0/24	Advantage clipping succeeds only with EMA $=0$ and cannot rescue experiments with large EMA decay rate.
S5: source KL $\beta$	$\beta\in\{0,10^{-4},10^{-3},10^{-2}\}$ $\times$ $w\in\{1,w_{\mathrm{sc}}\}$	0.993; 17/24	0.468; 0/24	0.346; 0/24	0.302; 0/24	Positive source KL penalties avoid the three $\beta=0$ non-finite failures, but do not rescue experiments with large EMA decay rate.
S6: EMA KL $\beta_{\mathrm{old}}$	$\beta_{\mathrm{old}}\in\{0,10^{-4},10^{-3},10^{-2}\}$ $\times$ $w\in\{1,w_{\mathrm{sc}}\}$	0.985; 19/24	0.408; 0/24	0.328; 0/24	0.297; 0/24	EMA-KL regularization also succeeds only with EMA $=0$ in this sweep.

Appendix

Table 22: Controlled 2D sweep for on-policy CFM-ratio training. Each cell reports “best final reward; passing configurations / tested configurations” aggregated over the three 2D datasets at seed 13. These fractions are not repeated-seed success probabilities.

Dataset	EMA $=0$	EMA $=0.5$	EMA $=0.9$	EMA $=1$
GMM	0.982 / 0.804	0.229 / 0.292	0.187 / 0.211	0.161 / 0.170
Banana	0.970 / 0.984	0.350 / 0.359	0.287 / 0.300	0.265 / 0.271
Two moons	0.983 / 0.899	0.349 / 0.388	0.305 / 0.327	0.277 / 0.289

Appendix

Table 23: Detailed S1 old-policy EMA-decay sweep. Each entry reports ordinary / score-calibrated train weight final reward; higher is better. EMA decay specifies the update rule and is not a measured policy distance.

Dataset	EMA $=0$	EMA $=0.5$	EMA $=0.9$	EMA $=1$
GMM	0.982 / 0.779	0.238 / 0.288	0.182 / 0.195	0.168 / 0.164
Banana	0.970 / 0.984	0.348 / 0.366	0.285 / 0.299	0.267 / 0.262
Two moons	0.983 / 0.941	0.371 / 0.381	0.307 / 0.326	0.283 / 0.294

Appendix

Table 24: Detailed S2 ratio-weight sweep. Each entry reports ordinary / score-calibrated train weight final reward; higher is better.

Dataset	Ratio clip $\delta$	EMA $=0$	EMA $=0.5$	EMA $=0.9$	EMA $=1$
GMM	$10^{-5}$	0.982 / 0.804	0.224 / 0.284	0.197 / 0.202	0.168 / 0.165
	0.05	0.995 / 0.945	0.985 / 0.976	0.339 / 0.434	0.187 / 0.194
	0.2	0.995 / 0.948	0.997 / 0.976	0.882 / 0.929	0.254 / 0.267
	None	0.995 / 0.997	0.997 / 0.998	0.995 / 0.972	0.712 / 0.861
Banana	$10^{-5}$	0.970 / 0.984	0.346 / 0.358	0.292 / 0.291	0.269 / 0.270
	0.05	0.995 / 0.997	0.991 / 0.992	0.726 / 0.636	0.316 / 0.317

Appendix

Table 25: Detailed S3 ratio-clip sweep. Each entry reports ordinary / score-calibrated train weight final reward; higher is better.

Dataset	Advantage clip $a$	EMA $=0$	EMA $=0.5$	EMA $=0.9$	EMA $=1$
GMM	0.5	0.962 / 0.806	0.238 / 0.278	0.200 / 0.195	0.165 / 0.165
	1	0.976 / 0.766	0.234 / 0.275	0.189 / 0.204	0.163 / 0.166
	2	0.981 / 0.764	0.237 / 0.273	0.191 / 0.203	0.167 / 0.169
	5	0.981 / 0.754	0.215 / 0.285	0.181 / 0.217	0.165 / 0.165
Banana	0.5	0.939 / 0.981	0.346 / 0.336	0.292 / 0.282	0.268 / 0.268
	1	0.959 / 0.983	0.342 / 0.351	0.283 / 0.295	0.266 / 0.268

Appendix

Table 26: Detailed S4 advantage-clip sweep. Each entry reports ordinary / score-calibrated train weight final reward; higher is better.

Dataset	Source VF $\beta$	EMA $=0$	EMA $=0.5$	EMA $=0.9$	EMA $=1$
GMM	0	0.986 / NF	0.219 / 0.311	0.198 / 0.212	0.168 / 0.168
	$10^{-4}$	0.986 / 0.793	0.215 / 0.322	0.192 / 0.216	0.164 / 0.169
	$10^{-3}$	0.986 / 0.779	0.235 / 0.290	0.195 / 0.214	0.169 / 0.174
	$10^{-2}$	0.981 / 0.754	0.215 / 0.285	0.181 / 0.217	0.165 / 0.165
Banana	0	0.978 / 0.993	0.337 / 0.385	0.295 / 0.297	0.270 / 0.270
	$10^{-4}$	0.976 / 0.993	0.342 / 0.379	0.288 / 0.299	0.267 / 0.270

Appendix

Table 27: Detailed S5 source KL regularization sweep. Each entry reports ordinary / score-calibrated train weight final reward; higher is better.

Dataset	Round-old VF $\beta_{\mathrm{old}}$	EMA $=0$	EMA $=0.5$	EMA $=0.9$	EMA $=1$
GMM	0	0.982 / 0.804	0.224 / 0.284	0.197 / 0.202	0.168 / 0.165
	$10^{-4}$	0.980 / 0.774	0.225 / 0.282	0.191 / 0.206	0.163 / 0.167
	$10^{-3}$	0.981 / 0.775	0.236 / 0.274	0.192 / 0.204	0.167 / 0.170
	$10^{-2}$	0.981 / 0.733	0.215 / 0.278	0.183 / 0.214	0.164 / 0.163
Banana	0	0.970 / 0.984	0.346 / 0.358	0.292 / 0.291	0.269 / 0.270
	$10^{-4}$	0.969 / 0.984	0.351 / 0.364	0.284 / 0.290	0.267 / 0.270

Appendix

Table 28: Detailed S6 EMA-KL regularization sweep. Each entry reports ordinary / score-calibrated train weight final reward; higher is better.

SETTING	TRAINING	FORMAL AUDIT	HIGH AUDIT
Protocol
Targets / seed	8-component GMMs in 4D, 8D, 16D, and 32D / 13
Path / $\varepsilon$	Linear Gaussian / 0.10
Train weights	$\{1,w_{\mathrm{sc}}\}$	–	–
Readout weights	–	$\{1,w_{\mathrm{sc}}\}$	$\{1,w_{\mathrm{sc}}\}$
Checkpoint scope	10 fractional	All saved	Final only

Appendix

Table 29: High-D off-policy settings for RQ1 and RQ2. The formal audit covers the saved-checkpoint trajectory, whereas the high-budget audit evaluates only the final checkpoint.

SETTING	ORDINARY	SCORE-CALIBRATED
Protocol
Targets / seed	GMM-4D, GMM-8D, GMM-16D, and GMM-32D / 13
Source checkpoint	Corresponding fixed-target $w=1$ , final
CFM-ratio weight	$w=1$	$w=w_{\mathrm{sc}}$
Audit readout weights	$\{1,w_{\mathrm{sc}}\}$
On-policy Optimization

Appendix

Table 30: High-D on-policy settings for RQ1 and RQ2. Ordinary and score-calibrated denote the CFM-ratio readout used by GRPO.

Dataset	Train weight	MMD $\downarrow$	$\mathrm{MAE}(\mathrm{NLL},\mathcal{J}_{w}+H)\downarrow$		$\mathrm{MAE}(\ell_{\varepsilon},\mathcal{J}_{w}+\mathcal{G}+H)\downarrow$
			$w=1$	$w=w_{\mathrm{sc}}$	$w=1$	$w=w_{\mathrm{sc}}$
GMM-4D	$w=1$	0.028	2.001	1.081	1.021	1.028
	$w=w_{\mathrm{sc}}$	0.026	1.967	1.088	1.000	1.033
GMM-8D	$w=1$	0.033	3.447	1.387	1.392	1.238
	$w=w_{\mathrm{sc}}$	0.037	3.485	1.386	1.392	1.226
GMM-16D	$w=1$	0.023	8.108	2.000	1.880	1.711

Appendix

Table 31: High-dimensional off-policy results at seed 13. Each checkpoint is evaluated with 128 points, 16 Monte Carlo samples per point, a 21-point time grid, and 160 ODE steps. The sample-quality column is an RBF-MMD proxy. Lower is better for all metrics. The final two columns report smoothed-identity closure

\mathrm{MAE}(\ell_{\varepsilon},\mathcal{J}_{w}+\mathcal{G}+H)

Dataset	Train weight	$w=1$ estimate		$w=w_{\mathrm{sc}}$ estimate
		$\mathcal{J}_{w}+H$	Closure	$\mathcal{J}_{w}+H$	Closure
GMM-4D	$w=1$	8.586	0.758	8.578	2.548
	$w=w_{\mathrm{sc}}$	9.946	2.944	9.461	2.191
GMM-8D	$w=1$	11.641	1.011	10.968	0.913
	$w=w_{\mathrm{sc}}$	$1.056{\times}10^{3}$	8.535	$4.324{\times}10^{3}$	6.843
GMM-16D	$w=1$	13.643	2.480	14.900	2.187

Appendix

Table 32: High-dimensional on-policy numerical audit at seed 13.

\mathcal{J}_{w}+H

is the CFM-only endpoint MAE; “Closure” is

\mathrm{MAE}(\ell_{\varepsilon},\mathcal{J}_{w}+\mathcal{G}+H)

. Lower is better.

Figure 6: High-D off-policy checkpoint trajectories at seed 13. The left axis reports the $\mathrm{MAE}(\mathrm{NLL},\mathcal{J}_{w}+H)$ ; color denotes the readout weight and line style denotes the off-policy training weight. Gray square curves report the RBF-MMD sample-quality proxy on the secondary axes, whose ranges are scaled separately by dimension. Each checkpoint is evaluated on samples generated by that checkpoint, so the curves are self-sample audits rather than fixed-test-set learning curves.

Dataset	CFM final reward $\uparrow$	SC-CFM final reward $\uparrow$	CFM source MAE $\downarrow$	CFM round-old MAE $\downarrow$
GMM-4D	0.997	0.998	9.454	0.815
GMM-8D	0.995	0.643	13.911	0.746
GMM-16D	0.991	0.481	18.298	1.574
GMM-32D	0.910	0.209	1382.955	2.617

Appendix

Table 33: High-dimensional on-policy RQ2 results at seed 13 after 1,000 rounds. Final reward uses 4,096 generated samples. Ratio MAE uses the ordinary CFM-ratio variant with

w=1

readout at the final checkpoint; “source” compares with the fixed-target source checkpoint and “round-old” with the current round’s old policy on the same rollout samples. Bold reward values pass the 0.90 gate.

Sweep	Settings	$\mathrm{EMA}=0$	$\mathrm{EMA}=0.5$	$\mathrm{EMA}=0.9$	$\mathrm{EMA}=1$
S1: EMA distance	$w\in\{1,w_{\mathrm{sc}}\}$	0.998; 4/8	0.995; 7/8	0.985; 6/8	0.113; 0/8
S3: 32D ratio clip $\delta$	$\delta\in\{10^{-5},0.05,0.2,\mathrm{none}\}$ $\times$ $w\in\{1,w_{\mathrm{sc}}\}$	0.910; 1/8	0.982; 3/8	0.977; 2/8	0.459; 0/8

Appendix

Table 34: Controlled High-D sweep for on-policy CFM-ratio training. Each cell reports “best final reward; passing configurations / tested configurations” at seed 13. Passing means final sampled reward at least

0.90

; the fractions are not repeated-seed success probabilities.

Dataset	EMA $=0$	EMA $=0.5$	EMA $=0.9$	EMA $=1$
GMM-4D	0.997 / 0.998	0.986 / 0.967	0.985 / 0.926	0.098 / 0.094
GMM-8D	0.996 / 0.464	0.995 / 0.969	0.979 / 0.914	0.080 / 0.090
GMM-16D	0.992 / 0.377	0.991 / 0.952	0.959 / 0.892	0.074 / 0.093
GMM-32D	0.590 / 0.188	0.971 / 0.083	0.951 / 0.211	0.078 / 0.113

Appendix

Table 35: Detailed High-D S1 old-policy EMA-decay sweep. Each entry reports ordinary / score-calibrated train-weight final reward at seed 13; higher is better. Cells use the setting-dependent random streams of the screening run.

Dataset	Ratio clip $\delta$	EMA $=0$	EMA $=0.5$	EMA $=0.9$	EMA $=1$
GMM-4D	$10^{-5}$	0.905 / 0.649	0.480 / 0.506	0.168 / 0.238	0.082 / 0.085
	0.05	0.997 / 0.999	0.987 / 0.969	0.980 / 0.930	0.101 / 0.092
	0.2	0.996 / 0.998	0.996 / 0.999	0.990 / 0.962	0.118 / 0.100
	None	0.996 / 0.998	0.996 / 0.998	0.997 / 0.995	0.379 / 0.645
GMM-8D	$10^{-5}$	0.993 / 0.752	0.701 / 0.647	0.216 / 0.385	0.078 / 0.091
	0.05	0.996 / 0.486	0.996 / 0.964	0.973 / 0.917	0.076 / 0.089

Appendix

Table 36: Detailed High-D S3 ratio-clip sweep. Each entry reports ordinary / score-calibrated train-weight final reward at seed 13; higher is better.

SETTING	SHARED	MNIST	CIFAR-10
Data and source model
Input / dimension	Raw pixels	$1\!\times\!28\!\times\!28$ / 784	$3\!\times\!32\!\times\!32$ / 3,072
Data used for source CFM	Full class mixture	Train and test splits	Train split
Velocity field	U-Net	Two resolution levels	Four resolution levels with attention
Model range	–	$[0,1]$	$[-1,1]$
Path / $\varepsilon$	Linear Gaussian / $10^{-4}$	–	–

Appendix

Table 37: Raw-image settings for RQ3. “Source checkpoint” denotes the off-policy checkpoint loaded by every on-policy run. Rollout and final evaluation sample counts are global across eight GPUs.

Sweep	EMA $=0$	EMA $=0.5$	EMA $=0.9$	EMA $=1$
S1: EMA distance	1.000 ; 2/2	0.984 ; 2/2	0.818; 0/2	0.541; 0/2
S3: Ratio clip	1.000 ; 6/8	1.000 ; 6/8	1.000 ; 3/8	0.992 ; 1/8

Appendix

Table 38: Raw-image RQ3 mechanism summary at seed 13. Each cell is “best final reward; successful settings / tested settings.” The success gate is final sampled reward mean

\geq 0.90

Dataset	EMA $=0$	EMA $=0.5$	EMA $=0.9$	EMA $=1$
MNIST	1.000	0.984	0.818	0.541
CIFAR-10	1.000	0.942	0.799	0.272

Appendix

Table 39: Detailed raw-image S1 EMA-decay sweep. Entries are final-checkpoint sampled reward means for ordinary CFM-ratio training at seed 13; higher is better. Bold values meet the

0.90

reward gate.

Dataset	Ratio clip $\delta$	EMA $=0$	EMA $=0.5$	EMA $=0.9$	EMA $=1$
MNIST	$10^{-5}$	1.000	0.933	0.738	0.702
	0.05	1.000	0.984	0.742	0.571
	0.2	1.000	1.000	0.847	0.812
	None	1.000	1.000	1.000	0.992
CIFAR-10	$10^{-5}$	1.000	0.815	0.918	0.673
	0.05	0.952	0.799	0.775	0.769

Appendix

Table 40: Detailed raw-image S3 ratio-clip sweep. Entries are final-checkpoint sampled reward means for ordinary CFM-ratio training at seed 13; higher is better. Bold values meet the

0.90

reward gate based on the unrounded reward.

Flow Matching

Flow matching learns continuous-time vector fields that transport simple noise distributions into data for efficient generative modeling.

See all 658 papers

FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models
Sep 9, 2026Yansen Han, Shengyi Liao, Peng Sun +4Pairwise Preference Learning Flow Matching
Westlake University · Zhejiang University · Kling Team, Kuaishou Technology
Co-Evolving Paths and Flows via Path-Flow Alignment
Oct 6, 2026Zeyu Michael Li, William Xingxu Chen, Xiang ChengFlow Matching
Duke University
Compatible Likelihoods for Flow Matching on Manifolds
Oct 22, 2026Lucas Ng, Georgios Batzolis, Mark GirolamiFlow Matching Riemannian Flow Matching
University of Cambridge · Devotion AI Labs Ltd.

Sep 9, 2026J/K move · Enter open · S save

FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models

Yansen Han, Shengyi Liao, Peng Sun, Deyuan Liu+3

Westlake University · Zhejiang University · Kling Team, Kuaishou Technology

Pairwise Preference Learning Flow Matching Contrastive Flow Matching Flow-Based Generative Modeling Preference Alignment Offline Preference Optimization

Open paper AskPDF arXiv

Preference alignment for flow and diffusion models now spans online reinforcement learning and offline preference optimization, but the relation between these methods remains unclear. In particular, existing forward-process alignment methods require fresh samples from the current model, while offline methods based on fixed preference pairs rely primarily on positive-only fine-tuning or DPO-style likelihood-ratio surrogates. We organize these approaches through a divergence-based framework and introduce FlowCPO, an offline forward-KL objective that uses both preferred and dispreferred samples without online rollouts. For linear interpolation, we show under explicit regularity conditions that the forward-KL objective is bounded by a contrastive flow matching loss, yielding a tractable surrogate on fixed data. We further show that this loss is nonnegative, whereas the signed regression loss of simplified FlowDPO can be unbounded below. In the in-domain setting, FlowCPO achieves higher mean GenEval and OCR scores than the evaluated baselines, reaching 0.84 and 0.87 versus 0.81 and 0.74 for FlowDPO at CFG 3.0. In the out-of-domain setting, the results are mixed, with the best GenEval result but lower reward scores than RFT on several metrics.

Abstract

Flow Matching

Explore similar work

FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models

Co-Evolving Paths and Flows via Path-Flow Alignment

Compatible Likelihoods for Flow Matching on Manifolds