Dual-Grained Policy Optimization for Hallucination Reduction in Flow-Based RL

Xiaofeng Tan; Jun Liu; Yuanting Fan; Bin-Bin Gao; Xi Jiang; Xiaochen Chen; Jinlong Peng; Chengjie Wang

cs.CVFeb 3, 2026

Dual-Grained Policy Optimization for Hallucination Reduction in Flow-Based RL

Authors: Xiaofeng Tan, Jun Liu, Yuanting Fan, Bin-Bin Gao, Xi Jiang, Xiaochen Chen, Jinlong Peng, Chengjie Wang,

Organizations: Southeast University · Tencent Youtu Lab · Southern University of Science and Technology

Abstract

Reinforcement Fine-Tuning (RFT) on flow-based models is crucial for alignment. However, they often introduce visual hallucinations like over-optimized details and semantic misalignment. This work explains why visual hallucinations arise and shows how to reduce them. We first investigate RFT methods from a unified perspective and trace hallucinations to two mechanisms. Limited exploration from stochastic differential equation (SDE) rollouts yields detail-dominated within-group differences, leading to fine-grained optimization that over-emphasizes local details and neglects global semantics. Trajectory imitation writes reward-selected SDE noise into the velocity field, leading to inconsistent optimization that degrades cross-step consistency. Building on this, we propose DGPO, a general framework to mitigate hallucinations. Specifically, we design Dynamic Granularity Rollout (DGR) to balance global semantics and local details by dynamically scheduling noise sources. We then introduce a Consistent Policy Gradient Optimization that preserves the consistency by aligning the policy with a more stable prior. Experiments show that DGPO reduces low-/high-level hallucinations by 49%/38% and improves the average of out-of-domain (OoD) metrics by +4.9% on FLUX.1.dev, compared with the baseline's -0.7%. Project page: https://xiaofeng-tan.github.io/projects/ConsistentRFT/

Figures & tables

Figure 1: The optimization process of existing flow-based RFT methods. Existing methods only perform fine- or coarse-grained optimization, leading to visual hallucinations. Prompt: “A baseball player pitching a baseball on a field.”

Figure 2: Visual hallucinations in flow-based RFT. Fig. (d,e) show unreliable rewards on diverse/OoD samples encourage irrelevant details, and (e) shows that limited exploration can trap the model in reward-specific local optima.

Figure 3: Relationship between rollout exploration and policy optimization. Coarse/fine-grained groups from rollout steer the model toward coarse/fine-grained optimization. See App. A.5 and App. A.4 for analysis of rollout granularity and coarse perception.

Figure 4: Trajectory imitation toy example. Pre-training establishes near-straight ODE paths, whereas SDE-based RFT regresses onto stochastic transitions, bending the paths and reducing velocity consistency.

Figure 5: Overview of DGPO. It consists of Dynamic Granularity Rollout and Consistent Policy Gradient Optimization.

DDPO-Based Methods
Method	Reward	Human Preference			Aesthetic	Semantic		Comprehensive
Method	Reward	HPS	IR	PS	Aes.	CLIP	UR-S	UR	Avg.
Flux Dev	-	0.312	1.089	0.226	5.837	0.388	3.365	3.520	2.404
DDPO	HPS	0.313 +0.3%	1.129 +3.7%	0.227 +0.4%	5.896 +1.0%	0.391 +0.8%	3.387 +0.7%	3.568 +1.4%	2.433 +1.2%
+ DGPO	HPS	0.324 +3.8%	1.197 +9.9%	0.228 +0.9%	5.998 +2.8%	0.396 +2.1%	3.476 +3.3%	3.593 +2.1%	2.481 +3.2%
DPO-Based Methods

Table 1: Comparison with SoTA RFT methods. Gray text marks in-domain training metrics, and gray rows are ours. Gains are in blue and drops in red ; best results within each block are in bold . Relative changes are against Flux Dev. Avg. is the mean of the OoD metrics, excluding in-domain training rewards: HPS throughout, and also CLIP in the CLIP+HPS block. HPS: Human Preference Score v2.1; IR: ImageReward; PS: PickScore; Aes.: Aesthetic Predictor v2.5; CLIP: CLIP Score; UR-S: Unified Reward-Semantic; UR: Unified Reward.

Method	Low $\downarrow$				High $\downarrow$				Cons. $\downarrow$
Method	LV	HF	Edge	Nse.	Sh.	Irr.	Gr.	Avg.	LC
Flux Dev	977	26	61	0.27	1.00	0.87	0.12	0.66	0.30
DDPO	3895	68	71	0.78	2.48	2.52	0.61	1.87	0.32
w/ DGPO	1028	40	66	0.62	1.07	0.92	0.12	0.70	0.30
OnlineDPO	2752	44	62	0.61	2.43	2.20	0.58	1.74	0.32
w/ DGPO	1131	37	59	0.59	1.50	1.34	0.29	1.04	0.30

Table 2: Evaluation on visual hallucination. Gray rows are ours. LV: Laplacian Variance; HF: High-Frequency Energy; Nse.: Noise; Sh.: Sharpness; Irr.: Irrelevant Details; Gr.: Grid; LC: Latent Consistency.

Method	Reward	Human Preference			Aesthetic	Semantic		Comprehensive		Time / s
Method	Reward	HPS	IR	PS	Aes.	CLIP	UR-S	UR	Avg.	Time / s
SDXL ( Podell et al., 2023 )	-	0.292	0.945	0.226	5.735	0.418	3.275	3.425	2.337	9
Hunyuan-DiT ( Li et al., 2024 )	-	0.302	1.094	0.225	5.616	0.411	3.296	3.458	2.350	23
Kolors ( Kolors Team, 2024 )	-	0.312	0.974	0.225	6.000	0.386	3.296	3.376	2.376	31
SD-3.5-M ( Esser et al., 2024 )	-	0.302	1.130	0.227	5.659	0.411	3.350	3.580	2.393	14
SD-3.5-L ( Esser et al., 2024 )	-	0.303	1.143	0.228	5.853	0.410	3.330	3.627	2.432	17

Table 3: Comparison with SoTA pretrained models. Best results are in bold , second-best results are underlined , and gray rows are ours. Avg. is the mean of the six metrics excluding HPS.

Figure 6: Complementary rollout granularities. Independent initial noise yields greater global semantic variation in coarse-grained groups, whereas shared initial noise isolates local appearance variations in fine-grained groups.

Method	Granularity	Time / s	HPS	IR	PS	Aes.	CLIP	UR-S	UR	Avg.
Flux Dev	-	-	0.312	1.09	0.226	5.84	0.388	3.37	3.52	2.41
DanceGRPO	Fine	665	0.353	1.16	0.226	5.90	0.361	3.30	3.38	2.39
-	Coarse	665	0.326	1.14	0.226	5.93	0.375	3.32	3.48	2.41
+ Inter DGR	Dynamic	665	0.349	1.19	0.230	6.14	0.376	3.35	3.50	2.46
+ CPGO	Dynamic	665	0.348	1.30	0.230	6.20	0.384	3.41	3.62	2.52
+ Intra DGR	Dynamic	541	0.348	1.28	0.231	6.16	0.373	3.39	3.53	2.49

Table 4: Ablation study of DGPO components. Gray text marks in-domain HPS, and gray rows are the full DGPO variants. Avg. is the mean of OoD metrics excluding HPS. IR: ImageReward; PS: PickScore; Aes.: Aesthetic Predictor v2.5; UR-S: Unified Reward-Semantic; UR: Unified Reward.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	DanceGRPO	MixGRPO
Learning Rate	$1e{-5}$	$1e{-5}$
Batch Size	2	2
Grad Accum Steps	12	3
Num Generations	12	12
Sampling Steps	16	25
Resolution	720 $\times$ 720	720 $\times$ 720

Appendix

Table A1 : Hyperparameter configurations for DanceGRPO and MixGRPO. MixGRPO-specific parameters are marked with

\dagger

Figure A1 : Visualization of noised images and their coarse perceptions. Coarse progress perception facilitates recognition of image semantics and quality by the reward model at earlier steps with smaller $t$ .

Figure A2 : Reliability of Coarse Progress Perception. Intermediate coarse predictions are compared with fully denoised images using 12 candidates per prompt. Left: ImageReward ranking correlation. Right: CLIP feature cosine similarity.

Figure A3 : Diversity and contrast of fine- and coarse-grained groups. Left: diversity. Right: contrast. The fine-grained group isolates randomness from progressive in-trajectory noise, while the coarse-grained group includes randomness from both initial and progressive noise.

Method	Period	Ratio	HPS	IR	PS	Aes.	CLIP	UR-S	UR	Avg.
Flux Dev	-	-	0.312	1.09	0.226	5.84	0.388	3.37	3.52	2.41
DanceGRPO ( Xue et al., 2025 )	-	-	0.353	1.16	0.226	5.90	0.361	3.30	3.38	2.39
DGPO	20	0.25	0.351	1.28	0.231	6.16	0.378	3.38	3.57	2.50
DGPO	50	0.25	0.346	1.29	0.229	6.13	0.387	3.43	3.60	2.51
DGPO	40	0.25	0.348	1.30	0.230	6.20	0.384	3.41	3.62	2.52
DGPO	40	0.75	0.342	1.25	0.228	6.04	0.389	3.46	3.55	2.49

Appendix

Table A2 : Parameter sensitivity analysis. Gray text marks in-domain HPS, and gray rows use the default period of 40 and coarse ratio of 0.25. HPS: HPSv2.1; IR: ImageReward; PS: PickScore; Aes.: Aesthetic Predictor v2.5; UR: Unified Reward; UR-S: Unified Reward-Semantic.

Figure A4 : Effect of the CPGO weight $\omega$ on optimization. A larger weight, such as $10^{-5}$ , can cause saturation, while moderate values of $10^{-6}$ or less strike a better balance between constraint strength and optimization flexibility.

Figure A5 : Reference examples used in the VH-Evaluator prompt. Example 1 versus Example 2 contrasts irrelevant detail accumulation with moderate detailing. Example 1 versus Example 3 contrasts over-sharpened details with natural sharpening. These comparisons correspond to the “Example Comparison” section in the prompt.

Figure A6 : Qualitative results from the VH-Evaluator demonstrating detection of over-optimization hallucinations at varying levels.

Figure A7 : Enlarged view of blurred image. These results are reproduced using official model weights or implementation settings of existing methods, with 8 GPUs.

Figure A8 : Enlarged view of Fig. 1 from the Sec. Introduction.

Figure A9 : Qualitative comparison of coarse-, fine-, and dynamic-grained optimization.

RL for Flow Matching Models

See all 22 papers

ReFPO: Reflow Regularization for Flow Matching Policy Gradients
Jun 19, 2026Ge Wang, Yibo Peng, Fan Feng +10Policy Gradient Reinforcement Learning
Ising AI · CUHK-Shenzhen · FNii-Shenzhen
BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation
Jun 14, 2026Ning Li, Zixuan Guo, Yan Xu +7RL for Language Models LLM Hallucination Mitigation
1Shanghai Jiao Tong University · 2Tencent · 3The
Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models
Oct 11, 2025Mingyang Lyu, Yinqian Sun, Erliang Lin +4Flow Matching Robotic Manipulation
Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · Beijing Institute of AI Safety and Governance, China. · University of Chinese Academy of Sciences (UCAS), Beijing, China. +3

Jun 19, 2026J/K move · Enter open · S save

ReFPO: Reflow Regularization for Flow Matching Policy Gradients

Ge Wang, Yibo Peng, Fan Feng, Shenhao Yan+9

Ising AI · CUHK-Shenzhen · FNii-Shenzhen

Policy Gradient Reinforcement Learning RL for Generative Models RL for Flow Matching Models

Open paper AskPDF arXiv

We present Reflow-regularized Flow Matching Policy Gradients (ReFPO), a simple online RL method that adds explicit Reflow regularization to FPO for efficient flow-based control. We uncover a key structural property: the gradient updates in Flow Matching Policy Gradients (FPO) can be interpreted as an implicit advantage-weighted Reflow process, providing a new geometric perspective on flow-based policy gradients. Building on this insight, ReFPO introduces an explicit geometric regularizer that can be implemented with a single line of code change without incurring additional computational overhead or auxiliary distillation stages. By synergizing advantage-guided updates with path rectification, our method reduces CFM proxy-ratio spikes, stabilizes PPO-style training, and enables high-fidelity one-step inference that often matches or exceeds multi-step performance. We experimentally demonstrate that ReFPO improves average performance and discretization robustness across GridWorld, MuJoCo Playground, and high-dimensional Humanoid Control tasks, providing a scalable and stable approach for generative policies in complex physical simulations.

Abstract

RL for Flow Matching Models

Explore similar work

ReFPO: Reflow Regularization for Flow Matching Policy Gradients

BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation

Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models