The Quadratic Geometry of Flow Matching: Semantic Granularity Alignment for Text-to-Image Synthesis

Zhinan Xiong; Shunqi Yuan

cs.CVMar 11, 2026

The Quadratic Geometry of Flow Matching: Semantic Granularity Alignment for Text-to-Image Synthesis

Authors: Zhinan Xiong, Shunqi Yuan

Organizations: Conservatoire National des Arts et Métiers, Paris, France · Sun Yat-sen University, Guangzhou, China

Abstract

In this work, we analyze the optimization dynamics of generative fine-tuning. We observe that under the Flow Matching framework, the standard MSE objective can be formulated as a Quadratic Form governed by a dynamically evolving Neural Tangent Kernel (NTK). This geometric perspective reveals a latent Data Interaction Matrix, where diagonal terms represent independent sample learning and off-diagonal terms encode residual correlation between heterogeneous features. Although standard training implicitly optimizes these cross-term interferences, it does so without explicit control; moreover, the prevailing data-homogeneity assumption may constrain the model's effective capacity. Motivated by this insight, we propose Semantic Granularity Alignment (SGA), using Text-to-Image synthesis as a testbed. SGA engineers targeted interventions in the vector residual field to mitigate gradient conflicts. Evaluations across DiT and U-Net architectures confirm that SGA advances the efficiency-quality trade-off by accelerating convergence and improving structural integrity.

Figures & tables

Figure 2: Structure of the Data Interference Matrix

\mathbf{\Omega}

. Diagonal entries (blue) represent independent learning within each sub-manifold. Off-diagonal entries encode cross-scale interactions: constructive (synergy, green) or destructive (conflict, red).

Figure 3: The optimization landscape

\mathcal{L}_{FM}=\boldsymbol{\alpha}^{\top}\mathbf{\Omega}\,\boldsymbol{\alpha}

of CFM fine-tuning. Path 1 (red): Underfitting Region. Path 3 (blue): OOD Region. Only Path 2 (green) navigates the narrow Goal Region.

Figure 4: Failure analysis of standard fine-tuning. All outputs exhibit dominant prior characteristics regardless of target domain—a prior capture phenomenon that traps the model in the Underfitting Region of Fig. 3 .

Figure 5: Overview of the Hierarchical Semantic Decomposition (H-SD) pipeline. Raw images are parsed by a general-purpose detector ( e.g . , multimodal models, YOLO, or Grounding DINO) into three granularities ( $Y_{\text{Macro}}$ , $Y_{\text{Meso}}$ , $Y_{\text{Micro}}$ ), filtered via IoU-based redundancy elimination, and post-processed into aspect-ratio-preserving resolution buckets.

Figure 6: Optimization trajectories on the H-SD-augmented loss landscape. H-SD + Standard Opt. (orange): after signal amplification via H-SD, vanilla CFM training still suffers from scale-biased gradient updates, causing persistent oscillation and drift toward the OOD boundary. SGA (blue): by adding Tuple-wise co-sampling and Scale-Adaptive Modulation, SGA stabilizes the descent and converges to the Goal Region. The residual correlation sign

\langle\Delta_{\xi},\Delta_{\eta}\rangle

determines whether cross-scale interactions are constructive (

>0

, stable) or destructive (

<0

, oscillation).

Panel of Figure 7 — Figure 7: Qualitative Comparison (FLUX-only). Domains $\delta$ , $\varepsilon$ , and $\zeta$ , evaluated exclusively on FLUX (portrait layout).

Panel of Figure 8 — Figure 8: Qualitative Comparison (Cross-Architecture). Domains $\alpha$ – $\gamma$ , evaluated on both SDXL and FLUX. Reference images (left) define each target domain.

Figure 9: Quantitative Evaluation — LLM Judge (GPT 5.2). Aggregate rank distribution (left) and per-dataset 1st-place rate heatmap (right). SGA variants achieve favourable rankings across both architectures and all stylistic domains.

Figure 10: Quantitative Evaluation — Human Evaluation. Matching layout to Fig. 9 . Human rankings corroborate the automated assessment, with SGA variants accumulating the majority of top rankings.

	FLUX (6 datasets)			SDXL (3 datasets)
Metric	Baseline	SGA		Baseline	SGA
CLIP-I	0.686	0.701	$\uparrow$	0.715	0.717	$\uparrow$
CLIP-T	0.248	0.253	$\uparrow$	0.296	0.298	$\uparrow$
DINO-I	0.425	0.427	$\uparrow$	0.492	0.539	$\uparrow$

Table 1: Calibration Metrics (averaged across datasets). Higher is better for all metrics.

\uparrow

denotes improvement of SGA over the Baseline.

	FLUX		SDXL
Variant	Human	LLM	Human	LLM
Full SGA	54	55	48	61
w/o Tuple-wise Opt.	27	25	16	14
w/o Scale-Adaptive Mod.	19	20	36	25

Table 2: Ablation Study: 1st-place rate (%) across evaluator types. Higher is better.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Backbone	FLUX.1-dev [ 6 ]
Adaptation Method	DoRA [ 37 ]
LoRA Rank	32
LoRA Alpha	16
Batch Size	8
Learning Rate	$1\times 10^{-4}$

Appendix

Table 3: FLUX training hyperparameters. All domains use the same configuration.

Domain	$\boldsymbol{N_{1}}$ (hours)	$\boldsymbol{N_{2}}$ (hours)
$\alpha$	14	21
$\beta$	12	18
$\gamma$	14	21
$\delta$	9	13.5
$\varepsilon$	8	12
$\zeta$	14	21

Appendix

Table 4: Estimated GPU hours per domain (FLUX).

N_{1}

and

N_{2}

denote the two evaluation checkpoints. Actual models used for evaluation are the closest saved checkpoints to the listed times.

Linear layers
Hyperparameter	Value
Backbone	Animagine XL 3.1 [ 47 ]
Optimizer	Lion
U-Net Learning Rate	$3\times 10^{-5}$
Text Encoder Learning Rate	$3\times 10^{-6}$
Batch Size	2

Appendix

Table 5: Animagine XL 3.1 (SDXL) training hyperparameters.

Domain	$\boldsymbol{N_{1}}$ (hours)	$\boldsymbol{N_{2}}$ (hours)
$\alpha$	2	3
$\beta$	2	3
$\gamma$	1.5	2.5

Appendix

Table 6: Estimated GPU hours per domain (Animagine XL 3.1). The same checkpoint selection protocol as FLUX applies: actual models used are the closest saved checkpoints to the listed times, with deviation not exceeding 30 minutes.

Panel of Figure 11 — Figure 11: Reference images defining the target domain for all ablation experiments.

Panel of Figure 12 — Figure 12: FLUX ablation (Group 1). Full SGA maintains domain fidelity; removing Scale-Adaptive Modulation introduces subtle but visible stylistic drift.

Panel of Figure 13 — Figure 13: FLUX ablation (Group 2). The w/o Scale-Adaptive Modulation variant exhibits increased OOD tendencies in fine-grained detail rendering.

Panel of Figure 14 — Figure 14: FLUX ablation (Group 3). All three variants produce structurally coherent outputs—consistent with FLUX’s global attention mechanism—but the Full SGA variant best preserves the target domain’s micro-level characteristics.

Panel of Figure 15 — Figure 15: LoRA Ablation (Group 1). LoRA without Modulation produces visible OOD drift.

Panel of Figure 16 — Figure 16: LoRA Ablation (Group 2). LoRA exhibits more pronounced domain deviation due to coupled updates.

Panel of Figure 17 — Figure 17: LoRA Ablation (Group 3). LoRA without Scale-Adaptive Modulation (right column) shows the most pronounced OOD deviation.

Panel of Figure 18 — Figure 18: SDXL ablation (Group 1). Removing SGA produces noticeable anatomical errors; removing Min-SNR yields subtler structural degradation.

Panel of Figure 19 — Figure 19: SDXL ablation (Group 2). Full SGA best preserves limb coherence, consistent with CNN-based models requiring explicit structural regularization.

Panel of Figure 20 — Figure 20: SDXL ablation (Group 3). SGA removal consistently causes the most significant anatomical degradation, supporting its critical role in compensating for CNN’s limited global awareness.

Architecture	Foundation Models	Community Derivatives
U-Net	SD 1.5 / 2.1 [ 51 ] ,	Pony Diffusion V6 [ 48 ] , Animagine XL [ 8 ] ,
U-Net	SDXL [ 47 ]	NoobAI-XL [ 32 ] , Illustrious XL [ 43 ]
MM-DiT	PixArt- $\alpha$ [ 11 ] , SD 3.5 [ 17 ] ,	—
MM-DiT	AuraFlow [ 18 ] , FLUX.1 [ 6 ] , FLUX.2 [ 7 ]

Appendix

Table 7: Open-source diffusion model ecosystem. Representative foundation models and their community-driven derivatives, organized by backbone architecture.

Flow Matching

Flow matching learns continuous-time vector fields that transport simple noise distributions into data for efficient generative modeling.

See all 656 papers

Flow-OPD: On-Policy Distillation for Flow Matching Models
May 8, 2026Zhen Fang, Wenxuan Huang, Yu Zeng +8Diffusion Model Alignment Flow Matching
University of Science and Technology of China · University of California, Los Angeles · 3The Chinese University of Hong Kong +1
LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectories
Apr 16, 2026Zhanhao Liang, Tao Yang, Jie Wu +2Diffusion Model Alignment Flow Matching
1The Australian National University · 2ByteDance Seed
Unlocking Spatial Grounding in Flow-Matching Models via SNR Tuning
Oct 21, 2026Martin Thißen, Elke HergenrötherFlow-Based Generative Modeling Vision-Language Grounding
Department of Computer Science, Darmstadt University of Applied Sciences, Darmstadt, Germany

May 8, 2026J/K move · Enter open · S save

Flow-OPD: On-Policy Distillation for Flow Matching Models

Zhen Fang, Wenxuan Huang, Yu Zeng, Yiming Zhao+7

University of Science and Technology of China · University of California, Los Angeles · 3The Chinese University of Hong Kong+1

Diffusion Model Alignment Flow Matching RL for Image Generation Flow-Based Generative Modeling On-Policy Distillation T2I Diffusion Models

Open paper AskPDF arXiv

Existing Flow Matching (FM) text-to-image models suffer from two critical bottlenecks under multi-task alignment: the reward sparsity induced by scalar-valued rewards, and the gradient interference arising from jointly optimizing heterogeneous objectives, which together give rise to a 'seesaw effect' of competing metrics and pervasive reward hacking. Inspired by the success of On-Policy Distillation (OPD) in the large language model community, we propose Flow-OPD, the first unified post-training framework that integrates on-policy distillation into Flow Matching models. Flow-OPD adopts a two-stage alignment strategy: it first cultivates domain-specialized teacher models via single-reward GRPO fine-tuning, allowing each expert to reach its performance ceiling in isolation; it then establishes a robust initial policy through a Flow-based Cold-Start scheme and seamlessly consolidates heterogeneous expertise into a single student via a three-step orchestration of on-policy sampling, task-routing labeling, and dense trajectory-level supervision. We further introduce Manifold Anchor Regularization (MAR), which leverages a task-agnostic teacher to provide full-data supervision that anchors generation to a high-quality manifold, effectively mitigating the aesthetic degradation commonly observed in purely RL-driven alignment. Built upon Stable Diffusion 3.5 Medium, Flow-OPD raises the GenEval score from 63 to 92 and the OCR accuracy from 59 to 94, yielding an overall improvement of roughly 10 points over vanilla GRPO, while preserving image fidelity and human-preference alignment and exhibiting an emergent 'teacher-surpassing' effect. These results establish Flow-OPD as a scalable alignment paradigm for building generalist text-to-image models. The codes and weights will be released in: https://github.com/CostaliyA/Flow-OPD .

Abstract

Flow Matching

Explore similar work

Flow-OPD: On-Policy Distillation for Flow Matching Models

LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectories

Unlocking Spatial Grounding in Flow-Matching Models via SNR Tuning