FlowInOne:Unifying Multimodal Generation as Image-in, Image-out Flow Matching
Authors: Junchao Yi, Rui Zhao, Jiahao Tang, Weixian Lei, Linjie Li, Qisheng Su, Zhengyuan Yang, Lijuan Wang, Xiaofeng Zhu, Alex Jinpeng Wang
Organizations: University of Electronic Science and Technology of China · National University of Singapore · Central South University · Microsoft · University of Science and Technology of China
Multimodal generation has long been dominated by text-driven pipelines where language dictates vision but cannot reason or create within it. We challenge this paradigm by asking whether all modalities, including textual descriptions, spatial layouts, and editing instructions, can be unified into a single visual representation. We present FlowInOne, a framework that reformulates multimodal generation as a purely visual flow, converting all inputs into visual prompts and enabling a clean image-in, image-out pipeline governed by a single flow matching model. This vision-centric formulation naturally eliminates cross-modal alignment bottlenecks, noise scheduling, and task-specific architectural branches, unifying text-to-image generation, layout-guided editing, and visual instruction following under one coherent paradigm. To support this, we introduce VisPrompt-5M, a large-scale dataset of 5 million visual prompt pairs spanning diverse tasks including physics-aware force dynamics and trajectory prediction, alongside VP-Bench, a rigorously curated benchmark assessing instruction faithfulness, spatial precision, visual realism, and content consistency. Extensive experiments demonstrate that FlowInOne achieves state-of-the-art performance among open-source models across all unified generation tasks while remaining competitive with leading commercial systems, thereby establishing a new foundation for fully vision-centric generative modeling, in which perception and creation coexist within a unified continuous visual space. Our code and models are released on https://csu-jpg.github.io/FlowInOne.github.io/
Figures & tables
Figure 1: Comparison of generation paradigms. Left: Traditional T2I only uses the text encoder to condition the Latent Diffusion Model(LDM); Middle: Traditional TI2I requires the joint conditioning of both the text and image encoders; Right: We unify the conditions as visual input and form a simple image-in, image-out framework with a single model.
Figure 2: VisPrompt-5M is a comprehensive dataset that comprises eight distinct data types, including class-to-image generation, text-to-image generation, text-in-image editing, text bounding box editing, visual marker editing, doodles editing, force understanding, and trajectory understanding. The dataset covers a wide spectrum of image-to-image generation, ranging from basic text-in-image generation to compositional editing, and further to physics-aware instruction following.
Figure 3: Overview of the FlowInOne architecture, a general and simple framework using flow matching for continuous evolution in only one modality. FlowInOne employs a Dual-Path Spatially-Adaptive Modulation to adapt computation by modality. For input image rendering with only text, the structural branch is bypassed to strictly follow semantic evolution. Conversely, for image editing, a spatially-adaptive gated network and cross attention activates to selectively inject source priors, dynamically balancing original image preservation with instruction-driven reconstruction.
Method
C2I
T2I
TIE
FU
TBE
TU
VME
DE
Total
Evaluator: Gemini3
Nano Banana Google (2025b)
.810
.980
.521
.500
.600
.020
.537
.740
.589
Omnigen2 Wu et al. (2025b)
.720
.760
.313
.013
.020
.000
.020
.140
.248
Kontext Labs et al. (2025)
.620
.700
.363
.027
.163
.020
.096
.180
.271
Qwen-IE-2509 Wu et al. (2025a)
.680
.690
.383
.047
.060
.000
.040
.160
.258
FlowInOne (Ours)
.890
.700
.355
.727
.302
.520
.292
.535
.540
Table 1: Evaluation on the VP-Bench visual instruction benchmark using three VLM evaluators and human judges. We use Gemini3 Google (2025a) , GPT5.2 OpenAI (2025) and Qwen3.5 Qwen Team (2026) to evaluate the success ratio of FlowInOne on all kinds of visual instruction tasks. C2I: class to image, T2I: text to image, TIE: text in image edit, FU: force understanding, TBE: text & bbox edit, TU: trajectory understanding, VME: visual marker edit, DE: doodles edit. Total denotes the average success rate across all sub-categories.
Method
Gemini3
GPT5.2
Qwen3.5
IF
CC
VR
SP
IF
CC
VR
SP
IF
CC
VR
SP
Nano Banana
3.43
3.10
4.43
2.99
3.28
2.78
4.06
2.86
3.30
3.01
4.43
2.94
Omnigen2
2.15
1.39
3.46
1.69
2.29
1.53
3.31
1.92
1.72
0.91
3.13
1.27
Kontext
2.31
1.62
3.15
2.25
2.40
1.71
3.25
2.21
1.97
1.87
3.11
1.76
Qwen-IE-2509
2.96
2.51
3.28
1.91
2.76
2.39
3.58
1.94
2.33
1.98
3.23
1.39
FlowInOne
3.38
2.94
3.12
3.42
3.16
2.81
2.96
3.24
3.31
2.87
3.20
3.30
Table 2: Overall four-dimensional breakdown. Each dimension has a maximum score of 5 points. IF: Instruction Faithfulness, CC: Content Consistency, VR: Visual Realism, SP: Spatial Precision.
Method
IQA ↑
CLIP Score ↑
Dir CLIP ↑
DINOv3 Sim ↑
Total
C2I
T2I
Avg.
TIE
VME
Avg.
DE
FU
TBE
TU
Avg.
Nano Banana Google (2025b)
0.688
0.281
0.302
0.291
0.106
0.103
0.105
0.430
0.474
0.501
0.486
0.473
Omnigen2 Wu et al. (2025b)
0.603
0.173
0.208
0.191
0.001
0.005
0.003
0.224
0.066
0.004
0.207
0.125
Kontext Labs et al. (2025)
0.621
0.164
0.172
0.168
0.010
0.008
0.009
0.283
0.240
0.118
0.004
0.161
Qwen-IE-2509 Wu et al. (2025a)
0.646
0.231
0.216
0.224
0.005
0.011
0.008
0.133
0.215
0.250
0.207
0.201
FlowInOne (Ours)
0.684
0.290
0.276
0.283
0.092
0.101
0.097
0.335
0.536
0.506
0.570
0.487
Table 3: Quantitative evaluation on VP-Bench across five models. CLIP-IQA is averaged over all categories to measure overall visual realism. CLIP Score evaluates semantic alignment for pure generation tasks. Directional CLIP Similarity assesses instruction-driven consistency in marker-based editing. DINOv3 Similarity measures fine-grained spatial and physical structural accuracy.
Figure 4: Visual instruction editing comparison across methods.
Table 4: Ablation study on compression method, modulation method, and training strategy. The average scores across Gemini3 Google (2025a) , GPT5.2 OpenAI (2025) and Qwen3.5 Qwen Team (2026) are used as the evaluation metric.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Overall error types in VP-Bench. For brevity in the figures, the labels Fidelity, Spatial, Realism, and Consistency correspond strictly to Instruction Fidelity, Spatial Precision, Visual Realism, and Content Consistency, respectively.
Figure 6: Error types for each subset in VP-Bench.
Figure 7: Qualitative results of the robustness analysis against visual instruction perturbations. We evaluate the model’s generation stability under five distinct conditions: the original unmodified input, variations in text style (e.g., size, color, font, and layout), changes in text length, text blurring, and random text corruption.
Figure 8: Generation results across different input image resolutions.
Figure 9: Qualitative results of the visual instruction ablation study. We compare generation outcomes across four input configurations: Blank (original image only), Text only, Visual prompt only, and the complete Text & visual prompt.
Model
Fleiss’ κ
Kendall’s W
FlowInOne
0.6537
0.7935
FLUX.1 Kontext
0.4540
0.5189
Nano Banana
0.4537
0.5251
OmniGen2
0.5361
0.5082
Qwen-IE-2509
0.4515
0.6115
Overall
0.6132
0.7422
Appendix
Table 5: Inter-evaluator Agreement.
Top-Level Task
Sub-Category / Grouping
Pairs
Total
Class-to-Image
-
-
100
Doodles Editing
-
-
50
Force Understanding
Point Force
100
150
Global Force
50
Text-to-Image
-
-
50
Text BBox Control
-
-
50
Appendix
Table 6: Detailed statistics of the human-curated VP-Bench . To provide a structured overview, highly fine-grained editing sub-categories are conceptually grouped into higher-level semantic macro-categories.
Task Category
Pairs
Avg. Words
Word Range
Avg. Chars
Class-to-Image
100
3.0
1 – 6
22.4
Text BBox Control
43
3.5
3 – 5
16.0
Doodles Editing
43
12.0
12 – 12
63.0
Trajectory Understanding
50
14.0
14 – 14
79.0
Visual Marker Editing
312
14.1
3 – 32
74.3
Force Understanding
150
17.7
13 – 24
96.0
Appendix
Table 7: Textual prompt statistics across different task categories in VP-Bench. The uniquely low average word counts prove the benchmark’s reliance on explicit visual markers rather than lengthy textual descriptions.
Figure 17
Figure 12: Additional qualitative results for text-to-image generation and text-in-image editing tasks.
Figure 13: Additional qualitative results for text bounding box (bbox) editing, doodle-guided editing, and visual marker-based editing.
Figure 14: Additional qualitative results demonstrating physical force understanding and trajectory understanding.
Task Category
Raw Source(s)
Raw Size
Retained Size
Construction Pipeline
Visual Instruction Format
Text to Image
text-to-image-2M zk [2024]
2.26M
2.26M
Direct Rendering
Text
Class to Image
ImageNet (Subset) Russakovsky et al. [2014]
2M
860K
Direct Rendering
Text (Class Labels)
Text in Image Edit
GPT-Image-Edit Wang et al. [2025b]
1.5M
1.04M
Extracted & Filtered
Text
Pico-Banana Qian et al. [2025]
400K
10K
UnicEdit Ye et al. [2025]
2M
585K
PixWizard Lin et al. [2024]
500K
315K
Extracted
Appendix
Table 8: Comprehensive Overview of the VisPrompt-5M Dataset. We detail the task categories, original data sources, data scale before and after the filtering pipeline, construction methodology, and the explicitly embedded visual instruction formats.
Figure 15: General data construct process.
Figure 16: Detailed instance distribution of VisPrompt-5M. Left: The Force & Trajectory Generation subset, highlighting strictly curated physics-aware categories (e.g., wind, object poking) designed to impart dynamic kinematic priors. Right: The Text-in-Image Editing subset (derived from GPT-Image-Edit), demonstrating a natural long-tailed distribution of semantic operations. Ranging from high-frequency attribute modifications to specialized style transfers, this structural diversity ensures robust generalization across both complex stylistic changes and physical constraints.
Figure 17: Fine-grained structure and stylized distributions of VisPrompt-5M. Left: The Structured Editing subset (derived from PixWizard), highlighting dense spatial translations such as Image-to-Sketch and Face Restoration. This subset trains the model to strictly adhere to geometric and structural conditions. Right: The Text-in-Image Editing subset (derived from PicoBanana), detailing 35 highly specialized, long-tailed aesthetic operations. This extreme categorical diversity ensures the model’s proficiency in handling nuanced, localized, and composite textual instructions.
Figure 18: Semantic diversity and region-aware distributions of VisPrompt-5M. Left: The UnicEdit10M Diverse Edits subset, showcasing a profound long-tailed distribution. It spans high-frequency semantic operations like Color Alteration and Subject Addition to rare edge cases such as Object Extraction. Right: The VisMarker Region-Aware Edits subset, featuring robust, high-volume spatial operations like Object Swap and Removal. This combination ensures the model learns broad semantic reasoning while maintaining strict adherence to localized visual markers.
Figure 19: Volume of ancillary generation and structural datasets. This logarithmic lollipop chart illustrates the extreme variance in scale and specialization across our supplementary sources. It highlights the massive foundation of over 2.26 million general Text-to-Image pairs, balanced by broad categorical coverage (ImageNet21K) and highly specialized, precise structural tasks such as Text Bbox Edit and Doodles. This multi-scale composition ensures the model achieves both robust open-domain generative capabilities and fine-grained execution of explicit visual constraints.
Task Category
Primary Source(s)
Initial Volume
Curated Pairs
Fundamental Gen. (Text)
Text-to-Image-2M zk [2024]
2,269,892
2,269,892
Fundamental Gen. (Class)
ImageNet Subset Russakovsky et al. [2014]
2,000,000
860,000
Text-in-Image Editing
GPT-Image-Edit, Pico-Banana, UnicEdit
2,100,000
1,601,000
Structured Editing
PixWizard Lin et al. [2024]
350,000
315,000
Bounding Box Editing
GPT-Image-Edit (Filtered)
45,000
24096
Visual Marker Editing
Qwen3-VL Synthesized
300,000
250,000
Appendix
Table 9: Comprehensive breakdown of the VisPrompt-5M composition. The dataset aggregates various public sources, strictly refined through our multi-stage OCR and VLM filtering pipeline.
Category
Primary Rejection Reason
VLM/OCR Filtering
Retention Rate
Text-in-Image Editing
Semantic Inconsistency
VLM Semantic Check
78.2%
Bounding Box Editing
Geometric Misalignment
VLM Spatial Check
53.3%
Doodles Editing
Generative Instability
VLM Realism Check
20.0%
Visual Marker Editing
Generative Instability
VLM Spatial Check
83.3%
Appendix
Table 10: Filtering survival rates across selected complex generative tasks, demonstrating the rigorousness of our automated OCR and VLM inspection.
Figure 20: Evaluation process.
Figure 21: Evaluation prompts used in VLM evaluators.
Component
Params (M)
Train.
Flow Backbone
1108.40
Yes
Text-Image VAE
103.65
Yes
Visual Encoder
322.58
No
Image VAE
83.65
No
Appendix
Table 11: Parameter Breakdown of FlowInOne.
(a) CFG Scale
(b) Sampling Steps
(c) Loss Weights
CFG
PASS (%)
Steps
PASS (%)
(Lclip,Lkld)
PASS (%)
1.1
38.5
10
25.7
(0.1, 0.001)
46.7
1.5
44.8
20
40.3
(0.1, 0.01)
45.2
3.0
47.7
30
46.3
(0.1, 0.1)
44.2
5.0
51.5
40
48.7
(1, 0.001)
46.0
7.0
54.0
50
54.0
(1, 0.01)
54.0
Appendix
Table 12: Hyperparameter ablation study on VP-Bench. We report the average PASS Rate (%) evaluated by three VLMs. Default optimal settings (CFG =7 , Steps =50 , β1=0.01 , β2=1 ) are marked in bold . The control variable method is applied; parameters not actively ablated are fixed to their default values.
Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.
While recent advancements in multimodal language models have enabled image generation from expressive multi-image instructions, existing methods struggle to maintain performance under complex interleaved instructions. This limitation stems from the structural separation of images and text in current paradigms, which forces models to bridge difficult long-range dependencies to match descriptions with visual targets. To address these challenges, we propose Images iNSEnTences (a.k.a, INSET), a unified generation model that seamlessly embeds images as native vocabulary within textual instructions. By positioning visual features directly at their corresponding semantic slots, INSET leverages the contextual locality of transformers for precise object binding, effectively treating images as dense, expressive language tokens. Furthermore, we introduce a scalable data engine that synthesizes 15M high-quality interleaved samples from standard image and video datasets, utilizing VLMs and LLMs to construct rich, long-horizon sequences. Evaluation results on InterleaveBench demonstrate that INSET significantly outperforms state-of-the-art methods in multi-image consistency and text alignment, with performance gaps widening as input complexity increases. Beyond standard generation, our approach inherently extends to multimodal image editing, integrating visual content as part of the instruction to facilitate highly expressive and creative visual manipulations.
Unified multimodal models can generate images and interpret them, but their two capabilities are usually optimized with separate supervision. We introduce Bidirectional Self-Refinement via Flow-Direction Feedback and Semantic Replay (BiSe), a post-training method that uses self-generated visual episodes to construct feedback for both pathways. Understanding-guided generation aligns the prompt-conditioned flow direction with an image-conditioned prediction and with the model's response to a repair instruction. Generation-guided understanding replays the source intent through captioning and image generation, and learns a stable semantic evidence profile across the replayed images while separating an independent episode. Flow-matching and source-prompt objectives remain active, and frozen initial-model predictions limit drift. BiSe requires no paired image--text data during post-training. It improves text-to-image compositionality across three unified backbones and visual understanding on the two backbones evaluated for that task.
Ke Hao, Yuanzhi Liang, Tingxi Chen +5
Shanghai Jiao Tong University, Shanghai, China · Institute of Artificial Intelligence, China Telecom (TeleAI), Shanghai, China · University of Science and Technology of China, Hefei, China
We present JoyAI-Image, a unified multimodal foundation model for visual understanding, text-to-image generation, and instruction-guided image editing. JoyAI-Image couples a spatially enhanced Multimodal Large Language Model (MLLM) with a Multimodal Diffusion Transformer (MMDiT), allowing perception and generation to interact through a shared multimodal interface. Around this architecture, we build a scalable training recipe that combines unified instruction tuning, long-text rendering supervision, spatially grounded data, and both general and spatial editing signals. This design gives the model broad multimodal capability while strengthening geometry-aware reasoning and controllable visual synthesis. Experiments across understanding, generation, long-text rendering, and editing benchmarks show that JoyAI-Image achieves state-of-the-art or highly competitive performance. More importantly, the bidirectional loop between enhanced understanding, controllable spatial editing, and novel-view-assisted reasoning enables the model to move beyond general visual competence toward stronger spatial intelligence. These results suggest a promising path for unified visual models in downstream applications such as vision-language-action systems and world models.