Understanding vs. Generation: Navigating Optimization Dilemma in Multimodal Models

Sen Ye; Mengde Xu; Shuyang Gu; Di He; Liwei Wang; Han Hu

cs.CVFeb 17, 2026

Understanding vs. Generation: Navigating Optimization Dilemma in Multimodal Models

Authors: Sen Ye, Mengde Xu, Shuyang Gu, Di He, Liwei Wang, Han Hu

Organizations: State Key Laboratory of General Artificial Intelligence, Peking University · Tencent · Center for Data Science, Peking University · Center for Machine Learning Research, Peking University

Abstract

Current research in multimodal models faces a key challenge where enhancing generative capabilities often comes at the expense of understanding, and vice versa. We analyzed this trade-off and identify the primary cause might be the potential conflict between generation and understanding, which creates a competitive dynamic within the model. To address this, we propose the Reason-Reflect-Refine (R3) framework. This innovative algorithm re-frames the single-step generation task into a multi-step process of "generate-understand-regenerate". By explicitly leveraging the model's understanding capability during generation, we successfully mitigate the optimization dilemma, achieved stronger generation results and improved understanding ability which are related to the generation process. This offers valuable insights for designing next-generation unified multimodal models. Code is available at https://github.com/sen-ye/R3.

Figures & tables

Figure 1: Fine-tuning BAGEL exclusively on generation or understanding degrades the complementary capability. Naive co-training shows minor gains, whereas our proposed method demonstrates significant improvement in both. Results are reported on counting subset of GenEval++.

Figure 2: The inference pipeline of our Reason-Reflect-Refine framework. The model starts by Reasoning to produce an initial plan and image. It then enters an iterative Reflect-Refine loop, assessing its output and making corrections until the image aligns with the user’s prompt or a stopping condition is met.

Figure 3: The training procedure, which alternates between optimizing the Reason policy and the Reflect-Refine policies. The replay buffer, populated by the Reason stage, provides on-policy data for training the subsequent stages.

Figure 4: Training reward curves of the Tree-RL versus Full Trajectory RL strategies. The reward curve for Full Trajectory RL is substantially lower than that of Tree-RL. This performance gap is attributed to the high variance and noise introduced by the long trajectories inherent in the full trajectory approach, which complicates the advantage assignment problem.

Method	Color	Count	Color/Count	Color/Pos	Pos/Count	Pos/Size	Multi-Count	Overall
GPT-4o ( OpenAI, 2025 )	0.900	0.675	0.725	0.625	0.600	0.800	0.850	0.739
FLUX.1-Kontext ( Labs, 2024 )	0.425	0.500	0.200	0.250	0.300	0.400	0.325	0.343
FLUX.1-dev ( Labs, 2024 )	0.350	0.625	0.150	0.275	0.200	0.375	0.225	0.314
Janus-Pro ( Chen et al., 2025 )	0.450	0.300	0.125	0.300	0.075	0.350	0.125	0.246
T2I-R1 ( Jiang et al., 2025 )	0.675	0.325	0.200	0.350	0.075	0.250	0.300	0.311
Echo-4o ( Ye et al., 2025 )	0.800	0.575	0.550	0.775	0.625	0.800	0.625	0.679

Table 1: Instruction-following generation ability on the GenEval++ benchmark, evaluated by GPT-4.1. Bold indicates the best result. †indicates our framework with only the reasoning stage. Green arrows indicate improvement over the BAGEL baseline.

ITA	Color	Count	Color/Count	Color/Pos	Pos/Count	Pos/Size	Multi-Count	Overall
BAGEL	60.63	58.54	45.42	63.54	53.96	80.83	61.50	60.60
BAGEL + Ours†	60.42	59.38	47.71	63.75	55.63	81.46	63.96	61.76 $\uparrow$ 1.16
BAGEL + Ours	69.58	67.50	69.79	72.29	76.04	83.33	75.00	73.37 $\uparrow$ 12.77

Table 2: Evaluation of understanding capabilities on our proposed ITA benchmarks. All scores are reported as accuracy (%). †indicates our framework with only the reasoning stage. Green arrows indicate improvement over the BAGEL baseline.

VQA	Color	Count	Color/Count	Color/Pos	Pos/Count	Pos/Size	Multi-Count	Overall
BAGEL	91.74	79.30	88.28	77.99	82.93	85.10	92.45	86.48
BAGEL + Ours†	91.67	76.12	88.76	78.71	83.29	84.98	93.45	86.72 $\uparrow$ 0.24
BAGEL + Ours	93.95	84.63	91.15	84.09	86.06	86.54	94.50	89.63 $\uparrow$ 3.15

Table 3: Evaluation of VQA capabilities. All scores are reported as accuracy (%). †indicates our framework with only the reasoning stage. Green arrows indicate improvement over the BAGEL baseline.

Figure 5: Qualitative comparison between Bagel and our results.

Figure 6: Inference-time scaling effect across the GenEval, GenEval++, and TIIF benchmarks (left to right). Performance is shown as a function of the maximum allowed reflection-refinement turns.

Figure 7: The evolution of generation and understanding abilities in the training process. The left figure shows the generation accuracy (measured by Qwen-2.5-VL-72B.) and the right figure shows the VQA accuracy.

Model	Overall	Basic Following				Advanced Following						Designer
Model	Overall	Avg	Attribute	Relation	Reasoning	Avg	Attribute +Relation	Attribute +Reasoning	Relation +Reasoning	Style	Text	Designer
GPT-4o ( OpenAI, 2025 )	84.19	85.30	81.00	86.16	88.74	81.24	81.95	80.03	80.88	76.67	92.76	89.55
FLUX.1-dev ( Labs, 2024 )	66.24	74.41	72.50	78.20	72.52	60.72	66.76	61.76	56.60	63.33	44.49	74.63
FLUX.1-Pro ( Labs, 2024 )	63.75	71.39	70.00	68.51	75.66	64.63	70.69	62.34	64.65	63.00	34.39	69.94
Janus-Pro-7B ( Chen et al., 2025 )	65.38	74.99	74.50	73.69	76.77	61.77	65.71	62.01	61.16	43.33	38.46	79.48
T2I-R1 ( Jiang et al., 2025 )	67.61	81.14	80.50	83.09	79.81	67.38	69.92	70.10	68.69	50.00	32.13	74.25

Table 6: Quantitative evaluation results on TIIF testmini benchmark ( Wei et al., 2025 ) . Results are evaluated by QwenVL2.5-72B. * indicates our reproduced results using the official repository and checkpoint. The best results are bolded .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparams	GenEval	GenEval++	TIIF
Learning Rate	5e-6	5e-6	5e-6
Batch Size	16	16	16
Group Size	16	16	16
Training Steps	400	300	200
CE Weight	1	1	1
MSE Weight	2	2	2

Appendix

Table 7: Hyperparameters for Different Datasets

Method	Single object	Two object	Counting	Colors	Position	Color attribution	Overall
FLUX.1-dev ( Labs, 2024 )	0.99	0.81	0.79	0.74	0.20	0.47	0.67
Qwen-Image-RL ( Wu et al., 2025a )	1.00	0.95	0.93	0.92	0.87	0.83	0.91
FlowGRPO ( Liu et al., 2025 )	1.00	0.99	0.95	0.92	0.99	0.86	0.95
BAGEL ( Deng et al., 2025 )	0.99	0.94	0.81	0.88	0.64	0.63	0.82
BAGEL + Ours	1.00	1.00	0.95	0.95	0.98	0.89	0.96

Appendix

Table 8: Text-to-image generation ability on the GenEval benchmark ( Ghosh et al., 2023 ) . The best scores are bolded .

Inference Strategy	Bagel	Ours
Reasoning only	0.399	0.593
Reason+RR $\times$ 1	0.432	0.675
Reason+RR $\times$ 2	0.436	0.689
Reason+RR $\times$ 3	0.439	0.682
Reason+RR $\times$ 4	0.439	0.689
Reason+RR $\times$ 5	0.439	-

Appendix

Table 9: Inference performance on GenEval++ comparison between Bagel and R3 (ours) under same inference strategies.

Figure 8: An illustration of our model’s two-stage generation process. For the prompt "The boy is on the side of the pond, not under a tree," the model’s initial reasoning stage erroneously generates a tree above the boy. The subsequent reflect-refine stage corrects this error, after which the process terminates automatically.

Figure 9: An illustration of our model’s two-stage generation process. For the prompt "A yellow grasshopper hops under the wooden fence as the yellow sun splashes warmth across the field," the model’s initial reasoning stage generates a grasshopper with an incorrect color. The subsequent reflect-refine stage corrects the color, after which the process terminates automatically.

Figure 10: An illustration of our model’s two-stage generation process. For the prompt ’A picture of a band performing on a stage, with the text on it: "SIMPLY", "Bullet", "Proof", "Nothing", "SAUCER"’, the model’s initial stage fails to correctly render some of the text. The subsequent reflect-refine stage successfully corrects the text to match the prompt, after which the process terminates automatically.

Figure 11: An illustration of our model’s two-stage generation process, showing a limitation. For the prompt requiring specific text on a tennis ball, the initial stage fails to render all the words. The reflect-refine stage adds most of the missing text but introduces a minor spelling error ("CLUB"). Despite this inaccuracy, the model terminates the process without attempting further corrections.

Figure 12: An illustration of our model’s two-stage generation process. For the prompt "A photo of a white bench, a yellow stop sign, and a pink tie," the model’s initial generation fails to correctly render the stop sign. The subsequent reflect-refine stage successfully generates the object as specified, and the process terminates automatically.

Figure 13: An illustration of our model’s two-stage generation process. For the prompt "A photo of a larger computer mouse on the above and a smaller sandwich," the model’s initial generation fails to correctly depict the specified size relationship. The subsequent reflect-refine stage adjusts the relative sizes of the objects to match the prompt, after which the process terminates.

Figure 14: An illustration of our model’s two-stage generation process. For the prompt "a photo of a white cat, a green bench, a red elephant, and a purple sandwich on the below," the model’s initial generation renders both the elephant and the sandwich with incorrect colors. The subsequent reflect-refine stage corrects the colors of both objects, and the process terminates automatically.

Figure 15: An illustration of our model’s three-stage generation process. For the prompt "A photo of two suitcases on the above and three donuts on the below," the initial generation incorrectly produces four donuts. In the first reflect-refine stage, the model identifies the count error but fails to correct it. A second reflect-refine stage is initiated, where the model successfully edits the image to show the correct number of donuts, after which the process terminates automatically.

Figure 16: An illustration of maze navigation results. The model is tested on a 12x10 maze, where it demonstrates the ability to accurately identify valid directions during the navigation process. When faced with multiple paths or dead ends, the model can effectively backtrack. Furthermore, it automatically recognizes when the destination has been reached and terminates the process accordingly.

Multimodal Generation

Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.

See all 243 papers

Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System
Sep 1, 2026Penghao Wu, Haiwen Diao, Weichen Fan +3Unified Multimodal Models Multimodal Generation
1S-Lab, Nanyang Technological University · 2SenseTime Research
Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation
Jun 25, 2026Jinyu Liu, Xincheng Shuai, Henghui Ding +1Unified Multimodal Models Multimodal Foundation Models
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China · Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China
Reversing the Flow: Generation-to-Understanding Synergy in Large Multimodal Models
May 15, 2026Yujun Tong, Dongliang Chang, Zijin Yin +3Unified Multimodal Models Multimodal Reasoning
Beijing University of Posts and Telecommunications · Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance

Sep 1, 2026J/K move · Enter open · S save

Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System

Penghao Wu, Haiwen Diao, Weichen Fan, Lewei Lu+2

1S-Lab, Nanyang Technological University · 2SenseTime Research

Unified Multimodal Models Multimodal Generation Multimodal Learning Multimodal Understanding

Open paper AskPDF arXiv

While unified multimodal models (UMMs) jointly perform visual understanding and generation within a single model, functional unification does not guarantee learning synergy: the two objectives may reinforce each other, compete for capacity, or merely coexist. We investigate their relationship at the representation, task, and system levels in a controlled, structurally native setting without pretrained vision priors. At the representation level, we find that each objective provides useful signal to the other: generation enriches the visual features learned for understanding, while understanding strengthens vision--language alignment for generation. However, when both objectives are forced through the same computation path, one tends to dominate. A task-decoupled architecture that specializes conflicting visual computation while preserving semantic interaction avoids this asymmetric degradation. At the task level, through three case studies, we find positive bidirectional transfer when understanding and generation tasks rely on shared knowledge. At the system level, we show that an end-to-end UMM outperforms a matched planner--executor pipeline on complex tasks that explicitly require both image understanding and generation. Together, these results show that the value of UMMs extends beyond a unified interface: appropriate specialization, shared task knowledge, and end-to-end optimization can turn coexistence into synergy.

Abstract

Multimodal Generation

Explore similar work

Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System

Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation

Reversing the Flow: Generation-to-Understanding Synergy in Large Multimodal Models