MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generation

Jihan Yao; Yushi Hu; Wenyuan Wang; Bin Han; Shangbin Feng; Guang Yang; Yujie Yi; Bingbing Wen

cs.AIMay 23, 2025

MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generation

Authors: Jihan Yao, Yushi Hu, Wenyuan Wang, Bin Han, Shangbin Feng, Guang Yang, Yujie Yi, Bingbing Wen, +5 more

Organizations: University of Washington · Allen Institute for AI

Abstract

Automatically evaluating multimodal generation presents a significant challenge, as automated metrics often struggle to align with human evaluation, especially for complex tasks that involve multiple modalities. We present MMMG, the first benchmark to bring the verifiable-task paradigm to multimodal generation, spanning 4 modality combinations (image, audio, interleaved text and image, interleaved text and audio). As few multimodal outputs can be checked by programs alone, MMMG targets tasks that are either verifiable or near-verifiable: by providing references, and constraining model judges with explicit rubrics. We keep tasks challenging for generation models while enabling reliable automatic evaluation through a combination of models and programs. MMMG encompasses 55 tasks (including 31 newly developed ones), each with a carefully designed evaluation pipeline, and 1288 instructions to systematically assess reasoning, controllability, and other key capabilities of multimodal generation models. Extensive validation demonstrates that MMMG is highly aligned with human judgment, achieving an average agreement of 94.4%. Benchmarking results on 29 models reveal that even though the state-of-the-art model, GPT Image, achieves 70.7% accuracy for image generation, it falls short on interleaved generation. Furthermore, results suggest considerable improvement space in audio generation, highlighting an important future direction.

Figures & tables

Figure 1: Examples of tasks and their evaluation in MMMG . For each task, we develop an evaluation metric using programs, models or their combinations. The tasks are either verifiable by programs or near-verifiable with big generation-evaluation gaps: generation is challenging, while strengthened task conditions allow automatic evaluations to easily align with human judgment. We show pseudo-code to demonstrate the evaluation process.

Task	Subtask	Description	In.	Out.	#	Evaluation
Object Generation	Inclusion	Include one or two unrelated objects in the scene.	$\mathbb{T}$		40	VLM
	Exclusion	Exclude one related object from the scene.	$\mathbb{T}$		40	VLM
	Count	Generate exactly N objects.	$\mathbb{T}$		40	VLM
	Attribution	Generate an object with uncommon attributes.	$\mathbb{T}$		40	VLM
	Knowledge	Reason the answer object to a multi-hop question.	$\mathbb{T}$		40	VLM
	Commonsense	Reason the answer object/scene by commonsense.	$\mathbb{T}$		40	VLM

Table 1: Detailed metadata for MMMG .

\mathbb{T}

denotes text modality, for image modality, for multiple images, for audio and for multiple audios. We evaluate each task with the method that most aligned with human judgment. green background indicates new tasks.

Dataset

# Samples

# Tasks

Generation Modality

Evaluation

Tested Capability

\mathbb{T}

\mathbb{T}

human

mllm

score

code

gen

edit

reason

GenEval ( Ghosh et al., 2023 )

553

✔

✘

✔

✘

DrawBench ( Saharia et al., 2022 )

200

✔

✘

✔

✘

✔

✘

GenAI-Bench ( Li et al., 2024 )

1,600

✔

✘

✔

✘

✔

✘

AudioTime ( Xie et al., 2024 )

500

✘

✔

✘

✔

✘

MusicEval ( Liu et al., 2025 )

384

✘

✔

✘

✔

✘

✔

✘

Table 2: MMMG compared with other multimodal generation benchmarks. , ,

\mathbb{T}

+ ,

\mathbb{T}

+ represent image, audio, interleaved image-text, and interleaved audio-text generation, respectively. “score” stands for embedding-based / rule-based similarity score, “code” for programmatically verification, and “reason” for multi-step reasoning. represents low human alignment or no human experiments. MMMG exceeds other benchmarks in the number of covered tasks and modalities while providing more reliable evaluation.

Panel of Figure 2 — Figure 2: Benchmark results of multimodal generation models on MMMG covering four modality combinations. Please refer to Table 1 for more detailed category information. We aggregate some sub-tasks for interleaved image-text generation. GPT Image beats all other models on most image generation tasks, and strongly competes other baselines in generating consistent image sequences and coherent interleaved image-text contents.

Model	Arena	GenEval	Draw	GenAI	MMMG
Imagen 3	1064	0.707	0.861	0.793	0.474
Recraft v3	1018	0.732	0.826	0.817	0.441
Luma Photon	997	0.738	0.766	0.804	0.587
Flux 1.1 Pro	992	0.588	0.725	0.736	0.431
Ideogram 2	1011	0.615	0.757	0.782	0.508
Dalle 3	978	0.627	0.809	0.811	0.352

Table 3: Correlation of automated image generation benchmarks with Chatbot Arena. Arena, Draw, GenAI represent Chatbot Arena, DrawBench, and GenAI-Bench. MMMG achieves the highest correlation with Chatbot Arena. This indicates even though our instructions are synthetic, the evaluation results are still highly human-aligned.

Figure 3: Two prevalent failure cases observed in interleaved image-text generation tasks for Gemini 2 Image : (1) models fail to accurately interpret the order of images in interleaved inputs; and (2) models frequently blend multiple images together, possibly due to limitations in encoding multiple images with continuous latent image representations.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Example	Input	Output
Table Generation	Create a 2x2 table image. In the first column, place the text ’apple’ in the top cell and ’pear’ in the bottom cell. In the second column, place an image of an apple in the top cell and an image of a pear in the bottom cell.	$\mathbb{T}$
Figure Generation	Create a histogram to visualize the given data. <data>	$\mathbb{T}$
Format Color	Create a watermelon farm using only varying shades of red.	$\mathbb{T}$
Format Symmetric	Generate an image of a futuristic cityscape. The image must be axisymmetric along the vertical center line.	$\mathbb{T}$
Art Style	Create a painting of a dandelion sea in Impressionist style.	$\mathbb{T}$
Photography	Create q zoomed out photo of a small bag of coffee beans from below.	$\mathbb{T}$

Appendix

Table 4: Tasks that are not included in MMMG .

\mathbb{T}

denotes text modality, for image modality, for multiple images, for audio and for multiple audios. We hope to incorporate these tasks when reliable evaluation methods are available.

Statistics	Number
Total number of modality combinations	4
Total number of tasks	55
- I : A : I-T : A-T	15 : 12 : 22 : 6
Total number of questions	1288
- I : A : I-T : A-T	410 : 238 : 480 : 160
Total number of images	542

Appendix

Table 5: Statistics of MMMG . I, A, I-T, A-T stands for image, audio, interleaved image-text and interleaved audio-text generations respectively.

You are a multimodal assistant capable of generating both text and images. When visual content would enhance your response or is specifically requested, you can generate or edit images through advanced diffusion models. To generate or edit an image: 1. Identify when visual content would be beneficial or requested. 2. Insert an image generation/editing placeholder using the following format: <image_start><image_prompt="Detailed image generation or editing prompt here."><image_ref=[reference identifiers]><image_end> 3. The post-processing system replaces this placeholder with an image created or edited based on your instructions. 4. Naturally incorporate references to the generated or edited image in your ongoing conversation. When crafting image prompts, follow these guidelines: For image prompts: • Provide detailed, specific descriptions (15-30 words) for optimal results. • Include artistic styles (photorealistic, cartoon, watercolor, etc.) or style transfers. • Specify key objects and their attributes (colors, textures, etc.), or modifications. • Detail composition elements (spatial relationships, perspective, lighting, etc.), or compositional changes. • Ensure instructions are clear and concise. For image references: Three reference types are available: 1. Image generation (no reference): <image_ref=[]> 2. Editing user-provided images: Format: <image_ref=[i]> where i is the index of the provided image (indices starting at 0). Example: <image_ref=[0]> references the first provided image. Multiple images example: <image_ref=[0,2]> references the first and third provided images. 3. Editing previously generated images: Format: <image_ref=[#N]> , where N is the sequential number of previously generated images (starting from 0). Example: <image_ref=[#3]> references the fourth generated image. Multiple images example: <image_ref=[#0,#2]> references the first and third generated images. Important: Use only one reference type within each placeholder. Different reference types may be used across multiple placeholders. Provide concise and direct responses following user instructions precisely. Always maintain the exact placeholder format for proper parsing, ensuring that both images and text appear in the required order. Do not omit any necessary text following image placeholders.

Appendix

Table 6: System prompt for interleaved image-text agent.

You are a multimodal assistant capable of generating both text and audio. When audio content would enhance your response or is specifically requested, you can generate audio through text-to-audio models. To generate audio: 1. Identify when audio content would be beneficial or requested. 2. Insert an audio generation placeholder using the format: <audio_start><audio_type="sound" OR "speech" OR "music"><audio_text="Text to be spoken here."><audio_style="Descriptive text here." OR audio reference ID><audio_end> 3. The post-processing system replaces this placeholder with generated audio based on your specifications. 4. Naturally incorporate references to the generated audio in your ongoing conversation. When crafting audio prompts, follow these guidelines: Audio Type: • Must be exactly one of: "sound" , "speech" , or "music" . • "speech" : For human speech. • "sound" : For environmental sounds or effects. • "music" : For musical compositions or instrumental pieces. Audio Text: • For "speech" : Provide the exact transcript. • For "sound" or "music" : Leave as empty string (""). • Keep speech concise (typically under 50 words). Audio Style: 1. Descriptive Text: • For "speech" : Specify voice characteristics (gender, emotion, pace, pitch, accent). • For "sound" : Specify sound source, environment, qualities. • For "music" : Specify genre, mood, tempo, instruments. 2. Reference Audio: • For consistency, particularly with speech: – Previously generated audio: <audio_style=#N> ( N is sequential number starting at 0). – User-provided audio: <audio_style=N> ( N is sequential number of provided audio starting at 0). • Important: Only reference audio that itself does not reference previous audio to avoid circular references. Provide concise, direct responses precisely following user instructions. In multi-speaker scenarios, maintain consistent and distinctive voice characteristics for each speaker. Always maintain the exact placeholder format for correct parsing

Appendix

Table 7: System prompt for interleaved audio-text agent.

Figure 4: Human annotation interface for instrument inclusion task. Typically, an inference will include reference audios/images, model’s generation, evaluation instruction, evaluation criteria and judgment radio boxes and next/previous button.

Task	GPT-4o		Gemini 2.5		Qwen2.5-VL		IAA
Task	agree	corr	agree	corr	agree	corr	agree	corr
Object Inclusion	0.925	0.776	0.900	0.715	0.888	0.657	1.000	1.000
Object Exclusion	0.963	0.924	0.913	0.823	0.925	0.855	1.000	1.000
Object Count	0.875	0.709	0.963	0.912	0.900	0.763	0.975	0.943
Object Knowledge	0.963	0.925	0.963	0.925	0.938	0.875	1.000	1.000
Object Commonsense	0.913	0.787	0.888	0.696	0.938	0.822	1.000	1.000

Appendix

Table 8: Agreement and Pearson correlation of MMMG evaluation with human annotations. “IAA” stands for inter-annotator agreement, “agree” stands for agreement and “corr” stands for Pearson correlation. We report Word Accuracy for text rendering, text editing and OCR tasks. Best results are in bold . MMMG achieves an average best human agreement of 0.944 with average inter-annotator agreement being 0.971. GPT-4o is the most human-aligned image evaluation model while

\mathrm{CLAPScore}_{\mathrm{audio}}

is the most human-aligned audio evaluation method.

Task	Imagen 3	Recraft v3	Luma Photon	Flux 1.1 Pro	Ideo -gram 2	Dalle 3	SD 3.5	Gemini 2 Image	GPT Image	BILP-o3	Janus Pro	Gemini 2.5 Image
Object Inclusion	0.838	0.688	0.831	0.444	0.863	0.788	0.544	0.844	0.869	0.475	0.706	0.919
Object Exclusion	0.338	0.300	0.425	0.325	0.469	0.244	0.013	0.281	0.819	0.138	0.063	0.681
Object Count	0.269	0.319	0.369	0.375	0.319	0.119	0.256	0.356	0.569	0.138	0.263	0.450
Object Knowledge	0.494	0.481	0.656	0.325	0.419	0.463	0.150	0.706	0.531	0.238	0.081	0.588
Object Commonsense	0.256	0.306	0.288	0.206	0.288	0.288	0.306	0.275	0.163	0.194	0.175	0.363
Object Attribution	0.325	0.206	0.319	0.256	0.275	0.294	0.244	0.375	0.619	0.331	0.363	0.475

Appendix

Table 9: Benchmarking results of 10 models on 15 image generation tasks. Best results are in bold . GPT-4o significantly outperforms other image generation models.

Task	Seed Llama	Anole	GPT-4o + GPT Image	Gemini 2.5 + GPT Image	Gemini 2 Image	GPT Image	Gemini 2.5 Image
Semantic Consistency	0.000	0.000	0.613	0.763	0.013	0.675	0.325
Multi-angle Consistency	0.000	0.000	0.230	0.461	0.352	0.448	0.367
Multi-view Consistency	0.000	0.000	0.064	0.221	0.143	0.188	0.191
Composition Consistency	0.000	0.000	0.800	0.738	0.000	0.075	0.225
Decomposition Consistency	0.000	0.000	0.600	0.875	0.013	0.575	0.638
Self Count	0.000	0.038	0.100	0.850	0.213	0.763	0.000

Appendix

Table 10: Benchmarking results of 6 models on 22 image-text interleaved generation tasks. Best results are in bold . Agent model Gemini 2.5 Pro + GPT Image is the best combination for consistent image sequence and coherent image-text pair generation. Gemini 2.5 Image as a modality-unified autoregressive model, performs best at image editing tasks.

Task	Stable Audio	Audio LDM 2	AudioGen	Make-An -Audio 2	Tango 2	MusicGen	Tango Music	Yue
Sound Begin-End	0.525	0.450	0.475	0.631	0.525	-	-	-
Sound Inclusion	0.700	0.413	0.450	0.575	0.513	-	-	-
Sound Reasoning	0.014	0.014	0.042	0.611	0.194	-	-	-
Sound Silence	0.063	0.019	0.019	0.131	0.006	-	-	-
Instrument Inclusion	0.817	0.833	-	-	-	0.833	0.950	0.600
Instrument Exclusion	0.225	0.163	-	-	-	0.200	0.050	0.525

Appendix

Table 11: Benchmarking results of 8 models on 9 sound and music generation tasks. Make-An-Audio 2 is the best audio generation model and the only model that can perform sound reasoning task; MusicGen is the best music generation model and the only model that can have tempo control.

Task	Gemini 2.5 + VoxInstruct	Gemini 2.5 + VoiceLDM	Spirit LM
Voice Attribution	0.684	0.568	0.000
Voice Replication	0.625	0.109	0.002
Speech Multi-lingual	0.654	-	-
Transcript Generation	0.638	0.438	0.200
Transcript Editing	0.200	0.350	0.000
Conversation Generation	0.788	0.375	0.000

Appendix

Table 12: Benchmarking results of 3 models on 9 speech-text interleaved generation tasks. Best results are in bold . Natural speech-text interleaved model Spirit LM does not have instruction following capability and get zero for most tasks. VoxInstruct is the best multi-functional speech synthesizer.

Task	Imagen 3	Recraft v3	Luma Photon	Flux 1.1 Pro	Ideo -gram 2	Dalle 3	SD 3.5	Gemini 2 Image	GPT Image	BILP-o3	Janus Pro	Average
Object Inclusion	4.24	3.16	5.05	5.43	4.24	4.24	5.79	3.08	3.67	4.90	3.08	4.26
Object Exclusion	4.24	5.29	7.21	5.29	6.44	2.35	2.45	6.44	1.22	4.24	1.41	4.24
Object Count	3.08	2.35	4.18	5.29	5.05	6.75	1.23	5.43	5.43	5.10	3.16	4.28
Object Knowledge	1.23	3.08	2.35	2.00	3.67	5.10	7.21	1.23	5.05	5.83	2.35	3.35
Object Commonsense	2.35	6.44	4.69	2.35	1.41	4.69	3.67	2.00	4.24	4.64	6.93	3.95
Object Attribution	3.46	3.08	5.05	5.79	7.75	1.22	3.67	4.47	3.08	2.35	2.45	3.85

Appendix

Table 13: 95% relative confidence intervals of 9 models on 15 image generation tasks. The numbers are in percentile. Highest CIs are in bold .

Task	Seed Llama	Anole	GPT-4o + GPT Image	Gemini 2.5 + GPT Image	Gemini 2 Image	GPT Image	Average
Semantic Consistency	0.00	0.00	7.35	2.45	2.45	6.33	3.10
Multi-angle Consistency	0.00	0.00	2.54	0.55	4.30	2.69	1.68
Multi-view Consistency	0.00	0.00	1.03	0.20	2.01	0.78	0.67
Composition Consistency	0.00	0.00	6.93	9.28	0.00	6.33	3.76
Decomposition Consistency	0.00	0.00	6.93	2.83	2.45	6.33	3.09
Self Count	0.00	2.45	6.93	6.93	4.69	8.37	4.89

Appendix

Table 14: 95 relative confidence intervals of 6 models on 22 image-text interleaved generation tasks. The numbers are in percentile. Highest CIs are in bold .

Task	Stable Audio	Audio LDM 2	AudioGen	Make-An -Audio 2	Tango 2	MusicGen	Tango Music	Yue	Average
Sound Begin-End	5.66	2.83	13.12	8.95	5.25	-	-	-	7.16
Sound Inclusion	10.59	8.37	5.66	2.45	5.13	-	-	-	6.44
Sound Reasoning	2.72	2.72	2.72	5.44	1.94	-	-	-	3.11
Sound Silence	2.45	3.67	1.23	1.23	0.63	-	-	-	1.84
Instrument Inclusion	6.26	3.77	-	0.00	-	3.77	3.27	0.00	2.84
Instrument Exclusion	2.83	7.35	-	0.00	-	5.66	5.66	2.83	4.05

Appendix

Table 15: 95 relative confidence intervals of 8 models on 9 sound and music generation tasks. The numbers are in percentile. Highest CIs are in bold .

Task	Gemini 2.5 + VoxInstruct	Gemini 2.5 + VoiceLDM	Spirit LM	Average
Voice Attribution	4.14	6.08	0.01	5.11
Voice Replication	5.91	2.80	0.13	4.35
Speech Multi-lingual	4.06	0.17	0.00	2.12
Transcript Generation	7.35	12.89	0.00	10.12
Transcript Editing	5.66	12.00	0.00	8.83
Conversation Generation	7.35	8.49	0.00	7.92

Appendix

Table 16: 95 relative confidence intervals of 3 models on 9 speech-text interleaved generation tasks. The numbers are in percentile. Highest CIs are in bold .

Task	Obj. Inc.	Obj. Exc.	Obj. Cou.	Obj. Kno.	Obj. Com.	Obj. Att.	Com. Rel.	Uni. Rel.	Rel. Spa.	Abs. Spa.	Reg. Fill	Bor. Fill	Sin. TR	Dou. TR	Mul. TR
Object Include	-	0.677	0.464	0.712	0.441	0.557	0.734	0.654	0.720	0.501	0.473	0.096	0.570	0.611	0.627
Object Exclude	0.677	-	0.859	0.671	0.232	0.742	0.884	0.930	0.889	0.540	0.691	0.389	0.601	0.799	0.656
Object Count	0.464	0.859	-	0.609	0.198	0.623	0.673	0.919	0.857	0.680	0.696	0.378	0.767	0.894	0.838
Object Knowl.	0.712	0.671	0.609	-	0.466	0.363	0.764	0.671	0.700	0.529	0.530	0.309	0.634	0.772	0.663
Object Common.	0.441	0.232	0.198	0.466	-	-0.120	0.304	0.298	0.196	-0.068	0.087	-0.458	0.477	0.453	0.220
Object Attribute	0.557	0.742	0.623	0.363	-0.120	-	0.664	0.664	0.740	0.615	0.823	0.612	0.188	0.449	0.550

Appendix

Table 17: Image generation task separability analysis based on model performance Pearson correlations. The lowest correlation are in bold .

You are a multimodal assistant capable of generating interleaved text and images based on user instructions. • Follow the required modality structure and number in user’s instruction exactly, especially when multiple images are implied or requested. • Generate separate images for each described part, do not combine multiple concepts into one image unless told to. • Interleave images and text in the order described. Your goal is to match the user’s intent with exact number and sequence of image and text.

Appendix

Table 18: System prompt used to make Gemini Image output correct modality order and number.

Task	Gemini Image w/ prompt	Gemini Image w/o prompt
Semantic Consistency	0.263	0.013
Multi-Angel Consistency	0.135	0.352
Multi-View Consistency	0.094	0.143
Compose Consistency	0.013	0.000
Decompose Consistency	0.000	0.013
Interleaved Object Adding	0.399	0.545

Appendix

Table 19: Comparison of Gemini Image performance with and without system prompt on image generation (right) and interleaved image-text generation (left) tasks. Best results are in bold . System prompt does not always have positive impact.

Error Category	% in All	Observation	Plausible Causes
Audio generation failure	20.0%	Generated audio consists of pure noise with no meaningful structure.	CLAPScore is not OOD-generalizable. When generation quality is low, it cannot make reliable judgments.
Audio generation quality unsatisfactory	26.7%	Generated audio contains recognizable and reasonable sounds but is overall of low quality with distortion. For example, the models repeatedly failed to generate proper reggae music, often producing tracks dominated by drums.	CLAPScore is not OOD-generalizable. When generation quality is low, it cannot make reliable judgments.
Lack of fine-grained understanding	26.7%	Some generated audio includes fine-grained details that are easily recognizable by humans, such as brief dog barks or brief sneezes, but their duration is too short. Others contain noisy background such as sirens in loud traffic, which is treated as a meaningless piece by the CLAP model.	Since CLAPScore computes the similarity of the whole generated audio with the reference audio, short duration or noisy background will underestimate such similarity. Future methods could attempt to locate the target sound first before computing similarity.
Underrepresented reference audio	26.7%	Some generated audio is not representative in the reference audio. A peaceful drum sequence is not considered as a drum piece by the CLAP model since all the reference drum music is metallic-like.	The reference datasets we use are standard and may not fully capture a sound, an instrument, or a music genre. Enriching the dataset diversity is important for more reliable evaluation.

Appendix

Table 20: Error analysis of CLAPScore. The total error rate is 7.7%.

Error Category	% in All	Observation	Plausible Causes
Generated speech with noise or distortion	21.4%	Some generated speeches contain noticeable background white noise. Sometimes, the noise is intermittent volume fluctuation (sometimes slightly lower, sometimes higher) or long pauses, which may interfere with the model’s judgment.	WavLM is not OOD-generalizable. When generation quality is low, it cannot make reliable judgments.
Generated speech with synthetic pattern	57.1%	Some generated speeches can be considered as coming from the same speaker, but with clear synthetic patterns, making it hard even for human judges to decide whether the generated speeches are from the same speaker.	WavLM is not OOD-generalizable. It is trained on real-world human voice and may not effectively judge synthetic speeches.
Computed similarity score close to threshold	21.4%	The generated speeches are very similar to the references, aside from slight white noise. This likely caused the model’s judgment to be only minimally affected, placing the scores near the decision threshold.	WavLM is not well-calibrated. It will only give high scores to the same speaker and low scores to different speakers. When the score is close to the threshold, it is likely to make mistakes.

Appendix

Table 21: Error analysis of WavLM . The total error rate is 13.7%.

Error Category	Examples	Possible Reasons	# (%) of All Err.	# (%) of Gem Err.	# (%) of GPT Err.
Fail to render required scene properly	Instruction: “Create an image of a waterfall flowing into the sky, containing one pumpkin and one suitcase.” Error: The waterfall is rendered normally, not flowing into the sky.	The model relies heavily on physical laws and realistic priors embedded in the training data, and struggles to synthesize scenes that violate common-sense physics.	3 (1.2%)	0 (0.0%)	3 (3.0%)
Fail to render objects of the required number	Instruction: “Generate an image of a historical site. Please include a single spaceship and a giraffe in the image.” Error: Two spaceships are generated even though the instruction asks for one.	The model demonstrates limited numeracy and counting capabilities. This may be due to the attention mechanism, which often fails to disentangle identical object instances and leads to superfluous objects.	49 (19.9%)	27 (18.5%)	22 (22.0%)
Object rendered with distortion or unrealistic appearance	Instruction: “Create an image of an endless mirrored hallway, with one cactus and one saxophone.” Error: The mirrored hallway is distorted.	This may be due to under-training. Models may not be trained on the target objects enough to reproduce full details.	15 (6.1%)	6 (4.1%)	9 (9.0%)
Required object not present, partially present, or wrong object present	Instruction: “Create an image of a whale flying through a sunset sky, featuring one suitcase and one mailbox.” Error: The image does not include a suitcase.	Failed instruction following may indicate under post-training. Instead of generating the required image, the model generates the most plausible image.	1 (0.4%)	0 (0.0%)	1 (1.0%)
Unable to remove the correlated object	Instruction: “Generate an image of people camping. Do not include tents in the image.” Error: The generated image still contains tents.	Negative constraints paradoxically activate associated concepts in the latent space. In addition, strong semantic co-occurrence between scenes and objects (spurious correlation) makes it difficult to decouple these associations.	31 (12.6%)	24 (16.4%)	7 (7.0%)
Fail to render required attribute (single object)	Instruction: “Generate an image of a single chair with 5 legs.” Error: The generated chair has 4 legs.	Strong visual priors, such as the normative four-legged chair, override textual prompts and create a conflict between internal knowledge and counterfactual instructions.	28 (11.4%)	20 (13.7%)	8 (8.0%)

Appendix

Table 22: Generation model failure analysis on image generation tasks (Part I). Analyzed models are Gemini Image 2.0 and GPT Image .

Error Category	Examples	Possible Reasons	# (%) of All Err.	# (%) of Gem Err.	# (%) of GPT Err.
Fail to apply attribute requirements to all objects (multiple objects)	Instruction: “Generate an image of birds on a wire where all birds are facing the same direction.” Error: Not all birds are facing the same direction.	The model has difficulty maintaining attribute consistency across multiple entities. Stochastic generation processes often fail to enforce constraints uniformly on every instance.	6 (2.4%)	4 (2.7%)	2 (2.0%)
Fail to apply attribute requirements only to the required objects (multiple objects)	Instruction: “Generate an image of a bookshelf where all books are standing vertically except one lying horizontally.” Error: More than one book is lying horizontally.	High logical complexity hampers the execution of exception logic, causing the model to over-generalize rules to excluded entities.	6 (2.4%)	5 (3.4%)	1 (1.0%)
Wrong / insufficient knowledge to reason the target object, wrong object generated	Instruction: “Create an image featuring only the national flag of a country that has a major city located on both the European and Asian continents.” Error: The model generates the French flag instead of the Turkish flag.	Unlike LLMs, image generation models are usually not trained on extensive world knowledge, so they can only follow the literal instruction and may fail to infer the intended target object.	20 (8.1%)	8 (5.5%)	12 (12.0%)
Wrong / insufficient knowledge to reason the target object, multiple objects generated	Instruction: “Generate an image of a small, handheld stringed instrument, commonly used in folk and country music and important in black American music.” Error: The model generates multiple instruments.	This shows signs of reward hacking in post-training. When the reward model does not punish guessing behavior, the model may generate multiple objects instead of following the instruction.	1 (0.4%)	0 (0.0%)	1 (1.0%)
Fail to render the required logical relation between objects	Instruction: “Create an image featuring a single nail and a single snake, and the nail is longer than the snake.” Error: The nail is still shorter than the snake.	The unrealistic logical relation in the prompt conflicts with the model’s internal semantic priors, and the model favors the prior.	25 (10.2%)	17 (11.6%)	8 (8.0%)
Fail to render the required spatial relation between objects	Instruction: “Generate an image of a peaceful garden, with a single watering can at the upper right quarter of the image.” Error: The watering can is only on the right side.	Models are trained on image-caption pairs that stress semantic consistency and overlook perception accuracy.	12 (4.9%)	6 (4.1%)	6 (6.0%)

Appendix

Table 23: Generation model failure analysis on image generation tasks (Part II). Analyzed models are Gemini Image 2.0 and GPT Image .

Error Category	Examples	Possible Reasons	# (%) of All Err.	# (%) of Gem Err.	# (%) of GPT Err.
Fail to render image format at all	Instruction: “Generate a space scene with planets and nebulae. The entire image should be surrounded by a simple and flat, solid and black border of approximately 10% of the image width on all sides.” Error: The scene is generated without any border.	Formatting constraints are ignored because the model interprets them as soft suggestions rather than hard rules. This is likely an OOD problem, where such image formats are rarely requested in training data.	1 (0.4%)	0 (0.0%)	1 (1.0%)
Rendered text with misspelling	Instruction: “Generate the text ‘always move forward’.” Error: The generated text is misspelled as “always move forwad.”	Text rendering is a known problem for image generation. Models operate at the pixel level rather than the character level, and without explicit orthographic verification they tend to generate pseudo-text that is visually similar but orthographically incorrect.	10 (4.1%)	6 (4.1%)	4 (4.0%)
Rendered text with duplication	Instruction: “Generate the text ‘capture the moment’.” Error: The generated text becomes “capture the the moment.”	Repetition artifacts arise from loops or redundancies in the text-generation attention mechanism.	3 (1.2%)	2 (1.4%)	1 (1.0%)
Required text not rendered	Instruction: “Generate the text ‘Washington WA 98105 Evergreen State’.” Error: No text is generated.	Lack of text rendering training makes the model neglect text requirements.	0 (0.0%)	0 (0.0%)	0 (0.0%)
Required text rendered in distortion	Instruction: “Generate the text ‘Enjoy The Little Things Always’.” Error: The word “Enjoy” has distorted glyphs.	Lack of text rendering training makes the model unable to generate high-quality text.	1 (0.4%)	1 (0.7%)	0 (0.0%)
Required text in the wrong place or overflow	Instruction: “Generate an image of exactly two posters on an office wall…” Error: The model generates three posters, and the second text appears on the middle poster.	Lack of text rendering training makes the model unable to place text correctly. Adding a layout controller or training on such data may help.	5 (2.0%)	2 (1.4%)	3 (3.0%)

Appendix

Table 24: Generation model failure analysis on image generation tasks (Part III). Analyzed models are Gemini Image 2.0 and GPT Image .

Error Category	Examples	Possible Reasons	# (%) of All Err.	# (%) of Gem Err.	# (%) of Age Err.
Generate wrong number of images	Instruction: “Using the provided image as the reference angle, create four additional images…” Error: The model generates only three images and misses the final view.	Difficulty in precise counting and planning; the model loses track of the count during the sequential generation loop.	40 (12.2%)	33 (13.9%)	7 (7.6%)
Only image description provided but no image	Instruction: “Create an image depicting a musician’s room that includes exactly one microphone, one guitar case, and one music stand…” Error: The model provides the text evaluation but does not generate the actual image.	Tool-use failure; the model prioritizes text reasoning instead of actually calling the image generation tool.	2 (0.6%)	2 (0.8%)	0 (0.0%)
Low image quality or distorted, unrealistic object	Instruction: “Based on the provided image showing a frontal view, create four more images depicting the scene from specific angles…” Error: Later views become severely distorted and blurry.	This may be due to under-training. Models may not be trained on the target objects enough to reproduce full details.	4 (1.2%)	4 (1.7%)	0 (0.0%)
Combine multiple images together instead of generating multiple images	Instruction: “Based on the provided image showing the frontal view, create four additional images…” Error: The model generates a single image grid instead of four separate images.	Models, especially modality-unified ARMs, are likely to entangle multiple images in one output due to continuous latent representations and a training bias toward single-turn unified outputs.	3 (0.9%)	3 (1.3%)	0 (0.0%)
Inconsistent image sequence: objects, style, or scene change during generation	Instruction: “Create four images that sequentially show the addition of a passport, a map, a camera, and a pair of sunglasses…” Error: Previously added objects disappear and the suitcase style changes.	Agent models do not show strong planning capabilities on image generation tasks. ARMs also lack cross-image consistency mechanisms, so state is not maintained across steps.	16 (4.9%)	8 (3.4%)	8 (8.7%)
Image sequence shows no variance: image repetition	Instruction: “Using the provided image as the reference angle, create four additional images…” Error: All generated images are identical to the original reference view.	Mode output collapse. This is likely related to the same phenomenon as repetitive outputs in LLMs, and may indicate lack of post-training.	11 (3.3%)	8 (3.4%)	3 (3.3%)

Appendix

Table 25: Generation model failure analysis on interleaved image-text generation tasks (Part I). Analyzed models are Gemini Image 2.0 and Gemini 2.5 Pro + GPT Image Agent.

Error Category	Examples	Possible Reasons	# (%) of All Err.	# (%) of Gem Err.	# (%) of Age Err.
Fail to render objects of the required number	Instruction: “Create an image of an office desk that includes a stapler, a mouse, and a pen…” Error: The generated image contains two pens, violating the “exactly once” constraint.	The model demonstrates limited numeracy and counting capabilities. This may be due to the attention mechanism, which often fails to disentangle identical object instances and leads to superfluous objects.	12 (3.6%)	10 (4.2%)	2 (2.2%)
Failed spatial generation: images are not in the position / angle asked	Instruction: “Using the provided image as the reference angle, create four additional images…” Error: The model generates two images at 30 degrees to the right and two at 30 degrees to the left.	Poor understanding of 3D position. Models are trained on image-caption pairs that stress semantic consistency and overlook other perception abilities.	4 (1.2%)	2 (0.8%)	2 (2.2%)
Failed temporal generation: images do not show the required adding / removing order	Instruction: “Create three images that sequentially show the addition of a coffee mug, a notebook, and a pen…” Error: The first image already shows the final result.	Agent models do not show strong planning capabilities on image generation tasks. ARMs also lack cross-image consistency mechanisms, so state is not maintained across steps.	21 (6.4%)	20 (8.4%)	1 (1.1%)
Failed logic generation: images do not show the required logical order	Instruction: “Create three images, each featuring a single balloon of a different color… Arrange the images in alphabetical order…” Error: The generated order is still red, yellow, and purple.	Agent models do not show strong planning capabilities on image generation tasks. ARMs also lack cross-image consistency mechanisms, so state is not maintained across steps.	11 (3.3%)	7 (3.0%)	4 (4.3%)
Wrong editing region	Instruction: “Create an image after the utensils have been removed from the photo.” Error: The model removes the food but leaves the utensils untouched.	Current training for image editing lacks verifiable and accurate reward, which makes the editing region inaccurate.	5 (1.5%)	3 (1.3%)	2 (2.2%)
Oversized / undersized editing region	Instruction: “Create an image that displays the result after removing the man’s wig…” Error: The edit also removes part of the forehead and background.	Current training for image editing lacks verifiable and accurate reward, which makes the editing region inaccurate.	2 (0.6%)	1 (0.4%)	1 (1.1%)

Appendix

Table 26: Generation model failure analysis on interleaved image-text generation tasks (Part II). Analyzed models are Gemini Image 2.0 and Gemini 2.5 Pro + GPT Image Agent.

Error Category	Examples	Possible Reasons	# (%) of All Err.	# (%) of Gem Err.	# (%) of Age Err.
Invalid edit: edit not applied to the required text / object	Instruction: “Create an image that shows the result after replacing the cop with a smiling clown…” Error: The output image is identical to the input image.	Under-training for image editing tasks. Models cannot understand the editing instruction well.	20 (6.1%)	16 (6.8%)	4 (4.3%)
Partial edit: edit only applies to a single object / text	Instruction: “Generate an image showing the editing result after making all the sprinkles on the cupcakes blue…” Error: Only the front cupcake is edited.	Under-training for image editing tasks. Models cannot understand the editing instruction well.	4 (1.2%)	3 (1.3%)	1 (1.1%)
Global changes outside the editing region	Instruction: “Create an image displaying the result after replacing the highest kite with an eagle…” Error: The eagle is added correctly, but the whole sky changes color.	Current training for image editing lacks verifiable and accurate reward, which makes the editing scope difficult to control.	11 (3.3%)	4 (1.7%)	7 (7.6%)
Generated image-text interleaved content not in the required modality order	Instruction: “For each phase, start with an image that illustrates the phase, followed by a written explanation…” Error: The model outputs all text first and all images at the end.	Lack of instruction-following training for interleaved image-text content. Models cannot maintain the required output structure and modality sequence.	6 (1.8%)	5 (2.1%)	1 (1.1%)
Parsing error: wrong function-calling format for image generation	Instruction: “Develop a 3-step guide…” Error: The model outputs raw tool code or no image at all.	Lack of training on agentic data. Models cannot reliably output the formatted text required by tool agents.	0 (0.3%)	1 (0.4%)	0 (0.0%)
Parsing error: failed to generate text in the required format	Instruction: “…return only a JSON object that maps each item to its corresponding color…” Error: The model returns a bulleted list instead of valid JSON.	Lack of training on agentic data. Models cannot reliably output the formatted text required by tool agents.	19 (5.8%)	19 (8.0%)	0 (0.0%)

Appendix

Table 27: Generation model failure analysis on interleaved image-text generation tasks (Part III). Analyzed models are Gemini Image 2.0 and Gemini 2.5 Pro + GPT Image Agent.

Error Category	Examples	Possible Reasons	# (%) of All Err.	# (%) of Gem Err.	# (%) of Age Err.
Image inconsistent with the self-generated object description in text	Instruction: “Create four images that sequentially show the result after removing the sunglasses, the camera, the map, and the passport…” Error: The text says the passport is removed, but the final image still shows the passport.	Lack of training on interleaved image-text tasks. Models cannot produce self-consistent image-text pairs. For agent models, this is often because the image generation tool does not strictly follow the plan, and the planning model cannot correct it with feedback.	23 (7.0%)	15 (6.3%)	8 (8.7%)
Image inconsistent with the self-generated relation description in text	Instruction: “…answer the following two questions…” Error: The model answers “Left,” but the generated image places the object on the right.	Lack of training on interleaved image-text tasks. Models cannot produce self-consistent image-text pairs. For agent models, this is often because the image generation tool does not strictly follow the plan, and the planning model cannot correct it with feedback.	45 (13.7%)	31 (13.1%)	14 (15.2%)
Image inconsistent with the self-generated OCR recognition results in text	Instruction: “…after generating the image, output only the announcement text in XML format…” Error: The XML output says “Science Fair,” but the rendered image contains gibberish.	Lack of training on interleaved image-text tasks. Models cannot produce self-consistent image-text pairs. For agent models, this is often because the image generation tool does not strictly follow the plan, and the planning model cannot correct it with feedback.	16 (4.9%)	14 (5.9%)	2 (2.2%)
No step-by-step reasoning analysis	Instruction: “Analyze the following SVG code step-by-step…” Error: The model gives only a short summary instead of the required step-by-step reasoning.	Under-training for multimodal reasoning tasks involving image generation. RLVR may improve such multimodal reasoning capabilities if more verifiable data like MMMG is released.	2 (0.6%)	2 (0.8%)	0 (0.0%)
Wrong reasoning analysis	Instruction: “What geometric shape does this SVG code describe?…” Error: The model reasons incorrectly and concludes that the code describes a circle.	The model hallucinates the function of the code or cannot mentally simulate the SVG geometry.	46 (14.0%)	22 (9.3%)	24 (26.1%)
Failed generation	Instruction: “…” Error: The output is a completely black image or a corrupted file that cannot be opened.	Activation of safety filters or internal system generation failures.	5 (1.5%)	4 (1.7%)	1 (1.1%)

Appendix

Table 28: Generation model failure analysis on interleaved image-text generation tasks (Part IV). Analyzed models are Gemini Image 2.0 and Gemini 2.5 Pro + GPT Image Agent.

Explore similar work

Date pendingcs.CL

MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation

Modern multimodal generative models can synthesize visually compelling images and videos, but it remains unclear whether this visual fluency reflects genuine reasoning: when prompted to generate a solution, can a model preserve the physical, logical, spatial, and temporal constraints a task requires, or does it merely produce plausible-looking media? To answer this question, we introduce MMGR (Multi-Modal Generative Reasoning Benchmark and Evaluation), a benchmark for evaluating generative reasoning across video, image, and language-based systems. MMGR covers 10 tasks from three domains (Abstract Reasoning, Embodied Navigation, and Physical Commonsense) and probes five reasoning abilities: Physical, Logical, 2D Spatial, 3D Spatial, and Temporal. Its evaluation emphasizes answer-verifiable tasks and, for video generation, process-aware chain-of-frame reasoning, where intermediate frames must form valid steps toward the target outcome rather than visually smooth but incorrect transitions. Evaluating state-of-the-art video generators, image generators, and LLM/VLM baselines reveals a sharp gap between visual quality and reasoning correctness: video models perform best on Physical Commonsense, but remain weak on symbolic tasks such as Sudoku, ARC, and Math, and brittle in cross-view embodied navigation. Image generators often outperform video generators on embodied navigation despite lacking temporal outputs, showing that longer visual generation does not automatically yield stronger reasoning. MMGR reframes evaluation of multimodal generation from whether outputs look realistic to whether they solve the underlying reasoning problem.

Zefan Cai, Haoyi Qiu, Tianyi Ma +13

University of Wisconsin–Madison · University of California, Los Angeles · Michigan State University +8

Multimodal Reasoning Benchmarks Video Generation Fluency Object Goal Navigation

Ask PDF

Feb 9, 2026cs.CL

UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Models

Unified multimodal models (UMMs) aim to integrate multimodal understanding and generation within a unified architecture, yet it remains unclear to what extent textual and visual modalities are aligned. To investigate this question, we use reasoning-guided image generation as a diagnostic task, where models produce textual reasoning first and then generate images. We introduce UReason, a benchmark for evaluating reasoning-to-generation alignment in this paradigm, consisting of 2,000 human-curated and human-verified instances spanning five reasoning-intensive tasks: Code, Arithmetic, Spatial, Attribute, and Text. To enable controlled analysis, we develop an evaluation framework that compares direct generation, reasoning-guided generation, and decontextualized generation, which conditions only on the refined prompt extracted from reasoning. Across eight widely used open-source UMMs, while we find that reasoning-guided generation yields improvements over direct generation, somewhat surprisingly, decontextualized generation consistently outperforms reasoning-guided generation by a large margin. Our further analyses suggest that the intended visual semantics in textual reasoning are not reliably reflected in the generated images, despite their unified design and training. Overall, UReason serves as a practical litmus test for reasoning-to-generation alignment and provides a challenging benchmark for developing next-generation, more tightly aligned UMMs.

Cheng Yang, Chufan Shi, Bo Shui +7

University of California San Diego · University of Southern California · University of Illinois Urbana-Champaign +1

Multimodal Generation Multimodal Model Large Language Model Alignment

Ask PDF

May 1, 2025cs.CV

Multimodal Language Models as Text-to-Image Model Evaluators

The steady improvements of text-to-image (T2I) generative models lead to slow deprecation of automatic evaluation benchmarks that rely on static datasets, motivating researchers to seek alternative ways to evaluate T2I progress. We present Multimodal Text-to-Image Eval (MT2IE), an evaluation framework in which a single multimodal large language model (MLLM) acts as an evaluator agent, iteratively generating the evaluation prompts and scoring the resulting images. We show that MT2IE's image-text consistency scores have higher correlation with human judgment than metrics previously introduced in the literature. MT2IE generates prompts that are efficient at probing T2I model performance: closely recovering the official T2I model rankings of three structurally distinct benchmarks from just 20 generated evaluation prompts, 28-105x fewer than the benchmarks' own prompt sets. When compared to existing evaluation metrics such as CLIPScore, VIEScore, and VQAScore, MT2IE's T2I model rankings are more faithful and far more consistent across multiple evaluation seeds when using the same number of prompts. MT2IE can also adapt evaluation to the model being tested: rewriting each prompt based on the model's own measured performance to produce a bespoke per-model benchmark that still recovers the official rankings and keeps the evaluated model in an informative scoring range. We hope that these results will encourage the development of dynamic and interactive evaluation frameworks, and mitigate the deprecation of automatic evaluation benchmarks.

Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat +5

1FAIR at Meta · University of Texas at Austin · ∗Work done at Meta +3

Modern Text-To-Image Models Text-To-Image Evaluation Benchmarks Multimodal Large Language Models Generative Models Similarity and Metrics

Ask PDF