A Survey of Generative Categories and Techniques in Multimodal Generative Models

Longzhen Han; Awes Mubarak; Almas Baimagambetov; Nikolaos Polatidis; Thar Baker

cs.MMMay 29, 2025

A Survey of Generative Categories and Techniques in Multimodal Generative Models

Authors: Longzhen Han, Awes Mubarak, Almas Baimagambetov, Nikolaos Polatidis, Thar Baker

Organizations: School of Architecture, Technology and Engineering, University of Brighton, Lewes Road, BN2 4GJ, United Kingdom · University of Khorfakkan, Sharjah, United Arab Emirates

Abstract

Multimodal generative models (MGMs) have rapidly expanded beyond text generation to encompass images, music, video, human motion, and 3D objects. This survey categorises six primary text-conditioned generative modalities and examines how four techniques, namely self-supervised learning, mixture of experts, reinforcement learning from human feedback, and chain-of-thought prompting, are adapted across them, distinguishing shared mechanisms from experimentally supported benefits. We organise evaluation around faithfulness, compositionality, and robustness, and interpret evidence within each study's protocol. Selected within-study comparisons reported in the reviewed literature support four findings: preference gains need not improve text relevance, and reward optimisation can reduce within-prompt diversity; intermediate plans require tests of their realisation in final outputs; expert granularity changes the mechanism and its storage, computation, and runtime costs; and gains against an optimisation score need confirmation outside that scoring path. We further analyse trustworthiness risks, including bias, privacy leakage, and misuse of generated media, and examine the evidence for mitigation strategies.

Figures & tables

Panel of Figure 1 — Figure 1: Taxonomy of the six generative categories of multimodal generative models (MGMs) covered in this survey, with example applications. Each category label defines its abbreviation.

Survey	Organising focus and coverage	Relation to this survey
Baltrušaitis et al., 2019 [ 19 ]	Multimodal machine learning organised by five challenges: representation, translation, alignment, fusion and co-learning	We focus on text-conditioned generation and compare how specific techniques are adapted across six output modalities
Caffagni et al., 2024 [ 20 ]	Visual multimodal LLMs: architectures, modality alignment and applications including visual grounding and image generation	We extend the comparison to music, human motion and 3D objects and organise it by the technique that changes the generator
Yu et al., 2025 [ 21 ]	Alignment algorithms for multimodal LLMs, emphasising human preference alignment	Preference optimisation is one of our four techniques; we relate it to representation learning, expert mixtures and planning, and read reported gains under their evaluation protocols
Dang et al., 2024 [ 22 ]	Interpretability and explainability of multimodal LLMs	We address trustworthiness through evaluation evidence and tested mitigations for misuse, privacy leakage and bias
Yin et al., 2024 [ 23 ]	Multimodal LLMs: granularity, modality and language coverage, applications, multimodal in-context learning and chain-of-thought	We examine generators whose outputs are images, music, video, human motion or 3D objects, including generators without a language-model backbone
Liu et al., 2024 [ 24 ]	Multimodal LLMs across modalities: concepts, applications and safety	We compare technique adaptation for each output modality and interpret evaluation evidence at the level of each study’s protocol

Table 1: Positioning against closely related surveys. Descriptions summarise each survey’s stated scope; topics not listed are not implied to be absent.

Model	$n_{params}$	$n_{layers}$	$d_{model}$	$n_{heads}$	$d_{head}$	$n_{vocab}$	Batch size (tokens)
GPT-3 (6.7B) [ 1 ]	6.7B	32	4,096	32	128	50K	2M
GPT-3 (175B) [ 1 ]	175B	96	12,288	96	128	50K	3.2M
GLaM 0.1B [ 40 ]	130M	12	768	12	64	256K	–
GLaM 0.1B/64E	1.9B	12	768	12	64	256K	–
GLaM 1.7B	1.7B	24	2,048	16	128	256K	–
GLaM 1.7B/32E	20B	24	2,048	16	128	256K	–

Table 2: Models and their hyperparameters for Text-to-Text generation.

Panel of Figure 3 — Figure 3: Conceptual pipeline combining self-supervised learning (SSL), mixture of experts (MoE), reinforcement learning from human feedback (RLHF), and chain-of-thought (CoT) prompting within a generic multimodal generative model. The order is illustrative; individual systems use any subset of the four techniques.

Modality	SSL	MoE	RLHF	CoT
T2T	Core, mature: next-token pretraining [ 15 ]	Advanced, efficient: GLaM, Mixtral [ 40 , 41 ]	Pioneered, refined: InstructGPT [ 3 ] ; verifiable rewards [ 4 ]	Rich, widely used: s1 [ 100 ] ; DeepSeek-R1 [ 4 ]
T2I	Helpful, limited: REPA [ 85 ]	Specialised, improving: ERNIE-ViLG 2.0, EC-DiT, DiffMoE [ 87 , 86 , 88 ]	Active, reward-rich: ImageReward, Diffusion-DPO [ 96 , 52 ]	Emerging, refinement: noise search [ 53 ] ; T2I-R1 [ 102 ]
T2M	Crucial, effective: MusicLM, Moûsai [ 59 , 29 ]	Rare, routing issues: StepAudio 3 Music [ 11 ]	Early, proxy rewards: MusicRL, MR-FlowDPO [ 97 , 62 ]	Sparse, sequentially hard: MusiCoT [ 103 ]
T2V	Essential for motion: Make-A-Video, VideoREPA [ 7 , 68 ]	Sparse, task-specific: Wan2.2 [ 91 ]	Rapid, multimodal rewards: VADER, LiFT [ 69 , 99 ]	Temporal, verifier-based: DirecT2V, Video-T1 [ 98 , 101 ]
T2HM	Emerging, motion priors: LMM, OMG [ 76 , 77 ]	Growing, control-based: OMG, GenM3 [ 77 , 93 ]	New, preference-driven: Sheng et al. [ 79 ]	Sparse, underexplored: Motion-R1 [ 78 ]
T2-3D	Minimal, data-limited	Rare, few large models: TripoSG (image-conditioned) [ 94 ]	Recent, viewer-aligned: DreamDPO, DreamReward, DreamAlign [ 82 , 83 , 84 ]	Nascent, inference search: HCoG [ 104 ]

Table 3: Trends across techniques and modalities. Labels summarise the qualitative assessment of the preceding subsections; the cited studies are representative examples, not performance rankings.

Model/study	Task and conditioning	Evaluation target	Metric/protocol and result versus comparator	Interpretation and limitation
Translation: supporting evidence
Attention NMT (2015) [ 13 ] [S1]	English $\to$ French; source sentence; WMT14 news-test-2014, full set	Reference overlap as a proxy for translation adequacy	RNNsearch-50: 26.75 BLEU versus RNNencdec-50: 17.82; source Table 1	Same-study model comparison; neither factual-response evidence nor an attention-only ablation
Transformer (2017) [ 12 ] [S2]	English $\to$ German; source sentence; WMT14 newstest2014	Reference overlap as a proxy for translation adequacy	Big: 28.4 BLEU versus base: 27.3; source Table 2	Capacity and training differ; not an isolated component comparison
Grounding and factual response: direct target, evaluator-dependent
InstructGPT (2022) [ 3 ] [S3]	Closed-domain API instructions with supplied source information	Human-judged unsupported additions	Reported hallucination rates: 21% versus GPT-3’s 41%; size-pooled evaluation, source Figure 4	Source-grounding result; neither a universal rate nor a 175B-only comparison
Llama 2 / Llama 2-Chat (2023) [ 2 ] [S4]	70B variants; TruthfulQA generation; six-shot QA prompt	Joint truthfulness and informativeness, judged automatically	Chat: 64.14% versus base: 50.18%; source Tables 11, 14 and 44	Joint outcome, not truthfulness alone; full post-training comparison, not an RLHF-only ablation

Table 4: Representative approaches to evaluating faithfulness in text-to-text (T2T) generation.

Model/study	Task and conditioning	Evaluation target	Metric/protocol and result versus comparator	Interpretation and limitation
AttnGAN (2018) [ 43 ] [I1]	CUB test descriptions; text-conditioned images	Caption matching: learned retrieval proxy	DAMSM weight 5: AttnGAN2 $67.82\pm 4.43\%$ versus AttnGAN1 $58.65\pm 5.41\%$ R-precision; source Table 2	Architecture comparison; evaluator shares the training alignment representation
DALL-E (2021) [ 45 ] [I2]	MS-COCO captions; contrastively reranked samples	Direct human judgement of caption matching	93.3% majority preference against DF-GAN; source Figure 7	Relative preference with candidate selection, not absolute adherence accuracy
DALL-E 2 / unCLIP (2022) [ 46 ] [I3]	MS-COCO validation captions; diffusion-prior unCLIP versus GLIDE	Direct human judgement of caption matching	unCLIP preferred $45.3\%\pm 3.0\%$ ; reported 95% interval, source Table 1	Does not show superior caption matching; diversity and photorealism are separate outcomes
ControlNet (2023) [ 48 ] [I4]	ADE20K segmentation-conditioned generation; additional spatial input	Supplied-segmentation adherence through a segmenter	Reconstructed IoU: $0.35\pm 0.14$ versus ControlNet-lite $0.32\pm 0.12$ ; source Table 2	Control fidelity, not caption faithfulness; limited by the segmentation evaluator
DALL-E 3 (2023) [ 49 ] [I5]	Short original MS-COCO 2014 evaluation captions	Caption similarity: learned embedding proxy	CLIP score 32.0 versus DALL-E 2 31.4 and SDXL 30.5; source Table 1	Full-system comparison; does not isolate recaptioning or check every requested relation

Table 5: Representative approaches to evaluating faithfulness in text-to-image (T2I) generation.

Model/study	Task and conditioning	Evaluation target	Metric/protocol and result versus comparator	Interpretation and limitation
MusicLM (2023) [ 59 ] [M1]	MusicCaps descriptions; hierarchical audio-token generation	Audio–text alignment proxy	MCC 0.51 versus 0.49 without the semantic modeling stage; source Section 5	Semantic-stage comparison under this evaluator; MuLan also supplies conditioning
Noise2Music (2023) [ 30 ] [M2]	MusicCaps captions; waveform versus spectrogram diffusion	Audio–text alignment proxy	MuLan audio–text similarity 0.478 versus 0.434; source Table 6	System-variant comparison; MuLan also contributes training pseudo-labels
MuseCoco (2023) [ 58 ] [M3]	Attribute-to-music stage of a text-to-symbolic-music pipeline; attributes supplied directly	Requested-attribute adherence: rule-based extraction and listener judgements	Prefix control: 67.40% average attribute control accuracy versus conditional LayerNorm 47.46% and embedding 36.94%; source Table 7	Same-study control-method comparison; bypasses text interpretation, so the full text-to-music pipeline is not tested
MusicGen (2023) [ 10 ] [M4]	Text conditioning; 300M codebook-pattern ablation on held-out music	Audio–text alignment proxy	Delay: CLAP 0.35, 1,500 steps; flattening: 0.37, 6,000 steps; published Table 4	Within-study score/sequence-length comparison; steps are not measured wall-clock speedups
Moûsai (2024) [ 29 ] [M5]	Text-conditioned audio; study-specific prompts	Audio–text alignment proxy	CLAP 0.13 versus Riffusion 0.06; source Table 4	Full-system comparison; evaluator checkpoint and scoring set not established
FluxMusic (2024) [ 60 ] [M6]	Small-model training comparison; held-out MTT prompts	Audio–text alignment proxy	Rectified flow: CLAP 0.312 versus DDIM 0.201; source Table 2	Evidence for this training comparison, not a universal architecture ranking

Table 6: Representative approaches to evaluating faithfulness in text-to-music (T2M) generation.

Model/study	Task and conditioning	Evaluation target	Metric/protocol and result versus comparator	Interpretation and limitation
TiVGAN (2020) [ 64 ] [V1]	Text conditioning; six-class Kinetics subset; 16 frames	Classifier-based quality/diversity: supporting evidence	Full schedule: IS 5.55 versus 5.13 without initial image-generation stage; source Table 5	Training-schedule comparison; IS does not check each video’s prompt match
CogVideo (2022) [ 65 ] [V2]	Class names from 30 UCF101 action classes; hierarchical versus one-stage generation	Direct human semantic-relevance ratings	Semantic relevance (1–5): 2.89 versus 2.57 for one-stage sequential generation; source Figure 5	Same-study design comparison; short class-name prompts test limited semantics
Make-A-Video (2022) [ 7 ] [V3]	DrawBench, 200 prompts; English-input CogVideo comparator	Direct human text correspondence	Reported faithfulness preference: 68.75%; source Table 3	Full-system comparison; percentages are described per rater although the protocol specifies majority votes
VideoLDM (2023) [ 8 ] [V4]	MSR-VTT test captions; SD 2.1 versus SD 1.4 variants	Frame–text embedding proxy	CLIPSIM 0.2929 versus 0.2848; source Table 10	System variants; frame averaging does not establish action ordering or temporal adherence
Emu Video (2023 preprint; ECCV 2024) [ 9 ] [V5]	Text-to-video through an internally generated image; 307 prompts	Direct human spatial/temporal text faithfulness	Factorised versus direct generation: 63.3% preference; source Table 1a	Matched design comparison; intermediate image is not an additional user-supplied condition
Movie Gen (2024 preprint) [ 67 ] [V6]	Movie Gen Video Bench; Runway Gen3 comparator	Direct human text alignment	+10.45 percentage points net win rate; source Table 6	Win% minus loss%, allowing ties; neither win probability nor absolute adherence

Table 7: Representative approaches to evaluating faithfulness in text-to-video (T2V) generation.

Model/study	Task and conditioning	Evaluation target	Metric/protocol and result versus comparator	Interpretation and limitation
TEMOS (2022) [ 70 ] [H1]	KIT-ML test descriptions and duration; Lin et al. also uses initial pose	Direct human text correspondence	90.5% semantic preference over Lin et al.; source Section 4.2/Figure 3	Unequal conditioning; realism can influence semantic preference
TM2T (2022) [ 72 ] [H2]	HumanML3D test descriptions; generated motion	Learned text–motion retrieval proxy	Top-3 R-precision $0.729\pm 0.002$ versus $0.635\pm 0.002$ without inverse alignment; source Table 2	GRU baseline comparison; not a direct check of each requested body-part action
MotionDiffuse (2022 preprint; 2024 journal) [ 31 ] [H3]	KIT-ML test descriptions with reference motion length	Learned text–motion retrieval proxy	With CLIP present: efficient attention $0.739\pm 0.004$ versus conventional attention $0.679\pm 0.005$ , top-3; source Table 3	Conditional attention ablation; not a universal benefit of diffusion
T2M-GPT (2023) [ 71 ] [H4]	HumanML3D test descriptions; predicted termination	Learned text–motion retrieval proxy	Training corruption $\tau=0.5$ : top-3 $0.775\pm 0.002$ versus $\tau=0$ : $0.685\pm 0.003$ ; source Table 1	Training ablation; $\tau$ is token-corruption probability, not sampling temperature
Fg-T2M (2023) [ 73 ] [H5]	Harder-HumanML3D subset selected for fine-grained wording	Learned text–motion retrieval proxy	Top-3 $0.763\pm 0.004$ versus MotionDiffuse $0.725\pm 0.005$ ; source Table 3	Subset-specific system comparison; retrieval does not certify left/right execution
MotionGPT, Zhang et al. (AAAI 2024) [ 74 ] [H6]	HumanML3D validation; 7B model evaluated with text only	Learned text–motion retrieval proxy	Joint multitask training: top-3 0.657 versus separate text-task training 0.577; source Table 4	Same evaluation condition, different training tasks; not human preference or an isolated LLM benefit

Table 8: Representative approaches to evaluating faithfulness in text-to-human-motion (T2HM) generation.

Model/study	Task and conditioning	Evaluation target	Metric/protocol and result versus comparator	Interpretation and limitation
DreamFusion (2022) [ 32 ] [D1]	Object-centric MS-COCO validation prompts; text-to-3D optimisation	Text–render retrieval proxy using textureless views	CLIP ViT-L/14 R-precision 58.5% versus reimplemented Dream Fields 1.4%; source Table 1	Different evaluation viewpoints and training objectives; not ground-truth geometric accuracy
Point-E (2022) [ 81 ] [D2]	Text to synthetic image to point cloud; same images across model sizes	Text–render retrieval proxy	CLIP ViT-L/14 R-precision: 300M 45.6% versus 40M 38.8%; source Table 1	Model-size comparison within this pipeline; mesh conversion can lose detail
Magic3D (2023; 2022 preprint protocol) [ 33 ] [D3]	Text prompts from DreamFusion gallery	Supporting evidence: perceived realism and detail	61.7% pairwise preference over DreamFusion; preprint Table 1	Visual-quality question, not explicit prompt adherence
Shap-E (2023) [ 34 ] [D4]	Direct text conditioning; 300M models with expanded 3D data	Text–render retrieval proxy	CLIP ViT-L/14 R-precision 40.9% versus text-only Point-E 35.5%; source Table 2	Base-model/data comparison with different auxiliary pipelines; not an isolated representation effect
Instant3D, Li et al. (2024) [ 35 ] [D5]	Text-conditioned assets; multiview videos for human assessment	Direct human text–3D correspondence	83.62% consistency selection share among six methods; source Table 1	Six-way choice, not pairwise win rate; human-study denominator not established

Table 9: Representative approaches to evaluating faithfulness in text-to-3D-object (T2-3D) generation.

Modality	Faithfulness	Compositionality	Robustness
T2T	BLEU, ROUGE, EM/F1 on QA and reasoning; judge-LLM preferences; limited sensitivity to hallucinations and long chains	SCAN, COGS, CFQ compositional splits; GSM8K/MATH; performance drops with concept count and nesting depth	Adversarial NLU and paraphrase suites; domain-shift tests; brittle to small lexical or structural perturbations
T2I	CLIPScore, captioning-back metrics, human preferences; high scores can mask object and attribute errors	T2I-CompBench, ConceptMix, relation/layout benchmarks; fidelity-composition divergence at higher realism	FID/IS and CLIP-based scores under corruptions, edits, and style shifts; sensitive to prompt phrasing and noise
T2M	MIR-inspired statistics, genre/mood classifiers, MOS; poor capture of high-level musical form	Multi-tag control of genre, mood, instrumentation, and structure; degradation for complex prompt combinations	Robustness to text perturbations, style interpolation, and length changes; vulnerable to prompt noise and style shift
T2V	CLIP-based video-text alignment, captioning metrics, action recognition; weak coverage of temporal coherence	Multi-entity and multi-event prompts, identity and state tracking over time; long-horizon consistency remains difficult	Robustness to resampling, compression, camera motion, and scene changes; failures accumulate over longer horizons
T2HM	Motion-language retrieval, pose error; limited nuance for rich descriptions	Multi-agent and multi-action prompts, spatial relations, temporal ordering; hard to coordinate interactions and constraints	Robustness to noisy text, viewpoint changes, trajectory perturbations; physical constraints often violated off-distribution
T2-3D	Text-shape retrieval, CLIP-based alignment, part-level IoU; struggles with fine-grained semantics	Multi-object scenes, part-whole relations, spatial layouts; limited support for binding several entities and relations	Robustness to lighting, rendering, and viewpoint changes; geometry may become implausible or non-manifold under edits

Table 10: Illustrative benchmarks and metrics for the three evaluation axes across six modalities. Entries are not exhaustive but highlight representative practices and typical blind spots.

Panel of Figure 4 — Figure 4: Heatmap view of Table 10 : qualitative maturity of evaluation practice per axis and modality. Medium denotes established benchmarks/metrics with noted caveats; Low denotes practice explicitly described as weak, poor, brittle, or difficult in the surveyed literature. No axis-modality pair reaches a fully mature ( High ) rating. T2T: text-to-text; T2I: text-to-image; T2M: text-to-music; T2V: text-to-video; T2HM: text-to-human-motion; T2-3D: text-to-3D objects.

Modality	Faithfulness	Compositionality	Robustness
T2T	Are source claims and task constraints satisfied?	Are combined constraints jointly satisfied, and are combinations held out?	Does correctness persist across equivalent prompts or a specified domain shift?
T2I	Are requested objects and attributes present?	Are attributes bound to the right objects and relations realised?	Does adherence persist under meaning-preserving wording changes across sampled seeds?
T2M	Is the description realised, separately from acoustic quality?	Are requested instruments and structural constraints jointly realised?	Does relevance persist under equivalent wording, and are structural constraints met at each tested duration?
T2V	Are requested entities and actions realised?	Are event order and identity preserved across frames?	Do adherence and temporal consistency persist under equivalent prompts, and are duration/control changes evaluated against their intended targets?
T2HM	Does the movement match the instruction?	Are body-part and temporal constraints jointly satisfied?	Do semantic and movement-validity measures persist under specified language or control perturbations?
T2-3D	Does the asset match requested attributes across views?	Are requested parts and their relationships realised?	Does prompt adherence persist under equivalent wording across views, with asset-edit and renderer/evaluator sensitivity assessed separately?

Table 11: Task-specific reporting questions for evaluating adherence, composition and robustness. Robustness requires an explicit baseline and changed condition; the suggested tests are not reported results.

Panel of Figure 5 — Figure 5: The five-dimensional trustworthiness lens for multimodal generative models (MGMs): truthfulness, safety, robustness, fairness, and privacy.

Mitigation	Targeted harm	Evidence (task)	Limitation	Capability cost
Misuse of generated content
Inference-time safety guidance: SLD [ 115 ]	Inappropriate imagery, such as nudity or violence	Reduced inappropriate content on the I2P prompts, without retraining (T2I)	An inference-time step can be removed when weights are available [ 116 ]	No adverse effect on overall quality or text alignment reported by the authors
Concept erasure in weights: ESD [ 116 ]	Explicit content; imitation of artistic styles	Comparable to SLD on explicit content; user study of style removal (T2I)	Learned word embeddings recover erased concepts [ 117 ]	Effects on related concepts require separate measurement
Prompt and output filters	Unsafe requests and outputs	Deployed as an additional layer	Vulnerable to increasingly sophisticated jailbreaks [ 118 ]	False refusals of benign prompts require measurement
Watermarking: Tree-Ring [ 119 ]	Undisclosed synthetic media	Detection after common image edits (T2I)	Identifies only the outputs of cooperating providers	Negligible FID change reported
Privacy leakage, memorisation and copyright

Table 12: Selected mitigations for misuse, privacy leakage and bias. Entries summarise the cited evidence and are not comparable effectiveness scores.

Paradigm	Examples	Training Loss	Inference	Strengths	Weaknesses
AR (Dense)	GPT-3/4; WaveNet (audio); MotionGPT (motion)	Cross-entropy (MLE)	$T$ sequential steps	High fidelity	High memory $O(T^{2})$ ; slow sampling; high computational cost
AR (MoE)	GLaM (text); Mixtral (text)	Same + sparsity losses	Sparse subnetwork	Larger capacity per flop; efficient inference (sparse)	Complex routing; large total parameter count; training complexity
Diffusion	Stable Diffusion (image); Lumiere (video); Noise2Music (audio)	Denoising score matching	$N$ iterative steps	Stable training; high output quality; inherent stochasticity	Slow sampling; many steps; larger inference cost
GAN	AttnGAN (image); MoCoGAN (video)	Minimax adversarial loss	Single generator pass	Very fast inference; sharp, high-resolution samples	Training instability; mode collapse; lower diversity
VAE	DALL-E1 (discrete VAE tokenizer); MusicVAE (music); TEMOS (motion)	ELBO	Encode-decode pass	Stable training; explicit likelihood bound	Blurry outputs; limited detail; KL trade-off loss
Flow/Score-based	Rectified Flow (image)	Flow matching	Few steps	Straight transport paths	Architectural constraints; numerical stability

Table 13: Comparison of major generative paradigms. Models and numbers are illustrative.

Multimodal Generation

Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.

See all 243 papers

May 23, 2025

MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generation

Automatically evaluating multimodal generation presents a significant challenge, as automated metrics often struggle to align with human evaluation, especially for complex tasks that involve multiple modalities. We present MMMG, the first benchmark to bring the verifiable-task paradigm to multimodal generation, spanning 4 modality combinations (image, audio, interleaved text and image, interleaved text and audio). As few multimodal outputs can be checked by programs alone, MMMG targets tasks that are either verifiable or near-verifiable: by providing references, and constraining model judges with explicit rubrics. We keep tasks challenging for generation models while enabling reliable automatic evaluation through a combination of models and programs. MMMG encompasses 55 tasks (including 31 newly developed ones), each with a carefully designed evaluation pipeline, and 1288 instructions to systematically assess reasoning, controllability, and other key capabilities of multimodal generation models. Extensive validation demonstrates that MMMG is highly aligned with human judgment, achieving an average agreement of 94.4%. Benchmarking results on 29 models reveal that even though the state-of-the-art model, GPT Image, achieves 70.7% accuracy for image generation, it falls short on interleaved generation. Furthermore, results suggest considerable improvement space in audio generation, highlighting an important future direction.

Jihan Yao, Yushi Hu, Wenyuan Wang +10

University of Washington · Allen Institute for AI

Multimodal Generation Multimodal Model Evaluation MLLM Evaluation

Ask PDF

Feb 9, 2026

UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Models

Unified multimodal models (UMMs) aim to integrate multimodal understanding and generation within a unified architecture, yet it remains unclear to what extent textual and visual modalities are aligned. To investigate this question, we use reasoning-guided image generation as a diagnostic task, where models produce textual reasoning first and then generate images. We introduce UReason, a benchmark for evaluating reasoning-to-generation alignment in this paradigm, consisting of 2,000 human-curated and human-verified instances spanning five reasoning-intensive tasks: Code, Arithmetic, Spatial, Attribute, and Text. To enable controlled analysis, we develop an evaluation framework that compares direct generation, reasoning-guided generation, and decontextualized generation, which conditions only on the refined prompt extracted from reasoning. Across eight widely used open-source UMMs, while we find that reasoning-guided generation yields improvements over direct generation, somewhat surprisingly, decontextualized generation consistently outperforms reasoning-guided generation by a large margin. Our further analyses suggest that the intended visual semantics in textual reasoning are not reliably reflected in the generated images, despite their unified design and training. Overall, UReason serves as a practical litmus test for reasoning-to-generation alignment and provides a challenging benchmark for developing next-generation, more tightly aligned UMMs.

Cheng Yang, Chufan Shi, Bo Shui +7

University of California San Diego · University of Southern California · University of Illinois Urbana-Champaign +1

Unified Multimodal Models Image Generation Multimodal Reasoning Multimodal Foundation Models Multimodal Model Evaluation Multimodal Understanding

Ask PDF

Dec 16, 2025

MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation

Modern multimodal generative models can synthesize visually compelling images and videos, but it remains unclear whether this visual fluency reflects genuine reasoning: when prompted to generate a solution, can a model preserve the physical, logical, spatial, and temporal constraints a task requires, or does it merely produce plausible-looking media? To answer this question, we introduce MMGR (Multi-Modal Generative Reasoning Benchmark and Evaluation), a benchmark for evaluating generative reasoning across video, image, and language-based systems. MMGR covers 10 tasks from three domains (Abstract Reasoning, Embodied Navigation, and Physical Commonsense) and probes five reasoning abilities: Physical, Logical, 2D Spatial, 3D Spatial, and Temporal. Its evaluation emphasizes answer-verifiable tasks and, for video generation, process-aware chain-of-frame reasoning, where intermediate frames must form valid steps toward the target outcome rather than visually smooth but incorrect transitions. Evaluating state-of-the-art video generators, image generators, and LLM/VLM baselines reveals a sharp gap between visual quality and reasoning correctness: video models perform best on Physical Commonsense, but remain weak on symbolic tasks such as Sudoku, ARC, and Math, and brittle in cross-view embodied navigation. Image generators often outperform video generators on embodied navigation despite lacking temporal outputs, showing that longer visual generation does not automatically yield stronger reasoning. MMGR reframes evaluation of multimodal generation from whether outputs look realistic to whether they solve the underlying reasoning problem.

Zefan Cai, Haoyi Qiu, Tianyi Ma +13

University of Wisconsin–Madison · University of California, Los Angeles · Michigan State University +8

Embodied Navigation Multimodal Reasoning Multimodal Model Evaluation Video Generation Evaluation

Ask PDF

Abstract

Multimodal Generation

Explore similar work

MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generation

UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Models

MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation