cs.MMMay 29, 2025
SaveA Survey of Generative Categories and Techniques in Multimodal Generative Models
Organizations: School of Architecture, Technology and Engineering, University of Brighton, Lewes Road, BN2 4GJ, United Kingdom · University of Khorfakkan, Sharjah, United Arab Emirates
Abstract
Multimodal generative models (MGMs) have rapidly expanded beyond text generation to encompass images, music, video, human motion, and 3D objects. This survey categorises six primary text-conditioned generative modalities and examines how four techniques, namely self-supervised learning, mixture of experts, reinforcement learning from human feedback, and chain-of-thought prompting, are adapted across them, distinguishing shared mechanisms from experimentally supported benefits. We organise evaluation around faithfulness, compositionality, and robustness, and interpret evidence within each study's protocol. Selected within-study comparisons reported in the reviewed literature support four findings: preference gains need not improve text relevance, and reward optimisation can reduce within-prompt diversity; intermediate plans require tests of their realisation in final outputs; expert granularity changes the mechanism and its storage, computation, and runtime costs; and gains against an optimisation score need confirmation outside that scoring path. We further analyse trustworthiness risks, including bias, privacy leakage, and misuse of generated media, and examine the evidence for mitigation strategies.
Figures & tables
Figure 1: Taxonomy of the six generative categories of multimodal generative models (MGMs) covered in this survey, with example applications. Each category label defines its abbreviation.
| Survey | Organising focus and coverage | Relation to this survey |
| Baltrušaitis et al., 2019 [ 19 ] | Multimodal machine learning organised by five challenges: representation, translation, alignment, fusion and co-learning | We focus on text-conditioned generation and compare how specific techniques are adapted across six output modalities |
| Caffagni et al., 2024 [ 20 ] | Visual multimodal LLMs: architectures, modality alignment and applications including visual grounding and image generation | We extend the comparison to music, human motion and 3D objects and organise it by the technique that changes the generator |
| Yu et al., 2025 [ 21 ] | Alignment algorithms for multimodal LLMs, emphasising human preference alignment | Preference optimisation is one of our four techniques; we relate it to representation learning, expert mixtures and planning, and read reported gains under their evaluation protocols |
| Dang et al., 2024 [ 22 ] | Interpretability and explainability of multimodal LLMs | We address trustworthiness through evaluation evidence and tested mitigations for misuse, privacy leakage and bias |
| Yin et al., 2024 [ 23 ] | Multimodal LLMs: granularity, modality and language coverage, applications, multimodal in-context learning and chain-of-thought | We examine generators whose outputs are images, music, video, human motion or 3D objects, including generators without a language-model backbone |
| Liu et al., 2024 [ 24 ] | Multimodal LLMs across modalities: concepts, applications and safety | We compare technique adaptation for each output modality and interpret evaluation evidence at the level of each study’s protocol |
Table 1: Positioning against closely related surveys. Descriptions summarise each survey’s stated scope; topics not listed are not implied to be absent.
| Model | Batch size (tokens) | ||||||
| GPT-3 (6.7B) [ 1 ] | 6.7B | 32 | 4,096 | 32 | 128 | 50K | 2M |
| GPT-3 (175B) [ 1 ] | 175B | 96 | 12,288 | 96 | 128 | 50K | 3.2M |
| GLaM 0.1B [ 40 ] | 130M | 12 | 768 | 12 | 64 | 256K | – |
| GLaM 0.1B/64E | 1.9B | 12 | 768 | 12 | 64 | 256K | – |
| GLaM 1.7B | 1.7B | 24 | 2,048 | 16 | 128 | 256K | – |
| GLaM 1.7B/32E | 20B | 24 | 2,048 | 16 | 128 | 256K | – |
Table 2: Models and their hyperparameters for Text-to-Text generation.
Figure 3: Conceptual pipeline combining self-supervised learning (SSL), mixture of experts (MoE), reinforcement learning from human feedback (RLHF), and chain-of-thought (CoT) prompting within a generic multimodal generative model. The order is illustrative; individual systems use any subset of the four techniques.
| Modality | SSL | MoE | RLHF | CoT |
| T2T | Core, mature: next-token pretraining [ 15 ] | Advanced, efficient: GLaM, Mixtral [ 40 , 41 ] | Pioneered, refined: InstructGPT [ 3 ] ; verifiable rewards [ 4 ] | Rich, widely used: s1 [ 100 ] ; DeepSeek-R1 [ 4 ] |
| T2I | Helpful, limited: REPA [ 85 ] | Specialised, improving: ERNIE-ViLG 2.0, EC-DiT, DiffMoE [ 87 , 86 , 88 ] | Active, reward-rich: ImageReward, Diffusion-DPO [ 96 , 52 ] | Emerging, refinement: noise search [ 53 ] ; T2I-R1 [ 102 ] |
| T2M | Crucial, effective: MusicLM, Moûsai [ 59 , 29 ] | Rare, routing issues: StepAudio 3 Music [ 11 ] | Early, proxy rewards: MusicRL, MR-FlowDPO [ 97 , 62 ] | Sparse, sequentially hard: MusiCoT [ 103 ] |
| T2V | Essential for motion: Make-A-Video, VideoREPA [ 7 , 68 ] | Sparse, task-specific: Wan2.2 [ 91 ] | Rapid, multimodal rewards: VADER, LiFT [ 69 , 99 ] | Temporal, verifier-based: DirecT2V, Video-T1 [ 98 , 101 ] |
| T2HM | Emerging, motion priors: LMM, OMG [ 76 , 77 ] | Growing, control-based: OMG, GenM3 [ 77 , 93 ] | New, preference-driven: Sheng et al. [ 79 ] | Sparse, underexplored: Motion-R1 [ 78 ] |
| T2-3D | Minimal, data-limited | Rare, few large models: TripoSG (image-conditioned) [ 94 ] | Recent, viewer-aligned: DreamDPO, DreamReward, DreamAlign [ 82 , 83 , 84 ] | Nascent, inference search: HCoG [ 104 ] |
Table 3: Trends across techniques and modalities. Labels summarise the qualitative assessment of the preceding subsections; the cited studies are representative examples, not performance rankings.
| Model/study | Task and conditioning | Evaluation target | Metric/protocol and result versus comparator | Interpretation and limitation |
| Translation: supporting evidence | ||||
| Attention NMT (2015) [ 13 ] [S1] | English French; source sentence; WMT14 news-test-2014, full set | Reference overlap as a proxy for translation adequacy | RNNsearch-50: 26.75 BLEU versus RNNencdec-50: 17.82; source Table 1 | Same-study model comparison; neither factual-response evidence nor an attention-only ablation |
| Transformer (2017) [ 12 ] [S2] | English German; source sentence; WMT14 newstest2014 | Reference overlap as a proxy for translation adequacy | Big: 28.4 BLEU versus base: 27.3; source Table 2 | Capacity and training differ; not an isolated component comparison |
| Grounding and factual response: direct target, evaluator-dependent | ||||
| InstructGPT (2022) [ 3 ] [S3] | Closed-domain API instructions with supplied source information | Human-judged unsupported additions | Reported hallucination rates: 21% versus GPT-3’s 41%; size-pooled evaluation, source Figure 4 | Source-grounding result; neither a universal rate nor a 175B-only comparison |
| Llama 2 / Llama 2-Chat (2023) [ 2 ] [S4] | 70B variants; TruthfulQA generation; six-shot QA prompt | Joint truthfulness and informativeness, judged automatically | Chat: 64.14% versus base: 50.18%; source Tables 11, 14 and 44 | Joint outcome, not truthfulness alone; full post-training comparison, not an RLHF-only ablation |
Table 4: Representative approaches to evaluating faithfulness in text-to-text (T2T) generation.
| Model/study | Task and conditioning | Evaluation target | Metric/protocol and result versus comparator | Interpretation and limitation |
| AttnGAN (2018) [ 43 ] [I1] | CUB test descriptions; text-conditioned images | Caption matching: learned retrieval proxy | DAMSM weight 5: AttnGAN2 versus AttnGAN1 R-precision; source Table 2 | Architecture comparison; evaluator shares the training alignment representation |
| DALL-E (2021) [ 45 ] [I2] | MS-COCO captions; contrastively reranked samples | Direct human judgement of caption matching | 93.3% majority preference against DF-GAN; source Figure 7 | Relative preference with candidate selection, not absolute adherence accuracy |
| DALL-E 2 / unCLIP (2022) [ 46 ] [I3] | MS-COCO validation captions; diffusion-prior unCLIP versus GLIDE | Direct human judgement of caption matching | unCLIP preferred ; reported 95% interval, source Table 1 | Does not show superior caption matching; diversity and photorealism are separate outcomes |
| ControlNet (2023) [ 48 ] [I4] | ADE20K segmentation-conditioned generation; additional spatial input | Supplied-segmentation adherence through a segmenter | Reconstructed IoU: versus ControlNet-lite ; source Table 2 | Control fidelity, not caption faithfulness; limited by the segmentation evaluator |
| DALL-E 3 (2023) [ 49 ] [I5] | Short original MS-COCO 2014 evaluation captions | Caption similarity: learned embedding proxy | CLIP score 32.0 versus DALL-E 2 31.4 and SDXL 30.5; source Table 1 | Full-system comparison; does not isolate recaptioning or check every requested relation |
Table 5: Representative approaches to evaluating faithfulness in text-to-image (T2I) generation.
| Model/study | Task and conditioning | Evaluation target | Metric/protocol and result versus comparator | Interpretation and limitation |
| MusicLM (2023) [ 59 ] [M1] | MusicCaps descriptions; hierarchical audio-token generation | Audio–text alignment proxy | MCC 0.51 versus 0.49 without the semantic modeling stage; source Section 5 | Semantic-stage comparison under this evaluator; MuLan also supplies conditioning |
| Noise2Music (2023) [ 30 ] [M2] | MusicCaps captions; waveform versus spectrogram diffusion | Audio–text alignment proxy | MuLan audio–text similarity 0.478 versus 0.434; source Table 6 | System-variant comparison; MuLan also contributes training pseudo-labels |
| MuseCoco (2023) [ 58 ] [M3] | Attribute-to-music stage of a text-to-symbolic-music pipeline; attributes supplied directly | Requested-attribute adherence: rule-based extraction and listener judgements | Prefix control: 67.40% average attribute control accuracy versus conditional LayerNorm 47.46% and embedding 36.94%; source Table 7 | Same-study control-method comparison; bypasses text interpretation, so the full text-to-music pipeline is not tested |
| MusicGen (2023) [ 10 ] [M4] | Text conditioning; 300M codebook-pattern ablation on held-out music | Audio–text alignment proxy | Delay: CLAP 0.35, 1,500 steps; flattening: 0.37, 6,000 steps; published Table 4 | Within-study score/sequence-length comparison; steps are not measured wall-clock speedups |
| Moûsai (2024) [ 29 ] [M5] | Text-conditioned audio; study-specific prompts | Audio–text alignment proxy | CLAP 0.13 versus Riffusion 0.06; source Table 4 | Full-system comparison; evaluator checkpoint and scoring set not established |
| FluxMusic (2024) [ 60 ] [M6] | Small-model training comparison; held-out MTT prompts | Audio–text alignment proxy | Rectified flow: CLAP 0.312 versus DDIM 0.201; source Table 2 | Evidence for this training comparison, not a universal architecture ranking |
Table 6: Representative approaches to evaluating faithfulness in text-to-music (T2M) generation.
| Model/study | Task and conditioning | Evaluation target | Metric/protocol and result versus comparator | Interpretation and limitation |
| TiVGAN (2020) [ 64 ] [V1] | Text conditioning; six-class Kinetics subset; 16 frames | Classifier-based quality/diversity: supporting evidence | Full schedule: IS 5.55 versus 5.13 without initial image-generation stage; source Table 5 | Training-schedule comparison; IS does not check each video’s prompt match |
| CogVideo (2022) [ 65 ] [V2] | Class names from 30 UCF101 action classes; hierarchical versus one-stage generation | Direct human semantic-relevance ratings | Semantic relevance (1–5): 2.89 versus 2.57 for one-stage sequential generation; source Figure 5 | Same-study design comparison; short class-name prompts test limited semantics |
| Make-A-Video (2022) [ 7 ] [V3] | DrawBench, 200 prompts; English-input CogVideo comparator | Direct human text correspondence | Reported faithfulness preference: 68.75%; source Table 3 | Full-system comparison; percentages are described per rater although the protocol specifies majority votes |
| VideoLDM (2023) [ 8 ] [V4] | MSR-VTT test captions; SD 2.1 versus SD 1.4 variants | Frame–text embedding proxy | CLIPSIM 0.2929 versus 0.2848; source Table 10 | System variants; frame averaging does not establish action ordering or temporal adherence |
| Emu Video (2023 preprint; ECCV 2024) [ 9 ] [V5] | Text-to-video through an internally generated image; 307 prompts | Direct human spatial/temporal text faithfulness | Factorised versus direct generation: 63.3% preference; source Table 1a | Matched design comparison; intermediate image is not an additional user-supplied condition |
| Movie Gen (2024 preprint) [ 67 ] [V6] | Movie Gen Video Bench; Runway Gen3 comparator | Direct human text alignment | +10.45 percentage points net win rate; source Table 6 | Win% minus loss%, allowing ties; neither win probability nor absolute adherence |
Table 7: Representative approaches to evaluating faithfulness in text-to-video (T2V) generation.
| Model/study | Task and conditioning | Evaluation target | Metric/protocol and result versus comparator | Interpretation and limitation |
| TEMOS (2022) [ 70 ] [H1] | KIT-ML test descriptions and duration; Lin et al. also uses initial pose | Direct human text correspondence | 90.5% semantic preference over Lin et al.; source Section 4.2/Figure 3 | Unequal conditioning; realism can influence semantic preference |
| TM2T (2022) [ 72 ] [H2] | HumanML3D test descriptions; generated motion | Learned text–motion retrieval proxy | Top-3 R-precision versus without inverse alignment; source Table 2 | GRU baseline comparison; not a direct check of each requested body-part action |
| MotionDiffuse (2022 preprint; 2024 journal) [ 31 ] [H3] | KIT-ML test descriptions with reference motion length | Learned text–motion retrieval proxy | With CLIP present: efficient attention versus conventional attention , top-3; source Table 3 | Conditional attention ablation; not a universal benefit of diffusion |
| T2M-GPT (2023) [ 71 ] [H4] | HumanML3D test descriptions; predicted termination | Learned text–motion retrieval proxy | Training corruption : top-3 versus : ; source Table 1 | Training ablation; is token-corruption probability, not sampling temperature |
| Fg-T2M (2023) [ 73 ] [H5] | Harder-HumanML3D subset selected for fine-grained wording | Learned text–motion retrieval proxy | Top-3 versus MotionDiffuse ; source Table 3 | Subset-specific system comparison; retrieval does not certify left/right execution |
| MotionGPT, Zhang et al. (AAAI 2024) [ 74 ] [H6] | HumanML3D validation; 7B model evaluated with text only | Learned text–motion retrieval proxy | Joint multitask training: top-3 0.657 versus separate text-task training 0.577; source Table 4 | Same evaluation condition, different training tasks; not human preference or an isolated LLM benefit |
Table 8: Representative approaches to evaluating faithfulness in text-to-human-motion (T2HM) generation.
| Model/study | Task and conditioning | Evaluation target | Metric/protocol and result versus comparator | Interpretation and limitation |
| DreamFusion (2022) [ 32 ] [D1] | Object-centric MS-COCO validation prompts; text-to-3D optimisation | Text–render retrieval proxy using textureless views | CLIP ViT-L/14 R-precision 58.5% versus reimplemented Dream Fields 1.4%; source Table 1 | Different evaluation viewpoints and training objectives; not ground-truth geometric accuracy |
| Point-E (2022) [ 81 ] [D2] | Text to synthetic image to point cloud; same images across model sizes | Text–render retrieval proxy | CLIP ViT-L/14 R-precision: 300M 45.6% versus 40M 38.8%; source Table 1 | Model-size comparison within this pipeline; mesh conversion can lose detail |
| Magic3D (2023; 2022 preprint protocol) [ 33 ] [D3] | Text prompts from DreamFusion gallery | Supporting evidence: perceived realism and detail | 61.7% pairwise preference over DreamFusion; preprint Table 1 | Visual-quality question, not explicit prompt adherence |
| Shap-E (2023) [ 34 ] [D4] | Direct text conditioning; 300M models with expanded 3D data | Text–render retrieval proxy | CLIP ViT-L/14 R-precision 40.9% versus text-only Point-E 35.5%; source Table 2 | Base-model/data comparison with different auxiliary pipelines; not an isolated representation effect |
| Instant3D, Li et al. (2024) [ 35 ] [D5] | Text-conditioned assets; multiview videos for human assessment | Direct human text–3D correspondence | 83.62% consistency selection share among six methods; source Table 1 | Six-way choice, not pairwise win rate; human-study denominator not established |
Table 9: Representative approaches to evaluating faithfulness in text-to-3D-object (T2-3D) generation.
| Modality | Faithfulness | Compositionality | Robustness |
| T2T | BLEU, ROUGE, EM/F1 on QA and reasoning; judge-LLM preferences; limited sensitivity to hallucinations and long chains | SCAN, COGS, CFQ compositional splits; GSM8K/MATH; performance drops with concept count and nesting depth | Adversarial NLU and paraphrase suites; domain-shift tests; brittle to small lexical or structural perturbations |
| T2I | CLIPScore, captioning-back metrics, human preferences; high scores can mask object and attribute errors | T2I-CompBench, ConceptMix, relation/layout benchmarks; fidelity-composition divergence at higher realism | FID/IS and CLIP-based scores under corruptions, edits, and style shifts; sensitive to prompt phrasing and noise |
| T2M | MIR-inspired statistics, genre/mood classifiers, MOS; poor capture of high-level musical form | Multi-tag control of genre, mood, instrumentation, and structure; degradation for complex prompt combinations | Robustness to text perturbations, style interpolation, and length changes; vulnerable to prompt noise and style shift |
| T2V | CLIP-based video-text alignment, captioning metrics, action recognition; weak coverage of temporal coherence | Multi-entity and multi-event prompts, identity and state tracking over time; long-horizon consistency remains difficult | Robustness to resampling, compression, camera motion, and scene changes; failures accumulate over longer horizons |
| T2HM | Motion-language retrieval, pose error; limited nuance for rich descriptions | Multi-agent and multi-action prompts, spatial relations, temporal ordering; hard to coordinate interactions and constraints | Robustness to noisy text, viewpoint changes, trajectory perturbations; physical constraints often violated off-distribution |
| T2-3D | Text-shape retrieval, CLIP-based alignment, part-level IoU; struggles with fine-grained semantics | Multi-object scenes, part-whole relations, spatial layouts; limited support for binding several entities and relations | Robustness to lighting, rendering, and viewpoint changes; geometry may become implausible or non-manifold under edits |
Table 10: Illustrative benchmarks and metrics for the three evaluation axes across six modalities. Entries are not exhaustive but highlight representative practices and typical blind spots.
Figure 4: Heatmap view of Table 10 : qualitative maturity of evaluation practice per axis and modality. Medium denotes established benchmarks/metrics with noted caveats; Low denotes practice explicitly described as weak, poor, brittle, or difficult in the surveyed literature. No axis-modality pair reaches a fully mature ( High ) rating. T2T: text-to-text; T2I: text-to-image; T2M: text-to-music; T2V: text-to-video; T2HM: text-to-human-motion; T2-3D: text-to-3D objects.
| Modality | Faithfulness | Compositionality | Robustness |
| T2T | Are source claims and task constraints satisfied? | Are combined constraints jointly satisfied, and are combinations held out? | Does correctness persist across equivalent prompts or a specified domain shift? |
| T2I | Are requested objects and attributes present? | Are attributes bound to the right objects and relations realised? | Does adherence persist under meaning-preserving wording changes across sampled seeds? |
| T2M | Is the description realised, separately from acoustic quality? | Are requested instruments and structural constraints jointly realised? | Does relevance persist under equivalent wording, and are structural constraints met at each tested duration? |
| T2V | Are requested entities and actions realised? | Are event order and identity preserved across frames? | Do adherence and temporal consistency persist under equivalent prompts, and are duration/control changes evaluated against their intended targets? |
| T2HM | Does the movement match the instruction? | Are body-part and temporal constraints jointly satisfied? | Do semantic and movement-validity measures persist under specified language or control perturbations? |
| T2-3D | Does the asset match requested attributes across views? | Are requested parts and their relationships realised? | Does prompt adherence persist under equivalent wording across views, with asset-edit and renderer/evaluator sensitivity assessed separately? |
Table 11: Task-specific reporting questions for evaluating adherence, composition and robustness. Robustness requires an explicit baseline and changed condition; the suggested tests are not reported results.
Figure 5: The five-dimensional trustworthiness lens for multimodal generative models (MGMs): truthfulness, safety, robustness, fairness, and privacy.
| Mitigation | Targeted harm | Evidence (task) | Limitation | Capability cost |
| Misuse of generated content | ||||
| Inference-time safety guidance: SLD [ 115 ] | Inappropriate imagery, such as nudity or violence | Reduced inappropriate content on the I2P prompts, without retraining (T2I) | An inference-time step can be removed when weights are available [ 116 ] | No adverse effect on overall quality or text alignment reported by the authors |
| Concept erasure in weights: ESD [ 116 ] | Explicit content; imitation of artistic styles | Comparable to SLD on explicit content; user study of style removal (T2I) | Learned word embeddings recover erased concepts [ 117 ] | Effects on related concepts require separate measurement |
| Prompt and output filters | Unsafe requests and outputs | Deployed as an additional layer | Vulnerable to increasingly sophisticated jailbreaks [ 118 ] | False refusals of benign prompts require measurement |
| Watermarking: Tree-Ring [ 119 ] | Undisclosed synthetic media | Detection after common image edits (T2I) | Identifies only the outputs of cooperating providers | Negligible FID change reported |
| Privacy leakage, memorisation and copyright | ||||
Table 12: Selected mitigations for misuse, privacy leakage and bias. Entries summarise the cited evidence and are not comparable effectiveness scores.
| Paradigm | Examples | Training Loss | Inference | Strengths | Weaknesses |
| AR (Dense) | GPT-3/4; WaveNet (audio); MotionGPT (motion) | Cross-entropy (MLE) | sequential steps | High fidelity | High memory ; slow sampling; high computational cost |
| AR (MoE) | GLaM (text); Mixtral (text) | Same + sparsity losses | Sparse subnetwork | Larger capacity per flop; efficient inference (sparse) | Complex routing; large total parameter count; training complexity |
| Diffusion | Stable Diffusion (image); Lumiere (video); Noise2Music (audio) | Denoising score matching | iterative steps | Stable training; high output quality; inherent stochasticity | Slow sampling; many steps; larger inference cost |
| GAN | AttnGAN (image); MoCoGAN (video) | Minimax adversarial loss | Single generator pass | Very fast inference; sharp, high-resolution samples | Training instability; mode collapse; lower diversity |
| VAE | DALL-E1 (discrete VAE tokenizer); MusicVAE (music); TEMOS (motion) | ELBO | Encode-decode pass | Stable training; explicit likelihood bound | Blurry outputs; limited detail; KL trade-off loss |
| Flow/Score-based | Rectified Flow (image) | Flow matching | Few steps | Straight transport paths | Architectural constraints; numerical stability |
Table 13: Comparison of major generative paradigms. Models and numbers are illustrative.
More in this topic
Multimodal Generation
Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.
See all 243 papers