Multimodal Generation

Ask about this topic

Overview

Multimodal generation studies models that create outputs such as images, video, speech, motion, or scientific signals from text, media, or combinations of them. Common approaches include diffusion and flow-matching decoders, autoregressive or discrete-token models, shared latent spaces, and systems that connect vision-language models to modality-specific generators; planning and reinforcement learning can add control for complex or interleaved outputs. A major direction is building models that both understand and generate, with benchmarks measuring prompt adherence, fidelity, temporal and cross-modal coherence, and task-specific validity. An open challenge is enabling shared models to turn understanding into faithful, controllable generation without modality imbalance or loss of specialized quality, especially over long sequences and varied real-world inputs.

Latest papers

CardsList

243 papers29 papers in the last four weeks, up 38% on the four weeks before. 0.2% of all new papers.

  1. Pseudo-Unification: Entropy Probing Reveals Divergent Information Patterns in Unified Multimodal Models

    Apr 13, 2026Songlin Yang, Xianghao Kong, Anyi RaoUnified Multimodal ModelsMultimodal Generation

  2. TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-training

    Apr 12, 2026Yinyi Luo, Wenwen Wang, Hayes Bai +6Unified Multimodal ModelsMultimodal Model Evaluation

    Carnegie Mellon University · William & Mary · Auburn University +1

  3. Deep-Reporter: Deep Research for Grounded Multimodal Long-Form Generation

    Apr 12, 2026Fangda Ye, Zhifei Xie, Yuxin Hu +5Multimodal GenerationMultimodal Grounding

    National University of Singapore · Nanyang Technological University · University of Edinburgh +1

  4. MAGE: Modality-Agnostic Music Generation and Target-Source Extraction

    Apr 10, 2026Muhammad Usama Saleem, Tejasvi Ravi, Tianyu Xu +4Controllable Music GenerationCross-Modal Alignment

    Google · University of North Carolina at Charlotte

  5. PolySLGen: Online Multimodal Speaking-Listening Reaction Generation in Polyadic Interaction

    Apr 9, 2026Zhi-Yi Lin, Thomas Markhorst, Jouh Yeong Chew +1Multimodal GenerationHuman Motion Generation

    Computer Vision Lab, Delft University of Technology · Honda Research Institute Japan

  6. Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding

    Apr 9, 2026Xiangyue Liu, Zijian Zhang, Miles Yang +3Cross-Modal Representation LearningUnified Multimodal Models

    HKUST · Tencent Hunyuan

  7. FlowInOne:Unifying Multimodal Generation as Image-in, Image-out Flow Matching

    Apr 8, 2026Junchao Yi, Rui Zhao, Jiahao Tang +7Multimodal GenerationImage Generation

    University of Electronic Science and Technology of China · National University of Singapore · Central South University +2

  8. Evolution of Video Generative Foundations

    Apr 7, 2026Teng Hu, Jiangning Zhang, Hongrui Huang +7Multimodal Video GenerationVideo Generation

  9. EduIllustrate: Towards Scalable Automated Generation Of Multimodal Educational Content

    Apr 6, 2026Shuzhen Bi, Mingzi Zhang, Zhuoxuan Li +3Generative AI in EducationMultimodal Generation

    Shanghai Innovation Institute · University of Science and Technology of China · East China Normal University

  10. A Generative Foundation Model for Multimodal Histopathology

    Apr 4, 2026Jinxi Xiang, Mingjie Li, Siyu Hou +9Computational PathologyPathology Foundation Models

    Department of Radiation Oncology, Stanford University School of Medicine, Stanford, CA · Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA · Department of Statistics and Data Science, Yale University, New Haven, CT +5

  11. Omni123: Exploring 3D Native Foundation Models with Limited 3D Data by Unifying Text to 2D and 3D Generation

    Apr 2, 2026Chongjie Ye, Cheng Cao, Chuanyu Pan +4Autoregressive 3D Scene Generation3D Foundation Models

    FNii-Shenzhen · SSE, CUHK(SZ) · Meshy AI

  12. LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model

    Apr 2, 2026Jiachun Jin, Zetong Zhou, Xiao Yang +4Unified Multimodal ModelsMultimodal Reasoning

    Shanghai Jiao Tong University · Tsinghua University · UCSD

  13. AromaGen: Interactive Generation of Rich Olfactory Experiences with Multimodal Language Models

    Apr 2, 2026Awu Chen, Silang Wang, Yunge Wen +6Multimodal GenerationMultimodal Large Language Models

    MIT Media Lab, Cambridge, MA, USA · University of Washington, Seattle, WA, USA · Harvard Graduate School of Design, Cambridge, MA, USA +3

  14. MMFace-DiT: A Dual-Stream Diffusion Transformer for High-Fidelity Multimodal Face Generation

    Mar 30, 2026Bharath Krishnamurthy, Ajita RattaniMultimodal GenerationControllable Image Generation

    University of North Texas, Denton, TX, USA

  15. GEMS: Agent-Native Multimodal Generation with Memory and Skills

    Mar 30, 2026Zefeng He, Siyuan Huang, Xiaoye Qu +4Multimodal GenerationMemory-Augmented Language Model Agents

    Shanghai AI Laboratory · Nanjing University · Shanghai Jiao Tong University +1

  16. LVRPO: Language-Visual Alignment with GRPO for Multimodal Understanding and Generation

    Mar 29, 2026Shentong Mo, Sukmin YunVision-Language AlignmentUnified Multimodal Models

    Department of Machine Learning, CMU, USA · Department of Machine Learning, MBZUAI, UAE · Department of Artificial Intelligence, Hanyang University ERICA, South Korea

  17. Customized Visual Storytelling with Unified Multimodal LLMs

    Mar 29, 2026Wei-Hua Li, Cheng Sun, Chu-Song ChenMultimodal GenerationControllable Video Generation

    National Taiwan University · NVIDA

  18. LongCat-Next: Lexicalizing Modalities as Discrete Tokens

    Mar 29, 2026Meituan LongCat Team, Bin Xiao, Chao Wang +86Unified Multimodal ModelsMultimodal Learning

  19. OmniColor: A Unified Framework for Multi-modal Lineart Colorization

    Mar 29, 2026Xulu Zhang, Haoqian Du, Xiaoyong Wei +1Multimodal GenerationUnified Multimodal Models

    The Hong Kong Polytechnic University · Sichuan University

  20. Spectral-Aware Text-to-Time Series Generation with Billion-Scale Multimodal Meteorological Data

    Mar 28, 2026Shijie ZhangFrequency-Domain Time Series ModelingTime Series Generation

    School of Computer Science and Engineering, Northeastern University, Shenyang, China

  21. Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Training

    Mar 26, 2026Jinbo Xing, Zeyinzi Jiang, Yuxiang Tuo +15Multimodal GenerationMultimodal Foundation Models

    Tongyi Lab · Tsinghua University

  22. OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning

    Mar 25, 2026Kaihang Pan, Qi Tian, Jianwei Zhang +12Multimodal GenerationVideo Generation

    Zhejiang University · Tencent Hunyuan · Nanyang Technological University

  23. M3T: Discrete Multi-Modal Motion Tokens for Sign Language Production

    Mar 24, 2026Alexandre Symeonidis-Herzig, Jianhe Low, Ozge Mercanoglu Sincan +1Human Motion GenerationMultimodal Generation

    CVSSP University of Surrey United Kingdom

  24. Seeing is Improving: Visual Feedback for Iterative Text Layout Refinement

    Mar 23, 2026Junrong Guo, Shancheng Fang, Yadong Qu +1Multimodal GenerationMultimodal Large Language Models

    University of Science and Technology of China · Shenzhen University

  25. Text-Image Conditioned 3D Generation

    Mar 22, 2026Jiazhong Cen, Jiemin Fang, Sikuang Li +83D Asset Generation3D Shape Generation

    MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University · Huawei Inc. · Huazhong University of Science and Technology

  26. OmniTrace: A Unified Framework for Generation-Time Source Attribution in Omni-Modal LLMs

    Mar 20, 2026Qianqi Yan, Yichen Guo, Ching-Chen Kuo +4Multimodal Model InterpretabilityMultimodal Large Language Models

    University of California, Santa Barbara · eBay

  27. Enhancing Alignment for Unified Multimodal Models via Semantically-Grounded Supervision

    Mar 20, 2026Jiyeong Kim, Yerim So, Hyesong Choi +2Unified Multimodal ModelsMultimodal Generation

    Ewha Womans University, South Korea · Soongsil University, South Korea

  28. Do Understanding and Generation Fight? A Diagnostic Study of DPO for Unified Multimodal Models

    Mar 17, 2026Abinav Rao, Sujan RachuriUnified Multimodal ModelsMulti-Objective Language Model Alignment

  29. Complementarity-Preserving Generative Theory for Multimodal ECG Synthesis: A Quantum-Inspired Approach

    Mar 16, 2026Timothy Oladunni, Farouk Ganiyu-Adewumi, Clyde Baidoo +1Multimodal GenerationElectrocardiography

    Department of Computer Science, Morgan State University, Baltimore, MD, USA

  30. HYDRA: Representation Harmonized Tokenization for Multimodal Generation and Understanding

    Mar 16, 2026Xuerui Qiu, Yutao Cui, Guozhen Zhang +9Unified Multimodal ModelsMultimodal Generation

    Institute of Automation, Chinese Academy of Sciences · Tencent Hunyuan · Zhongguancun Academy +2