Multimodal Generation

Follow this topic Ask about this topic

Multimodal generation studies models that create outputs such as images, video, speech, motion, or scientific signals from text, media, or combinations of them. Common approaches include diffusion and flow-matching decoders, autoregressive or discrete-token models, shared latent spaces, and systems that connect vision-language models to modality-specific generators; planning and reinforcement learning can add control for complex or interleaved outputs. A major direction is building models that both understand and generate, with benchmarks measuring prompt adherence, fidelity, temporal and cross-modal coherence, and task-specific validity. An open challenge is enabling shared models to turn understanding into faithful, controllable generation without modality imbalance or loss of specialized quality, especially over long sequences and varied real-world inputs.

Latest papers

243 papers26 papers in the last four weeks, up 4% on the four weeks before. 0.1% of all new papers.

Drift Says When, Budget Says How Much: Rate-Controlled KV Re-selection for Long-Form Multimodal Generation
Oct 22, 2026Bin Hu, Yingbo XiaoMultimodal Large Language Models KV-Cache Compression
Tsinghua University · The Chinese University of Hong Kong, Shenzhen
Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation
Oct 22, 2026Songlin Yang, Zhe Wang, Xiaotong Zhao +8Multimodal Video Generation Video Generation
Tencent Video · LIX, École Polytechnique
JUICE: Unified Multimodal Modeling in Pretrained Representation Spaces
Oct 20, 2026Jiahan Li, Huangyuan Su, Shengbang Tong +9Unified Multimodal Models Multimodal Generation
New York University · Harvard University · University of Illinois Urbana-Champaign +2
Drive2Gauss: FeedForward 4D Gaussian Reconstruction from Multimodal Generative Driving Videos
Oct 19, 2026Tong Zhao, Zhiyuan Han, Lin Chen +10Multimodal Video Generation Dynamic Scene Reconstruction
Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China. · Tongji University, Shanghai, China. · Beijing Technology and Business University, Beijing, China. +8
MIRA: A Multimodal Illustrated Deep Research Agent for Controllable Reports
Oct 19, 2026Hongrui Wu, Kaiwen Tuo, Congcong Wang +7Deep Research Agents Multimodal Agents
Stanford University · The Hong Kong University of Science and Technology · Fudan University +2
Modaliff: A Unified Token Diffusion Model for Any-to-Any Generation
Oct 18, 2026Duc Kieu, Kien Do, Minh Khoa Le +3Multimodal Generation Unified Multimodal Models
Applied Artificial Intelligence Initiative, Deakin University · VinSmart Future · Aalborg University
Beyond Reward Scoring: Online Multi-Agent Preference Learning for Multimodal Response Generation
Oct 16, 2026Ziyun Zou, Jiwoo Hong, Yifan Lu +5Preference Alignment Preference-Based RL
Mohamed bin Zayed University of Artificial Intelligence · Nubank · Dartmouth College
Can Visual Generation Expand What Unified Multimodal Models Know?
Oct 15, 2026Wenkai Li, Wenyi Hong, Zhuoyi Yang +1Unified Multimodal Models Transfer Learning
Tsinghua University
Full-Duplex Interaction with Speech, Expressions and Gestures
Oct 15, 2026Siyin Wang, Wenyi Yu, Terumi Chiba +5Facial Expression Synthesis Multimodal Generation
Tsinghua University · ByteDance
MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data
Oct 14, 2026Robert Moseley, Md Kaykobad Reza, Ameya Patil +2Multimodal Learning Multimodal Foundation Models
University of California, Riverside · Amazon
ReasonUni: Reasoning-Augmented Reconstruction Advances Unified Multimodal Models
Oct 14, 2026Feize Wu, Zhiyuan Yan, Yuhuan Hong +10Unified Multimodal Models Multimodal Generation
Peking University · Meituan · Tsinghua University +1
WorldCast: Exploring Intelligent Visual Generation in World State Prediction
Oct 14, 2026Wenhao Yan, Haodong Li, Xiaotong Li +9World Models Multimodal Generation
THU · CUHK · Meituan LongCat +4
Scaling Unified Multimodal Models to Long-Horizon Interleaved Generation
Oct 12, 2026Ziqiao Peng, Yutao Cui, Wang Lin +12Multimodal Generation Unified Multimodal Models
Renmin University of China · Tencent Hy · Tsinghua University
OmniRSI: Taming Latent Diffusion for Unified Multimodal Remote Sensing Image Generation
Oct 11, 2026Hebaixu Wang, Jing Zhang, Haonan Guo +3Multimodal Generation Unified Multimodal Models
Wuhan University & Zhongguancun Academy
Understanding Unified VLMs: From Attention Probing to Guidance Tuning for Multimodal Generation
Oct 10, 2026Zhengyang Luo, Xiaoyu Qiu, Zhiheng Tan +3Unified Multimodal Models Multimodal Generation
University of Michigan, Ann Arbor · The Ohio State University · Indiana University Indianapolis
Multimodal Co-Generation With Continuous Bitstream Diffusion
Oct 9, 2026Georgios Batzolis, Georgios Kontos, Bruno Trentini +4Multimodal Generation Multimodal Diffusion Models
Radboud University
UniData: Universal Multimodal Instruction Generation Pipeline
Oct 8, 2026Jiaqi Tang, Yi-Feng Wu, Yuting Zhang +12Synthetic Data Generation for Language Models Multimodal Generation
The Hong Kong University of Science and Technology · ATH, Alibaba Group · The Hong Kong University of Science and Technology (Guangzhou) +2
Self-correction Optimization for Interleaved Multimodal Generation
Oct 7, 2026Xin You, Zhiwei Ning, Zukai Chen +7Multimodal Generation Temporal Consistency in Video Generation
Shanghai Jiao Tong University · SenseTime Research · Technical University of Munich
MUNITE: Unified Multimodal Latent Inference for Any-to-Any Multimodal Generation
Oct 7, 2026Kyeongmin Yeo, Minhyuk SungMultimodal Generation Unified Multimodal Models
KAIST
The Poisoned Conversation: Privacy-Leaking Watermarks in Unified Multimodal Models
Oct 4, 2026Tobias Braun, Jonas Henry Grebe, Emil Sivic +4Unified Multimodal Models Multimodal Generation
TU Darmstadt, Germany · hessian.AI, Germany · Zuse School
Structural-Functional Brain Connectivity Generation via Multimodal Hypergraph-based Flow Matching
Oct 2, 2026Chyong Yi Poh, Hwa Hui Tew, Junn Yong Loo +4Multimodal Generation Functional Connectivity
School of Information Technology, Monash University Malaysia · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Department of Radiology, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
Gestalt: Large Multimodal Interplay Model
Sep 30, 2026Zequn Yang, Yu Miao, Haotian Ni +8Unified Multimodal Models Cross-Modal Alignment
Gaoling School of Artificial Intelligence, Renmin University of China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Beihang University +3
ScaffoldM3C: A Multimodal Sequential Monte Carlo Framework for Generative Stable Construction Planning
Sep 30, 2026Gadiel Sznaier Camps, Chengyang He, Guillaume Sartoretti +2Robotic Construction Multimodal Generation
Multi-robot Systems Lab (MSL), Stanford University, Stanford, CA, USA · Multi-Agent Robotic Motion Lab (MARMot), National University of Singapore, Singapore · Perception-Oriented Control Team (POC), Universidad de Zaragoza, Zaragoza, Spain
Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces
Sep 30, 2026Hongyuan Tao, Xinggang Wang, Lianghui Zhu +7Unified Multimodal Models Multimodal Pretraining
Huazhong University of Science and Technology · Beijing Jiaotong University · Horizon Robotics
PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation
Sep 29, 2026Cong Wei, Xuanchi Ren, Bryan Chu +8Unified Multimodal Models Multimodal Generation
NVIDIA
Beyond Layers: Position-Resolved Gradient Conflict and Position-Aware Modulation for Unified Multimodal Models
Sep 29, 2026Shuyang Jiang, Fucheng Deng, Yuchuan Luo +1Unified Multimodal Models Gradient Interference
University of California, Los Angeles · Aimakj · College of Computer Science and Technology National University of Defense Technology +1
What Comes Next? Omni-StoryBench for Evaluating Story-Grounded Omnimodal Generation
Sep 29, 2026Sieun Hyeon, Yejoon Lee, Mintaek Lim +3Multimodal Generation Multimodal Model Evaluation
AIDAS Laboratory, 1ECE & 2IPAI, Seoul National University
Mutually Adversarial Self-Training with Evolving Data for Unified Multimodal Models
Sep 28, 2026Wentao Zhou, Weijie Gan, Jiayun WangAdversarial Training Unified Multimodal Models
Georgia Institute of Technology · Washington University in St. Louis
Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomy
Sep 28, 2026Abhinav Sharma, Sai Karthik Navuluru, Wang Wei +21Audio-Video Generation Multimodal Generation
University of Massachusetts Amherst · University of Texas at Dallas · Virginia Tech +10
Prompting Particle Physics: Tokenized Multi-modal Foundation Models for Combinatorially Many Tasks
Sep 25, 2026Nilotpal Kakati, Daniel Murnane, Baran Hashemi +5Unified Multimodal Models High-Energy Physics
Weizmann Institute of Science, Rehovot, Israel · Niels Bohr Institute, University of Copenhagen, Copenhagen, Denmark · Lawrence Berkeley National Laboratory, Berkeley, CA, USA +3

Oct 22, 2026J/K move · Enter open · S save

Drift Says When, Budget Says How Much: Rate-Controlled KV Re-selection for Long-Form Multimodal Generation

Bin Hu, Yingbo Xiao

Tsinghua University · The Chinese University of Hong Kong, Shenzhen

Multimodal Large Language Models KV-Cache Compression Multimodal Generation LLM Inference Acceleration

Open paper AskPDF arXiv

KV cache compression for multimodal LLMs usually selects the visual tokens to keep once, at prefill, and reuses that selection for the whole generation. We show that this one-shot selection is severely limited on long-form generation: on detailed image description, even an oracle that selects with the attention of the uncompressed answer reaches only 0.44-0.52 ROUGE-L agreement with the uncompressed output, because the evidence the model reads moves as the description unfolds. Re-selecting whenever attention drifts out of the kept set is the natural remedy, but we find that a fixed drift threshold fails systematically: drift magnitudes are not comparable across samples, so low-drift samples are almost never refreshed although they benefit the most. We propose Rate-Controlled Refresh (RCR), which lets drift decide when and where to re-select while a per-sample controller regulates how often, combined with a low-rank drift detector, candidate-restricted re-selection, and a decision that stays on the GPU. With one setting for all models and tasks, RCR outperforms the per-step retrieval method InfiniGen by 1.6-3.8 ROUGE-L points on five of six model-task settings with no significant difference on the sixth, while staying within 1.7 points of exact per-step re-selection, with comparable estimated data movement per step and a 4x smaller GPU-resident detector. On DetailCaps, drift-chosen timing beats random timing at the same refresh rate and periodic timing even at a higher rate, by 2.0-5.0 points. On pre-registered video description, where drift is homogeneous across samples, RCR is on par with InfiniGen on Qwen2-VL and exceeds it on Qwen3-VL but no longer beats a fixed threshold at the nearest tested rates, consistent with the diagnosis; object-level evaluation shows that RCR achieves higher object F1 against the uncompressed output than InfiniGen on two of three models.