Unified Multimodal Models
Overview
Unified multimodal models study how one system can process and generate text, images, video, audio, and sometimes spatial or action data. Common approaches use shared tokenizers and autoregressive or diffusion-based backbones, often alongside modality-specific encoders, decoders, or experts; training may combine supervised instruction tuning, alignment, and reinforcement learning. Evaluation spans tasks such as visual question answering, text-to-image generation, video understanding, editing, and cross-modal consistency. A central challenge is that shared architecture does not guarantee useful transfer: understanding and generation can interfere or remain semantically misaligned, while efficiency, safety, and reliable evaluation across interacting modalities remain open concerns.
Latest papers
314 papers26 papers in the last four weeks, down 33% on the four weeks before. 0.2% of all new papers.
AV-Unified: A Unified Framework for Audio-visual Scene Understanding
Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models
UniPAR: A Unified Framework for Pedestrian Attribute Recognition
A Simple Baseline for Unifying Understanding, Generation, and Editing via Vanilla Next-token Prediction
Crab: A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperation
Modeling Cross-vision Synergy for Unified Large Vision Model
Beyond Language Modeling: An Exploration of Multimodal Pretraining
UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?
ACE-Brain-0: Spatial Intelligence as a Shared Scaffold for Universal Embodiments
Towards an Incremental Unified Multimodal Anomaly Detection: Augmenting Multimodal Denoising From an Information Bottleneck Perspective
LLaDA-o: An Effective and Length-Adaptive Omni Diffusion Model
U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation
Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encoder
Can Unified Generation and Understanding Models Maintain Semantic Equivalence Across Different Output Modalities?
The Trinity of Consistency as a Defining Principle for General World Models
OmniZip: Learning a Unified and Lightweight Lossless Compressor for Multi-Modal Data
Synergizing Understanding and Generation with Interleaved Analyzing-Drafting Thinking
Mobile-O: Unified Multimodal Understanding and Generation on Mobile Device
MICON-Bench: Benchmarking and Enhancing Multi-Image Context Image Generation in Unified Multimodal Models
Multimodal Crystal Flow: Any-to-Any Modality Generation for Unified Crystal Modeling
Continual-NExT: A Unified Comprehension And Generation Continual Learning Framework
TimeOmni-VL: Unified Models for Time Series Understanding and Generation
Understanding vs. Generation: Navigating Optimization Dilemma in Multimodal Models
UniTAF: A Modular Framework for Joint Text-to-Speech and Audio-to-Face Modeling
LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Models
Kelix Technical Report
UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Models
Superman: Unifying Skeleton and Vision for Human Motion Perception and Generation
perception'' models that understand motion from video but only output text, and generation'' models that cannot perceive from raw visual input. Second, generative MLLMs are often limited to single-frame, static poses using dense, parametric SMPL models, failing to handle temporal motion. Third, existing motion vocabularies are built from skeleton data alone, severing the link to the visual domain. To address these challenges, we introduce Superman, a unified framework that bridges visual perception with temporal, skeleton-based motion generation. Our solution is twofold. First, to overcome the modality disconnect, we propose a Vision-Guided Motion Tokenizer. Leveraging the natural geometric alignment between 3D skeletons and visual data, this module pioneers robust joint learning from both modalities, creating a unified, cross-modal motion vocabulary. Second, grounded in this motion language, a single, unified MLLM architecture is trained to handle all tasks. This module flexibly processes diverse, temporal inputs, unifying 3D skeleton pose estimation from video (perception) with skeleton-based motion prediction and in-betweening (generation). Extensive experiments on standard benchmarks, including Human3.6M, demonstrate that our unified method achieves state-of-the-art or competitive performance across all motion tasks. This showcases a more efficient and scalable path for generative motion analysis using skeletons.