Multimodal Understanding

Follow this topic Ask about this topic

Momentum

10 papers in the last four weeks, up 233% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 413

All topics

JoyAI-Image: Awaking Spatial Intelligence in Unified Multimodal Understanding and Generation
May 5, 2026Lin Song, Wenbo Li, Guoqing Ma +16Unified Multimodal Models Multimodal Foundation Models
Joy Future Academy, JD
ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval
May 5, 2026Honglei Zhang, Yuting Chen, Chenpeng Hu +2Audio Reasoning Audio Understanding
School of Software, Nanjing University Nanjing, China · School of Software, Northwestern Polytechnical University Xi’an, China · College of Computing and Data Science, Nanyang Technological University Singapore, Singapore
Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Experts
May 5, 2026Hahyeon Choi, Nojun KwakCross-Modal Representation Learning Sparse Mixture-of-Experts
Seoul National University, Seoul, South Korea.
Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs
May 4, 2026Xin Zhang, Qiqi Tao, Jiawei Du +2Inference-Time Optimization Test-Time Optimization
Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · Singapore University of Technology and Design +1
HY-Himmel Technical Report: Hierarchical Interleaved Multi-stream Motion Encoding for Long Video Understanding
May 4, 2026Haopeng Jin, Hongzhu Yi, Wenlong Zhao +6Video Understanding Video-Language Models
Tencent · University of Chinese Academy of Sciences · Beijing Forestry University
VIDA: A Dataset for Visually Dependent Ambiguity in Multimodal Machine Translation
May 3, 2026Jingheng Pan, Xintong Wang, Longyue Wang +3Large Vision-Language Models Multimodal Model Evaluation
Department of Informatics, Universität Hamburg · Alibaba Group
Referring Multiple Regions with Large Multimodal Models via Contextual Latent Steering
May 3, 2026Yun Xing, Hanyuan Liu, Jiahao Nie +1Multimodal Large Language Models Vision-Language Grounding
Nanyang Technological University
Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Models
May 3, 2026Yuriel Ryan, Hei Man Ip, Adriel Kuek +2VLM Robustness Multimodal Hallucination
Singapore University of Technology and Design · DSO National Laboratories · Massachusetts Institute of Technology
OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder
May 2, 2026Detao Bai, Shimin Yao, Weixuan Chen +4Cross-Modal Learning Video Understanding
1Tongyi Lab Alibaba Group · Shenzhen University of Advanced Technology
Valley3: Scaling Omni Foundation Models for E-commerce
May 2, 2026Zeyu Chen, Guanghao Zhou, Min Yang +6Unified Multimodal Models Multimodal Pretraining
Valley Team, ByteDance Group
LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations
May 1, 2026Huangbiao Xu, Huanqi Wu, Xiao Ke +1Missing-Modality Learning Multimodal Reasoning
Fujian Provincial Key Laboratory of Networking Computing and Intelligent Information Processing, College of Computer and Data Science, Fuzhou University, Fuzhou 350108, China · Engineering Research Center of Big Data Intelligence, Ministry of Education, Fuzhou 350108, China · Wangxuan Institute of Computer Technology, Peking University, Beijing 100871, China
GaMMA: Towards Joint Global-Temporal Music Understanding in Large Multimodal Models
May 1, 2026Zuyao You, Zhesong Yu, Mingyu Liu +3Multimodal Pretraining Multimodal Large Language Models
1Fudan University · 2ByteDance
Group Cognition Learning: Making Everything Better Through Governed Two-Stage Agents Collaboration
May 1, 2026Chunlei Meng, Pengbin Feng, Rong Fu +7Cross-Modal Learning Multi-Agent Collaboration
The College of Intelligent Robotics and Advanced Manufacturing, Fudan University · University of Southern California · University of Macau +5
Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL
Apr 30, 2026Sudong Wang, Weiquan Huang, Xiaomin Yu +9Multimodal Large Language Models Multimodal Reasoning
Hong Kong University of Science and Technology (Guangzhou) · Nanyang Technological University · Tsinghua University +3
MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
Apr 30, 2026Junbo Cui, Bokai Xu, Chongyi Wang +33Efficient Multimodal Inference Unified Multimodal Models
MiniCPM-o Team, OpenBMB
COHERENCE: Benchmarking Fine-Grained Image-Text Alignment in Interleaved Multimodal Contexts
Apr 30, 2026Bingli Wang, Huanze Tang, Haijun Lv +5Vision-Language Models Vision-Language Grounding
1Southeast University · 2Shanghai AI Laboratory
GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents
Apr 29, 2026GLM-V Team, :, Wenyi Hong +95Multimodal Large Language Models Multimodal Reasoning
Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning
Apr 29, 2026Junwon You, Mihyun Jang, Sangwoo Mo +1Cross-Modal Learning Cross-Modal Representation Learning
1KAIST · 2POSTECH
M $^3$ -VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering
Apr 28, 2026Jiatong Ma, Longteng Guo, Yuchen Liu +4Multimodal RAG Visual Question Answering
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Models
Apr 27, 2026Weixing Wang, Liudvikas Zekas, Anton Hackl +5Unified Multimodal Models Cross-Modal Alignment
Hasso Plattner Institute / University of Potsdam, Potsdam, Germany
Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence
Apr 27, 2026NVIDIA, :, Amala Sanjay Deshmukh +216Efficient Multimodal Inference Multimodal Large Language Models
Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation
Apr 27, 2026Zhiheng Liu, Weiming Ren, Xiaoke Huang +12Unified Multimodal Models Multimodal Pretraining
Meta AI · University of Waterloo · The University of Hong Kong
X2SAM: Any Segmentation in Images and Videos
Apr 27, 2026Hao Wang, Limeng Qiao, Chi Zhang +4Image Segmentation Video Object Segmentation
Sun Yat-Sen University · 2 Peng Cheng Laboratory · 3 Meituan Inc
Omni-o3: Deep Nested Omnimodal Deduction for Deliberative Audio-Visual Reasoning
Apr 27, 2026Zhicheng Zhang, Wentao Gu, Weicheng Wang +5Audio-Visual Understanding Tree Search
Nankai University · Kuaishou Technology · Pengcheng Laboratory +1
Multimodal QUD: Inquisitive Questions from Scientific Figures
Apr 26, 2026Yating Wu, William Rudman, Venkata S Govindarajan +2Multimodal Grounding Scientific QA
The University of Texas at Austin · Ithaca College · UC Berkeley, BespokeLabs.ai
PushupBench: Your VLM is not good at counting pushups
Apr 25, 2026Shengzhi Li, Jiarun Chen, Karun Sharma +2VLM Evaluation Video-Language Model Evaluation
EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMs
Apr 25, 2026He Hu, Tengjin Weng, Zebang Cheng +5Multimodal Large Language Models Multimodal Model Evaluation
Shenzhen University Shenzhen, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) Shenzhen, China · Queen Mary University of London +3
StoryTR: Narrative-Centric Video Temporal Retrieval with Theory of Mind Reasoning
Apr 25, 2026Xuanyue Zhong, Yuqiang Xie, Guanqun Bi +2Temporal Video Grounding Video Understanding
Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong · SkyReels, Skywork AI · Independent Researcher
CGC: Compositional Grounded Contrast for Fine-Grained Multi-Image Understanding
Apr 24, 2026Lihao Zheng, Zhenwei Shao, Yu Zhou +5Multimodal Grounding Multimodal Large Language Models
School of Computer Science and Technology, Hangzhou Dianzi University · Base Model, Li Auto
Can Multimodal Large Language Models Truly Understand Small Objects?
Apr 24, 2026Fujun Han, Junan Chen, Xintong Zhu +4Multimodal QA Multimodal Model Evaluation
Shanghai AI Laboratory · The Chinese University of Hong Kong, Shenzhen · University of Science and Technology of China +2

May 5, 2026, cs.GRJ/K move · Enter open · S save

JoyAI-Image: Awaking Spatial Intelligence in Unified Multimodal Understanding and Generation

Lin Song, Wenbo Li, Guoqing Ma, Wei Tang+15

Joy Future Academy, JD

Unified Multimodal Models Multimodal Foundation Models Image Editing Controllable Image Generation Spatial Reasoning Multimodal Diffusion Models+1

Open paper AskPDF arXiv

We present JoyAI-Image, a unified multimodal foundation model for visual understanding, text-to-image generation, and instruction-guided image editing. JoyAI-Image couples a spatially enhanced Multimodal Large Language Model (MLLM) with a Multimodal Diffusion Transformer (MMDiT), allowing perception and generation to interact through a shared multimodal interface. Around this architecture, we build a scalable training recipe that combines unified instruction tuning, long-text rendering supervision, spatially grounded data, and both general and spatial editing signals. This design gives the model broad multimodal capability while strengthening geometry-aware reasoning and controllable visual synthesis. Experiments across understanding, generation, long-text rendering, and editing benchmarks show that JoyAI-Image achieves state-of-the-art or highly competitive performance. More importantly, the bidirectional loop between enhanced understanding, controllable spatial editing, and novel-view-assisted reasoning enables the model to move beyond general visual competence toward stronger spatial intelligence. These results suggest a promising path for unified visual models in downstream applications such as vision-language-action systems and world models.

Momentum

Related topics

Latest papers 413