Multimodal Understanding

Follow this topic Ask about this topic

The field studies how AI can combine evidence from different kinds of input to answer questions, recognize events, retrieve information, and reason about scenes over time. Common approaches include vision-language models and multimodal large language models, trained with contrastive alignment, cross-attention, instruction tuning, and reinforcement learning; some systems add modality-specific experts, retrieval, or tools. Benchmarks test image and video question answering, document and chart understanding, audio-visual reasoning, grounding, and performance across languages and domains. A central challenge is ensuring that answers genuinely depend on relevant evidence from each modality, while handling long inputs efficiently and making capabilities measurable with reliable tests.

Latest papers

479 papers15 papers in the last four weeks, up 25% on the four weeks before. 0.1% of all new papers.

Zooming without Zooming v2: From Seeing Details to Connecting Evidence for Generalizable Multimodal Perception
Oct 27, 2026Lai Wei, Yatai Li, Liangbo He +10Multimodal Understanding Multimodal Large Language Models
School of Computer Science, Shanghai Jiao Tong University · Zhongguancun Academy · Ant Group +2
Generalized CycleGRPO for Fine-Grained Multimodal Understanding and Grounding
Oct 21, 2026Xin Zhang, Haochen Wang, Beibei Lin +4Multimodal Grounding Multimodal Understanding
JUICE: Unified Multimodal Modeling in Pretrained Representation Spaces
Oct 20, 2026Jiahan Li, Huangyuan Su, Shengbang Tong +9Unified Multimodal Models Multimodal Generation
New York University · Harvard University · University of Illinois Urbana-Champaign +2
SenseAlign: Understanding Vision and Touch through Language at Test Time
Oct 20, 2026Chuyang Ye, Qinting Jiang, Jingyan JiangMultimodal Understanding Cross-Modal Alignment
New York University · Tsinghua Shenzhen International Graduate School, Tsinghua University · Shenzhen Technology University
Beyond Omni: A Unified MLLM for Embodied Sensory Understanding
Oct 19, 2026Shiyun Xiong, Dongming Wu, Peiwen Sun +5Unified Multimodal Models Multimodal Large Language Models
MMLab, The Chinese University of Hong Kong · Shenzhen Loop Area Institute · The Chinese University of Hong Kong, Shenzhen +2
VISTA: A Multimodal Egocentric Benchmark for Goal-Oriented Perception and Assistance to Blind and Low-Vision Users
Oct 16, 2026Noé Mosca, Mohamed Rayan Barhdadi, Muhammad Hamza +11Assistive Technology for Blind and Low-Vision People Visually Impaired Navigation Assistance
Harvard AI and Robotics Lab, Harvard University · Télécom Paris, Institut Polytechnique de Paris · Texas A&M University +2
What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models
Oct 14, 2026Tengju Ru, Weitong Lian, Zecong Tang +6Multimodal Understanding Unified Multimodal Models
Zhejiang University · Shenzhen Loop Area Institute · The University of Hong Kong +1
GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?
Oct 10, 2026Zonghuan Li, Hayley Hung, Chirag RamanMultimodal Model Evaluation Multimodal Understanding
Department of EEMCS, Delft University of Technology, Delft, Netherlands
UkrMeme: Multimodal Explainable Persuasion Detection in Ukrainian Memes
Oct 9, 2026Nazar Andrushko, Vera Schmitt, Veronika SolopovaVLM Evaluation VLM Interpretability
Ukrainian Catholic University, Lviv, Ukraine · Johannes Gutenberg University Mainz, Mainz, Germany · German Research Center for Artificial Intelligence (DFKI), Berlin, Germany +1
Addressing Overcommitment in the Reasoning of Gendered Economic Memes under Multimodal Ambiguity
Oct 8, 2026Kushal Kanwar, Dushyant Singh Chauhan, Kapil Rana +2Multimodal Understanding Epistemic Uncertainty
Jaypee University of Information Technology · Mohamed bin Zayed University of Artificial Intelligence · Thapar Institute of Engineering & Technology +1
MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface
Oct 8, 2026Jianpeng Cheng, Guangyu Sun, Aashu Singh +11Text Embedding Models Cross-Modal Retrieval
Meta AI
Omni-Diffusion-Distill: Few-Step Distillation of Unified Multimodal Diffusion Large Language Models
Oct 7, 2026Hong Huang, Chenhongyi Yang, Junzhe Sun +3Diffusion Language Model Distillation Diffusion Model Inference Acceleration
Simon Fraser University · Meta
Gestalt: Large Multimodal Interplay Model
Sep 30, 2026Zequn Yang, Yu Miao, Haotian Ni +8Unified Multimodal Models Cross-Modal Alignment
Gaoling School of Artificial Intelligence, Renmin University of China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Beihang University +3
PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation
Sep 29, 2026Cong Wei, Xuanchi Ren, Bryan Chu +8Unified Multimodal Models Multimodal Generation
NVIDIA
Beyond Layers: Position-Resolved Gradient Conflict and Position-Aware Modulation for Unified Multimodal Models
Sep 29, 2026Shuyang Jiang, Fucheng Deng, Yuchuan Luo +1Unified Multimodal Models Gradient Interference
University of California, Los Angeles · Aimakj · College of Computer Science and Technology National University of Defense Technology +1
Mutually Adversarial Self-Training with Evolving Data for Unified Multimodal Models
Sep 28, 2026Wentao Zhou, Weijie Gan, Jiayun WangAdversarial Training Unified Multimodal Models
Georgia Institute of Technology · Washington University in St. Louis
Refinement Symmetry in Multimodal Transformers
Sep 26, 2026Yuhao Du, Shunian ChenAttention Mechanisms Multimodal Robustness
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute
Multimodal Voice Activity Projection for Social Robot Mediation: Expected Behavior and Deployment Constraints
Sep 23, 2026Antonio Cano, Guillermo Perez, Luis Merino +1Turn-Taking Prediction Human-Robot Interaction
4i Intelligent Insights, Seville, Spain · Universidad de Sevilla, Seville, Spain · Universidad Pablo de Olavide, Seville, Spain +1
Virtual Encoders in Multimodal Transformers
Sep 22, 2026Katsuya Ogata, Yuta NakashimaVisual Representation Learning Multimodal Large Language Models
The University of Osaka
Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interaction
Sep 18, 2026Qi Chen, Yunfei Chu, Haolin He +15Multimodal Intent Recognition Multimodal Model Evaluation
Shanghai Jiao Tong University · Shanghai Innovation Institute · Alibaba Token Hub, Alibaba Group +5
Efficient Unified Multimodal Understanding (EUMU): Winning Solution for the MUMU Track at the 8th LSVOS Challenge
Sep 16, 2026Dayoung Kil, Seong-heum KimUnified Multimodal Models Efficient Multimodal Inference
Department of Intelligent Semiconductors, Soongsil University, Seoul, Republic of Korea
Negation Beyond the Verbal Channel: Temporal Multimodal Correlates in Dialogue
Sep 14, 2026Leon Hammerla, Patrick Schrottenbacher, Alexander MehlerMultimodal Understanding
Goethe University, Frankfurt am Main, Germany
MultiHuSE: A Multimodal Dataset for Humour Styles and Emotions
Sep 10, 2026Mary Ogbuka Kenneth, Foaad Khosmood, Abbas EdalatText Classification Multimodal Understanding
Algorithmic Human Development group, Department of Computing Imperial College London London, United Kingdom · Computer Engineering Department California Polytechnic State University San Luis Obispo, United States
DYAD: A Multimodal Dataset of Co-Located Human Assistance
Sep 8, 2026Akhil Ajikumar, Mahya Qorbani, Sakib Reza +2Human-AI Interaction Assistive Robotics
Georgia Institute of Technology, Atlanta, GA, USA · Northeastern University, Boston, MA, USA · Microsoft Research, Redmond, WA, USA
BanglaMemeX: Advancing Cultural Metaphoric Image Interpretation in Bangla with a Multimodal Explainable Dataset
Sep 7, 2026Md. Sadman Sakib, Zisan Mahmud, Md. Fahim Arefin +1Figurative Language Understanding Multilingual VLM Evaluation
University of Dhaka · University of Texas at Dallas
Generalist Open-World Temporal Perception
Sep 6, 2026Cristian SminchisescuWorld Model Learning Multimodal Understanding
Google DeepMind · Lund University · IMAR
Video2Reaction: Training Foundation Video Models to Predict Audience Reaction
Sep 1, 2026Sidong Zhang, Trang Nguyen, Shiv Shankar +4VLM Adaptation Video-Language Models
UMass Amherst 130 Governors Drive, Amherst · Dolby Laboratories 1275 Market Street, San Francisco
Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System
Sep 1, 2026Penghao Wu, Haiwen Diao, Weichen Fan +3Unified Multimodal Models Multimodal Generation
1S-Lab, Nanyang Technological University · 2SenseTime Research
Do Multimodal LLMs See Before They Read? Diagnosing Contextual Sycophancy
Aug 30, 2026Yi-Cheng Lai, Hen-Hsen HuangLLM Sycophancy Evidence-Grounded Reasoning
Institute of Information Science, Academia Sinica Taipei, Taiwan
Order Matters: A Chinese Multi-Panel Meme Benchmark for Vision-Language Reasoning
Aug 27, 2026Haihan Li, Haihao Li, Zhenfei Xu +2Multimodal Reasoning VLM Reasoning
Shanghai Maritime University, Shanghai, China · Dalian Maritime University, Dalian, China · Oxford Suzhou Centre for Advanced Research

Oct 27, 2026J/K move · Enter open · S save

Zooming without Zooming v2: From Seeing Details to Connecting Evidence for Generalizable Multimodal Perception

Lai Wei, Yatai Li, Liangbo He, Jun Lan+9

School of Computer Science, Shanghai Jiao Tong University · Zhongguancun Academy · Ant Group+2

Multimodal Understanding Multimodal Large Language Models Synthetic Data Generation Multimodal Spatial Reasoning

Open paper AskPDF arXiv

Multimodal perception forms the foundation of multimodal intelligence. In particular, spatial perception and multimodal long-context understanding are two pivotal tasks that underpin embodied AI and multimodal agents. However, synthesizing targeted, high-quality training data for these tasks remains difficult and costly. Our key insight is that these tasks share an underlying capability, which we call connecting evidence: identifying relevant visual facts, binding them to the correct entities, recovering their relations, and composing them into an answer. Crucially, targeted training data are far easier to synthesize for this capability than for these tasks themselves. To strengthen this capability, we propose EvidenceConnect, an easy-to-implement data synthesis framework that uses zoomed views of related regions within a single image to generate questions requiring comparison, aggregation, or chaining of visual evidence. To ensure quality, a reference answer is retained only when independent VLMs agree on it, and region-ablation tests discard questions whose answer survives the removal of selected regions. We then train base VLMs on the resulting data with RLVR, obtaining Zooming without Zooming v2 (ZwZ2). Although the synthesized data are not tailored to any downstream benchmark, ZwZ2 improves fine-grained and spatial perception, with further gains on embodied spatial reasoning, multimodal long-context understanding, multimodal agent tasks, and general visual reasoning. These results establish EvidenceConnect as an effective framework for supervising evidence connection and suggest that strengthening this shared capability leads to more generalizable multimodal perception.