Multimodal Large Language Models

Also known as MLLM

Multimodal large language models connect language generation and reasoning to visual, audio, video, and sometimes sensor or structured data. Common approaches align pretrained modality encoders with a language model using projectors or adapters, or train models to process modalities jointly; instruction tuning, reinforcement learning, retrieval, and tool use extend their capabilities. Evaluation spans visual question answering, document reading, long-video and spatial reasoning, and specialized settings such as medical imaging, where models still often miss fine-grained evidence or fail to use it consistently. A central tension is how to improve cross-modal grounding, temporal and physical reasoning, and safety while reducing the high inference costs of processing long or detailed inputs.

Latest papers

1,096 papers106 papers in the last four weeks, up 23% on the four weeks before. 0.6% of all new papers.

On the Generalization Capacities of MLLMs for Spatial Intelligence
Mar 5, 2026Gongjie Zhang, Wenhao Li, Quanhao Qian +4Multimodal Large Language Models 3D Spatial Reasoning
DAMO Academy, Alibaba Group · HuPan Lab · Nanyang Technological University
Mario: Multimodal Graph Reasoning with Large Language Models
Mar 5, 2026Yuanfu Sun, Kang Li, Pengkang Guo +2LLM Reasoning with Graphs Multimodal Reasoning
New York University Shanghai · New York University · Tsinghua University +1
VisionPangu: A Compact and Fine-Grained Multimodal Assistant with 1.7B Parameters
Mar 5, 2026Jiaxin Fan, Wenpo SongVision-Language Models Multimodal Large Language Models
National Key Laboratory for Novel Software Technology, Nanjing University
Multi-Paradigm Collaborative Adversarial Attack Against Multi-Modal Large Language Models
Mar 5, 2026Yuanbo Li, Tianyang Xu, Cong Hu +3Adversarial Attacks on Unified Multimodal Models Transfer-Based Adversarial Attacks
School of Artificial Intelligence and Computer Science, Jiangnan University · Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey
MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Models
Mar 5, 2026Lulu Hu, Wenhu Xiao, Xin Chen +4Post-Training Quantization Neural Network Quantization
Alibaba Cloud Computing, Alibaba Group
Are Multimodal LLMs Ready for Surveillance? A Reality Check on Zero-Shot Anomaly Detection in the Wild
Mar 5, 2026Shanle Yao, Armin Danesh Pazho, Narges Rashvand +1Video Anomaly Detection Zero-Shot Anomaly Detection
Electrical and Computer Engineering Department, University of North Carolina at Charlotte, Charlotte, USA
Visuospatial Perspective Taking in Multimodal Language Models
Mar 4, 2026Jonathan Prunty, Seraphina Zhang, Patrick Quinn +3Multimodal Large Language Models Visual Spatial Reasoning
Leverhulme Centre for the Future of Intelligence, University of Cambridge · Department of Psychology, University of Cambridge · Microsoft Research Asia
Social Norm Reasoning in Multimodal Language Models: An Evaluation
Mar 3, 2026Oishik Chowdhury, Anushka Debnath, Bastin Tony Roy SavarimuthuSocial Reasoning Multimodal Large Language Models
School of Computing, University of Otago, New Zealand
MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization
Mar 3, 2026Ashutosh Chaubey, Jiacheng Pang, Mohammad SoleymaniMultimodal Large Language Models Multimodal Hallucination
University of Southern California
OCR or Not? Rethinking Document Information Extraction in the MLLMs Era with Real-World Large-Scale Datasets
Mar 3, 2026Jiyuan Shen, Peiyue Yuan, Atin Ghosh +2Document Information Extraction Multimodal Large Language Models
SAP · Stanford University
Seeing Clearly without Training: Mitigating Hallucinations in Multimodal LLMs for Remote Sensing
Mar 3, 2026Yi Liu, Jing Zhang, Di Wang +3Remote Sensing VQA VLM Hallucination Mitigation
School of Computer Science, Wuhan University, China · Zhongguancun Academy, China · School of Computer Science, Chongqing University, China
Real-Time Generation of Game Video Commentary with Multimodal LLMs: Pause-Aware Decoding Approaches
Mar 3, 2026Anum Afzal, Yuki Saito, Hiroya Takamura +5Language Model Decoding Multimodal Large Language Models
School of CIT, Technical University of Munich · National Institute of Advanced Industrial Science and Technology (AIST) · The University of Tokyo +3
On Discriminative vs. Generative classifiers: Rethinking MLLMs for Action Understanding
Mar 3, 2026Zhanzhong Pang, Dibyadip Chatterjee, Fadime Sener +1Classification Multimodal Classification
National University of Singapore
MUSE: A Run-Centric Platform for Multimodal Unified Safety Evaluation of Large Language Models
Mar 3, 2026Zhongxi Wang, Yueqian Lin, Jingyang Zhang +3LLM Safety Benchmarks LLM Safety
1Duke University · 2Virtue AI
Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy
Mar 2, 2026Jiahao Huang, Fengyan Lin, Xuechao Yang +4Affective Computing Multimodal Large Language Models
Fujian Normal University · RMIT University · Independent Researcher +1
MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning
Mar 2, 2026Jiachun Li, Shaoping Huang, Zhuoran Jin +5Multi-Image Reasoning Multimodal Reasoning
School of Artificial Intelligence, University of Chinese Academy of Sciences · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences
DeLo: Dual Decomposed Low-Rank Experts Collaboration for Continual Missing Modality Learning
Mar 2, 2026Xiwei Liu, Yulong Li, Feilong Tang +1Missing-Modality Learning Multimodal Large Language Models
Mohamed bin Zayed University of Artificial Intelligence · Monash University
Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMs
Mar 2, 2026Hengwei Ye, Yuanting Guan, Yuxuan Ge +7Multimodal Model Evaluation Multimodal Large Language Models
ShanghaiTech University
Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning
Mar 2, 2026Jiahan Chen, Da Li, Hengran Zhang +6Multimodal Embedding Multimodal Learning
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China · Baidu Inc., Beijing, China
Can Thinking Models Think to Detect Hateful Memes?
Mar 1, 2026Mohamed Bayan Kmainasi, Mucahid Kutlu, Ali Ezzat Shahroor +2Harmful Meme Detection Multimodal Large Language Models
Qatar University Doha, Qatar · Qatar Computing Research Institute Doha, Qatar · APAVI.AI France +1
FCN-LLM: Empower LLM for Brain Functional Connectivity Network Understanding via Graph-level Multi-task Instruction Tuning
Mar 1, 2026Xingcan Hu, Wei Wang, Li XiaoBrain Network Analysis Functional Connectivity
University of Science and Technology of China
From Intuition to Investigation: A Tool-Augmented Reasoning MLLM Framework for Generalizable Face Anti-Spoofing
Mar 1, 2026Haoyuan Zhang, Keyao Wang, Guosheng Zhang +11Face Presentation Attack Detection Face Anti-Spoofing
SAI, UCAS · MAIS, CASIA · Baidu Inc +2
Seeing Beyond 8bits: Subjective and Objective Quality Assessment of HDR-UGC Videos
Mar 1, 2026Shreshth Saini, Bowen Chen, Neil Birkbeck +3Video Quality Assessment HDR Imaging
Laboratory for Image and Video Engineering (LIVE), UT Austin. · Google/YouTube
MARS: Harmonizing Multimodal Convergence via Adaptive Rank Search
Feb 28, 2026Minkyoung Cho, Insu Jang, Shuowei Jin +5Multimodal Large Language Models Multimodal Optimization
University of Michigan · NVIDIA
Mema: Memory-Augmented Adapter for Enhanced Vision-Language Understanding
Feb 28, 2026Ying Liu, Yudong Han, Kean Shi +1Memory-Augmented VLMs Multimodal Large Language Models
Beijing Institute of Technology · Peking University · Yangtze Delta Region Academy of Beijing Institude of Technology, Jiaxing, China
UNICBench: UNIfied Counting Benchmark for MLLM
Feb 28, 2026Chenggang Rong, Tao Han, Zhiyuan Zhao +5Multimodal Large Language Models Multimodal Dataset
Northwestern Polytechnical University · Hong Kong University of Science and Technology · Institute of Artificial Intelligence (TeleAI), China Telecom +2
MIDAS: Multi-Image Dispersion and Semantic Reconstruction for Jailbreaking MLLMs
Feb 28, 2026Yilian Liu, Xiaojun Jia, Guoshun Nan +6Multimodal Jailbreak Attacks Adversarial Attacks on Unified Multimodal Models
Beijing University of Posts and Telecommunications, China · Nanyang Technological University, Singapore · Guilin University of Electronic Technology, China
Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation
Feb 28, 2026Zeyu Chen, Huanjin Yao, Ziwang Zhao +1LLM Evaluation LLM-as-a-Judge
Pragma-VL: Towards a Pragmatic Arbitration of Safety and Helpfulness in MLLMs
Feb 28, 2026Ming Wen, Kun Yang, Xin Chen +4Multimodal Large Language Models LLM Safety Alignment
Fudan University · Shanghai Innovation Institute · Ant Group +3
MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence
Feb 28, 2026Xingyilang Yin, Chengzhengxu Li, Jiahao Chang +2Spatiotemporal Reasoning Multimodal Large Language Models
University of Macau · GVC Lab, Great Bay University · Xi’an Jiaotong University +1

Mar 5, 2026J/K move · Enter open · S save

On the Generalization Capacities of MLLMs for Spatial Intelligence

Gongjie Zhang, Wenhao Li, Quanhao Qian, Jiuniu Wang+3

DAMO Academy, Alibaba Group · HuPan Lab · Nanyang Technological University

Multimodal Large Language Models 3D Spatial Reasoning

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) that directly process RGB inputs for tasks like 3D localization and navigation have shown remarkable potential. However, we argue that these RGB-only approaches are fundamentally flawed in their ability to generalize across cameras. By ignoring camera parameters, they entangle an object's physical properties with the camera's perspective, creating an irresolvable ambiguity. We show this leads MLLMs to overfit to the training camera distribution, rather than learning true and generalizable 3D geometric principles. To address this, we propose Camera-Aware MLLM framework for spatial MLLMs. It learns generalizable spatial reasoning by: (i) injecting camera intrinsics via a dense embedding that conditions each visual token; (ii) introducing a camera-aware data augmentation strategy that synthetically varies camera parameters, forcing the model to disentangle camera properties from scene content; and (iii) distilling geometric priors from a 3D vision foundation model. Extensive experiments demonstrate that camera-aware MLLMs substantially outperform their naive counterparts, particularly in cross-camera generalization tests on spatially-grounded tasks, indicating that camera-awareness is not only beneficial but also a prerequisite for robust and generalizable spatial intelligence in MLLMs.