Multimodal Large Language Models

Also known as MLLM

Multimodal large language models connect language generation and reasoning to visual, audio, video, and sometimes sensor or structured data. Common approaches align pretrained modality encoders with a language model using projectors or adapters, or train models to process modalities jointly; instruction tuning, reinforcement learning, retrieval, and tool use extend their capabilities. Evaluation spans visual question answering, document reading, long-video and spatial reasoning, and specialized settings such as medical imaging, where models still often miss fine-grained evidence or fail to use it consistently. A central tension is how to improve cross-modal grounding, temporal and physical reasoning, and safety while reducing the high inference costs of processing long or detailed inputs.

Latest papers

1,096 papers106 papers in the last four weeks, up 23% on the four weeks before. 0.6% of all new papers.

LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models
Apr 8, 2026Shi-Yu Tian, Zhi Zhou, Kun-Yang Yu +5Visual Spatial Reasoning Multimodal Large Language Models
National Key Laboratory for Novel Software Technology, Nanjing University, China
Steering the Verifiability of Multimodal AI Hallucinations
Apr 8, 2026Jianhong Pang, Ruoxi Cheng, Ziyi Ye +4Hallucination in Language Models Multimodal Hallucination
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI
Decoupling and Regulating Modality Heads for Fake News Detection under Missing Modality
Apr 8, 2026Kai Qian, Dongjie Xu, Weijie Shi +7Fake News Detection Multimodal Robustness
Soochow University · Hong Kong University of Science and Technology · Computer Science and Engineering, The Chinese University of Hong Kong +2
ValueGround: Evaluating Culture-Conditioned Visual Value Grounding in MLLMs
Apr 7, 2026Zhipin Wang, Christoph Leiter, Christian Frey +3Cultural Knowledge in Language Models VLM Evaluation
University of Technology Nuremberg (UTN), Germany
HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model Inference
Apr 7, 2026Bowen Zeng, Feiyang Ren, Jun Zhang +4KV-Cache Compression Efficient Multimodal Inference
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · Hangzhou Dianzi University, Hangzhou, China
Let Geometry GUIDE: Layer-wise Unrolling of Geometric Priors in Multimodal LLMs
Apr 7, 2026Chongyu Wang, Ting Huang, Chunyu Sun +3Multimodal Large Language Models 3D VLMs
Xi’an Jiaotong University · Peking University · Institute of Automation, Chinese Academy of Sciences +1
Cross-Modal Coreference Alignment: Enabling Reliable Information Transfer in Omni-LLMs
Apr 7, 2026Hongcheng Liu, Yuhao Wang, Zhe Chen +5Cross-Modal Alignment Multimodal Large Language Models
Shanghai Jiao Tong University
A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Models
Apr 6, 2026Tianmeng Fang, Yong Wang, Zetai Kong +4Backdoor Defense in LLMs LLM Security
School of Computing and Information Systems, Singapore Management University, Singapore, 178902, , Singapore. · School of Artificial Intelligence, China University of Mining and Technology, Beijing, 102206, Beijing, PR China. · Faculty of Arts, The University of Melbourne, Melbourne, Carlton VIC 3053, Australia. +3
MolDA: Molecular Understanding and Generation via Large Language Diffusion Model
Apr 6, 2026Seohyeon Shin, HanJun Choi, Jun-Hyung Park +2Multimodal Large Language Models Molecular Graph Generation
Gwangju Institute of Science and Technology, Gwangju, South Korea · Hankuk University of Foreign Studies, Seoul, South Korea
A Systematic Study of Cross-Modal Typographic Attacks on Audio-Visual Reasoning
Apr 5, 2026Tianle Chen, Deepti GhadiyaramAdversarial Attacks on Unified Multimodal Models Adversarial Attacks
Department of Computer Science, Boston University
Can LLMs Reason About Attention? Towards Zero-Shot Analysis of Multimodal Classroom Behavior
Apr 3, 2026Nolan Platt, Sehrish Nizamani, Alp Tural +5Spatial Reasoning in Language Models Multimodal Large Language Models
Virginia Tech, Blacksburg, Virginia, USA · Incoming PhD student, University of Pennsylvania. · University of Virginia, Charlottesville, Virginia, USA
MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMs
Apr 3, 2026Jiameng Li, Aleksei Tiulpin, Matthew B. BlaschkoEfficient Multimodal Inference Visual Token Pruning
ESAT-PSI, KU Leuven, Belgium · Faculty of Medicine, University of Oulu, Finland · Weill Cornell Medicine, USA
PolyReal: A Benchmark for Real-World Polymer Science Workflows
Apr 3, 2026Wanhao Liu, Weida Wang, Jiaqing Xie +12Multimodal Model Evaluation Multimodal Large Language Models
University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · Fudan University +5
Token Warping Helps MLLMs Look from Nearby Viewpoints
Apr 3, 2026Phillip Y. Lee, Chanho Park, Mingue Park +3Multimodal Large Language Models VLM Reasoning
KAIST
QAPruner: Quantization-Aware Vision Token Pruning for Multimodal Large Language Models
Apr 3, 2026Xinhao Wang, Zhonyu Xia, Zhiwei Lin +2Visual Token Pruning LVLM Pruning
Wangxuan Institute of Computer Technology, Peking University
MixAtlas: Uncertainty-aware Data Mixture Optimization for Multimodal LLM Midtraining
Apr 3, 2026Bingbing Wen, Sirajul Salekin, Feiyang Kang +4Data Mixture Optimization Multimodal Large Language Models
University of Washington · Work done during an internship at Apple · Apple +1
V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views
Apr 3, 2026Junwei You, Pei Li, Zhuoyu Jiang +7Autonomous Driving Benchmarks VLMs for Autonomous Driving
Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI, 53706, USA · Department of Civil and Architectural Engineering and Construction Management, University of Wyoming, Laramie, WY, 82071, USA · Kuaishou Technology, Beijing, 100085, China +1
Do Audio-Visual Large Language Models Really See and Hear?
Apr 3, 2026Ramaneswaran Selvakumar, Kaousheik Jayakumar, S Sakshi +3Multimodal Large Language Models Audio-Visual Understanding
University of Maryland, College Park
Impact of Multimodal and Conversational AI on Learning Outcomes and Experience
Apr 2, 2026Karan Taneja, Anjali Singh, Ashok K. GoelEducational Dialogue Systems AI in Education
Georgia Institute of Technology, Atlanta GA 30332, USA · University of Texas at Austin, Austin TX 78712, USA
Ego-Grounding for Personalized Question-Answering in Egocentric Videos
Apr 2, 2026Junbin Xiao, Shenglang Zhang, Pengxiang Zhu +1Egocentric Video QA Question Answering
University of Science and Technology of China · National University of Singapore
AromaGen: Interactive Generation of Rich Olfactory Experiences with Multimodal Language Models
Apr 2, 2026Awu Chen, Silang Wang, Yunge Wen +6Multimodal Generation Multimodal Large Language Models
MIT Media Lab, Cambridge, MA, USA · University of Washington, Seattle, WA, USA · Harvard Graduate School of Design, Cambridge, MA, USA +3
MM-ReCoder: Advancing Chart-to-Code Generation with Reinforcement Learning and Self-Correction
Apr 2, 2026Zitian Tang, Xu Zhang, Jianbo Yuan +4Chart-to-Code Generation Code Generation
Brown University · Amazon AGI
EgoMind: Activating Spatial Cognition through Linguistic Reasoning in MLLMs
Apr 1, 2026Zhenghao Chen, Huiqun Wang, Di HuangSpatial Reasoning in Language Models Multimodal Spatial Reasoning
State Key Laboratory of Complex and Critical Software Environment, Beihang University · School of Computer Science and Engineering, Beihang University
Multimodal Language Models Cannot Spot Spatial Inconsistencies
Apr 1, 2026Om Khangaonkar, Hadi J. Rad, Hamed PirsiavashMulti-View Consistency 3D Spatial Reasoning
University of California, Davis · Shell · TU Delft
MOON3.0: Reasoning-aware Multimodal Representation Learning for E-commerce Product Understanding
Apr 1, 2026Junxian Wu, Chenghan Fu, Zhanheng Nie +6Multimodal Learning Multimodal Large Language Models
Alibaba Group, Hangzhou, China
CASA: Classification Augmented with Safety Attention for Robust Multimodal Alignment
Mar 31, 2026Anurag Kumar, Raghuveer Peri, Jon Burnsky +4MLLM Safety LLM Safety Alignment
The Ohio State University · AWS
Diagnosing Long-Video Quantitative Reasoning in Multimodal LLMs via Enumeration and Counting
Mar 31, 2026Fumihiko Tsuchiya, Taiki Miyanishi, Shunsuke Yasuki +5Long-Video QA Multimodal Large Language Models
The University of Tokyo, Japan · Institute of Science Tokyo, Japan · National Institute of Informatics, Japan
Covert Visual Prompt Injection against Commercial Multimodal Large Language Models
Mar 31, 2026Meiwen Ding, Song Xia, Chenqi Kong +1Adversarial Attacks on VLMs Prompt Injection Attacks
Rapid-Rich Object Search Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore.
Advancing Multi-Robot Networks via MLLM-Driven Sensing, Communication, and Computation: A Comprehensive Survey
Mar 31, 2026Hyun Jong Yang, Howon Lee, Kyuhong Shim +10Multi-Agent Orchestration Multi-Robot Coordination
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea · Department of Electrical and Computer Engineering, Ajou University, Suwon, South Korea · Department of Intelligent Software, College of Computing and Informatics, Sungkyunkwan University, Suwon, South Korea +4
Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanism
Mar 31, 2026Tao Chen, Kun Zhang, Qiong Wu +5Long-Video Understanding Memory-Augmented Video Understanding
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · National University of Defense Technology, 230000, P.R. China.

Apr 8, 2026J/K move · Enter open · S save

LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models

Shi-Yu Tian, Zhi Zhou, Kun-Yang Yu, Ming Yang+4

National Key Laboratory for Novel Software Technology, Nanjing University, China

Visual Spatial Reasoning Multimodal Large Language Models Multimodal Reasoning Tool-Augmented Language Model Agents

Open paper AskPDF arXiv

Spatial reasoning is a cornerstone capability for intelligent systems to perceive and interact with the physical world. However, multimodal large language models (MLLMs) frequently suffer from hallucinations and imprecision when parsing complex geometric layouts. As data-driven scaling struggles to internalize structured geometric priors and spatial constraints, integrating mature, specialized vision models presents a compelling alternative. Despite its promise, applying this paradigm to spatial reasoning is hindered by two key challenges: The difficulty of invoking heterogeneous, parameter-rich tools, as well as the challenge of understanding and effectively leveraging their diverse low-level outputs (e.g., segmentation masks, depth maps) in high-level reasoning. To address these challenges, we propose LAST, a unified framework for tool-augmented spatial reasoning. LAST features an extensible interactive sandbox, termed LAST-Box, which abstracts heterogeneous tool invocations into atomic instructions and reusable spatial skills, returning multimodal hints (e.g., annotated images and textual descriptions) that can be directly consumed by LLMs. We further design a three-stage progressive training strategy that guides models from understanding tool outputs to proficient and adaptive tool invocation. Experiments on four datasets show that LAST-7B achieves around 20% performance gains over its backbone and outperforms strong proprietary closed-source LLMs, substantially enhancing reasoning on complex spatial tasks.