Multimodal Large Language Models

Also known as MLLM

Ask about this topic

Overview

Multimodal large language models connect language generation and reasoning to visual, audio, video, and sometimes sensor or structured data. Common approaches align pretrained modality encoders with a language model using projectors or adapters, or train models to process modalities jointly; instruction tuning, reinforcement learning, retrieval, and tool use extend their capabilities. Evaluation spans visual question answering, document reading, long-video and spatial reasoning, and specialized settings such as medical imaging, where models still often miss fine-grained evidence or fail to use it consistently. A central tension is how to improve cross-modal grounding, temporal and physical reasoning, and safety while reducing the high inference costs of processing long or detailed inputs.

Latest papers

CardsList

1,096 papers106 papers in the last four weeks, up 23% on the four weeks before. 0.6% of all new papers.

  1. Temporal Gains, Spatial Costs: Revisiting Video Fine-Tuning in Multimodal Large Language Models

    Mar 18, 2026Linghao Zhang, Jungang Li, Yonghua Hei +12Multimodal Large Language ModelsTemporal Video Understanding

    SJTU · CityU · HKUST(GZ) +4

  2. SaFeR-Steer: Evolving Multi-Turn MLLMs via Synthetic Bootstrapping and Feedback Dynamics

    Mar 18, 2026Haolong Hu, Hanyu Li, Tiancheng He +6Language Model SteeringMultimodal Large Language Models

    Huazhong University of Science and Technology · Beijing University of Posts and Telecommunications · West China Biomedical Big Data Center, Sichuan University +2

  3. Omni-I2C: A Holistic Benchmark for High-Fidelity Image-to-Code Generation

    Mar 18, 2026Jiawei Zhou, Chi Zhang, Xiang Feng +6Image-to-Code GenerationCode Generation

    School of Computer Science, Wuhan University, Wuhan, China · Independent Researcher, China · Xidian University, Xi’an, China +1

  4. From Drop-off to Recovery: A Mechanistic Analysis of Segmentation in MLLMs

    Mar 18, 2026Boyong Wu, Sanghwan Kim, Zeynep AkataMultimodal Large Language ModelsSemantic Segmentation

    Technical University of Munich · Helmholtz Munich · Munich Center for Machine Learning (MCML)

  5. Generalist Multimodal LLMs Gain Biometric Expertise via Human Salience

    Mar 17, 2026Jacob Piland, Byron Dowling, Christopher Sweet +1Multimodal Large Language Models

    Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, IN 46556, USA · Center for Research Computing, University of Notre Dame, Notre Dame, IN 46556, USA

  6. OpenQlaw: An Agentic AI Assistant for Analysis of 2D Quantum Materials

    Mar 17, 2026Sankalp Pandey, Thanh-Dat Truong, Xuan-Bac Nguyen +5Tool-Using Vision-Language AgentsAI Agents for Scientific Discovery

    Quantum AI Lab, University of Arkansas, Fayetteville, AR 72701, USA · MonArk NSF Quantum Foundry, USA · Department of Physics and Astronomy, University of Utah, Salt Lake City, UT 84112, USA +1

  7. Caption First, VQA Second: Knowledge Density, Not Task Format, Drives Multimodal Scaling

    Mar 17, 2026Hongjian Zou, Yue Ge, Qi Ding +2Multimodal Large Language ModelsMultimodal Learning

    vivo AI Lab, Shenzhen, China · Wuhan University, Wuhan, China

  8. MLLM-based Textual Explanations for Face Comparison

    Mar 17, 2026Redwan Sony, Anil K Jain, Arun RossFaithfulness of Language Model ExplanationsFace Recognition

    Department of Computer Science and Engineering, Michigan State University, East Lansing, MI, USA

  9. ExpressMind: A Multimodal Pretrained Large Language Model for Expressway Operation

    Mar 17, 2026Zihe Wang, Yihuan Wang, Haiyang Yu. Zhiyong Cui +4Intelligent Transportation SystemsMultimodal Large Language Models

    Beihang University Beijing, China · Shandong Hi-speed Group Co., Ltd Jinan, China · Institute of automation, Chinese Academy of Sciences Beijing, China

  10. Follow the Clues, Frame the Truth: Hybrid-evidential Deductive Reasoning in Open-Vocabulary Multimodal Emotion Recognition

    Mar 17, 2026Yu Liu, Lei Zhang, Haoxun Li +4Multimodal Emotion RecognitionMultimodal Reasoning

    Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China.

  11. GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Models

    Mar 17, 2026Jiaxin Zhang, Junjun Jiang, Haijie Li +3Multimodal Large Language ModelsMultimodal Pretraining

    Harbin Institute of Technology, China · School of Electronic and Computer Engineering, Peking University, China · Huawei, China

  12. Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory

    Mar 16, 2026Ce Zhang, Jinxi He, Junyi He +2Multimodal Large Language ModelsLanguage Model Safety Evaluation

    Robotics Institute, Carnegie Mellon University

  13. MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledge

    Mar 16, 2026Baochen Fu, Yuntao Du, Cheng Chang +6Multimodal Knowledge EditingMultimodal Large Language Models

    Shandong University, Jinan, China · Jinzhong Group, Jinan, China.

  14. Decision-Level Ordinal Modeling for Multimodal Essay Scoring with Large Language Models

    Mar 16, 2026Han Zhang, Jiamin Su, Li liuAutomated Essay ScoringOrdinal Regression

  15. CAMD: Coverage-Aware Multimodal Decoding for Efficient Reasoning of Multimodal Large Language Models

    Mar 16, 2026Huijie Guo, Jingyao Wang, Lingyu Si +3Efficient Multimodal InferenceMultimodal Large Language Models

    Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences · Wangxuan Institute of Computer Technology, Peking University

  16. Fine-tuning MLLMs Without Forgetting Is Easier Than You Think

    Mar 15, 2026He Li, Yuhui Zhang, Xiaohan Wang +2LLM Fine-TuningMultimodal Large Language Models

    Stanford University · Tsinghua University

  17. Zoom to Essence: Trainless GUI Grounding by Inferring upon Interface Elements

    Mar 15, 2026Ziwei Liu, Tao Feng, Borui Kang +2GUI AgentsGUI Grounding

    Department of Computer Science and Technology, Sichuan University, China · Department of Computer Science and Technology, Tsinghua University, China · School of Computer Science, Nanjing University, China +1

  18. ES-Merging: Biological MLLM Merging via Embedding Space Signals

    Mar 15, 2026Wonbin Lee, Dongki Kim, Sung Ju HwangLanguage Model MergingMultimodal Large Language Models

    KAIST

  19. AgroOmni: A Large-Scale Multi-view Agricultural Dataset for Cross-Scale Multimodal Reasoning

    Mar 15, 2026Jiarui Zhang, Junqi Hu, Zurong Mai +10Multimodal ReasoningMultimodal Large Language Models

    Sun Yat-sen University · The Chinese University of Hong Kong · HuanTian Wisdom Technology Co., Ltd. +4

  20. On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs

    Mar 15, 2026Suho Yoo, Youngjoon Jang, Joon Son ChungAttention SinksMultimodal Large Language Models

    KAIST · VGG, University of Oxford

  21. ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation

    Mar 15, 2026Jungwoo Oh, Hyunseung Chung, Junhee Lee +6Electrocardiogram InterpretationClinical Reasoning

    KAIST · Ajou University School of Medicine · Yonsei University College of Medicine +1

  22. Deeper Thought, Weaker Aim: Understanding and Mitigating Perceptual Impairment during Reasoning in Multimodal Large Language Models

    Mar 15, 2026Ruiying Peng, Xueyu Wu, Jing Lei +3Multimodal Large Language ModelsVLM Reasoning

    Tsinghua Shenzhen International Graduate School · Huawei Technologies

  23. When Visual Privacy Protection Meets Multimodal Large Language Models

    Mar 14, 2026Xiaofei Hui, Qian Wu, Haoxuan Qu +3Privacy Protection in VLMsMultimodal Large Language Models

    School of Computing and Communications, Lancaster University, South Dr,, Lancaster LA1 4WA, United Kingdom · Singapore University of Technology and Design, 8 Somapah Rd, Singapore 487372, Singapore · Department of Computer Science, University of Bristol, Queens Road, Bristol BS8 1QU, United Kingdom

  24. Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Space

    Mar 14, 2026Quoc-Huy Trinh, Xi Ding, Yang Liu +73D Spatial ReasoningMultimodal Large Language Models

  25. HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks

    Mar 13, 2026Xiaoyu Li, Yuhang Liu, Xuanshuo Kang +4Multimodal ICLMultimodal Large Language Models

    University of Electronic Science and Technology of China · Institute of Electronics and Information Industry Technology of Kashgar

  26. Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World

    Mar 13, 2026Yuzhi Huang, Kairun Wen, Rongxin Gao +14Spatiotemporal ReasoningMultimodal Large Language Models

    XMU · SUSTech · THU +4

  27. Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity

    Mar 13, 2026Donglin YuEfficient Multimodal InferenceLLM Serving

  28. MIBench: Evaluating LMMs on Multimodal Interaction

    Mar 13, 2026Yu Miao, Zequn Yang, Yake Wei +5Multimodal Large Language ModelsMultimodal Model Evaluation

    Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance, Beijing, China · Beihang University, Beijing, China +1

  29. SPARROW: Learning Spatial Precision and Temporal Referential Consistency in Pixel-Grounded Video MLLMs

    Mar 12, 2026Mohamad Alansari, Naufal Suryanto, Divya Velayudhan +3Multimodal Large Language ModelsVision-Language Grounding

    Khalifa University, Abu Dhabi, UAE

  30. MM-CondChain: A Programmatically Verified Benchmark for Visually Grounded Deep Compositional Reasoning

    Mar 12, 2026Haozhan Shen, Shilin Yan, Hongwei Xue +5Visually Grounded ReasoningVLM Evaluation

    Accio Team, Alibaba Group · Zhejiang University · ZJU-BJ