Multimodal Large Language Models

Also known as MLLM

Overview

Multimodal large language models connect language generation and reasoning to visual, audio, video, and sometimes sensor or structured data. Common approaches align pretrained modality encoders with a language model using projectors or adapters, or train models to process modalities jointly; instruction tuning, reinforcement learning, retrieval, and tool use extend their capabilities. Evaluation spans visual question answering, document reading, long-video and spatial reasoning, and specialized settings such as medical imaging, where models still often miss fine-grained evidence or fail to use it consistently. A central tension is how to improve cross-modal grounding, temporal and physical reasoning, and safety while reducing the high inference costs of processing long or detailed inputs.

Latest papers

CardsList

1,147 papers50 papers in the last four weeks, down 57% on the four weeks before. 0.4% of all new papers.

  1. SpurLens: Automatic Detection of Spurious Cues in Multimodal LLMs

    Mar 11, 2025Parsa Hosseini, Sumit Nawathe, Mazda Moayeri +2VLM EvaluationVLM Robustness

  2. Histopathology Multi-modal Embedding for Pathology Composed Retrieval

    Feb 11, 2025Qifeng Zhou, Wenliang Zhong, Thao M. Dang +4Computational PathologyHealthcare

  3. Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models

    Dec 21, 2024Yanxu Mao, Peipei Liu, Tiehan Cui +3Adversarial Attacks on LLMsMultimodal Large Language Models

  4. Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotators

    Nov 6, 2024Claire Wonjeong Jo, Miki Wesołowska, Magdalena WojcieszakVideo Content ModerationLLM-Assisted Annotation

  5. A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks

    Aug 2, 2024Jiaqi Wang, Hanqi Jiang, Yiheng Liu +23Multimodal Large Language ModelsMultimodal Understanding

  6. RS-Agent: Automating Remote Sensing Tasks through Intelligent Agent

    Jun 11, 2024Wenjia Xu, Zijian Yu, Boyang Mu +3Multimodal RAGMultimodal Large Language Models

  7. Multimodal Large Language Model driven Radiology Report Generation with Clinical Knowledge Enhancement

    Mar 11, 2024Miaojing Shi, Tianyu Cen, Zijie Yue +3Radiology Report GenerationRadiology VLMs