Multimodal Large Language Models

Also known as MLLM

Ask about this topic

Overview

Multimodal large language models connect language generation and reasoning to visual, audio, video, and sometimes sensor or structured data. Common approaches align pretrained modality encoders with a language model using projectors or adapters, or train models to process modalities jointly; instruction tuning, reinforcement learning, retrieval, and tool use extend their capabilities. Evaluation spans visual question answering, document reading, long-video and spatial reasoning, and specialized settings such as medical imaging, where models still often miss fine-grained evidence or fail to use it consistently. A central tension is how to improve cross-modal grounding, temporal and physical reasoning, and safety while reducing the high inference costs of processing long or detailed inputs.

Latest papers

CardsList

1,096 papers106 papers in the last four weeks, up 23% on the four weeks before. 0.6% of all new papers.

  1. Xuanwu: Evolving General Multimodal Models into an Industrial-Grade Foundation for Content Ecosystems

    Mar 31, 2026Zhiqian Zhang, Xu Zhao, Xiaoqing Xu +5Multimodal Content ModerationMultimodal Large Language Models

    Computational Intelligence Dept, Hello Group Inc.

  2. AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understanding

    Mar 30, 2026Haozhe Qi, Kevin Qu, Mahdi Rad +3Long-Video UnderstandingMultimodal Large Language Models

    Microsoft Spatial AI Lab · EPFL · ETH Zurich

  3. Seeing with You: Perception-Reasoning Coevolution for Multimodal Reasoning

    Mar 30, 2026Ziqi Miao, Haonan Jia, Lijun Li +4Multimodal ReasoningMultimodal Large Language Models

  4. ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning

    Mar 30, 2026Huanxuan Liao, Zhongtao Jiang, Yupu Hao +6Efficient Multimodal InferenceMultimodal Large Language Models

    Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences

  5. AutoCut: End-to-end advertisement video editing based on multimodal discretization and controllable generation

    Mar 30, 2026Milton Zhou, Sizhong Qin, Yongzhi Li +2Video EditingMultimodal Large Language Models

    Tsinghua University · Kuaishou Technology

  6. Beyond Static Vision: Scene Dynamic Field Unlocks Intuitive Physics Understanding in Multi-modal Large Language Models

    Mar 30, 2026Nanxi Li, Xiang Wang, Yuanjie Chen +3Physical ReasoningMultimodal Large Language Models

    Shanghai Jiao Tong University · Tianjin University · Shanghai Innovation Institute

  7. CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs

    Mar 30, 2026Yongkang Du, Xiaohan Zou, Minhao Cheng +1Analogical ReasoningMultimodal Reasoning

    Pennsylvania State University

  8. Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Survey

    Mar 30, 2026Bhavuk Jain, Sercan Ö. Arık, Hardeo K. ThakurAdversarial Attacks on Unified Multimodal ModelsLLM Security

    Google · Bennett University, India

  9. Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models

    Mar 29, 2026Baoheng Zhang, Jiahui Liu, Gui Zhao +7Vision-Language ModelsMultimodal Large Language Models

    The University of Hong Kong

  10. RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMs

    Mar 27, 2026Logan Lawrence, Mustafa Chasmai, Rangel Daroya +8Fine-Grained Image ClassificationMultimodal Model Evaluation

    Computer Vision Lab, UMass Amherst

  11. MA-Bench: Towards Fine-grained Micro-Action Understanding

    Mar 27, 2026Kun Li, Jihao Gu, Fei Wang +3MLLM EvaluationMultimodal Large Language Models

    CVLab, College of Information Technology, United Arab Emirates University · University College London · Hefei University of Technology +2

  12. Rocks, Pebbles and Sand: Modality-aware Scheduling for Multimodal Large Language Model Inference

    Mar 27, 2026Konstantinos Papaioannou, Thaleia Dimitra DoudaliLLM ServingLLM Inference Scheduling

    IMDEA Software Institute, Universidad Politécnica de Madrid

  13. Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification

    Mar 27, 2026Shuai Lv, Chang Liu, Feng Tang +5Multimodal Large Language ModelsMultimodal Reasoning

    University of Science and Technology of China, Hefei, China · Huawei Foundation Model Department, Shanghai, China · The Chinese University of Hong Kong, Hong Kong, China +1

  14. FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants

    Mar 27, 2026Mahesh Bhosale, Abdul Wasi, Shantam Srivastava +5Multimodal Large Language ModelsMedical VLMs

    University at Buffalo · University of Kashmir · Accenture +1

  15. GazeQwen: Lightweight Gaze-Conditioned LLM Modulation for Streaming Video Understanding

    Mar 26, 2026Trong Thang Pham, Hien Nguyen, Ngan LeMultimodal Large Language ModelsStreaming Video Understanding

    University of Arkansas · University of Houston

  16. Demographic Fairness in Multimodal LLMs: A Benchmark of Gender and Ethnicity Bias in Face Verification

    Mar 26, 2026Ünsal Öztürk, Hatef Otroshi Shahreza, Sébastien MarcelAlgorithmic FairnessMultimodal Large Language Models

    Idiap Research Institute, Switzerland

  17. AnyDoc: Enhancing Document Generation via Large-Scale HTML/CSS Data Synthesis and Height-Aware Reinforcement Optimization

    Mar 26, 2026Jiawei Lin, Wanrong Zhu, Vlad I Morariu +1Text GenerationSynthetic Data Generation for Language Models

    Xi’an Jiaotong University · Adobe Research

  18. Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMs

    Mar 26, 2026Jinda Lu, Junkang Wu, Jinghan Li +6RL for VLMsVLM Reasoning

    University of Science and Technology of China · Peking University · Independent Researcher

  19. ReLope: From Hidden-State Probing to a Decision Module for Multimodal LLM Routing

    Mar 25, 2026Yaopei Zeng, Congchao Wang, Blake JianHang Chen +1Multimodal Large Language ModelsLLM Routing

    Pennsylvania State University · Google · Amazon AGI

  20. ReDiPrune: Relevance-Diversity Pre-Projection Token Pruning for Efficient Multimodal LLMs

    Mar 25, 2026An Yu, Ting Yu Tsai, Zhenfei Zhang +3Efficient VLM InferenceToken Pruning

    University at Albany, SUNY · Peking University

  21. Lightweight Multimodal LLM-Enabled Cost-Effective Defect Grading of Power Transmission Equipment

    Mar 25, 2026Tao Wang, Lipeng Zhu, Jiayong Li +2Multimodal Large Language ModelsLLM Fine-Tuning

  22. Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models

    Mar 25, 2026Siqi Liu, Xinyang Li, Bochao Zou +3Theory of MindMultimodal Large Language Models

    University of Science and Technology Beijing

  23. When Understanding Becomes a Risk: Authenticity and Safety Risks in the Emerging Image Generation Paradigm

    Mar 25, 2026Ye Leng, Junjie Chu, Mingjie Li +5MLLM SafetyImage Generation

    CISPA Helmholtz Center for Information Security · Xi’an Jiaotong University · Flexera

  24. A Step Toward Federated Pretraining of Multimodal Large Language Models

    Mar 25, 2026Baochen Xiong, Yifan Xu, Xiaoshan Yang +3Multimodal Federated LearningMultimodal Pretraining

    MAIS, Institute of Automation, Chinese Academy of Sciences · Pengcheng Laboratory · School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS) +1

  25. A3^3: Towards Advertising Aesthetic Assessment

    Mar 25, 2026Kaiyuan Ji, Yixuan Gao, Lu Sun +7Image Quality AssessmentMultimodal Large Language Models

    Shanghai Artificial Intelligence Laboratory · School of Information and Electronic Engineering, East China Normal University · Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University +1

  26. Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning

    Mar 25, 2026Kun-Yang Yu, Zhi Zhou, Shi-Yu Tian +6Multimodal UnderstandingMultimodal Large Language Models

    State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · School of Intelligence Science and Technology, Nanjing University, China

  27. Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training

    Mar 25, 2026Gengluo Li, Pengyuan Lyu, Chengquan Zhang +7Document ParsingMultimodal Large Language Models

    Institute of Information Engineering, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Tencent +2

  28. 3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding

    Mar 24, 2026Yiping Chen, Jinpeng Li, Wenyu Ke +63D VLMs3D Scene Understanding

    School of Geospatial Engineering and Science, Sun Yat-sen University, Zhuhai, 519082, China. · College of Electronic Science, National University of Defense Technology, Changsha, 410000, China. · Department of Civil and Environmental Engineering, Norwegian University of Science and Technology, Trondheim, Norway. +1

  29. Unleashing Spatial Reasoning in Multimodal Large Language Models via Textual Representation Guided Reasoning

    Mar 24, 2026Jiacheng Hua, Yishu Yin, Yuhang Wu +33D Spatial ReasoningMultimodal Large Language Models

    College of AI, Tsinghua University, Beijing, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · The University of Tokyo, Tokyo, Japan

  30. SIMART: Decomposing Monolithic Meshes into Sim-ready Articulated Assets via MLLM

    Mar 24, 2026Chuanrui Zhang, Minghan Qin, Yuang Wang +3Articulated 3D Asset Generation3D Asset Generation

    ByteDance Seed · NTU