Multimodal Large Language Models

Also known as MLLM

Multimodal large language models connect language generation and reasoning to visual, audio, video, and sometimes sensor or structured data. Common approaches align pretrained modality encoders with a language model using projectors or adapters, or train models to process modalities jointly; instruction tuning, reinforcement learning, retrieval, and tool use extend their capabilities. Evaluation spans visual question answering, document reading, long-video and spatial reasoning, and specialized settings such as medical imaging, where models still often miss fine-grained evidence or fail to use it consistently. A central tension is how to improve cross-modal grounding, temporal and physical reasoning, and safety while reducing the high inference costs of processing long or detailed inputs.

Latest papers

1,096 papers106 papers in the last four weeks, up 23% on the four weeks before. 0.6% of all new papers.

UnAC: Adaptive Visual Prompting with Abstraction and Stepwise Checking for Complex Multimodal Reasoning
May 5, 2026Yifan Wang, Yun FuMultimodal Large Language Models Multimodal Reasoning
Northeastern University
CC-OCR V2: Benchmarking Large Multimodal Models for Literacy in Real-world Document Processing
May 5, 2026Zhipeng Xu, Junhao Ji, Zulong Chen +10Document Understanding Multimodal Large Language Models
Alibaba Group · Northeastern University
Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
May 5, 2026Quanxing Xu, Ling Zhou, Xian Zhong +3Multimodal RAG Visual Question Answering
School of Computer Science and Engineering, Macau University of Science and Technology, Macao SAR 999078, China · Hubei Key Laboratory of Transportation Internet of Things, School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan, Hubei 430070, China · State Key Laboratory of Maritime Technology and Safety, Wuhan University of Technology, Wuhan 430063, China +3
Can Multimodal Large Language Models Understand Pathologic Movements? A Pilot Study on Seizure Semiology
May 5, 2026Lina Zhang, Tonmoy Monsoor, Mehmet Efe Lorasdagi +8Video Understanding Multimodal Large Language Models
University of California, Los Angeles · Mercedes-Benz AG · Zhejiang University
Bolek: A Multimodal Language Model for Molecular Reasoning
May 4, 2026Frederic Grabowski, Jacek Szczerbiński, Maciej Jaśkowski +4LLM Grounding Scientific Reasoning
1Ingenix.ai, Warsaw, Poland · Laboratory of Bioinformatics and Computational Genomics, Warsaw University of Technology, Warsaw, Poland
Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs
May 4, 2026Xin Zhang, Qiqi Tao, Jiawei Du +2Inference-Time Optimization Test-Time Optimization
Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · Singapore University of Technology and Design +1
From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMs
May 4, 2026Le Zhang, Jihan Yang, Soundarya Krishnan +11Spatial Reasoning Benchmarks Visual Spatial Reasoning
1Mila - Qu´ebec AI Institute, UdeM · 2NYU · 3Apple +1
Referring Multiple Regions with Large Multimodal Models via Contextual Latent Steering
May 3, 2026Yun Xing, Hanyuan Liu, Jiahao Nie +1Referring Expression Comprehension Multimodal Large Language Models
Nanyang Technological University
Mitigating Multimodal LLMs Hallucinations via Relevance Propagation at Inference Time
May 3, 2026Itai Allouche, Joseph KeshetMultimodal Grounding Inference-Time Optimization
Department of Electrical and Computer Engineering, Technion, Haifa, Israel
Injecting Distributional Awareness into MLLMs via Reinforcement Learning for Deep Imbalanced Regression
May 2, 2026Yao Du, Shanshan Song, Xiaomeng LiMultimodal Large Language Models RL Fine-Tuning
The Hong Kong University of Science and Technology, Hong Kong SAR, China.
DiagramNet: An End-to-End Recognition Framework and Dataset for Non-Standard System-Level Diagrams
May 2, 2026Jincheng Lou, Ruohan Xu, Jiapeng Li +6Multi-Agent LLM Systems Electronic Design Automation
School of IC, Peking University, Beijing, China · College of Artificial Intelligence, Xi’an Jiaotong University, Xi’an, China · Department of Precision Instruments, Tsinghua University, Beijing, China +4
OralMLLM-Bench: Evaluating Cognitive Capabilities of Multimodal Large Language Models in Dental Practice
May 2, 2026Rongyang Wang, Shuang Zhou, Jiashuo Wang +2Medical Imaging Cognitive Modeling
Department of Orthodontics, Beijing Stomatological Hospital, School of Stomatology, Capital Medical University, Fanjiacun Road #9, Fengtai District, Beijing 100070, China · Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, MN, USA · Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China +1
Valley3: Scaling Omni Foundation Models for E-commerce
May 2, 2026Zeyu Chen, Guanghao Zhou, Min Yang +6Unified Multimodal Models Multimodal Pretraining
Valley Team, ByteDance Group
EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness
May 1, 2026Yueru Sun, Yimeng Zhang, Haoyu Gu +5Missing-Modality Learning Multimodal Large Language Models
School of Future Technology, South China University of Technology, Guangzhou, China
Let ViT Speak: Generative Language-Image Pre-training
May 1, 2026Yan Fang, Mengcheng Lan, Zilong Huang +7Cross-Modal Learning Vision Transformer
1Beijing Jiaotong University · 2ByteDance · 3Nanyang Technological University
Generating Statistical Charts with Validation-Driven LLM Workflows
May 1, 2026Pavlin G. Poličar, Andraž Pevcin, Blaž ZupanData Visualization Multimodal Large Language Models
University of Ljubljana · Faculty of Computer and Information Science · Veˇcna pot 113, 1000 Ljubljana, Slovenia
DIYHealth Suite: Dataset, Model, and Benchmark for Health Management at Home
May 1, 2026Changshuo Liu, Junran Wu, Zhongle Xie +9Healthcare Multimodal Large Language Models
School of Computing, National University of Singapore, Singapore
Multimodal Large Language Models as Synthetic Participants in Video-Based Studies: An Evaluation
May 1, 2026Prabal Shrestha, Bohan Jiang, Haoning Xue +2Multimodal Large Language Models Language Model Bias Evaluation
Department of Computer Science, Boise State University, Boise, ID, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA · Department of Communication, University of Utah, Salt Lake City, UT, USA
GaMMA: Towards Joint Global-Temporal Music Understanding in Large Multimodal Models
May 1, 2026Zuyao You, Zhesong Yu, Mingyu Liu +3Music Understanding Multimodal Pretraining
1Fudan University · 2ByteDance
Are Multimodal LLMs Ready for Clinical Dermatology? A Real-World Evaluation in Dermatology
May 1, 2026Roy Jiang, Hyunjae Kim, Zhenyue Qin +8Differential Diagnosis Clinical Triage
Department of Dermatology, Yale School of Medicine, Yale University, New Haven, CT, USA · Department of Bioinformatics and Data Science, Yale School of Medicine, Yale University, New Haven, CT, USA · Yale School of Medicine, Yale University, New Haven, CT, USA
Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL
Apr 30, 2026Sudong Wang, Weiquan Huang, Xiaomin Yu +9Multimodal Large Language Models Multimodal Reasoning
Hong Kong University of Science and Technology (Guangzhou) · Nanyang Technological University · Tsinghua University +3
From Mirage to Grounding: Towards Reliable Multimodal Circuit-to-Verilog Code Generation
Apr 30, 2026Guang Yang, Xing Hu, Xiang Chen +1Multimodal Grounding Code Generation
State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security, Hangzhou, China · School of Artificial Intelligence and Computer Science, Nantong University, Nantong, China
MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
Apr 30, 2026Junbo Cui, Bokai Xu, Chongyi Wang +33Efficient Multimodal Inference Unified Multimodal Models
MiniCPM-o Team, OpenBMB
GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents
Apr 29, 2026GLM-V Team, :, Wenyi Hong +95Multimodal Large Language Models Multimodal Reasoning
State Beyond Appearance: Diagnosing and Improving State Consistency in Dial-Based Measurement Reading
Apr 29, 2026Yuanze Hu, Gen Li, Yuqin Lan +5Multimodal Robustness Cross-Modal Alignment
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University, Beijing, China · Beihang University, Beijing, China · Fudan University, Shanghai, China +1
Assessing Y-Axis Influence: Bias in Multimodal Language Models on Chart-to-Table Translation
Apr 27, 2026Seok Hwan Song, Azher Ahmed Efat, Wallapak TavanapongData Visualization Multimodal Large Language Models
Department of Computer Science, Iowa State University, Ames, Iowa, USA
Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence
Apr 27, 2026NVIDIA, :, Amala Sanjay Deshmukh +216Efficient Multimodal Inference Multimodal Large Language Models
X2SAM: Any Segmentation in Images and Videos
Apr 27, 2026Hao Wang, Limeng Qiao, Chi Zhang +4Image Segmentation Video Object Segmentation
Sun Yat-Sen University · 2 Peng Cheng Laboratory · 3 Meituan Inc
Aligned Multi-View Scripts for Universal Chart-to-Code Generation
Apr 27, 2026Zhihan Zhang, Lizi LiaoCode Generation Parameter-Efficient Fine-Tuning
School of Computing and Information Systems, Singapore Management University
ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction
Apr 26, 2026Zichun Guo, Yuling Shi, Wenhao Zeng +6Image Reconstruction Multimodal Large Language Models
Xidian University · Shanghai Jiao Tong University · Alibaba Qwen +1

May 5, 2026J/K move · Enter open · S save

UnAC: Adaptive Visual Prompting with Abstraction and Stepwise Checking for Complex Multimodal Reasoning

Yifan Wang, Yun Fu

Northeastern University

Multimodal Large Language Models Multimodal Reasoning VLM Reasoning Visually Grounded Reasoning Multimodal Foundation Models Visual Prompt Learning+1

Open paper AskPDF arXiv

Although recent LMMs have become much stronger at visual perception, they remain unreliable on problems that require multi-step reasoning over visual evidence. In this paper, we present UnAC (Understanding, Abstracting, and Checking), a multimodal prompting method that strengthens reasoning for complex multimodal tasks in LMMs (e.g., GPT-4o, Gemini 1.5, and GPT-4V). To improve image understanding and capture fine details, we propose an adaptive visual prompting strategy that enables LMMs to focus on salient regions. We further design an image-abstraction prompt to effectively extract key information from images. In addition, we introduce a gradual self-checking scheme that improves reasoning by verifying each decomposed subquestion and its answer. Extensive experiments on three public benchmarks-MathVista, MM-Vet, and MMMU.