Multimodal Large Language Models

Also known as MLLM

Multimodal large language models connect language generation and reasoning to visual, audio, video, and sometimes sensor or structured data. Common approaches align pretrained modality encoders with a language model using projectors or adapters, or train models to process modalities jointly; instruction tuning, reinforcement learning, retrieval, and tool use extend their capabilities. Evaluation spans visual question answering, document reading, long-video and spatial reasoning, and specialized settings such as medical imaging, where models still often miss fine-grained evidence or fail to use it consistently. A central tension is how to improve cross-modal grounding, temporal and physical reasoning, and safety while reducing the high inference costs of processing long or detailed inputs.

Latest papers

1,096 papers106 papers in the last four weeks, up 23% on the four weeks before. 0.6% of all new papers.

Raincheck: Deferred Multimodal Prefill for Efficient Multi-Turn MLLM Inference
Oct 29, 2026Wenjie Huang, Jun Zhang, Hongyaoxing Gu +10Efficient Multimodal Inference Multimodal Large Language Models
Zhejiang University · Alibaba Group · University of the Chinese Academy of Sciences +4
LatentWeave: Test-Time Latent Optimization for Multimodal Reasoning
Oct 29, 2026Fanci Meng, Kun Wang, Guibin Zhang +4MLLM Reasoning Inference-Time Adaptation of Language Models
University of Science and Technology of China · Nanyang Technological University · National University of Singapore +2
Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI
Oct 28, 2026NVIDIA, :, Aastha Jhunjhunwala +556LLM Agent Training Mixture-of-Experts Language Models
Where Can We Fool the Model's Mind? Response-Guided Semantic Probing and Adaptive Routing for Transferable Attacks on Black-Box MLLMs
Oct 27, 2026Ahmed Abdelnaby, Mohamed ElmahallawyAdversarial Attacks on VLMs Transfer-Based Adversarial Attacks
School of Engineering and Applied Sciences, Washington State University, WA 99354, USA
When Visual Evidence Is Insufficient: How Instructions, Post-Training, and Internal Activations Shape Refusal in Multimodal Language Models
Oct 27, 2026Ananth K Suresh, Aveen Dayal, Gouthaman KV +2Refusal Mechanisms in Language Models Abstention in VLMs
Indian Institute of Technology Hyderabad
Zhengrong-Spatial-Aware 1.0: Advancing 3D Multimodal Intelligence
Oct 27, 2026Yu Zhang, Xiaogang Wang, Yuhang Cheng +23D VQA Multimodal Spatial Reasoning
Beijing Freedo Technology Co., Ltd. · Southwest University
VGIQA: Fine-Grained Image Quality Assessment with Visual Grounding Rewards
Oct 27, 2026Weichang Wang, Liyu Chen, Shuyue Guan +4Image Quality Assessment Multimodal Large Language Models
State Key Laboratory of Integrated Services Networks, Xidian University · Huawei Foundation Model Department
Zooming without Zooming v2: From Seeing Details to Connecting Evidence for Generalizable Multimodal Perception
Oct 27, 2026Lai Wei, Yatai Li, Liangbo He +10Multimodal Understanding Multimodal Large Language Models
School of Computer Science, Shanghai Jiao Tong University · Zhongguancun Academy · Ant Group +2
Live-OPSD: On-Policy Self-Distillation for Proactive Streaming Large Video Models
Oct 27, 2026Kaixuan Du, Xin Wan, Dai Guan +3On-Policy Distillation Multimodal Agents
Qwen Business Unit of Alibaba
Decoupling Consistency and Conflict across MLLM Layers for Fake News Detection
Oct 27, 2026Kai Qian, Dongjie Xu, Weijie Shi +4Fake News Detection Multimodal Large Language Models
School of Computer Science and Technology, Soochow University, China · Soochow University, China · Hong Kong University of Science and Technology, China +3
OneEEG: A Unified Signal-Visual Multimodal Language Model with Task-Aware Mixture of Projectors for Multi-Task EEG Understanding
Oct 26, 2026Jiacheng Zhang, Xueyi Zhang, Xin Lu +2Electroencephalography Multimodal Large Language Models
School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · Language, Intelligence, and Machine Center, Shenzhen Loop Area Institute, Shenzhen, China · School of Computing, National University of Singapore, Singapore +2
HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget
Oct 25, 2026Harish Santhanalakshmi Ganesan, Hariharan M. S., Prabhurajhan RMixture-of-Experts Language Models Efficient Language Model Training
CASE: Confusion-Aware Skill Evolution for Frozen Multimodal Emotion Models
Oct 23, 2026Siyu Wu, Zeyu Liang, Linghao Wang +4Multimodal Emotion Recognition Multimodal Large Language Models
Jilin University · National Chiao Tung University · National Tsing Hua University
A Broader Look into MLLM's Retrieval Heads
Oct 23, 2026Jia Wang, Yuqing Li, Jiangnan Li +2Multimodal Large Language Models Multimodal IR
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Tencent
Efficient Omni-modal Large Language Model Inference via Per-Query Modality Selection
Oct 22, 2026Shuqi Liu, Yuzhou Cao, Luke OngEfficient Multimodal Inference Multimodal Large Language Models
College of Computing and Data Science, Nanyang Technological University
Drift Says When, Budget Says How Much: Rate-Controlled KV Re-selection for Long-Form Multimodal Generation
Oct 22, 2026Bin Hu, Yingbo XiaoMultimodal Large Language Models KV-Cache Compression
Tsinghua University · The Chinese University of Hong Kong, Shenzhen
Predicting Music Valence and Arousal via Textual and Multimodal In-Context Learning in Large Language Models
Oct 22, 2026Chi-En Dai, Chih-Pin Tan, Yi-Hsuan YangValence-Arousal Modeling Emotion Recognition
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan
Asking a Multimodal LLM Whether Someone Fell: Zero-Shot Screening of Low-Resolution Thermal Video in a Laboratory and an Occupied Home
Oct 21, 2026Marcos Junca, Mauro JuncaFall Detection Healthcare
Senecta Inc.
Melange: A Synthetic Visual Bias Benchmark for Multi-modal Large Language Models
Oct 21, 2026Razieh Rezaei, Ehsaneddin Asgary, Masoud Jalili SabetMLLM Evaluation Multimodal Large Language Models
Technical University of Munich · Helmholtz Munich · Munich Center for Machine Learning +2
Omnilingual-MsASR: Benchmarking Multimodal, Multilingual, Long-Form Multi-Speaker ASR of Omni Large Language Models
Oct 21, 2026Yuan Ge, Pei Zhang, Xize Cheng +8Multi-Speaker ASR Speech Recognition Benchmarks
Northeastern University, China · Alibaba Token Hub, Alibaba · Kunming University of Science and Technology +1
DIVE: Benchmarking Full-Duplex Interaction with Visual Engagement in Continuous Streams
Oct 20, 2026Yuhang Chen, Guangyue Liu, Shilong LiuMultimodal Model Evaluation Multimodal Large Language Models
Huazhong University of Science and Technology · Princeton University
PI-MLLM: Psychophysics-Inspired Multimodal Large Language Model for Oddly Satisfying Video Emotion Analysis
Oct 20, 2026Yuheng Zhang, Jie Zhang, Yuecong Min +4Multimodal Emotion Recognition Multimodal Large Language Models
Institute of Computing Technology, Chinese Academy of Sciences · CNCERT/CC · Nanyang Technological University +1
Testing Associative Creativity in Humans and Multimodal LLMs with Difficult-to-Name Visual Stimuli
Oct 20, 2026Manar Ali, Lisa Gottschalk, Simeon Junker +6Creativity Assessment Multimodal Large Language Models
CRC 1646 ‘Linguistic Creativity in Communication’ Bielefeld University, Bielefeld, Germany
CARLAR Influence: A Structured Hallucination Risk in Multimodal Cheque Extraction and an Exploratory Compressed-Image Countermeasure
Oct 19, 2026Joong Ho, Choi, Aditya Paul +1Document Information Extraction Multimodal Large Language Models
Joshua · BNY, Pittsburgh, PA, USA
Beyond Omni: A Unified MLLM for Embodied Sensory Understanding
Oct 19, 2026Shiyun Xiong, Dongming Wu, Peiwen Sun +5Unified Multimodal Models Multimodal Large Language Models
MMLab, The Chinese University of Hong Kong · Shenzhen Loop Area Institute · The Chinese University of Hong Kong, Shenzhen +2
OmniGround: Strengthening Omni-Modal Temporal Grounding via Attention-Consistency Training
Oct 19, 2026Zixuan Chen, Depeng Wang, Hao Lin +5Temporal Video Grounding Multimodal Grounding
Shanghai Jiao Tong University · Ant Group · Independent Researcher
Think, Compress, Answer: Reinforcement Learning for Multimodal Grounding via Reasoning Compression
Oct 19, 2026Jinbae Seo, Hyeongjun Kwon, Jiyoung Lee +4Multimodal Grounding Multimodal Large Language Models
Yonsei University · School of AI and Software, Ewha Womans University · LIG Defense & Aerospace +1
From Label Priors to Task Evidence: Efficient Long-Video Frame Selection via Bayesian GRPO
Oct 18, 2026Shuochen Chang, Bingjie Gao, Qingyang Liu +7Long-Video Understanding Query-Aware Video Frame Selection
Shanghai Jiao Tong University · Tongji University
Local Evidence, Global Failure: Diagnosing Speech-Vision Temporal Composition in Audio-Visual Language Models
Oct 18, 2026Jaehong Myung, Seunghun Oh, Unsang ParkAudio-Visual Understanding Multimodal Large Language Models
Department of Artificial Intelligence, Sogang University, Seoul, Republic of Korea · Department of Computer Science and Engineering, Sogang University, Seoul, Republic of Korea
The Bitter Lesson Comes to Robotics
Oct 18, 2026Christopher Leet, Achyutha Menon, Sravanthi Machcha +3Language Model-Based Control Robotic Manipulation
Robocurve · University of Southern California · University of California, Irvine

Oct 29, 2026J/K move · Enter open · S save

Raincheck: Deferred Multimodal Prefill for Efficient Multi-Turn MLLM Inference

Wenjie Huang, Jun Zhang, Hongyaoxing Gu, Haoyang Huang+9

Zhejiang University · Alibaba Group · University of the Chinese Academy of Sciences+4

Efficient Multimodal Inference Multimodal Large Language Models LLM Inference Scheduling

Open paper AskPDF arXiv

Multi-turn conversations with multimodal large language models (MLLMs) reuse input context, but long multimodal inputs increase time to first token (TTFT). Query-aware hard pruning reduces prefill tokens but permanently excludes information that later questions may require. We propose Raincheck, a framework for deferred multimodal prefill that decouples initial token selection from later information availability. An existing pruning method selects tokens for initial prefill; Raincheck preserves and progressively prefills unselected tokens in source order using unused compute capacity during decode. Unfinished prefill continues after answer return during available scheduling gaps. Attention visibility control and affine state mapping enable context updates after generation begins in full-attention and hybrid recurrent models. Prefill-decode fusion reduces repeated weight reads. We evaluate Qwen3-VL-8B and Qwen3.5-9B with 30% initial multimodal-token retention. Across both models and four pruning methods, Raincheck recovers on average 77.7% and 60.4% of the quality lost to matched hard pruning on MultiVerse and ConvBench, respectively. On MultiVerse, the SGLang implementation reduces mean first-turn TTFT by up to 24.1% relative to full prefill, with less than 0.5% increase in mean time per output token (TPOT).