Multimodal Large Language Models

Also known as MLLM

Multimodal large language models connect language generation and reasoning to visual, audio, video, and sometimes sensor or structured data. Common approaches align pretrained modality encoders with a language model using projectors or adapters, or train models to process modalities jointly; instruction tuning, reinforcement learning, retrieval, and tool use extend their capabilities. Evaluation spans visual question answering, document reading, long-video and spatial reasoning, and specialized settings such as medical imaging, where models still often miss fine-grained evidence or fail to use it consistently. A central tension is how to improve cross-modal grounding, temporal and physical reasoning, and safety while reducing the high inference costs of processing long or detailed inputs.

Latest papers

1,096 papers106 papers in the last four weeks, up 23% on the four weeks before. 0.6% of all new papers.

What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models
Feb 28, 2026Yingqi Fan, Junlong Tong, Anhao Zhao +1Multimodal Large Language Models Multimodal Model Interpretability
Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative · Shanghai Jiao Tong University +1
Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Models
Feb 28, 2026Hulingxiao He, Zhi Tan, Yuxin PengHierarchical Classification Multimodal Large Language Models
Wangxuan Institute of Computer Technology, Peking University
How Well Do Multimodal Models Reason on ECG Signals?
Feb 27, 2026Maxwell A. Xu, Harish Haresamudram, Catherine W. Liu +11Electrocardiogram Interpretation Multimodal Model Evaluation
University of Illinois Urbana Champaign · Google Inc · Rush University +4
Induced Numerical Instability: Hidden Costs in Multimodal Large Language Models
Feb 27, 2026Wai Tuck Wong, Jun Sun, Arunesh SinhaMultimodal Large Language Models Multimodal Robustness
School of Computing and Information Systems, Singapore Management University, Singapore · Management Science and Information Systems Department, Rutgers Business School, New Jersey, USA
Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume
Feb 27, 2026Gregory Kang Ruey Lau, Hieu Dao, Nicole Kan Hui Lin +1LLM Uncertainty Estimation Multimodal Large Language Models
Department of Computer Science, National University of Singapore
Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation
Feb 27, 2026Xingyu Zhu, Kesen Zhao, Liang Yi +4VLM Hallucination Mitigation Multimodal Large Language Models
MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University
Steering and Rectifying Latent Representation Manifolds in Frozen Multi-modal LLMs for Video Anomaly Detection
Feb 27, 2026Zhaolin Cai, Fan Li, Huiyu Duan +2Video Anomaly Detection Multimodal Anomaly Detection
Xinjiang University · Xi’an Jiao Tong University · Shanghai Jiao Tong University +1
EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models
Feb 27, 2026Yiyang Fang, Wenke Huang, Pei Fu +5Multimodal Large Language Models Multimodal Reasoning
School of Computer Science, Wuhan University. · MiLM Plus, Xiaomi Inc.
Reasoning-Driven Multimodal LLM for Domain Generalization
Feb 27, 2026Zhipeng Xu, Zilong Wang, Xinyang Jiang +3Domain Generalization Multimodal Domain Generalization
Xidian University · Microsoft Research Asia · National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application
Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognition
Feb 27, 2026Jae Young Choi, Seon Gyeom Kim, Hyungjun Yoon +7Human Activity Recognition Multimodal Large Language Models
KAIST · Adobe Research
ThinkOmni: Lifting Textual Reasoning to Omni-modal Scenarios via Guidance Decoding
Feb 26, 2026Yiran Guan, Sifan Tu, Dingkang Liang +6Multimodal Reasoning MLLM Reasoning
Huazhong University of Science and Technology · MiLM Plus, Xiaomi Inc.
Large Multimodal Models as General In-Context Classifiers
Feb 26, 2026Marco Garosi, Matteo Farina, Alessandro Conti +2Multimodal Classification Classification
University of Trento · Fondazione Bruno Kessler
Modality Collapse as Mismatched Decoding: Information-Theoretic Limits of Multimodal LLMs
Feb 26, 2026Jayadev BillaMultimodal Large Language Models Mutual Information
MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understanding
Feb 26, 2026Wenhui Tan, Xiaoyi Yu, Jiaze Li +5Long-Video QA Multimodal Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China · Tongji University · MiLM Plus, Xiaomi Inc.
From Blind Spots to Gains: Diagnostic-Driven Iterative Training for Large Multimodal Models
Feb 26, 2026Hongrui Jia, Chaoya Jiang, Yongrui Heng +2Multimodal Large Language Models Continual Learning for VLMs
Peking University · Shandong University
Instruction-based Image Editing with Planning, Reasoning, and Generation
Feb 26, 2026Liya Ji, Chenyang Qi, Qifeng ChenText-Guided Image Editing Diffusion-Based Image Editing
HKUST
Exploring Multimodal LMMs for Online Episodic Memory Question Answering on the Edge
Feb 25, 2026Giuseppe Lando, Rosario Forte, Antonino FurnariMultimodal QA Episodic Memory
Department of Mathematics and Computer Science, University of Catania, Italy
SimpleOCR: Rendering Visualized Questions to Teach MLLMs to Read
Feb 25, 2026Yibo Peng, Peng Xia, Ding Zhong +6Multimodal Large Language Models Optical Character Recognition
UNC-Chapel Hill · Carnegie Mellon University · University of Michigan +1
Global-Local Dual Perception for MLLMs in High-Resolution Text-Rich Image Translation
Feb 25, 2026Junxin Lu, Tengfei Song, Zhanglin Wu +9Image Text Translation Machine Translation
School of Communication and Information Engineering, Shanghai University, Shanghai 200444, China · Huawei Translation Services Center, China · School of Computer Science and Technology, East China Normal University, Shanghai 200062, China +1
EmoOmni: Bridging Emotional Understanding and Expression in Omni-Modal LLMs
Feb 25, 2026Wenjie Tian, Zhixian Zhao, Jingbin Hu +4Multimodal Large Language Models Emotion-Aware Dialogue Generation
Northwestern Polytechnical University, Xi’an, China · University of Surrey, Guildford, United Kingdom
BrepCoder: A Unified Multimodal Large Language Model for Multi-task B-rep Reasoning
Feb 25, 2026Mingi Kim, Yongjun Kim, Jungwoo Kang +1Multimodal Large Language Models
Chungnam National University, Daejeon, Republic of Korea
FewMMBench: A Benchmark for Multimodal Few-Shot Learning
Feb 25, 2026Mustafa Dogan, Ilker Kesen, Iacer Calixto +2Multimodal Model Evaluation Multimodal Large Language Models
Aselsan Research · Department of Computer Science, University of Copenhagen · Department of Medical Informatics, Amsterdam UMC, University of Amsterdam +4
RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrieval
Feb 25, 2026Dawei Su, Dongsheng WangMultimodal IR Multimodal Reranking
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China
Enhancing Multi-Modal LLMs Reasoning via Difficulty-Aware Group Normalization
Feb 25, 2026Jinghan Li, Junfeng Fang, Jinda Lu +5Multimodal Large Language Models Multimodal Reasoning
University of Science and Technology of China · National University of Singapore
Leveraging large multimodal models for audio-video deepfake detection: a pilot study
Feb 25, 2026Songjun Cao, Yuqi Li, Yunpeng Luo +2Multimodal Deepfake Detection Deepfake Detection
Tencent YouTu Lab, China · Tencent Youtu Lab, China · Fudan University, China
One Brain, Omni Modalities: Towards Unified Non-Invasive Brain Decoding with Large Language Models
Feb 25, 2026Changli Tang, Shurui Li, Junliang Wang +8Neural Decoding Multimodal Fusion
Tsinghua University · Shanghai AI Laboratory · ShanghaiTech University
VecGlypher: Unified Vector Glyph Generation with Language Models
Feb 25, 2026Xiaoke Huang, Bhavul Gauri, Kam Woh Ng +12Vector Graphics Generation Text-to-SVG Generation
Meta AI · UC, Santa Cruz
Causal Decoding for Hallucination-Resistant Multimodal Large Language Models
Feb 24, 2026Shiwei Tan, Hengyi Wang, Weiyi Qin +3Object Hallucination in VLMs VLM Hallucination Mitigation
Rutgers University · Meta Ranking AI
Are Multimodal Large Language Models Good Annotators for Image Tagging?
Feb 24, 2026Ming-Kun Xie, Jia-Hao Xiao, Zhiqiang Kou +3LLM-Assisted Annotation Multimodal Large Language Models
RIKEN Center for Advanced Intelligence Project, Japan · Southeast University, China · China University of Mining and Technology, China +1
Physics-based phenomenological characterization of cross-modal bias in multimodal models
Feb 24, 2026Hyeongmo Kim, Sohyun Kang, Yerin Choi +5Multimodal Large Language Models Algorithmic Fairness
Brain Science Institute, Korea Institute of Science and Technology, Seoul, Republic of Korea · Department of Physics and Astronomy, Seoul National University, Seoul, Republic of Korea · Department of Computer Science and Engineering, Korea University, Seoul, Republic of Korea +2

Feb 28, 2026J/K move · Enter open · S save

What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models

Yingqi Fan, Junlong Tong, Anhao Zhao, Xiaoyu Shen

Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative · Shanghai Jiao Tong University+1

Multimodal Large Language Models Multimodal Model Interpretability Visual Token Pruning

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) project visual tokens into the embedding space of language models, yet the internal structuring and processing of visual semantics remain poorly understood. In this work, we introduce a two-fold analytical framework featuring a novel probing tool,

\textbf{EmbedLens}

, to conduct a fine-grained analysis. We uncover a pronounced semantic sparsity at the input level: visual tokens consistently partition into sink, dead, and alive categories. Remarkably, only the alive tokens, comprising

\approx60\%

of the total input, carry image-specific meaning. Furthermore, using a targeted patch-compression benchmark, we demonstrate that these alive tokens already encode rich, fine-grained cues (e.g., objects, colors, and OCR) prior to entering the LLM. Internal visual computations (such as visual attention and feed-forward networks) are redundant for most standard tasks. For the small subset of highly vision-centric tasks that actually benefit from internal processing, we reveal that alive tokens naturally align with intermediate LLM layers rather than the initial embedding space, indicating that shallow-layer processing is unnecessary and that direct mid-layer injection is both sufficient. Ultimately, our findings provide a unified mechanistic view of visual token processing, paving the way for more efficient and interpretable MLLM architectures through selective token pruning, minimized visual computation, and mid-layer injection. The code is released at: https://github.com/EIT-NLP/EmbedLens.