Multimodal Large Language Models

Also known as MLLM

Follow this topic Ask about this topic

Momentum

67 papers in the last four weeks, up 91% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 676

All topics

LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models
Sep 28, 2025Shubhang Bhatnagar, Andy Xu, Kar-Han Tan +1Mixed-Precision Quantization LLM Quantization
University of Illinois Urbana-Champaign · Work done as an intern at HP Inc. · University of California, Los Angeles +1
Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Models
Sep 26, 2025Jiawei Liang, Jianjie Huang, Ruoyu Chen +4Feature Attribution Multimodal Large Language Models
Shenzhen Campus of Sun Yat-sen University, Shenzhen 518107, China · Zhongguan-cun Academy, Beijing 100094, China · University of Chinese Academy of Sciences, Beijing 100049, China +2
Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping
Aug 17, 2025Xuhui Zhan, Tyler DerrCross-Modal Learning Cross-Modal Representation Learning
Data Science Institute, Vanderbilt University, 2201 West End Ave, Nashville, 37235, Tennessee, United States. · Computer Science Department, Vanderbilt University, 2201 West End Ave, Nashville, 37235, Tennessee, United States.
SyncLoop: A Multimodal Dual-Loop Framework for Self-Improving Mathematical Reasoning
Jul 22, 2025Xiuwei Chen, Wentao Hu, Hanhui Li +9Synthetic Data Generation Multimodal Large Language Models
Sun Yat-sen University · The Hong Kong Polytechnic University · MBZUAI +2
Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey
Jul 21, 2025Jindong Li, Yali Fu, Jiahong Liu +5Multimodal Large Language Models Vector Quantization
Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Jilin University, Changchun, China · The Chinese University of Hong Kong, Hong Kong, China +2
Region-Aware Multimodal Large Language Model via SlowFast Tokenization and Pseudo-Mask Guidance for 3D CT Report Generation
Jun 29, 2025Sunggu Kyung, Jinyoung Seo, Hyunseok Lim +7Radiology Report Generation 3D Medical Imaging
Department of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea · University of Ulsan College of Medicine, Seoul, Republic of Korea · Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea
Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding
May 22, 2025Runpeng Yu, Xinyin Ma, Xinchao WangMultimodal Large Language Models LLM Inference Acceleration
National University of Singapore
HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation
May 16, 2025Shaina Raza, Aravind Narayanan, Vahid Reza Khazaie +6Algorithmic Fairness Multimodal Large Language Models
Vector Institute for Artificial Intelligence, Toronto, Canada · University of Central Florida, Orlando, USA
FORGE: Forensic Reasoning with Grounded Evidence
Mar 20, 2025Rohit Kundu, Shan Jia, Vishal Mohanty +2VLM Adaptation Explainable Deepfake Detection
University of California, Riverside · Google LLC
SpurLens: Automatic Detection of Spurious Cues in Multimodal LLMs
Mar 11, 2025Parsa Hosseini, Sumit Nawathe, Mazda Moayeri +2VLM Evaluation VLM Robustness
Department of Computer Science University of Maryland
Histopathology Multi-modal Embedding for Pathology Composed Retrieval
Feb 11, 2025Qifeng Zhou, Wenliang Zhong, Thao M. Dang +4Computational Pathology Healthcare
The University of Texas at Arlington, Arlington TX 76010, USA
Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models
Dec 21, 2024Yanxu Mao, Peipei Liu, Tiehan Cui +3Adversarial Attacks on LLMs Multimodal Large Language Models
School of Computer Science and Technology, Shanghai Jiao Tong University, Shanghai, China · Zhongguancun Laboratory, Beijing, China · School of Software, Henan University, Kaifeng, China +1
RS-Agent: Automating Remote Sensing Tasks through Intelligent Agent
Jun 11, 2024Wenjia Xu, Zijian Yu, Boyang Mu +3Multimodal RAG Multimodal Large Language Models
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China · City University of HongKong, Hong Kong 999077, China · School of Geographic Sciences, Hunan Normal University, Changsha 410081, China
Multimodal Large Language Model driven Radiology Report Generation with Clinical Knowledge Enhancement
Mar 11, 2024Miaojing Shi, Tianyu Cen, Zijie Yue +3Radiology Report Generation Radiology VLMs
College of Electronic and Information Engineering, Tongji University, Shanghai, China
MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models
Date pendingSangyun Chung, Se Yeon Kim, Youngchae Chee +1Multimodal Robustness Hallucination in Language Models
Integrated Vision Language Lab, KAIST, South Korea
SaFeR-Steer: Evolving Multi-Turn MLLMs via Synthetic Bootstrapping and Feedback Dynamics
Date pendingHaolong Hu, Hanyu Li, Tiancheng He +6Language Model Steering Multimodal Large Language Models
Huazhong University of Science and Technology · Beijing University of Posts and Telecommunications · West China Biomedical Big Data Center, Sichuan University +2

Sep 28, 2025, cs.CVJ/K move · Enter open · S save

LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models

Shubhang Bhatnagar, Andy Xu, Kar-Han Tan, Narendra Ahuja

University of Illinois Urbana-Champaign · Work done as an intern at HP Inc. · University of California, Los Angeles+1

Mixed-Precision Quantization LLM Quantization VLM Quantization Multimodal Large Language Models Multimodal Foundation Models Post-Training Quantization+1

Open paper AskPDF arXiv

Large Language Models (LLMs) with multimodal capabilities have revolutionized vision-language tasks, but their deployment often requires huge memory and computational resources. Post-training quantization (PTQ) has successfully compressed language models to as low as 1-bit precision, its effectiveness for multimodal LLMs (MLLMs) remains unexplored. In this paper, we present the first method for ultra-low-bit (<4-bit) quantization of MLLMs. Our analysis reveals that multimodal tokens and intermediate layer activations produced by them exhibit significantly higher entropy compared to text tokens, indicating greater functional complexity that makes MLLMs less tolerant to ultra-low bit quantization. However, this entropy varies significantly across layers, with some layers producing lower-entropy activation distributions that we empirically show can better tolerate ultra-low bit quantization. Existing PTQ methods optimize weight quantization within each layer but apply the same target precision uniformly, ignoring this variation in complexity across layers. Building on this insight, we propose LUQ: Layerwise Ultra-Low Bit Quantization, which characterizes each transformer layer's functional complexity via its output activation entropy and selectively applies ultra-low bit quantization to layers encoding simpler, more compressible functions. We also show that multimodal calibration (image and text tokens) boosts VQA performance in the ultra-low bit regime. Evaluated on LLaVA-1.5 and Qwen-2.5-VL across 9 VQA benchmarks, LUQ models use 40% and 31% less memory than their 4-bit counterparts while exhibiting less than 10% degradation on MME.

Momentum

Related topics

Latest papers 676