Multimodal Large Language Models

Also known as MLLM

Follow this topic Ask about this topic

Momentum

67 papers in the last four weeks, up 91% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 676

All topics

Mind the Heads: Topological Representation Alignment for Multimodal LLMs
Jun 22, 2026Davide Caffagni, Alberto Compagnoni, Federico Melis +5Cross-Modal Alignment Multimodal Large Language Models
University of Modena and Reggio Emilia · University of Pisa · AMD Silo AI
AIR: Adaptive Interleaved Reasoning with Code in MLLMs
Jun 22, 2026Cong Han, Xiaohan Lan, Haibo Qiu +1Multimodal Large Language Models Mathematical Reasoning
1Independent Researcher.
PIVOTSBench: Evaluating Fine-Grained Interpersonal Relationship Reasoning in Multimodal Large Language Models
Jun 22, 2026Shuxiang Zhang, Yiting Yin, Wenxuan Song +2Multimodal Large Language Models Multimodal Reasoning
Sun Yat-sen University · University of Michigan · Tsinghua University
Physiology-Aware CNN and Zero-Shot Multimodal LLMs for ECG Image Classification: A Comparative Study
Jun 22, 2026Khalil Ahammad, Derek Abbott, Mohsen DorrakiElectrocardiography Multimodal Large Language Models
School of Computer Science and Information Technology, Adelaide University, SA 5005, Australia · Australian Institute for Machine Learning (AIML), Adelaide, Australia · Pi MedTech, Adelaide, Australia +1
SingGuard: A Policy-Adaptive Multimodal LLM Guardrail with Dynamic Reasoning
Jun 22, 2026SingGuard TeamLanguage Model Safety Evaluation LLM Guardrails
1AI Security Lab, Ant Group
VideoLatent: Video-Language Learning via Latent Self-Forcing
Jun 22, 2026Zi-Yuan Hu, Zicong Tang, Shijia Huang +3Multimodal Large Language Models Latent Visual Reasoning
The Chinese University of Hong Kong · Weitu AI
Bayesian Adaptation Gym: A Benchmark for the Bayesian Low-Rank Adaptation of Multi-Modal Language Models
Jun 20, 2026Colin Samplawski, Ramneet Kaur, Manoj Acharya +2Distribution Shift Robustness Multimodal Large Language Models
Neuro-Symbolic Computing and Intelligence Research Group, Computer Science Laboratory, SRI International
Safe responses matter: Output-aware safety guardrail mitigate over-refusal in MLLMs
Jun 20, 2026Jiayi Li, Kun ZhanLLM Guardrails Multimodal Large Language Models
Lanzhou University
StylisticBias: A Few Human Visual Cues Drive Most Social Biases in MLLMs
Jun 18, 2026Shaghayegh Kolli, Timo Cavelius, Nafiseh Nikeghbal +2Social Bias in Language Models Multimodal Large Language Models
Technical University of Munich · Munich Center for Machine Learning · Princeton Center for Information and Technology Policy
ELVA: Exploring Ranking-Driven Universal Multimodal Retrieval
Jun 18, 2026Yuhan Liu, Pei Fu, Hang Li +8Reinforcement Learning Multimodal Large Language Models
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · MiLM Plus, Xiaomi Inc · Zhongguancun Academy, Beijing, China
Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMs
Jun 18, 2026Haochen Han, Jue Wang, Alex Jinpeng Wang +1Test-Time Adaptation Multimodal Large Language Models
Peng Cheng Laboratory, Shenzhen, China · Tsinghua University, Beijing, China · Central South University, Changsha, China
Timage: A Generative Text-in-Image Paradigm for Fine-Tuning Vision-Language Models
Jun 18, 2026Yifeng Wu, Huimin Huang, Ruiluo Wu +5Visual Spatial Reasoning Multimodal Large Language Models
Fudan University · Shenzhen University of Advanced Technology · Tencent Jarvis Lab +1
Exploring Multi-Modal Large Language Models and Two-Stage Fine-Tuning for Fashion Image Retrieval
Jun 18, 2026Nguyen Cao Hoang, Hoang Bui Le, Nam Vo Hoang +1Multimodal Large Language Models Image-Text Retrieval
University of Science, VNU-HCM, Ho Chi Minh, Vietnam · Vietnam National University, Ho Chi Minh, Vietnam
Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation
Jun 17, 2026Sihan Wang, Xiyao Liu, Lianqing Liu +1Vision-Language Models On-Policy Self-Distillation
State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences
ThinkDeception: A Progressive Reinforcement Learning Framework for Interpretable Multimodal Deception Detection
Jun 17, 2026Jinhao Song, Shan Liang, Yiqun Yue +2RL for Language Model Reasoning Multimodal Large Language Models
Xi’an Jiaotong-Liverpool University · Suzhou, China
Evaluating Large Language Models Abilities for Addressee, Turn-change, and Next Speaker Prediction in Meetings
Jun 16, 2026Ryo Fukuda, Takatomo Kano, Siddhant Arora +7LLM Evaluation Multimodal Large Language Models
NTT, Inc., Japan · Language Technologies Institute, Carnegie Mellon University, USA
DifferAD-R1: A Difference-Guided IndustrialAnomaly Localization with Multimodal LargeLanguage Models
Jun 15, 2026Dingrong Wang, Xian Tao, Zhen Qu +5Reinforcement Learning Industrial Anomaly Detection
Institute of Automation, Chinese Academy of Sciences (CAS), Beijing 100190, China · School of Artificial Intelligence, University of Chinese Sciences, Beijing 100049, China · Institute of Automation, CAS, Beijing 100190, China +4
LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models
Jun 15, 2026Zhou Tao, Fang Zhang, Zewen Ding +5Multimodal Large Language Models Vision-Language Grounding
University of Science and Technology of China · State Key Laboratory of Cognitive Intelligence
MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs
Jun 15, 2026Yuanteng Chen, Peisong Wang, Zhilei Liu +9Mixed-Precision Quantization LLM Quantization
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · 3Zhongguancun Academy
SAE++: Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMs
Jun 15, 2026Yusong Zhao, Hengyi Wang, Tanuja Ganu +2Hierarchical Representation Learning Multimodal Large Language Models
Rutgers University. · Microsoft.
Multimodal LLM-Empowered Re-Ranking for Generalizable Person Re-Identification
Jun 15, 2026Jiachen Li, Xiaojin GongMultimodal Reranking Multimodal Large Language Models
College of Information Science and Electronic Engineering Zhejiang University
VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA
Jun 15, 2026Young Rok Jang, Hyesoo Kong, Kyunghwan An +3VLM Evaluation Multimodal Large Language Models
LG AI Research
Stepwise Token Selection for Efficient Multimodal Large Language Models
Jun 14, 2026Landi He, Shawn Young, Lijian XuEfficient Multimodal Inference Multimodal Large Language Models
Shenzhen University of Advanced Technology
Mind the Gap: Diagnosing Constraint Discovery Failures in Text-in-Image Editing
Jun 14, 2026Rui GuiText-Guided Image Editing Multimodal Large Language Models
Central South University
OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computing
Jun 14, 2026Zebang Cheng, Shuimu Chen, Boxue Yang +7On-Policy Self-Distillation Multimodal Large Language Models
Shenzhen University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Tsinghua University +6
UrbanWell: Benchmarking Multimodal Large Language Models for Spatio-Temporal Urban Wellbeing Analytics
Jun 14, 2026Yanxin Xi, Xiang Su, Jie Feng +3Spatiotemporal Reasoning Multimodal Large Language Models
Department of Computer Science, University of Helsinki, Helsinki, Finland · Department of Computer Science, Department of Agricultural Sciences, University of Helsinki, Helsinki, Finland · Zhongguancun Academy, Beijing, China +2
Deep Residual Injection for Full-Spectrum Forensic Signal Perception in Multimodal Large Language Models
Jun 14, 2026Kaiqing Lin, Zhiyuan Yan, Ruoxin Chen +8Digital Image Forensics Multimodal Large Language Models
Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen Key Laboratory of Media Security, and SZU-AFS Joint Innovation Center for AI Technology, Shenzhen University · Tencent Youtu Lab · Peking University +1
The Truth Stays in the Family: Enhancing Contextual Grounding via Inherited Truthful Heads in Model Lineages
Jun 14, 2026Miso Choi, Seonga Choi, Mincheol Kwon +3LLM Grounding LLM Interpretability
Korea University, Seoul, Republic of Korea · Kakao Mobility Corp., Seongnam, Republic of Korea
MAF: Multimodal Adaptive Few-shot Prompting for Sentiment Analysis with MLLMs
Jun 14, 2026Hangling XieMultimodal ICL Multimodal Large Language Models
Nanjing University of Posts and Telecommunications, Wenyuan Road No.9, Xixia District, Nanjing City, Jiangsu Province, China
Adapting Reinforcement Learning with Chain-of-Thought Supervision for Explainable Detection of Hateful and Propagandistic Memes
Jun 13, 2026Mohamed Bayan Kmainasi, Mucahid Kutlu, Ali Ezzat Shahroor +2Social Media Analysis Multimodal Large Language Models
Qatar Computing Research Institute, Doha, Qatar · Qatar University, Doha, Qatar · APAVI.AI, France +1

Jun 22, 2026, cs.CVJ/K move · Enter open · S save

Mind the Heads: Topological Representation Alignment for Multimodal LLMs

Davide Caffagni, Alberto Compagnoni, Federico Melis, Sara Sarto+4

University of Modena and Reggio Emilia · University of Pisa · AMD Silo AI

Cross-Modal Alignment Multimodal Large Language Models Multimodal Foundation Models Multimodal Contrastive Learning VLM Hallucination Mitigation

Open paper AskPDF arXiv

Representation alignment has emerged as an effective approach to improve Multimodal Large Language Models (MLLMs) by regularizing their internal representations toward those of an external vision encoder. However, existing methods typically align a fixed layer of the language backbone, overlooking the fine-grained structure of Transformer models. In this work, we propose Head-Wise Representation Alignment (HeRA), a method that enforces cross-modal alignment at the level of individual attention heads. Our approach is grounded in the Platonic Representation Hypothesis, focusing on preserving the topological structure of representations (i.e., their local neighborhood relationships) across modalities. Following the Mutual K-Nearest Neighbor (MKNN) alignment metric, we introduce a contrastive objective that acts as a differentiable proxy for matching local structures. HeRA applies this objective during multimodal training to specific attention heads in the LLM, selected by their alignment score according to the MKNN metric. Counterintuitively, we find that aligning the least aligned heads yields the largest gains. Extensive evaluations across multiple MLLMs and 18 benchmarks demonstrate that HeRA consistently improves performance on challenging vision-centric tasks and serves as an effective regularizer against visual hallucinations by naturally curbing the over-reliance on linguistic priors. Our code is publicly released.

Momentum

Related topics

Latest papers 676