Multimodal Large Language Models

Also known as MLLM

Follow this topic Ask about this topic

Momentum

67 papers in the last four weeks, up 91% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 676

All topics

Causal Evidence for Attention Head Imbalance in Modality Conflict Hallucination
May 19, 2026Jinrui Jiang, Zhangtai Wu, Zhen Wu +1Hallucination in Language Models Multimodal Large Language Models
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation
May 18, 2026Qianhao Yuan, Jie Lou, Xing Yu +4Multimodal Large Language Models On-Policy Distillation
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Xiaohongshu Inc.
CrossView Suite: Harnessing Cross-view Spatial Intelligence of MLLMs with Dataset, Model and Benchmark
May 18, 2026Wei Wang, Yuqian Yuan, Tianwei Lin +4VLM Evaluation Multi-View Consistency
Zhejiang University China
Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models
May 18, 2026Kunyu Peng, Zhikun Zhou, Kailun Yang +9Visual Spatial Reasoning Multi-Robot Systems
Karlsruhe Institute of Technology · INSAIT, Sofia University “St. Kliment Ohridski” · Hunan University +4
RAVE: Re-Allocating Visual Attention in Large Multimodal Models
May 18, 2026Xi Leng, Xinhong Ma, Ziqiang Dong +4Visual Attention Self-Attention
Qwen Business Unit of Alibaba · The Chinese University of Hong Kong, Shenzhen · Beijing Institute of Technology
Beyond the Cartesian Illusion: Testing Two-Stage Multi-Modal Theory of Mind under Perceptual Bottlenecks
May 18, 2026Yajing Zhou, Xiangyu KongAudio-Visual Understanding Visual Spatial Reasoning
College of Computer Science, Beijing Information Science and Technology University, No.55 Taihang Road, Changping District, Beijing, 102206, China
Visualizing the Invisible: Generative Visual Grounding Empowers Universal EEG Understanding in MLLMs
May 18, 2026Jun-Yu Pan, Yansen Wang, Enze Zhang +3EEG Decoding EEG-Based Visual Decoding
Shanghai Jiao Tong University Shanghai, China · Microsoft Research Asia Shanghai, China
Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models
May 18, 2026Xinpeng Dong, Min Zhang, Kairong Han +3Object Hallucination in VLMs Multimodal Large Language Models
Zhejiang University · East China Normal University · Zhejiang University of Science and Technology
Safety Geometry Collapse in Multimodal LLMs and Adaptive Drift Correction
May 18, 2026Jiahe Guo, Xiangran Guo, Jiaxuan Chen +6Multimodal Large Language Models LLM Safety
Harbin Institute of Technology · Huawei Technologies Co., Ltd
OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models
May 18, 2026Morunliu Yang, Ruotao Xu, Le Li +6Efficient Multimodal Inference Multimodal Large Language Models
1Soochow University · Peking University
A More Word-like Image Tokenization for MLLMs
May 18, 2026Hyun Lee, Hyemin Jeong, Yejin Kim +4Image Tokenization Efficient VLM Inference
Seoul National University · Ewha Womans University
DMN: A Compositional Framework for Jailbreaking Multimodal LLMs with Multi-Image Inputs
May 18, 2026Wenzhuo Xu, Zhipeng Wei, Zonghao Ying +4Multimodal Large Language Models LLM Safety
AI Security Lab, Beijing, China · International Computer Science Institute, CA, USA · UC Berkeley, CA, USA +1
Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction
May 17, 2026Chaoqun He, Mingyang Xiang, Yingjing Xu +5Multimodal Large Language Models Multimodal Agents
Tsinghua University · Shanghai Qi Zhi Institute · ModelBest Inc.
EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning
May 17, 2026Zeyu Wang, Chang Liu, Eduardus Tjitrahardja +22Multimodal Large Language Models Egocentric Video Understanding
Tsinghua University · Shanghai Qi Zhi Institute · Tongji University +5
CatalyticMLLM: A Graph-Text Multimodal Large Language Model for Catalytic Materials
May 17, 2026Yanjie Li, Jian Xu, Xu-Yao Zhang +4Multimodal Large Language Models Materials Property Prediction
AnnLab, Institute of Semiconductors, Chinese Academy of Sciences, Beijing, China. · Zhongguancun Academy, Zhongguancun Academy, Beijing, China. · School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing, China. +3
Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages
May 16, 2026Firoj Alam, Shammur Absar Chowdhury, Enamul Hoque PrinceCross-Modal Alignment Multimodal Large Language Models
Qatar Computing Research Institute, HBKU, Qatar · York University, Canada
Navigating the Emotion Tree: Hierarchical Hyperbolic RAG for Multimodal Emotion Recognition
May 16, 2026Zeheng Wang, Bo Zhao, Yijie Zhu +6Multimodal RAG Hierarchical Classification
Great Bay University Guangdong, Dongguan, China · Tencent Youtu Lab Shanghai, China · CUHK, shenzhen Guangdong, Shenzhen, China
EVA01: Unified Native 3D Understanding and Generation via Mixture-of-Transformers
May 16, 2026Zongyuan Yang, Mingjing Yi, Wanli Ma +83D Mesh Generation 3D Shape Editing
SeeleAI Team
MAgSeg: Segmentation of Agricultural Landscapes in High-Resolution Satellite Imagery using Multimodal Large Language Models
May 15, 2026Piyush Tiwary, Utkarsh Ahuja, Depanshu Sani +5Remote Sensing Image Segmentation Agricultural Remote Sensing
Google DeepMind · Indian Institute of Science · Google
Attribute-Grounded Selective Reasoning for Artwork Emotion Understanding with Multimodal Large Language Models
May 15, 2026Cheng Zhang, Yuer Liu, Zhiyu Zhou +2Feature Attribution Multimodal Large Language Models
Department of Computer Science and Technology, Jilin University, Changchun, China · Department of Computer Science, National Taiwan University, Taipei, Taiwan
ASRU: Activation Steering Meets Reinforcement Unlearning for Multimodal Large Language Models
May 15, 2026Jiahui Guang, Haiyan Wang, Yingjie Zhu +4VLM Unlearning Multimodal Large Language Models
Harbin Institute of Technology, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China · The Hong Kong Polytechnic University, Hong Kong, China +1
MultiEmo-Bench: Multi-label Visual Emotion Analysis for Multi-modal Large Language Models
May 14, 2026Tianwei Chen, Takuya Furusawa, Yuki Hirakawa +3Multi-Label Classification Multimodal Large Language Models
ZOZO NEXT Inc.
OmniDrop: Layer-wise Token Pruning for Omni-modal LLMs via Query-Guidance
May 14, 2026Yeo Jeong Park, Hyemi Jang, Minseo Choi +3Efficient Multimodal Inference Multimodal Large Language Models
Samsung Research
Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs
May 13, 2026Trung Nguyen Quang, Yiming Gao, Fanyi Pu +3Multimodal Grounding Multimodal Large Language Models
Nanyang Technological University · LMMs-Lab Team · Johns Hopkins University
MILM: Large Language Models for Multimodal Irregular Time Series with Informative Sampling
May 13, 2026Hsing-Huan Chung, Shijun Li, Yoav Wald +3Irregular Time-Series Modeling Healthcare
University of Texas at Austin · Technion-IIT · Johns Hopkins University
Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models
May 13, 2026Qinwu Xu, Yifan Jiang, Haoyu RenMultimodal Grounding CoT Reasoning
Meta AI · UT Austin · Current independent researcher
Robust Checkpoint Selection for Multimodal LLMs via Agentic Evaluation and Stability-Aware Ranking
May 13, 2026Qinwu Xu, Zhuoheng Li, Jessie SalasMultimodal Large Language Models Multimodal Model Evaluation
Meta AI · Independent Researcher, previously Meta AI
Learning to See What You Need: Gaze Attention for Multimodal Large Language Models
May 13, 2026Junha Song, Byeongho Heo, Geonmo Gu +3Efficient Multimodal Inference Visual Attention
1KAIST · 2NAVER AI Lab
DiM\textsuperscript{3}: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging
May 13, 2026Zijing Wang, Mingyang Wang, Ercong Nie +6Multilingual Language Models Multimodal Large Language Models
Northeastern University, China · CIS, LMU Munich, Germany · Munich Center for Machine Learning (MCML), Germany +1
Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation
May 13, 2026Chao Hao, Jun Xu, Ji Du +6Text-Guided Image Segmentation Promptable Image Segmentation
School of Computing and Information Technology, Great Bay University, Dongguan, Guangdong, 523000, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, 311115, China · Department of Computing, The Hong Kong Polytechnic University, Hong Kong

May 19, 2026, cs.AIJ/K move · Enter open · S save

Causal Evidence for Attention Head Imbalance in Modality Conflict Hallucination

Jinrui Jiang, Zhangtai Wu, Zhen Wu, Xinyu Dai

National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China

Hallucination in Language Models Multimodal Large Language Models VLM Interpretability Multimodal Hallucination Causal Interventions in Language Models Hallucination Mitigation+1

Open paper AskPDF arXiv

Modality-conflict hallucination occurs when multimodal large language models (MLLMs) prioritize erroneous textual premises over contradictory visual evidence. To understand why visual evidence fails to prevail during generation, we take a mechanistic perspective and examine which internal components drive or resist this failure. We perform head-level causal analysis using path patching across five open-source MLLMs and identify two groups of attention heads with opposing causal roles: hallucination-driving heads and hallucination-resisting heads. We find a consistent asymmetry: driving effects are more broadly distributed and carry greater aggregate weight, whereas resisting effects concentrate in a small number of high-importance heads. Ablation experiments further confirm that these groups exert opposing effects during generation: distributed driving influence and localized resistance together form an imbalanced routing structure that biases generation toward the erroneous premise. Motivated by this finding, we propose MACI (Modality-conflict-Aware Causal Intervention), a conditional intervention that suppresses causally identified hallucination-driving heads only when conflict is detected. Across five MLLMs, MACI achieves the largest hallucination reduction among compared inference-time baselines on the MMMC benchmark with a favorable hallucination-accuracy trade-off, and transfers zero-shot to the SCI-SemanticConflict test.

Momentum

Related topics

Latest papers 676