Multimodal Large Language Models

Also known as MLLM

Follow this topic Ask about this topic

Momentum

67 papers in the last four weeks, up 91% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 684

All topics

Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs
Sep 29, 2026Yueran Ma, Ronghao LinAudio QA Multimodal Large Language Models
The University of Queensland, Brisbane, Australia · Shenzhen University, Shenzhen, China
Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning
Sep 29, 2026Cheng Ye, Weidong Chen, Zhaobo Qi +2Multimodal Large Language Models Multimodal Emotion Recognition
University of Science and Technology of China, Hefei · Harbin Institute of Technology, Weihai
AdaKerNet: Neural Kernel Decoding for Task-Adaptive Prediction with Multimodal Large Models
Sep 28, 2026Konstantinos D. Polyzos, Eleni Oikonomou, Tara JavidiMultimodal Large Language Models Kernel Methods
University of California San Diego
Render Before Reading: Visual Rendering as a Prompt Injection Defense
Sep 28, 2026Jie Zhang, Andrei Baroian, Jan N. van Rijn +2Multimodal Large Language Models Modality Gap in VLMs
ETH Zurich · Leiden University
How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining
Sep 28, 2026Lin Chen, Bolin Ni, Qi Yang +6Multimodal Pretraining Multimodal Large Language Models
CASIA · UCAS · Foundation Model Department, Tencent
SPIDER: Multi-Layer Semantic Token Pruning and Adaptive Sub-Layer Skipping in Multimodal Large Language Models
Sep 28, 2026Tianxiang Chen, Zhentao Tan, Zi Ye +9Efficient Multimodal Inference Transformer
End-User Intelligent Computing BU, Alibaba Cloud · Alibaba Group · Department of Computer Science, Maynooth University +2
Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models
Sep 28, 2026Jingdi lei, Junxian Li, Di Zhang +3Efficient Multimodal Inference Multimodal Large Language Models
Nanyang Technological University · Shanghai Jiao Tong University · Fudan University +2
Resolution as a First-Class Decision: Task-Conditioned Routing for Efficient Multimodal Large Language Models
Sep 28, 2026Zhiqiang Xia, Yang Li, Xinyuan Zhang +5Efficient Multimodal Inference Efficient VLM Inference
Xiaomi Corporation, Beijing, China
Revisit to Segment: Working Memory Distillation for Reasoning Segmentation
Sep 28, 2026Cilin Yan, Yilun Qiu, Wanyang Zhang +4Text-Guided Image Segmentation Multimodal Large Language Models
Xiaohongshu · NTU · PKU
OmniTide: Co-Designing Algorithms and Systems for Efficient On-Device Omni-LLM Streaming
Sep 28, 2026Zongshang Shen, Wangsong Yin, Daliang Xu +2Efficient Multimodal Inference On-Device Language Model Inference
Key Lab of High Confidence Software Technologies (Peking University), Beijing, China · State Key Laboratory of Networking and Switching Technology (BUPT), Beijing, China
Counterfactual Attention Policy Distillation for Temporal Video Grounding
Sep 28, 2026Shaobo Ju, Haiyang Yu, Xuecheng Wu +5Temporal Video Grounding VLM Distillation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · Fudan University. · MMLab, The Chinese University of Hong Kong. +1
MiCo: Mutual Information Coverage Optimization through Semantic Erasure Modeling for Efficient MLLM Inference
Sep 28, 2026Tinghao Wang, Yichen Guo, Qizhe Zhang +11Efficient Multimodal Inference Multimodal Large Language Models
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · University of Electronic Science and Technology of China · Nanyang Technological University +2
Uncovering Ordinal-Matching Bias in Audio-Visual LLMs
Sep 28, 2026Jihoo Jung, Youngjoon Jang, Hyebin Cho +2Audio-Visual Understanding Multimodal Large Language Models
Korea Advanced Institute of Science and Technology, South Korea · University of Oxford, United Kingdom
PhysFieldBench: Can Multimodal Models Understand Physical Fields?
Sep 28, 2026Yuezhou Ma, Huikun Weng, Jialong Wu +5Physical Reasoning Multimodal Large Language Models
School of Software, BNRist, Tsinghua University, China
MetaSampling: Making Frame Samplers Efficient for Long-Video Question Answering
Sep 27, 2026Ashim Dahal, Bikramjit BanerjeeMultimodal Large Language Models Adaptive Sampling
University of Southern Mississippi Hattiesburg, MS, USA
One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMs
Sep 27, 2026Sen Nie, Jie Zhang, Zhongqi Wang +2Adversarial Attacks on VLMs Adversarial Attacks on LLMs
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, and the University of Chinese Academy of Sciences.
Learning Multimodal Embeddings with Evidence-Aligned Readout
Sep 27, 2026Zirong Chen, Fuda Ye, Enjun Du +8Multimodal Large Language Models Multimodal Embedding
The Hong Kong University of Science and Technology (Guangzhou) · Tencent Yuanbao · Tsinghua University +3
IVT-Guard: All-in-One Reasoning Model for AI-Generated Content Detection
Sep 27, 2026Hongwei Niu, Yunpeng Luo, Hanjun Li +6AI-Generated Text Detection Multimodal Large Language Models
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, Xiamen 361005, P.R. China · Tencent YouTu Lab
Multimodal LLMs Outperform Pathology Foundation Models in Cross-Domain Histological Similarity
Sep 26, 2026Yishu Zhang, Yun Li, Daiwei ZhangMultimodal Robustness Multimodal Large Language Models
University of North Carolina at Chapel Hill
The Alignment Illusion in Multimodal Large Language Models
Sep 24, 2026Hong-Han Wang, Yuntao Wang, Hu DingRepresentational Similarity Analysis Multimodal Large Language Models
University of Science and Technology of China, Hefei, China
Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentration
Sep 24, 2026Jiaqi Deng, Zonghan Wu, Zhan Heng +3Multimodal Grounding Multimodal Large Language Models
University of Technology Sydney · East China Normal University · The University of New South Wales +2
MoVISA: Multi-Token Reasoning for Video Object Segmentation
Sep 24, 2026Ruining Zhao, Ho Kei Cheng, Alexander G SchwingVideo Object Segmentation Referring Video Object Segmentation
University of Illinois Urbana-Champaign
DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMs
Sep 23, 2026Yingxuan Zhuang, Miao Pan, Wangjie Gan +6Multimodal Large Language Models VLM Hallucination Mitigation
Zhejiang University · Shanghai Artificial Intelligence Laboratory
Don't Read the Log: Execution Traces Contaminate Verifiers in Video-Generation Agents
Sep 23, 2026Jian XuLLM-as-a-Judge AI Agent Auditing
RIKEN
Pistis Technical Report
Sep 23, 2026Heyun Chen, Xiaohan Lan, Jiaxi Li +17Multimodal Large Language Models Multimodal Reasoning
Pistis Team, ByteDance
Virtual Encoders in Multimodal Transformers
Sep 22, 2026Katsuya Ogata, Yuta NakashimaVisual Representation Learning Multimodal Large Language Models
The University of Osaka
BAS-OPD: Budget-Aware Selective On-Policy Self-Distillation for Fine-Grained Multimodal Perception
Sep 22, 2026Zihan Chen, Hengguang Zhou, Yuan Kang +5VLM Distillation Multimodal Large Language Models
HFIPS, Chinese Academy of Sciences · University of Science and Technology of China · University of California, Los Angeles
LingLan: An Advancing Traditional Chinese Medicine Diagnosis LLM with Multimodal Data
Sep 22, 2026Zheng Chen, Zhicheng Du, Haoxuan Li +2Medical Diagnosis Multimodal Large Language Models
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Guangdong Provincial Laboratory of Traditional Chinese Medicine Hengqin, Guangdong, China
Qwen3.8-Omni: Towards Native Omni-Modal Agents
Sep 22, 2026Qwen TeamMultimodal Pretraining Multimodal Large Language Models
Qwen Team
SLICEChat: Progressive In-Encoder Token Pruning for Whole-Slide Pathology Language Models
Sep 21, 2026Ali Kerem Bozkurt, Baris Cem Bakay, Ibrahim Kulac +3Efficient Multimodal Inference Multimodal Large Language Models
Kocı University, Department of Computer Engineering · Kocı University, Department of Pathology · Hacettepe University, Department of Artificial Intelligence and Data Engineering

Sep 29, 2026, cs.CVJ/K move · Enter open · S save

Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs

Yueran Ma, Ronghao Lin

The University of Queensland, Brisbane, Australia · Shenzhen University, Shenzhen, China

Audio QA Multimodal Large Language Models Multimodal Model Interpretability Multimodal Learning

Open paper AskPDF arXiv

Omni-modal large language models (LLMs) are expected to answer a question using the modality it explicitly refers to. However, existing training paradigms rarely verify whether models actually follow this modality, because multimodal inputs from the same sample often provide redundant evidence for the same answer. In this work, we uncover a pervasive cross-modal shortcut in omni-modal LLMs: when asked an audio-related question, models rely on the image as much as on the audio, and sometimes even more. To systematically diagnose this behavior, we introduce the Factorized Modality Diagnostic, which independently swaps audio and images between samples to isolate each modality's causal contribution. Across two model families in different settings, we find that this shortcut persists throughout supervised fine-tuning and reinforcement learning post-training, while judge-based RL may further amplify such reliance on irrelevant visual information. Based on this finding, we propose DMC-Repair, which trains models on the same kind of cross-modal swapped samples while assigning supervision according to the modality specified by the question. This prevents models from exploiting the spurious correspondence between modalities within the same clip. Experiments demonstrate that DMC-Repair reduces the image-induced share of the answer effect by 59.9%, effectively suppressing the cross-modal shortcut without compromising audio-question answering performance. The reduction in shortcut reliance generalizes across two model families and zero-shot to an unseen dataset and an unseen benchmark, and persists through subsequent post-training. Code is available at https://anonymous.4open.science/r/DMC-Repair.

Momentum

Related topics

Latest papers 684