Multimodal Large Language Models

Also known as MLLM

Follow this topic Ask about this topic

Momentum

67 papers in the last four weeks, up 91% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 676

All topics

SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models
May 31, 2026Chenshuang Zhang, Kyeong Seon Kim, Chengxin Liu +1Audio-Visual Understanding Multimodal Large Language Models
KAIST
Large Language Models in Transportation Systems Management and Operations: From Text Reasoning to Multi-modal Decision Support
May 31, 2026Siyan Li, Zehao Wang, Jiachen Li +3AI-Assisted Decision Making Intelligent Transportation Systems
Bourns College of Engineering, Center for Environmental Research and Technology, University of California at Riverside, CA, USA · Bourns College of Engineering, Computer Science and Engineering Department, University of California at Riverside, CA, USA · Bourns College of Engineering, Electrical and Computer Engineering Department, University of California at Riverside, CA, USA
Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
May 31, 2026Wanlong Fang, Tianle Zhang, Wen Tao +1Multimodal Grounding Multimodal Large Language Models
College of Computing and Data Science, Nanyang Technological University, Singapore · Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore · Centre of AI in Medicine (C-AIM), Nanyang Technological University, Singapore
MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models
May 30, 2026Ravil Mussabayev, Rustam MussabayevRepresentation Geometry Multimodal Large Language Models
Satbayev University
MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue
May 30, 2026Yue Jiang, Xue Jiang, Lihua Zhang +6Multimodal Large Language Models Multimodal Hallucination
College of Intelligent Robotics and Advanced Manufacturing, Fudan University · Southern University of Science and Technology · TMLR Group, Hong Kong Baptist University +3
MindZero: Learning Online Mental Reasoning With Zero Annotations
May 29, 2026Shunchi Zhang, Jin Lu, Chuanyang Jin +3Self-Supervised Learning Theory of Mind
Johns Hopkins University · Peking University
ERGeoBench:A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models
May 29, 2026Kaiwen Xue, Tao Wei, Guoxin Zhang +5Spatial Reasoning Benchmarks Visual Spatial Reasoning
Beijing University of Posts and Telecommunications, Beijing, China · State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China · School of Materials Science and Engineering, Shanghai Jiao Tong University, Shanghai, China +2
The Regularizing Power of Language-Training Deepfake Detectors
May 29, 2026Benedikt Hopf, Zongwei Wu, Radu TimofteVLM Robustness Explainable Deepfake Detection
Computer Vision Lab, CAIDAS, University of Würzburg, Germany
AMix-2: Establishing Protein as a Native Modality in Large Language Models
May 29, 2026Keyue Qiu, Yixin Wu, Lihao Wang +19Multimodal Large Language Models Protein Language Models
1Shanghai Artificial Intelligence Laboratory · Institute for AI Industry Research (AIR), Tsinghua University
BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs
May 29, 2026Ben Wang, Xiaogang Li, Ruochen Gao +6Physical Reasoning Multimodal Large Language Models
Alibaba Group Beijing, China
Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages
May 29, 2026Fan Wu, Lishuai Dong, Cuiyun Gao +4Web Application Generation Code Generation
Computer Science and Technolog, Harbin Institute of Technology. · Institute of Information Engineering, Chinese Academy of Sciences.
MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding
May 29, 2026Qian Kou, Xiaofeng Shi, Yulin Li +4Visual Spatial Reasoning Multimodal Large Language Models
Beijing Academy of Artificial Intelligence (BAAI), China · Institute of Information Engineering, Chinese Academy of Sciences, China · Beijing University of Technology, China
Towards Localized and Disentangled Knowledge Editing for Multimodal Large Language Models
May 28, 2026Leijiang Gu, Zhen Zeng, Feng Li +2Knowledge Editing Multimodal Large Language Models
1Hefei University of Technology, Hefei, China
ReactBench: A Cause-Driven Benchmark for Multimodal Hallucination via Systematic Evaluation
May 28, 2026Shizhe Zhou, Bohan Jia, Kai Wu +4VLM Robustness Hallucination Detection
East China Normal University
DMC-CF: Dynamic Multimodal CounterFactual QA benchmark for Causal Reasoning
May 28, 2026Junzhe Zhang, Huixuan Zhang, Guirong Wang +5Counterfactual Evaluation Counterfactual Reasoning in Language Models
Wangxuan Institute of Computer Technology, Peking University · Alibaba Group
SalsaAgent: A multimodal embodied language model for interactive dance generation
May 28, 2026Payam Jome Yazdian, Zoe Stanley, Angelica LimMultimodal Large Language Models Human Motion Generation
Simon Fraser University, Burnaby, Canada
Persona Prompting in Multimodal Urban Perception: Descriptive Convergence and Interpretive Variation
May 27, 2026Neemias da Silva, Matt Ratto, Myriam Delgado +3Multimodal Large Language Models Language Model Bias Evaluation
University of Toronto, Toronto, Canada · Universidade Tecnologica Federal do Parana, Curitiba, Brazil
OmniVerifier-M1: Multimodal Meta-Verifier with Explicit Structured Recalibration
May 27, 2026Xinchen Zhang, Bowei Liu, Jiale Liu +7Multimodal Large Language Models Reinforcement Learning with Verifiable Rewards
Tsinghua University · Pennsylvania State University · University of Southern California +2
Explaining is Harder Than Predicting Alone: Evaluating Concept-based Explanations of MLLMs as ICL Visual Classifiers
May 27, 2026Carmen Quiles-Ramírez, Leticia L. Rodríguez, Nicolás Martorell +1VLM Evaluation Multimodal Large Language Models
Universidad de Granada, Spain · Universidad de Buenos Aires, Argentina
SMILE-Next: Teaching Large Language Models to Detect, Classify, and Reason about Laughter
May 27, 2026Lee Jung-Mok, Kim Sung-Bin, Joohyun Chang +2Mixture of Experts Multimodal Large Language Models
School of EE, KAIST · 2Dept. of EE, POSTECH · School of Computing, KAIST
KSAFE-MM: A Multimodal Safety Benchmark via Localized Contextualization for Korean Cultural Risks
May 27, 2026Yongwoo Kim, Sojung An, Yunjin Park +8Adversarial Attacks on VLMs Language Model Safety Evaluation
Korea University · KT Corporation
Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigation
May 27, 2026Jingwen Wu, Xijun Zhang, Ge SongLLM Hallucination Mitigation Hallucination in Language Models
School of Computer and Electronic Information, Nanjing Normal University, China
Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization
May 27, 2026Jiawei Kong, Hao Fang, Shunxiang Liao +5Multimodal Grounding Direct Preference Optimization
Tsinghua Shenzhen International Graduate School, Tsinghua University · Harbin Institute of Technology, Shenzhen
Heterogeneous Parallelism for Multimodal Large Language Model Training
May 26, 2026Yashaswi Karnati, Kamran Jafari, Akash Mehra +10Communication-Efficient Distributed Training Multimodal Large Language Models
NVIDIA
OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs
May 26, 2026Guangzhi Sun, Yixuan Li, Yudong Yang +1Streaming Video Understanding Multimodal Large Language Models
ByteDance · Department of Engineering, University of Cambridge · Tsinghua University
Hallucination Behavior in Multimodal LLMs Across Agricultural Image Interpretation and Generation Tasks
May 26, 2026Partho Ghose, Al Bashir, Prem Raj +1Agricultural Image Analysis VLM Hallucination
Department of Biological & Agricultural Engineering, Texas A&M AgriLife Research, Texas A&M University System, Dallas, TX 75252, USA.
Touch-R1: Reinforcing Touch Reasoning in MLLMs
May 26, 2026Yingxin Lai, Yafei Zhou, Fucai Zhu +2Tactile Sensing Multimodal Large Language Models
1Xiamen University · 5Daimon Robotics · 2Great Bay University +2
Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning
May 26, 2026Junkai Chen, Yuhao He, Junxiang You +3VLM Unlearning Multimodal Large Language Models
Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, UCAS
LiveK12Bench: Have Large Multimodal Models Truly Conquered High School-level Examinations?
May 26, 2026Xiaohan Wang, Mingze Yin, Yilin Zhao +2Mathematical Reasoning Benchmarks Educational Assessment
Tencent PCG · College of Computer Science and Technology, Zhejiang University
OnceSelect: Reusable Data Selection for Efficient Multimodal Instruction Tuning
May 26, 2026Mingkang Dong, Muxin Pu, Hongyi Cai +5Instruction-Tuning Data Selection Multimodal Large Language Models
Faculty of Computer Science Universiti Malaya Kuala Lumpur, Malaysia · School of Information Technology Monash University Malaysia Subang Jaya, Selangor, Malaysia · Faculty of Computer Science University of Science and Technology Beijing Beijing, China

May 31, 2026, eess.ASJ/K move · Enter open · S save

SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models

Chenshuang Zhang, Kyeong Seon Kim, Chengxin Liu, Tae-Hyun Oh

KAIST

Audio-Visual Understanding Multimodal Large Language Models Multimodal Hallucination Audio-Visual Alignment

Open paper AskPDF arXiv

Despite the success of audio-visual large-language models (LLMs), they can produce plausible but ungrounded outputs, termed hallucination. Existing benchmarks focus on environmental sounds (e.g., dog barking) to indicate event occurrence. In contrast, human speech carries fundamentally different, rich semantics and temporal structures, yet it remains unexplored whether current models can accurately align speech content with corresponding visual signals. In this work, we show that speech content can induce hallucinations in audio-visual LLMs. To systematically study this, we introduce SVHalluc, the first comprehensive benchmark for evaluating speech-vision hallucination in audio-visual LLMs. Our benchmark diagnoses speech-vision hallucinations from two critical and complementary aspects: semantic and temporal. Experimental results demonstrate that state-of-the-art open-source audio-visual LLMs struggle with aligning speech content with corresponding visual signals, with a near-random accuracy on multiple tasks. In contrast, Gemini 2.5 Pro significantly outperforms the open-source models. Our analysis suggests that their failures stem from limited ability in cross-modality understanding, despite strong performance in single-modality perception. Our work uncovers a new and fundamental limitation of current audio-visual LLMs and highlights the need for speech-grounded video comprehension. Project page: https://chenshuang-zhang.github.io/projects/svhalluc/.

Momentum

Related topics

Latest papers 676