Multimodal Model Evaluation

Follow this topic Ask about this topic

Momentum

49 papers in the last four weeks, up 44% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 496

All topics

Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models
Jun 24, 2026Akshay Paruchuri, Sanmi Koyejo, Ehsan AdeliMultimodal Robustness LLM Auditing
Stanford University
TriViewBench: Controlled Complexity Scaling for Multi-View Structural Reasoning in MLLMs
Jun 24, 2026Yu-Yang Chen, Lan-Zhe GuoSpatial Reasoning Benchmarks Visual Reasoning
School of Intelligence Science and Technology, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China
Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models
Jun 24, 2026Davie ChenScientific Visualization Multimodal Model Evaluation
Falcon: Functional Assembly and Language for Compositional Reasoning in X-ray
Jun 24, 2026Yonathan Michael, Mohamad Alansari, Natnael Takele +2Multimodal Grounding Multimodal Reasoning
Computer Science Department Khalifa University · Center of Cyber-physical Systems (C2PS) Khalifa University
SSMNBench: Diagnosing Image-based Cross-View Human-Object Understanding via Single-View Sufficiency and Multi-View Necessity
Jun 24, 2026Tianchen Guo, Chen Liu, Ling Chen +1Visual Question Answering Multi-View Consistency
The University of Queensland · Follow Me AI Pty LTD · University of Technology Sydney +1
Yuvion VL: A Multimodal Foundation Model for Adversarial Content and AI Safety
Jun 23, 2026Shikai Qiu, Xiaowen Xu, Benlei Cui +55VLM Robustness Multimodal Large Language Models
MedBench v5: A Dynamic, Process-Oriented, and Hallucination-Aware Benchmark for Clinical Multimodal Models
Jun 23, 2026Jinru Ding, Chuchu Jiang, Lu Lu +12AI Agent Benchmarks Medical VLMs
1Shanghai Artificial Intelligence Laboratory
ReMMD: Realistic Multilingual Multi-Image Agentic Verification for Multimodal Misinformation Detection
Jun 23, 2026Chenhao Dang, Dantong Zhu, Jun Yang +2Multimodal Agents Multimodal Model Evaluation
Shanghai Jiaotong University · Shanghai Artificial Intelligence Laboratory · Central South University +2
PIVOTSBench: Evaluating Fine-Grained Interpersonal Relationship Reasoning in Multimodal Large Language Models
Jun 22, 2026Shuxiang Zhang, Yiting Yin, Wenxuan Song +2Multimodal Large Language Models Multimodal Reasoning
Sun Yat-sen University · University of Michigan · Tsinghua University
SingGuard: A Policy-Adaptive Multimodal LLM Guardrail with Dynamic Reasoning
Jun 22, 2026SingGuard TeamLanguage Model Safety Evaluation LLM Guardrails
1AI Security Lab, Ant Group
Catching Lies Without Sending the Video: Privacy-Preserving Multimodal Deception Detection
Jun 21, 2026Nikita Sharma, Pranav Sara, Karan SinglaPrivacy-Preserving ML Multimodal Model Evaluation
University of California, San Diego · Case Western Reserve University, Ohio · WhissleAI, USA
Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do
Jun 21, 2026Zhuoran Jin, Kejian Zhu, Hongbang Yuan +5Multimodal Reasoning VLM Reasoning
The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Efficient Multimodal Clinical Question Answering for Pulmonary Embolism Risk Assessment
Jun 21, 2026Xiangyuan Xue, Yang Yu, Yan Gao +5Efficient Multimodal Inference Multimodal QA
University of Auckland, Auckland, New Zealand · University of Cambridge, Cambridge, United Kingdom · University of Adelaide, Adelaide, Australia +1
T-IMPACT: A Severity-Aware Benchmark for Contextual Image-Text Manipulation
Jun 21, 2026Gagandeep Singh, Aaditya Yadav, Priyanka SinghMultimodal Model Evaluation
The University of Queensland · Brisbane, Australia
Bayesian Adaptation Gym: A Benchmark for the Bayesian Low-Rank Adaptation of Multi-Modal Language Models
Jun 20, 2026Colin Samplawski, Ramneet Kaur, Manoj Acharya +2Distribution Shift Robustness Multimodal Large Language Models
Neuro-Symbolic Computing and Intelligence Research Group, Computer Science Laboratory, SRI International
SAGE: An Expert-Annotated South Asian GI Endoscopy Dataset for Multimodal Learning and Hallucination Analysis
Jun 20, 2026Niyoj Oli, Sachin Acharya, Sandesh Pokhrel +7Endoscopy Medical Imaging
Nepal Applied Mathematics and Informatics Institute for Research, Nepal · University of Utah, USA · GastroIntestinal Department, Dhulikhel Hospital, Nepal +3
Chehre: An Emoji-Prompted Dataset to Explore Perceptual Flexibility in Video Language Models
Jun 19, 2026Bita Azari, Zoe Stanley, Avneet Batra +4VLM Evaluation Video-Language Model Evaluation
Simon Fraser University, Canada
CulMind: Benchmarking Multimodal Understanding and Reasoning in Chinese Cultural Heritage
Jun 19, 2026Zhangwei Cao, Shuhan Fan, Yuting Wei +5Benchmark Construction Reasoning Evaluation
University of International Relations · 2Kedge Business School · 3Peking University +3
Cross-Modal Corroboration for Annotation-Free Wildlife Monitoring
Jun 19, 2026Bharath Pillai, Varun Viswapriyan, Christopher Stewart +2Animal Behavior Analysis Multimodal Model Evaluation
The Ohio State University
StylisticBias: A Few Human Visual Cues Drive Most Social Biases in MLLMs
Jun 18, 2026Shaghayegh Kolli, Timo Cavelius, Nafiseh Nikeghbal +2Social Bias in Language Models Multimodal Large Language Models
Technical University of Munich · Munich Center for Machine Learning · Princeton Center for Information and Technology Policy
ROSE: Benchmarking the Perception-to-Action Gap in Multimodal Models
Jun 18, 2026Yihao Wang, Zijian He, Jie Ren +1Multimodal Grounding Multimodal Reasoning
Sun Yat-sen University · Shaanxi Normal University
Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games
Jun 17, 2026Shengyuan Ding, Xilin Wei, Xinyu Fang +4Multimodal Memory Sequential Decision Making
1Fudan University · 2Shanghai Innovation Institute · 3Shanghai Artificial Intelligence Laboratory +2
Million-scale multimodal pollen microscopy with expert-guided foundation models
Jun 16, 2026András Biricz, Björn Gedda, Donát Magyar +4Domain Adaptation Multimodal Model Evaluation
Department of Physics of Complex Systems, ELTE Eötvös Lor´and University, Budapest, Hungary · 2The Palynological Laboratory at the Swedish Museum of Natural History, Stockholm, Sweden · 3National Centre for Public Health and Pharmacy, Budapest, Hungary +4
The Slop Paradox: How Synthetic Standardization Erodes Clinical Uncertainty and Cross-Modal Alignment in AI-Rewritten Radiology Reports
Jun 16, 2026Samar AnsariMedical VLMs Vision-Language Alignment
School of Computing and Engineering Sciences University of Chester Chester, CH1 4BJ, United Kingdom
MODE-RAG: Manifold Outlier Diagnosis and Energy-based Retrieval-Augmented Generation Evaluation
Jun 16, 2026Zehang Wei, Jiaxin Dai, Jiamin Yan +1Multimodal RAG Multi-Agent LLM Systems
School of AI and Automation, Huazhong University of Science and Technology, China · School of Computer Science & Tech, Huazhong University of Science and Technology
Multimodal Evaluator Preference Collapse: Cross-Modal Coupling in Self-Evolving Agents
Jun 15, 2026Zewen LiuAI Agent Evaluation Multimodal Model Evaluation
Qilu Institute of Technology, School of Software Engineering · Tai’an, Shandong, China
Mind the Gap: Diagnosing Constraint Discovery Failures in Text-in-Image Editing
Jun 14, 2026Rui GuiText-Guided Image Editing Multimodal Large Language Models
Central South University
UrbanWell: Benchmarking Multimodal Large Language Models for Spatio-Temporal Urban Wellbeing Analytics
Jun 14, 2026Yanxin Xi, Xiang Su, Jie Feng +3Spatiotemporal Reasoning Multimodal Large Language Models
Department of Computer Science, University of Helsinki, Helsinki, Finland · Department of Computer Science, Department of Agricultural Sciences, University of Helsinki, Helsinki, Finland · Zhongguancun Academy, Beijing, China +2
OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning
Jun 14, 2026Maonan Wang, Zhengyan Huang, Kemou Jiang +13Autonomous Driving Datasets Spatiotemporal Reasoning
The Chinese University of Hong Kong, Shenzhen · Shanghai AI Lab · Beihang University +3
On the Adversarial Robustness of Multimodal LLM Judges
Jun 14, 2026Zihan Wang, Guansong Pang, Zelin Liu +3LLM-as-a-Judge Adversarial Robustness
School of Computer Science and Engineering, Beihang University, Beijing, China · School of Computing and Information Systems, Singapore Management University, Singapore · State Key Laboratory of Virtual Reality Technology and System, Beihang University, Beijing, China +1

Jun 24, 2026, cs.CLJ/K move · Enter open · S save

Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models

Akshay Paruchuri, Sanmi Koyejo, Ehsan Adeli

Stanford University

Multimodal Robustness LLM Auditing Multimodal Model Evaluation Multimodal Understanding

Open paper AskPDF arXiv

Standard benchmarks for multimodal large language models (MLLMs) score each item on one canonical ordering and miss whether order-irrelevant shuffling changes the answer, a baseline reliability property called for by emerging AI evaluation guidelines. We introduce Facet-Probe, a five-facet audit (option, evidence-chunk, document-rank, image-set, and mixed-modality ordering) of 18 frontier and open-weight MLLMs. A Bayesian item-response model separates ordering noise from per-facet bias, and a same-ordering control estimates the decoder-stochastic floor for observed flips. We find that none of the 18 MLLMs we audit are order-invariant: screened per-facet panel-mean flip rates span 24-50%. A Gemini same-ordering control at temperature 0 estimates a substantial ordering excess over a same-input decoder-noise floor in verified cells. Capability predicts but does not eliminate flips; the best model still flips on 13.4% of trials. In our Gemini mitigation tests, training-free prompt changes are modality-conditional and do not transfer from text to visual reasoning. These results suggest that prompt-level mitigation alone is unlikely to provide general order robustness, motivating future work on training-time and architectural approaches. We propose cross-ordering flip rate as a standard reporting axis for MLLMs.

Momentum

Related topics

Latest papers 496