Multimodal Model Evaluation

Follow this topic Ask about this topic

Momentum

49 papers in the last four weeks, up 48% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 496

All topics

Right Answer, Wrong Heat: Explanation-Aware Evaluation and Thermal-Grounded Feedback for MLLMs on Infrared Images
Aug 10, 2026Yongsong Huang, Xiaofeng Liu, Tomo Miyazaki +2Vision-Language Models Vision-Language Grounding
Tohoku University · Yale University
FitAQA: A Benchmark of Fitness Action Quality Assessment for Multimodal Large Language Models
Aug 9, 2026Kaili Zheng, Kaiwen Wang, Xun Zhu +3Multimodal Large Language Models Action Quality Assessment
Department of Electronic Engineering, Tsinghua University · Xinjiang Region Sports Science Research Center · College of AI, Tsinghua University +1
TomaMMU: A Comprehensive Multimodal Understanding Benchmark for Tomato Leaf Diseases
Aug 9, 2026Gia-Han Truong, Khang Nguyen Quoc, Luyl-Da QuachVision-Language Models Plant Disease Classification
Department of Information Technology, FPT University, Can Tho, Vietnam · School of Electrical Engineering, Korea University, Seoul, South Korea
Large Multimodal Agents for Intelligent Transportation Systems: Architectures, Evidence, and Deployment Challenges
Aug 8, 2026Muhammad Ayub Sabir, Shaohong Zheng, Zhiyu Qu +2Intelligent Transportation Systems Multimodal Agents
Locating Failure in Multi-Page Visually Rich Document Understanding: An Empirical Attribution
Aug 8, 2026Lewei Xu, Yihao Ding, Zihan Xu +5Document Understanding Multimodal Reasoning
The University of Western Australia · The University of Melbourne · Weill Cornell Medicine
Forged Peer Judgments Mislead Multimodal LLM Judge Panels: Source-Blind Anchoring and Panel-Consensus Verification
Aug 8, 2026Yang ShuLLM-as-a-Judge Adversarial Attacks on LLMs
Zhejiang University, Hangzhou, China
LMM Modality Transfer: A Pre-requisite for Autonomous GIS Agents
Aug 7, 2026Ivan Majic, Zexian Huang, Franziska Hübl +5Cross-Modal Alignment Multimodal Large Language Models
Graz University of Technology, Austria · University of Vienna, Austria · University of Liverpool, UK
Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery
Aug 7, 2026Taolin Han, Yuchen Zhang, Jinghang Wang +22Scientific Reasoning Scientific Discovery
1Alibaba Group · University of Chinese Academy of Sciences · Taolin Han1,3* +28
Can MLLMs Decode the Creative Leap? Introducing C4 for Cross-Concept Understanding
Aug 6, 2026Ming Wang, Yuqing Zhang, Tingna Xie +5Creativity Assessment Multimodal Large Language Models
School of Computer Science and Engineering, Northeastern University, Liaoning, China · School of Computing and Information Systems, Singapore Management University, Singapore · School of Computer Science and Engineering, Shandong University of Science and Technology, Shandong, China
Seeing Is Not Deciding: Can Multimodal LLMs Act as Effective CEOs?
Aug 6, 2026Yuyang Dai, Xueqing Peng, Yuxia Wang +2Multimodal Grounding LLM Decision-Making
INSAIT, Sofia University “St. Kliment Ohridski”, Bulgaria · Yale University, New Haven, CT, USA · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
M $^3$ R-Bench: A Unified Benchmark for Evidence-Grounded Multimodal Metaphor Understanding
Aug 6, 2026Hong Jiang, Junnan Zhu, Jingwang Huang +9Figurative Language Understanding Multimodal Reasoning
School of Computer Science, Chongqing University, Chongqing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore +3
C $^3$ PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models
Aug 5, 2026Swapnanil Mukherjee, Agyeya Negi, Tanuja Ganu +1Modality Imbalance Multimodal Reasoning
Microsoft Research India · IIIT Hyderabad, India
The First EgoCross Challenge at EgoVis 2026: Cross-Domain Egocentric Video Question Answering
Aug 5, 2026Yuqian Fu, Tianwen Qian, Yanjun Li +30Egocentric Video QA Video QA
RESPClinBench: Benchmarking Multimodal Clinical Decision-Making and Longitudinal Disease Management in Respiratory Specialty Care
Aug 5, 2026Mouxiao Bian, Zhi Chen, Ruiyao Chen +8LLM Safety Benchmarks Healthcare
Shanghai Artificial Intelligence Laboratory, Shanhai, China · Macau University of Science and Technology, Macau, China · First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China +1
SIGNPOST-Bench: Benchmarking Text-Vision Conflict Resolution in Multimodal Large Language Models
Aug 4, 2026Sirun Li, Minghao Liu, Ling Dai +4VLM Robustness Multimodal Large Language Models
Peking University, Beijing, China · 22077AI · The Hong Kong University of Science and Technology, Hong Kong, China
GEB-Bench: Abstract Structures Told in Many Voices
Aug 4, 2026Tong Zhang, Zhiyuan Shi, Yun Peng +1Analogical Reasoning Multimodal Reasoning
Fudan University · Peking University
Geo-Embed: Towards Unified Multimodal Embeddings for Urban Understanding
Aug 4, 2026Jiapeng Li, Yong Li, Junjie Zhou +2Geospatial Representation Learning Cross-Modal Retrieval
1Peking University · 2Beijing University of Posts and Telecommunications
KnowHal: A Knowledge-Driven Benchmark for Comprehensive Multimodal Hallucination Evaluation
Aug 4, 2026Ruihan Li, Jiyang Tan, Kailin Jiang +5Multimodal Hallucination Multimodal Model Evaluation
1C-FAIR & School of software, Shandong University · University of Science and Technology of China · 3Jiangnan University
Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation
Aug 4, 2026Khai-Nguyen Nguyen, Oscar Chaparro, Antonio MastropaoloCode Generation Multimodal Large Language Models
William & Mary Williamsburg, USA · University of Virginia
ChartAnno: Benchmarking Multimodal Large Language Models for Chart Annotation Generation
Aug 4, 2026Zhenghan Chen, Zekai Shao, Lidan Tan +10Data Visualization Multimodal Large Language Models
Fudan University · Sun Yat-sen University · The Hong Kong University of Science and Technology (Guangzhou) +1
ArtECulture: Benchmarking Culture-Conditioned Visual Emotion Understanding in Multimodal Large Language Models
Aug 4, 2026Xiaolin Chen, Xuemeng Song, Wenhao Shi +3Multimodal Large Language Models Multimodal Emotion Recognition
National University of Singapore · Southern University of Science and Technology · Nanyang Technological University
LDU-Bench: Multimodal LLM Evaluation for Lithography Defect Understanding under Layout-Varying Circuit Backgrounds
Aug 4, 2026Huanglong Ji, Botong Zhao, Shujing Lv +1Semiconductor Manufacturing Multimodal Large Language Models
MIEScore: Human-Aligned Evaluation for Multi-Source Image Editing
Aug 3, 2026Zitong Xu, Huiyu Duan, Xinyun Zhang +7Image Editing Evaluation Text-Guided Image Editing
Shanghai Jiao Tong University · Vivo Mobile Communication Co., Ltd · University of Electronic and Science Technology of China
SVGEval: A Vision-Grounded Framework for Perceptual-Quality Benchmarking and Evaluation in Text-to-SVG Generation
Aug 3, 2026Yiming Wang, Ye Chen, Hanqi Chen +1Multimodal Model Evaluation
School of Integrated Circuits (School of Information Science and Electronic Engineering), Shanghai Jiao Tong University, Shanghai, China · Zhiyuan College, Shanghai Jiao Tong University, Shanghai, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen 518107, China
Decoupling semantics from vision: A framework for faithful visual-text compression evaluation
Aug 3, 2026Yonghan Gao, Zehong Chen, Lijian Xu +3Multimodal Model Evaluation Visual Token Compression
Shenzhen University of Advanced Technology · Shenzhen Technology University
Loud or Silent? A Reusable Framework for Per-Modality Failure Analysis in Multimodal Clinical AI
Aug 2, 2026Quang Bui, Shlok Jaiswal, Samuel Paik-Heintz +14Missing-Modality Learning Multimodal Robustness
LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning
Aug 2, 2026Ziyan Xiao, Yinghao Zhu, Wenting Zhang +2Data Visualization Visual Reasoning
The University of Hong Kong
Where did the ambiguity go? Examining how multimodal models interpret polysemous words
Aug 1, 2026Jasin Cekinmez, Addison J. Wu, Raja Marjieh +1Multimodal Generation Modality Gap in VLMs
Princeton University
FriendBench: Benchmarking Dyadic Familiarity Inference in Humans and Multimodal Large Language Models
Jul 31, 2026Jeffrey M. Girard, Jason Z. Zheng, Jacqueline R. Vertino +2Classification Multimodal Model Evaluation
Fluid Concepts Research
LEDGERMIND: Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger
Jul 30, 2026Enjun Du, Hange Zhou, Chenxu Du +4Multimodal Grounding Multimodal QA
The Hong Kong University of Science and Technology (Guangzhou) · The University of Hong Kong · Tsinghua University +1

Aug 10, 2026, cs.CVJ/K move · Enter open · S save

Right Answer, Wrong Heat: Explanation-Aware Evaluation and Thermal-Grounded Feedback for MLLMs on Infrared Images

Yongsong Huang, Xiaofeng Liu, Tomo Miyazaki, Yaohou Fan+1

Tohoku University · Yale University

Vision-Language Models Vision-Language Grounding Explainability Evaluation Multimodal Model Evaluation

Open paper AskPDF arXiv

General-purpose multimodal large language models (MLLMs) are increasingly applied to infrared images, where they are commonly scored by answer accuracy alone. However, a correct answer does not ensure that the model's explanation is grounded in infrared thermal evidence. We introduce an explanation-aware evaluation framework that separates answer correctness, output-level explanation groundedness, and thermal grounding for infrared visual questions. Using a Dual-LLM Consensus Judge with a preliminary human-anchor calibration check, we find that correct answers can still rely on weak or visible-light evidence; withholding the original infrared image and showing only a visible-like rendering erodes thermal grounding with little accuracy change; and this erosion is observed most strongly for more capable models but disappears when infrared remains available. We further propose Thermal-Grounded Feedback (TGF), a training-free feedback loop that diagnoses explanation-side failures and revises the explanation while preserving the selected answer. On local paired-input validation, TGF improves explanation-side grounding without changing answers. These findings suggest that future trustworthy MLLMs for infrared scene understanding should be evaluated and developed to produce thermally grounded explanations rather than merely accurate answers.

Momentum

Related topics

Latest papers 496