Multimodal Model Evaluation

Follow this topic Ask about this topic

Momentum

49 papers in the last four weeks, up 44% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 496

All topics

Towards Verifiable Multimodal Deep Research: A Multi-Agent Harness for Interleaved Report Generation
May 28, 2026Chenghao Zhang, Guanting Dong, Yufan Liu +3Multimodal Generation Multimodal Agents
Gaoling School of Artificial Intelligence, Renmin University of China
OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields
May 28, 2026Wanhao Liu, Jiaqing Xie, Qian Tan +10Benchmark Design Scientific QA
University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · Fudan University +4
CardioLens: Revealing the Clinical Reality Gap of MLLMs via Multi-Sequence Cardiac MRI Evaluations
May 28, 2026Zixian Su, Hongkai Zhang, Fan Gao +12Magnetic Resonance Imaging Medical VLMs
Beijing Academy of Artificial Intelligence · Beijing Anzhen Hospital · Beihang University +1
ReactBench: A Cause-Driven Benchmark for Multimodal Hallucination via Systematic Evaluation
May 28, 2026Shizhe Zhou, Bohan Jia, Kai Wu +4VLM Robustness Hallucination Detection
East China Normal University
Benchmarking Large Vision-Language Models on CFMME: A Comprehensive Chinese Financial Multimodal Evaluation Dataset
May 28, 2026Qian Chen, Xianyin Zhang, Yanzhi Liu +3VLM Evaluation Vision-Language Models
Qwen DianJin Team, Alibaba Cloud Computing
OmniVerifier-M1: Multimodal Meta-Verifier with Explicit Structured Recalibration
May 27, 2026Xinchen Zhang, Bowei Liu, Jiale Liu +7Multimodal Large Language Models Reinforcement Learning with Verifiable Rewards
Tsinghua University · Pennsylvania State University · University of Southern California +2
IPO-Mine: A Toolkit and Dataset for Section-Structured Analysis of Long, Multimodal IPO Documents
May 27, 2026Michael Galarnyk, Siddharth Lohani, Vidhyakshaya Kannan +8Document Information Extraction Multimodal Model Evaluation
Georgia Institute of Technology Atlanta, GA, USA · Sai University Chennai, Tamil Nadu, India · Duke University Durham, NC, USA
Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation
May 27, 2026Niantong Li, Guangzheng Hu, Weixu Qiao +35Image Generation Multimodal Model Evaluation
Alibaba Inc.
MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generation
May 27, 2026Haitian Li, Yanghao Zhou, Heyan Huang +15Audio-Video Generation Multimodal Model Evaluation
Shanghai University · Tsinghua University · Beijing Institute of Technology +7
KSAFE-MM: A Multimodal Safety Benchmark via Localized Contextualization for Korean Cultural Risks
May 27, 2026Yongwoo Kim, Sojung An, Yunjin Park +8Adversarial Attacks on VLMs Language Model Safety Evaluation
Korea University · KT Corporation
Pop-Up Distractions Reveal Bag-of-Events Behavior in Video Large Language Models
May 26, 2026Oscar Chew, Serhii Honcharenko, Qian-Hui Chen +4Temporal Video Grounding Video-Language Model Evaluation
Texas A&M University · National Taiwan University · Stanford University +1
ChartAct: A Benchmark for Dynamic Chart Understanding
May 26, 2026Muye Huang, Lin Wu, Lingling Zhang +5Data Visualization Chart QA
School of Computer Science and Technology, Xi’an Jiaotong University · 2MOE KLNN Lab, Xi’an Jiaotong University
LiveK12Bench: Have Large Multimodal Models Truly Conquered High School-level Examinations?
May 26, 2026Xiaohan Wang, Mingze Yin, Yilin Zhao +2Mathematical Reasoning Benchmarks Educational Assessment
Tencent PCG · College of Computer Science and Technology, Zhejiang University
OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants
May 26, 2026Xudong Lu, Xueying Li, Annan Wang +8Streaming Video Understanding Multimodal Foundation Models
CUHK MMLab · SJTU · NTU +3
Advancing Creative Physical Intelligence in Large Multimodal Models
May 25, 2026Cheng Qian, Hyeonjeong Ha, Jiayu Liu +10Multimodal Grounding Object Hallucination in VLMs
1UIUC · 2Amazon
RAPTOR+: A Visually Grounded Vision-Language Framework to Improve Clinical Trust and Auditability in Automated Cancer Referral Processing
May 25, 2026Sofiat Abioye, Ufaq Khan, Shazad Ashraf +4Document Understanding Healthcare
Birmingham City University, Birmingham, England, UK · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE · University Hospitals Birmingham NHS Foundation Trust, Birmingham, England, UK +1
Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning
May 25, 2026Longteng Guo, Yifan Wang, Pengkang Huo +4Visual Reasoning VLM Reasoning
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
OmniEgo-R $^2$ : A Routed Reasoning Framework for the 1st Cross-Domain EgoCross Challenge at CVPR 2026
May 23, 2026Zixu Li, Zhiwei Chen, Zhiheng Fu +4Egocentric Video QA Egocentric Video Understanding
Shandong University · Harbin Institute of Technology (Shenzhen)
ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models
May 22, 2026Fen Wang, Zekai Shao, Qiman Kang +5Data Visualization Multimodal Large Language Models
School of Data Science, Fudan University · Zhengzhou Zhongke Institute of Integrated Circuit and System Application · School of Computer Science, Fudan University
DDX-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMs
May 22, 2026Jiazhen Pan, Weixiang Shen, Jun Li +7Medical Imaging Radiology VLMs
Technical University of Munich (TUM) · TUM University Hospital · Munich Center for Machine Learning (MCML) +3
RE-TRIANGLE: Does TRIANGLE Enable Multimodal Alignment Beyond Cosine Similarity in Retrieval?
May 22, 2026Arijit Ghosh, Aritra Bandyopadhyay, Chiranjeev Bindra +1Cross-Modal Learning Cross-Modal Alignment
University of Amsterdam Amsterdam, The Netherlands
Emotion Recognition in Sign Language Conversation
May 22, 2026Yusong Wang, Keyu Mao, Takao Obi +2Emotion Recognition in Conversations Sign Language Recognition
Institute of Science Tokyo, Japan · New York University, New York, 11201, USA
Same Model, Different Weakness: How Language and Modality Reshape the Jailbreak Attack Surface in Frontier MLLMs
May 22, 2026Casey Ford, Madison Van Doren, Sicheng Jin +1Multilingual Language Model Evaluation Adversarial Attacks on VLMs
Appen
Pointwise Metrics Mislead: An Evaluation Protocol for Multimodal Inverse Problems
May 21, 2026Mads H. Baattrup, Jörn Bach, Laurids Jeppe +4Bayesian Inverse Problems Multimodal Model Evaluation
Deutsches Elektronen-Synchrotron Hamburg, Germany · CERN Geneva, Switzerland · University of Hamburg Hamburg, Germany +1
AgroTools: A Benchmark for Tool-Augmented Multimodal Agents in Agriculture
May 21, 2026Zi Ye, Yibin Wen, Xiaoya Fan +10AI Agent Benchmarks Tool-Using Vision-Language Agents
1Sun Yat-Sen University · 2Southwest University · 3Northeastern University +4
Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality?
May 21, 2026Caixin Kang, Tianyu Yan, Sitong Gong +8LLM Grounding Personality Modeling in Language Models
1The University of Tokyo · 2Shanda AI Research Tokyo · 3Dalian University of Technology
Seizure-Semiology-Suite (S3): A Clinically Multimodal Dataset, Benchmark, and Models for Seizure Semiology Understanding
May 21, 2026Lina Zhang, Tonmoy Monsoor, Peizheng Li +23Video Understanding Multimodal Large Language Models
University of California, Los Angeles · Mercedes-Benz AG · University of Pittsburgh +4
EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs
May 19, 2026Yang Dai, Dian Jiao, Tianwei Lin +1Egocentric Video QA CoT Reasoning
Zhejiang University · Hangzhou, Zhejiang, China
MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systems
May 19, 2026Quanxing Xu, Yuhao Tian, Ling Zhou +4Visual Question Answering Automated Software Testing
School of Computer Science and Engineering, Macau University of Science and Technology, Macao SAR 999078, China · Hubei Key Laboratory of Transportation Internet of Things, School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan 430070, China · State Key Laboratory of Maritime Technology and Safety, Wuhan University of Technology, Wuhan 430063, China +3
HAVEN: Hierarchically Aligned Multimodal Benchmark for Unified Video Understanding
May 19, 2026Mengqi Shi, Haopeng ZhangMultimodal Grounding Video Summarization
Department of Information and Computer Sciences · University of Hawaii at Manoa

May 28, 2026, cs.CLJ/K move · Enter open · S save

Towards Verifiable Multimodal Deep Research: A Multi-Agent Harness for Interleaved Report Generation

Chenghao Zhang, Guanting Dong, Yufan Liu, Tong Zhao+2

Gaoling School of Artificial Intelligence, Renmin University of China

Multimodal Generation Multimodal Agents Multi-Agent LLMs Multimodal Model Evaluation Multimodal Tool Use

Open paper AskPDF arXiv

Large Language Models (LLMs) have advanced autonomous agents from deep search, which retrieves concise factual answers, to deep research, which synthesizes scattered evidence into long-form reports. However, verifiable multimodal deep research remains challenging due to open-ended synthesis without deterministic ground truth and the need to interleave textual arguments with visual evidence. We propose Ptah, a multi-agent harness for interleaved report generation. Ptah orchestrates the lifecycle from user query to rendered web report through planning, research, and writing stages, where specialized agents construct visual-aware plans, collect claim-grounded evidence, maintain source-aligned images in a Visual Working Memory, and compose reports through declarative multimodal tool use. A verifier agent serves as the harness's acceptance function, enforcing factual grounding, citation fidelity, and cross-modal consistency throughout the workflow. We further introduce PtahEval, an evaluation protocol that augments existing benchmarks with image-level and presentation-level assessments. Experiments on deep research benchmarks show that Ptah produces more reliable, visually informative, and usable human-facing multimodal reports than strong baselines. Our code is released at https://github.com/SnowNation101/Ptah

Momentum

Related topics

Latest papers 496