Multimodal Model Evaluation

Follow this topic Ask about this topic

Momentum

49 papers in the last four weeks, up 48% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 496

All topics

OmniVerifier-M1: Multimodal Meta-Verifier with Explicit Structured Recalibration
May 27, 2026Xinchen Zhang, Bowei Liu, Jiale Liu +7Multimodal Large Language Models Reinforcement Learning with Verifiable Rewards
Tsinghua University · Pennsylvania State University · University of Southern California +2
IPO-Mine: A Toolkit and Dataset for Section-Structured Analysis of Long, Multimodal IPO Documents
May 27, 2026Michael Galarnyk, Siddharth Lohani, Vidhyakshaya Kannan +8Document Information Extraction Multimodal Model Evaluation
Georgia Institute of Technology Atlanta, GA, USA · Sai University Chennai, Tamil Nadu, India · Duke University Durham, NC, USA
Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation
May 27, 2026Niantong Li, Guangzheng Hu, Weixu Qiao +35Image Generation Multimodal Model Evaluation
Alibaba Inc.
MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generation
May 27, 2026Haitian Li, Yanghao Zhou, Heyan Huang +15Audio-Video Generation Multimodal Model Evaluation
Shanghai University · Tsinghua University · Beijing Institute of Technology +7
KSAFE-MM: A Multimodal Safety Benchmark via Localized Contextualization for Korean Cultural Risks
May 27, 2026Yongwoo Kim, Sojung An, Yunjin Park +8Adversarial Attacks on VLMs Language Model Safety Evaluation
Korea University · KT Corporation
Pop-Up Distractions Reveal Bag-of-Events Behavior in Video Large Language Models
May 26, 2026Oscar Chew, Serhii Honcharenko, Qian-Hui Chen +4Temporal Video Grounding Video-Language Model Evaluation
Texas A&M University · National Taiwan University · Stanford University +1
ChartAct: A Benchmark for Dynamic Chart Understanding
May 26, 2026Muye Huang, Lin Wu, Lingling Zhang +5Data Visualization Chart QA
School of Computer Science and Technology, Xi’an Jiaotong University · 2MOE KLNN Lab, Xi’an Jiaotong University
LiveK12Bench: Have Large Multimodal Models Truly Conquered High School-level Examinations?
May 26, 2026Xiaohan Wang, Mingze Yin, Yilin Zhao +2Mathematical Reasoning Benchmarks Educational Assessment
Tencent PCG · College of Computer Science and Technology, Zhejiang University
OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants
May 26, 2026Xudong Lu, Xueying Li, Annan Wang +8Streaming Video Understanding Multimodal Foundation Models
CUHK MMLab · SJTU · NTU +3
Advancing Creative Physical Intelligence in Large Multimodal Models
May 25, 2026Cheng Qian, Hyeonjeong Ha, Jiayu Liu +10Multimodal Grounding Object Hallucination in VLMs
1UIUC · 2Amazon
RAPTOR+: A Visually Grounded Vision-Language Framework to Improve Clinical Trust and Auditability in Automated Cancer Referral Processing
May 25, 2026Sofiat Abioye, Ufaq Khan, Shazad Ashraf +4Document Understanding Healthcare
Birmingham City University, Birmingham, England, UK · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE · University Hospitals Birmingham NHS Foundation Trust, Birmingham, England, UK +1
Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning
May 25, 2026Longteng Guo, Yifan Wang, Pengkang Huo +4Visual Reasoning VLM Reasoning
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
OmniEgo-R $^2$ : A Routed Reasoning Framework for the 1st Cross-Domain EgoCross Challenge at CVPR 2026
May 23, 2026Zixu Li, Zhiwei Chen, Zhiheng Fu +4Egocentric Video QA Egocentric Video Understanding
Shandong University · Harbin Institute of Technology (Shenzhen)
ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models
May 22, 2026Fen Wang, Zekai Shao, Qiman Kang +5Data Visualization Multimodal Large Language Models
School of Data Science, Fudan University · Zhengzhou Zhongke Institute of Integrated Circuit and System Application · School of Computer Science, Fudan University
DDX-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMs
May 22, 2026Jiazhen Pan, Weixiang Shen, Jun Li +7Medical Imaging Radiology VLMs
Technical University of Munich (TUM) · TUM University Hospital · Munich Center for Machine Learning (MCML) +3
RE-TRIANGLE: Does TRIANGLE Enable Multimodal Alignment Beyond Cosine Similarity in Retrieval?
May 22, 2026Arijit Ghosh, Aritra Bandyopadhyay, Chiranjeev Bindra +1Cross-Modal Learning Cross-Modal Alignment
University of Amsterdam Amsterdam, The Netherlands
Emotion Recognition in Sign Language Conversation
May 22, 2026Yusong Wang, Keyu Mao, Takao Obi +2Emotion Recognition in Conversations Sign Language Recognition
Institute of Science Tokyo, Japan · New York University, New York, 11201, USA
Same Model, Different Weakness: How Language and Modality Reshape the Jailbreak Attack Surface in Frontier MLLMs
May 22, 2026Casey Ford, Madison Van Doren, Sicheng Jin +1Multilingual Language Model Evaluation Adversarial Attacks on VLMs
Appen
Pointwise Metrics Mislead: An Evaluation Protocol for Multimodal Inverse Problems
May 21, 2026Mads H. Baattrup, Jörn Bach, Laurids Jeppe +4Bayesian Inverse Problems Multimodal Model Evaluation
Deutsches Elektronen-Synchrotron Hamburg, Germany · CERN Geneva, Switzerland · University of Hamburg Hamburg, Germany +1
AgroTools: A Benchmark for Tool-Augmented Multimodal Agents in Agriculture
May 21, 2026Zi Ye, Yibin Wen, Xiaoya Fan +10AI Agent Benchmarks Tool-Using Vision-Language Agents
1Sun Yat-Sen University · 2Southwest University · 3Northeastern University +4
Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality?
May 21, 2026Caixin Kang, Tianyu Yan, Sitong Gong +8LLM Grounding Personality Modeling in Language Models
1The University of Tokyo · 2Shanda AI Research Tokyo · 3Dalian University of Technology
Seizure-Semiology-Suite (S3): A Clinically Multimodal Dataset, Benchmark, and Models for Seizure Semiology Understanding
May 21, 2026Lina Zhang, Tonmoy Monsoor, Peizheng Li +23Video Understanding Multimodal Large Language Models
University of California, Los Angeles · Mercedes-Benz AG · University of Pittsburgh +4
EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs
May 19, 2026Yang Dai, Dian Jiao, Tianwei Lin +1Egocentric Video QA CoT Reasoning
Zhejiang University · Hangzhou, Zhejiang, China
MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systems
May 19, 2026Quanxing Xu, Yuhao Tian, Ling Zhou +4Visual Question Answering Automated Software Testing
School of Computer Science and Engineering, Macau University of Science and Technology, Macao SAR 999078, China · Hubei Key Laboratory of Transportation Internet of Things, School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan 430070, China · State Key Laboratory of Maritime Technology and Safety, Wuhan University of Technology, Wuhan 430063, China +3
HAVEN: Hierarchically Aligned Multimodal Benchmark for Unified Video Understanding
May 19, 2026Mengqi Shi, Haopeng ZhangMultimodal Grounding Video Summarization
Department of Information and Computer Sciences · University of Hawaii at Manoa
OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understanding
May 18, 2026Ruixiang Zhao, Jie Yang, Zijie Xin +4Audio-Visual Understanding Streaming Video Understanding
Renmin University of China · WeChat Vision, Tencent Inc.
Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction
May 17, 2026Chaoqun He, Mingyang Xiang, Yingjing Xu +5Multimodal Large Language Models Multimodal Agents
Tsinghua University · Shanghai Qi Zhi Institute · ModelBest Inc.
EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning
May 17, 2026Zeyu Wang, Chang Liu, Eduardus Tjitrahardja +22Multimodal Large Language Models Egocentric Video Understanding
Tsinghua University · Shanghai Qi Zhi Institute · Tongji University +5
PluRule: A Benchmark for Moderating Pluralistic Communities on Social Media
May 16, 2026Zoher Kachwala, Bao Tran Truong, Rasika Muralidharan +3Multilingual Language Model Evaluation Content Moderation
Observatory on Social Media, Indiana University, USA · Center Synergy of Systems, TUD Dresden University of Technology, Germany
GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions
May 15, 2026Junho Kim, Xu Cao, Houze Yang +6Multimodal QA Multimodal Model Evaluation
University of Illinois Urbana-Champaign · Georgia Institute of Technology · Amazon AGI +1

May 27, 2026, cs.CLJ/K move · Enter open · S save

OmniVerifier-M1: Multimodal Meta-Verifier with Explicit Structured Recalibration

Xinchen Zhang, Bowei Liu, Jiale Liu, Chufan Shi+6

Tsinghua University · Pennsylvania State University · University of Southern California+2

Multimodal Large Language Models Reinforcement Learning with Verifiable Rewards Multimodal Foundation Models Multimodal Model Evaluation

Open paper AskPDF arXiv

Visual outcomes are increasingly central to multimodal large language models, making reliable and fine-grained verification essential for scaling generalist foundation models. In this work, we investigate multimodal meta-verification, which leverages verifier-generated rationales rather than decision-only signals, and explore how to effectively incorporate meta-verification feedback into multimodal verifier training. We identify two key findings. First, symbolic verifier outputs (e.g., bounding boxes) outperform textual explanations as meta-verification rationales, enabling efficient rule-based reinforcement learning rewards while avoiding reliance on model-based rewards from auxiliary judge models. Second, decoupling reinforcement learning objectives for binary judgment and meta-verification substantially outperforms joint reward optimization, due to intrinsic differences in output structure and learning dynamics. Based on these insights, we train OmniVerifier-M1, a generalist visual verifier leveraging symbolic meta-verification and decoupled reinforcement learning. OmniVerifier-M1 provides robust verification and fine-grained error localization, and further enables M1-TTS, a verifier-driven agentic generation system achieving dynamic region-level self-correction. This approach paves the way for more reliable, interpretable, and fine-grained multimodal verification, supporting safer and more controllable foundation model deployment.

Momentum

Related topics

Latest papers 496