Multimodal Model Evaluation

Follow this topic Ask about this topic

Momentum

49 papers in the last four weeks, up 44% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 503

All topics

Conditional Multi-Event Temporal Grounding in Long-Form Video
Jun 13, 2026Yuanhao Zou, Arthad Kulkarni, Lucas Tonanez +12Temporal Video Grounding Long-Video Understanding
University of Central Florida · 2Qualcomm AI Research
MVEB: Massive Video Embedding Benchmark
Jun 12, 2026Adnan El Assadi, Roman Solomatin, Isaac Chung +13Multimodal Embedding Video Representation Learning
1Harvard University · 2SaluteDevices · 3MIRAI +11
ClinHallu: A Benchmark for Diagnosing Stage-Wise Hallucinations in Medical MLLM Reasoning
Jun 12, 2026Sicheng Yang, Hangjie Yuan, Wenjun Zhang +5Multimodal Hallucination Clinical Reasoning
The Hong Kong University of Science and Technology (Guangzhou) · DAMO Academy, Alibaba Group · Hupan Lab +1
Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity
Jun 12, 2026Daniel Lee, Harsh Sharma, Eunkyu Park +5LLM-as-a-Judge Language Model Calibration
Salesforce AI Research · University of Colorado Boulder · Carnegie Mellon University +2
IndustryBench-MIPU: Benchmarking Multi-Image Attribute Value Extraction for Industrial Products
Jun 12, 2026Haonan Qi, Jin Cao, Yongqi Zhang +9Multimodal Large Language Models Multimodal Model Evaluation
Multimodal and Industrial AI TeamTaobao&Tmall, Alibaba Group
Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach
Jun 11, 2026Ruichao Mao, Zhou Fang, Teng Guo +9Multimodal Reasoning Multimodal Model Evaluation
Ant Group
Multi-Modal Agents for Power Distribution Defect Detection: An Evaluation of Foundation Models
Jun 11, 2026Quan QuanIndustrial Anomaly Detection Multimodal Reasoning
Emerging Flexible Designs for Geospatial Multimodal Foundation Models
Jun 10, 2026Philipe Dias, Waqwoya Abebe, Abhishek Potnis +4Geospatial Foundation Models Multimodal Foundation Models
Oak Ridge National Laboratory USA
MMClima: A Framework for Multimodal Climate Science Data and Evaluation
Jun 8, 2026Muhammad Umer Sheikh, Hassan Abid, Khawar Shehzad +2Climate Science Scientific QA
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates · University of Missouri, Columbia, Missouri, United States
TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs
Jun 8, 2026Momina Ahsan, Sarfraz Ahmad, Ming Shan Hee +2Document Understanding Table QA
1Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · 2Singapore University of Technology and Design (SUTD)
Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?
Jun 8, 2026Apratim Bhattacharyya, Shweta Mahajan, Sanjay Haresh +5LLM Fine-Tuning Synthetic Data Generation
Qualcomm AI Research · York University · Vector Institute for AI
IMUG-Bench: Benchmarking Unified Multimodal Models on Interleaved Understanding and Generation
Jun 8, 2026Lingyi Meng, Zecong Tang, Haoran Li +12Unified Multimodal Models Multimodal Generation
Zhejiang University · Institute of Automation, Chinese Academy of Sciences · The University of Hong Kong +1
Unveiling Privacy Risks in Multi-modal Large Language Models: Task-specific Vulnerabilities and Mitigation Challenges
Jun 8, 2026Tiejin Chen, Pingzhi Li, Kaixiong Zhou +2Multimodal Large Language Models Privacy Leakage in Language Models
Arizona State University · University of North Carolina at Chapel Hill · North Carolina State University
OmniCap-IF: Benchmarking and Improving Instruction Following Abilities for Omni-Video Captioning
Jun 7, 2026Jiahao Wang, An Ping, Yanghai Wang +13Video Captioning Multimodal Generation
NJU-LINK Team, Nanjing University · Kling Team, Kuaishou Technology
When Correct Decisions Hide Internal Stress: Decision-State Probing in Multimodal Language Models
Jun 7, 2026Haoran Zhao, Soyeon Caren Han, Eduard HovyMultimodal Robustness Multimodal Large Language Models
1The University of Melbourne
When No Answer Is Correct: Diagnosing Absent Answer Detection for MLLMs in Video Understanding
Jun 6, 2026Yiheng Wang, Yueqian Lin, Lichen Zhu +3Multimodal Large Language Models Question Answering
Duke University, Durham, North Carolina, USA
ChronoPhyBench: Do MLLMs Truly Understand the World or Merely Exploit Language Priors?
Jun 6, 2026Bin Zhu, Yanhao Jia, Kexin Zhao +12Physical Reasoning Multimodal QA
Peking University, Shenzhen Graduate School · 2Peng Cheng Laboratory · Rabbitpre Intelligence +3
REACT 2026: The Fourth Multiple Appropriate Facial Reaction Generation Challenge: Personalised MAFRG and Appropriate EEG Reaction Prediction
Jun 6, 2026Siyang Song, Micol Spitale, Zijian Wu +11Conditional Image Generation Multimodal Generation
University of Exeter United Kingdom · Politecnico di Milano Milan, Italy · Nanjing University of Science and Technology China +8
VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?
Jun 5, 2026Didi Zhu, Changrui Chen, Stefanos Zafeiriou +1VLM Evaluation Multimodal Grounding
Imperial College London
From Vision to Text: A Compact Multimodal Approach for Robust, Cross-Domain Presentation Attack Detection on ID Cards
Jun 5, 2026Qingwen Zeng, Juan E. Tapia, Sneha Das +1Multimodal Robustness Domain Generalization
Technical University of Denmark (DTU), Denmark · da/sec-Biometrics and Security Research Group, Hochschule Darmstadt, Germany
MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios
Jun 5, 2026Haitian Wang, Ruoxi Sun, Quantong Qiu +5Multimodal IR Multimodal Embedding
Soochow University, Suzhou, China · Ant Group, Beijing, China
MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models
Jun 4, 2026Ryan D'Cunha, Alejandro Lozano, Xiaoxiao Sun +17Medical Image Analysis Medical VLMs
Stanford University, Stanford, CA, USA · Instituto Tecnológico de Monterrey, Mexico · Monash University, Melbourne, VIC, Australia +4
DisasterBench: A Multimodal Benchmark for UAV-Based Disaster Response in Complex Environments
Jun 4, 2026Tan Zhang, Quanyou Li, Lu Zhang +3Efficient Multimodal Inference Multimodal Reasoning
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, China · School of Information and Communication Engineering, Dalian University of Technology, Dalian, 116024, China · School of Computing and Communications, Lancaster University, Lancaster, LA1 4YW, England +1
Evaluating Stochastic Collapse and Implicit Bias in Multimodal Large Language Models
Jun 4, 2026Huiyuan Zheng, Houtao Zhang, Boyang Wang +2Multimodal Large Language Models Language Model Bias Evaluation
Fudan University · Beihang University
WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark
Jun 4, 2026Yida Yin, Harish Krishnakumar, Chung Peng Lee +9Multimodal Large Language Models Multimodal Reasoning
Princeton University · NYU · University of Waterloo +1
UniCAD: A Unified Benchmark and Universal Model for Multi-Modal Multi-Task CAD
Jun 3, 2026Jingyuan Chen, Sheng Jin, Haopeng Sun +23D Reconstruction Multimodal Large Language Models
SenseTime Research and Tetras.AI
M $^3$ Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks
Jun 3, 2026Jie Huang, Ruixun Liu, Sirui Sun +4Multimodal Memory Multimodal Model Evaluation
School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University · Yuanpei College, Peking University +3
A Systematic Evaluation of Positional Bias in Multi-Video Summarization with MLLMs
Jun 3, 2026Huangchen Xu, Yuan Wu, Yi ChangMultimodal Large Language Models Multimodal Model Evaluation
School of Artificial Intelligence, Jilin University · 2Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University · 3International Center of Future Science, Jilin University
FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs
Jun 2, 2026Eshika Khandelwal, Jingjing Pan, Mingfang Zhang +3Multimodal Grounding Video Object Detection
Tuebingen AI Center, University of Tuebingen · Woven by Toyota, Inc., Tokyo, Japan · Toyota Motor Europe, Brussels, Belgium
VAMPS: Visual-Assisted Mathematical Problem Solving Benchmark
Jun 2, 2026Amirhossein Dabiriaghdam, Shayan Vassef, Mohammadreza Bakhtiari +5Mathematical Reasoning Benchmarks Mathematical Reasoning
University of British Columbia · University of Illinois Chicago · Stony Brook University

Jun 13, 2026, cs.CVJ/K move · Enter open · S save

Conditional Multi-Event Temporal Grounding in Long-Form Video

Yuanhao Zou, Arthad Kulkarni, Lucas Tonanez, Lincoln Spencer+11

University of Central Florida · 2Qualcomm AI Research

Temporal Video Grounding Long-Video Understanding Multimodal Model Evaluation Multimodal Understanding

Open paper AskPDF arXiv

Multimodal large language models have made rapid progress in video temporal grounding, yet real-world applications routinely require localizing every event that satisfies compositional temporal and spatial conditions. Existing benchmarks fall short: they localize only a single moment per query, count without temporal conditions, or treat grounding and counting as disjoint tasks. We introduce CoMET-Bench for Conditional Multi-Event Temporal Grounding in long-form video, comprising 2789 queries over 600 videos averaging 33.8 minutes across five real-world domains, with each query composed from 4 temporal conditions, 3 spatial conditions, and a dedicated negative-query subset. We further propose a unified evaluation protocol jointly measuring counting, grounding, and negative-query recognition, including a new Rejection-F1 metric that prevents trivial gaming by lazy "always-empty" models. Benchmarking a broad suite of MLLMs, agent-based, and grounding-specialized methods reveals that existing approaches remain far from solving this task. Building on these findings, we propose CoMET-Agent, a training-free agentic framework that reformulates the task as structured search-and-aggregate, improving [email protected] by 6.1% over GPT-5 purely through structural reasoning. Failure analysis further surfaces three open directions: fine-grained entity tracking, position-uniform retrieval, and causal event pairing.

Momentum

Related topics

Latest papers 503