Multimodal Model Evaluation

Follow this topic Ask about this topic

Momentum

49 papers in the last four weeks, up 44% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 496

All topics

OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understanding
May 18, 2026Ruixiang Zhao, Jie Yang, Zijie Xin +4Audio-Visual Understanding Streaming Video Understanding
Renmin University of China · WeChat Vision, Tencent Inc.
Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction
May 17, 2026Chaoqun He, Mingyang Xiang, Yingjing Xu +5Multimodal Large Language Models Multimodal Agents
Tsinghua University · Shanghai Qi Zhi Institute · ModelBest Inc.
EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning
May 17, 2026Zeyu Wang, Chang Liu, Eduardus Tjitrahardja +22Multimodal Large Language Models Egocentric Video Understanding
Tsinghua University · Shanghai Qi Zhi Institute · Tongji University +5
PluRule: A Benchmark for Moderating Pluralistic Communities on Social Media
May 16, 2026Zoher Kachwala, Bao Tran Truong, Rasika Muralidharan +3Multilingual Language Model Evaluation Content Moderation
Observatory on Social Media, Indiana University, USA · Center Synergy of Systems, TUD Dresden University of Technology, Germany
GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions
May 15, 2026Junho Kim, Xu Cao, Houze Yang +6Multimodal QA Multimodal Model Evaluation
University of Illinois Urbana-Champaign · Georgia Institute of Technology · Amazon AGI +1
MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory
May 14, 2026Minghao Guo, Qingyue Jiao, Zeru Shi +14Multimodal Memory AI Agent Evaluation
Rutgers · Notre Dame · Princeton +1
Do Composed Image Retrieval Benchmarks Require Multimodal Composition?
May 14, 2026Matteo Attimonelli, Alessandro De Bellis, Aryo Pradipta Gema +8Benchmark Auditing Image Retrieval
Politecnico di Bari
MultiEmo-Bench: Multi-label Visual Emotion Analysis for Multi-modal Large Language Models
May 14, 2026Tianwei Chen, Takuya Furusawa, Yuki Hirakawa +3Multi-Label Classification Multimodal Large Language Models
ZOZO NEXT Inc.
ViMU: Benchmarking Video Metaphorical Understanding
May 14, 2026Qi Li, Xinchao WangVideo Understanding Video QA
National University of Singapore
MuteBench: Modality Unavailability Tolerance Evaluation for Incomplete Multimodal Fusion
May 13, 2026Wugeng Zheng, Ziwen Kan, Tianlong Chen +2Missing-Modality Learning Multimodal Robustness
University of Central Florida · University of North Carolina at Chapel Hill
Physics-R1: An Audited Olympiad Corpus and Recipe for Visual Physics Reasoning
May 13, 2026Shan YangVisual Reasoning RL for Language Model Reasoning
Independent Researcher
A Multi-Probe Audit of Clinical-Interview Depression Detection Benchmarks
May 13, 2026Takehiro Ishikawa, Jon DukeDepression Detection Mental Health
College of Computing, Georgia Institute of Technology, Atlanta, GA, USA · Georgia Tech Research Institute, Georgia Institute of Technology, Atlanta, GA, USA
Robust Checkpoint Selection for Multimodal LLMs via Agentic Evaluation and Stability-Aware Ranking
May 13, 2026Qinwu Xu, Zhuoheng Li, Jessie SalasMultimodal Large Language Models Multimodal Model Evaluation
Meta AI · Independent Researcher, previously Meta AI
ViDR: Grounding Multimodal Deep Research Reports in Source Visual Evidence
May 13, 2026Zhuofan Shi, Peilun Jia, Baoqin Sun +4Multimodal RAG Multimodal Grounding
School of Software and Microelectronics, Peking University, Beijing 100871, China · 2National Key Laboratory of Data Space Technology and System, Beijing 100871, China · School of Software Engineering, Beijing Jiaotong University, Beijing, 100044, China
WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments
May 13, 2026Haoren Zhao, Tianyi Chen, Zhen WangMultimodal Robustness GUI Agents
School of Cyberspace, Hangzhou Dianzi University, Hangzhou, China · 2Microsoft
MMCL-Bench: Multimodal Context Learning from Visual Rules, Procedures, and Evidence
May 12, 2026Yifan Chen, Fei Yin, Qingyan Bai +2Multimodal ICL Multimodal Reasoning
University of Cambridge · HKUST · Tsinghua University
Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?
May 12, 2026Yichen Feng, Yuetai Li, Chunjiang Liu +14Multimodal Large Language Models Image Quality Assessment
MM-OptBench: A Solver-Grounded Benchmark for Multimodal Optimization Modeling
May 12, 2026Zhong Li, Qi Huang, Yuxuan Zhu +6Multimodal Optimization Synthetic Benchmark Generation
Great Bay University · Leiden University · Rensselaer Polytechnic Institute +1
Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation
May 12, 2026Che Liu, Lichao Ma, Xiangyu Tony Zhang +4Cross-Modal Learning Multimodal Large Language Models
StepFun-Audio Team
Measuring What Matters Beyond Text: Evaluating Multimodal Summaries by Quality, Alignment, and Diversity
May 12, 2026Abid Ali, Diego Molla-Aliod, Usman NaseemLLM Evaluation Multimodal Large Language Models
School of Computing, Macquarie University, Sydney, Australia
Human-Grounded Multimodal Benchmark with 900K-Scale Aggregated Student Response Distributions from Japan's National Assessment of Academic Ability
May 12, 2026Kyosuke Takami, Yuka Tateisi, Satoshi Sekine +1Educational Assessment Multimodal Reasoning
Osaka Kyoiku University · University of Tokyo · NII LLMC
GeoR-Bench: Evaluating Geoscience Visual Reasoning
May 12, 2026Yushuo Zheng, Zicheng Zhang, Huiyu Duan +7Remote Sensing Image Understanding Geospatial Reasoning
Shanghai Jiao Tong University · Beijing University of Technology
EvoStreaming: Your Offline Video Model Is a Natively Streaming Assistant
May 11, 2026Zichen Wen, Boxue Yang, Junlong Ke +5Streaming Video Understanding Self-Training
EPIC Lab, Shanghai Jiao Tong University · Tsinghua University · The Hong Kong University of Science and Technology (Guangzhou) +1
SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation
May 11, 2026Longteng Guo, Xuanxu Lin, Dongze Hao +5Multimodal Large Language Models Scientific QA
Institute of Automation, Chinese Academy of Sciences (CASIA) · School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS) · OPPO AI Center
Omni-Persona: Systematic Benchmarking and Improving Omnimodal Personalization
May 11, 2026Yeongtak Oh, Dongwook Lee, Sangkwon Park +2Multimodal Grounding Multimodal Large Language Models
Department of Electrical and Computer Engineering, Seoul National University · Interdisciplinary Program in Artificial Intelligence, Seoul National University · Department of Artificial Intelligence, University of Seoul
SpecX: A Large-Scale Benchmark for Multi-Modal Spectroscopy and Cross-Paradigm Evaluation
May 11, 2026Chengrui Xiang, Tengfei Ma, Yujie Chen +3Multimodal Model Evaluation
College of Computer Science and Technology, Hunan University, Changsha, China
TOC-Bench: A Temporal Object Consistency Benchmark for Video Large Language Models
May 11, 2026Junzhe Chen, Siyuan Meng, Yuxi Chen +3Video-Language Model Evaluation Video-Language Models
Tianjin University
Perception Without Engagement: Dissecting the Causal Discovery Deficit in LMMs
May 10, 2026Jiafeng Liang, Zhihao Zhu, Zihan Zhang +7Causal Discovery Multimodal Large Language Models
Harbin Institute of Technology · Harvard University · Peking University +2
DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification
May 10, 2026Rui Liu, Dian Yu, Zhenwen Liang +6Multimodal Large Language Models Multimodal Reward Modeling
1Tencent Hunyuan · University of Maryland, College Park · University of North Carolina, Chapel Hill
SeePhys Pro: Diagnosing Modality Transfer and Blind-Training Effects in Multimodal RLVR for Physics Reasoning
May 10, 2026Kun Xiang, Terry Jingchen Zhang, Zirong Liu +15Cross-Modal Learning Multimodal Grounding
Sun Yat-sen University · ETH Zurich · Huawei Technologies Ltd. +1

May 18, 2026, cs.CVJ/K move · Enter open · S save

OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understanding

Ruixiang Zhao, Jie Yang, Zijie Xin, Tianyi Wang+3

Renmin University of China · WeChat Vision, Tencent Inc.

Audio-Visual Understanding Streaming Video Understanding Multimodal Model Evaluation Multimodal Understanding

Open paper AskPDF arXiv

Omni-proactive streaming video understanding, i.e., autonomously deciding when to speak and what to say from continuous audio-visual streams, is an emerging capability of omni-modal large language models. Existing benchmarks fall short in three key aspects: they rely primarily on visual signals, adopt polling or fixed-timestamp protocols instead of true proactive evaluation, and cover only a limited range of tasks, preventing reliable assessment and differentiation of omni-proactive streaming models. We present OmniPro, the first benchmark to jointly evaluate omni-modal perception, proactive responding, and diverse video understanding tasks. It comprises 2,700 human-verified samples spanning 9 sub-tasks and 3 cognitive levels, covering 6 basic video understanding capabilities. Notably, 84% of samples require audio signals (speech or non-speech), and each sample is annotated with modality-isolation labels to enable fine-grained multimodal analysis. We further introduce a dual-mode evaluation protocol: Probe mode assesses content understanding by querying the model before and after each ground-truth trigger, while Online mode evaluates full proactive ability by requiring models to autonomously decide when to respond in streaming input. Evaluating 11 representative models reveals three key findings: (1) audio provides consistent gains but with highly variable utilization across models, (2) performance degrades significantly over time, indicating limited long-horizon robustness, and (3) non-speech audio perception remains the weakest dimension.

Momentum

Related topics

Latest papers 496