Multimodal Agent Evaluation

Follow this topic Ask about this topic

Latest papers

22 papers4 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

VibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video Generation
Oct 26, 2026Muxi Diao, Wenbin Xu, Lele Yang +21Video Generation Evaluation Multimodal Agent Evaluation
Profiling a Frontier Agent with EBench: From Direct Control to Harness with Policy
Oct 26, 2026Yifei Yao, Yude Zou, Guanbo Jiao +12Language Model-Based Control Robotic Policy Evaluation
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Fudan University +1
Omni-InteractScope: Benchmarking Proactive and Agentic Interaction in Continuous Omnimodal Streams
Oct 26, 2026Yinsong Yan, Ling Wang, Yuxuan Wang +11Multimodal Agent Evaluation MLLM Evaluation
The Hong Kong Polytechnic University · Alibaba Token Hub, Alibaba Group · Shanghai Jiaotong University +2
Multimodal Socially Aware Framework for Human-Robot Interaction in Virtual Reality
Oct 25, 2026Richard Xu, Dakarai Crowder, Alexis E. Block +1Human-Robot Interaction Social Robotics
University of Illinois at Urbana-Champaign, Champaign, IL, USA · Case Western Reserve University, Cleveland, OH, USA
AndroidFlux: Evaluating and Reward Modeling Failure Recovery Capabilities in Mobile Use Multimodal Agents
Oct 25, 2026Gyubeum Lim, Yemo Koo, Ghunwoo Jang +1Multimodal Agent Evaluation Reward Model Evaluation
Georgia Institute of Technology · Konkuk University
InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants
Oct 25, 2026Qinsi Wang, Wei Wen, Chong Zhou +7Multimodal Agent Evaluation Human-in-the-Loop AI
Meta · Duke University
Dual Exam: Controlled Evaluation of Multimodal Agents and World Models
Oct 22, 2026Zhefan Rao, Meng Chu, Xuan Zhang +80Multimodal Agent Evaluation Agent Evaluation
Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents
Sep 15, 2026Srishti Palani, Vidya SetlurAgent Evaluation Visual Analytics
Tableau Research, Palo Alto, CA, USA
OmniStepBench: Deployment-Aware Agent Evaluation across Modalities and Task Complexity
Sep 5, 2026Yu Liu, Zhilin Liu, Zhiwei Yang +7Multimodal Agent Evaluation LLM Evaluation
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · MiLM Plus, Xiaomi Inc. +1
Paint What You See: Benchmarking Dexterous Visual Tool Use in Multimodal Agents
Aug 26, 2026Shudong Liu, Dongyang Chen, Enci Zhang +3AI Agent Benchmarks Tool Use
Peking University · SenseTime Research · Tsinghua University
M $^3$ Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions
Jun 5, 2026Zhengjun Huang, Wenxuan Liu, Zhoujin Tian +6Multimodal Understanding Multimodal Grounding
The Hong Kong University of Science and Technology · Peng Cheng Laboratory · Beijing University of Chemical Technology +4
WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction
May 28, 2026Chengzhi Liu, Yuzhe Yang, Sophia Xiao Pu +14Multimodal Agent Evaluation Multimodal Memory
University of California, Santa Barbara · Johns Hopkins University · Carnegie Mellon University +3
TOBench: A Task-Oriented Omni-Modal Benchmark for Real-World Tool-Using Agents
May 16, 2026Zhiqiang Liu, Wenhui Dong, Yilang Tan +3Tool-Use Evaluation AI Agent Benchmarks
Nanjing University · Huazhong University of Science and Technology · Southwest Jiaotong University +1
MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory
May 14, 2026Minghao Guo, Qingyue Jiao, Zeru Shi +14Multimodal Agent Evaluation Multimodal Memory
Rutgers · Notre Dame · Princeton +1
Do Joint Audio-Video Generation Models Understand Physics?
May 8, 2026Zijun Cui, Xiulong Liu, Hao Fang +8Multimodal Agent Evaluation Physical Consistency in Video Generation
University of Texas at Dallas · University of Washington · University of California, Los Angeles
Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?
Apr 3, 2026Qianshan Wei, Yishan Yang, Siyi Wang +12Multimodal Agent Evaluation Agent Evaluation
CASIA · UCAS · SEU +5
MM-tau-p $^2$ : Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings
Mar 10, 2026Anupam Purwar, Aditya ChoudharyMultimodal Agent Evaluation Agent Evaluation
Sprinklr AI, Gurugram, India
ConflictBench: Evaluating Human-AI Conflict via Interactive and Visually Grounded Environments
Mar 9, 2026Weixiang Zhao, Haozhen Li, Yanyan Zhao +5Multimodal Agent Evaluation AI Agent Safety Benchmarks
Harbin Institute of Technology · SERES
AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarios
Feb 26, 2026Zhaochen Su, Jincheng Gao, Hangyu Guo +10Multimodal Agent Evaluation AI Agent Benchmarks
Hong Kong University of Science and Technology · University of North Carolina at Chapel Hill · Zhejiang University +1
MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents
Feb 24, 2026Zhenyu Wang, Xiaofen Xing, Yirong Chen +1Role-Playing Agent Evaluation Multimodal Agent Evaluation
South China University of Technology, Guangzhou 510641, China · Foshan University, Foshan 528225, China
GameDevBench: Evaluating Agentic Capabilities Through Game Development
Feb 11, 2026Wayne Chi, Yixiong Fang, Arnav Yayavaram +8Multimodal Agent Evaluation Software Engineering Agents
Carnegie Mellon University, Pittsburgh, PA, USA · Princeton University, Princeton, NJ, USA
DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems
Jan 20, 2026Maojun Sun, Yifei Xie, Yue Wu +5Multimodal Agent Evaluation LLM Agent Evaluation
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University · Department of Applied Mathematics, The Hong Kong Polytechnic University

Oct 26, 2026J/K move · Enter open · S save

VibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video Generation

Muxi Diao, Wenbin Xu, Lele Yang, Xuyang Ma+20

Video Generation Evaluation Multimodal Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

Diffusion video models produce striking clips, but they cannot reliably render an exact formula, a correct chart, a ruleconsistent game or narration synchronized to on-screen events. A different route is emerging: multimodal agents that write code to render every frame and synthesize every sound. We introduce VibeVideoBench, a benchmark for evaluating agents on code-driven video generation. Its 239 self-contained tasks in English and Chinese span 12 categories, from data stories and music-driven pieces to classroom lessons and paper explainers; each asks an agent, running in its native harness inside a sandbox, to deliver a finished MP4 with sound within one hour. Evaluation is hierarchical and evidence-based: program checks of the delivery spec, about 13k task-specific rubric checks traced to the requirements they enforce, and five quality dimensions anchored per task, judged by three VLM judge panels and validated with blind human preferences. Across nine frontier agent systems and 2,151 scored rollouts, the benchmark is far from solved: GPT-6 Astra and Claude Opus 5.5 lead clearly, yet even the best system fails a core requirement on about a quarter of tasks. Whether a video arrives at all matters as much as its quality, and three quarters of the lost videos are final renders started too late, run too slowly or never started. The judges agree with each other and with human preferences on system rankings, but only moderately on individual videos.