Evaluation Benchmarks

Ask about this topic

Momentum

9 papers in the last four weeks, up 125% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 91

All topics
CardsList

  1. PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence

    Oct 5, 2026Dheeraj Varghese, Anna Vettoruzzo, Walter Simoncini +5Evaluation BenchmarksVideo World Models

    University of Amsterdam · Eindhoven University of Technology · Eberhard Karls Universität Tübingen

  2. VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision

    Sep 30, 2026Vu Dinh Xuan, Duc-Hai Nguyen, Minh-Dung Dao +6Visual Question Answering BenchmarksEvaluation Benchmarks

    University of Information Technology, VNU-HCM, Vietnam · University College Cork, Ireland · Hanoi University of Science and Technology, Vietnam +1

  3. Eval4DiRec: A Unified and Systematic Evaluation Framework for Diffusion-based Recommender Systems

    Sep 28, 2026Cong Wang, Shoujin Wang, Yishuo Li +3Real-World Content Recommendation ProblemEvaluation Benchmarks

    Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences); Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing, Shandong Fundamental Research Center for Computer Science, Jinan, Shandong, China · University of Technology Sydney, Sydney, NSW, Australia · Department of Computer Science and Technology, Tongji University, Shanghai, China +1

  4. PlaylistEval: Can Video-Language Judges Be Trusted at Day Scale and Beyond?

    Sep 28, 2026Shayekh Bin Islam, Hwanjun SongLarge Language Model JudgesLong-Video Benchmarks

    KAIST

  5. AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents

    Sep 27, 2026Tianzhuo Yang, Zirui Mi, Yantao Huang +4RefusalsEvaluation Benchmarks

    Peking University · Beijing Academy of Artificial Intelligence

  6. ReVeal: A Reconstruction-Aware Real-to-Sim Framework for VLA Policy Evaluation

    Sep 20, 2026Xinyi Wang, Heng Hao, Wenjun Hu +5Simulation-Based Reinforcement LearningEvaluation Benchmarks

    Purdue University · Samsung SDS Research America

  7. Towards Reliable Underwater Diver-Robot Interaction: Gesture Design, Interaction Logic, and Real-World Evaluation

    Sep 20, 2026Yingqi Liu, Kanzhong Yao, Zimeng Peng +4Underwater RobotsUnderwater

    University of Science and Technology of China, Hefei, China. · Institute of Artificial Intelligence (TeleAI), China Telecom, Shanghai 200232, China.

  8. Behavior2Value: Benchmarking and Empowering LLMs for Consumer Value Measurement from E-commerce Behaviors

    Sep 16, 2026Peixuan Hou, Bin Chen, Li He +4ConsumptionValue

    State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Alibaba Group

  9. Do LLMs Trust the Accuser or the Accusation? Measuring Belief Shifts in Werewolf

    Sep 14, 2026Yu-Yu Yang, Ti-Rong Wu, Hung Guei +2Evaluation Benchmarks

    Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan · College of Artificial Intelligence, National Yang Ming Chiao Tung University, Taiwan +1

  10. APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents

    Sep 7, 2026Jintian Feng, Long Chen, Xiao Yu +7Graphical User Interface AgentsDevice

    Laboratory for Artificial Intelligence and New Forms of Education, Central China Normal University · Faculty of Artificial Intelligence in Education, Central China Normal University · Agentic Labs, Acrab AI

  11. R2S-Eval: Robot Evaluation with Real-to-Sim Calibration via Vision-Language Models

    Sep 3, 2026Yidi Wang, Feixiang Ruan, Ruoqu Chen +4Evaluation BenchmarksSim-To-Real Gap

    Sharpa · Nanjing University · Tongji University +1

  12. TRUST: Threshold-Recalibrated Uncertainty-Safe Training for Certified Dismissal in Breast Cancer Screening

    Aug 31, 2026Parham Hajishafiezahramini, Matthew Hamilton, Edward Kendall +2ScreeningDecision Thresholds

    Department of Computer Science, Memorial University of Newfoundland, Canada · 2Division of Biomedical Sciences, Faculty of Medicine, Memorial University of2026 Newfoundland, Canada · 3Cancer Care, Newfoundland and Labrador Health Services, St. John’s, NL, CanadaAug 31

  13. Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenes

    Aug 11, 2026Zhaoyang Wei, Bowen Jiang, Xumeng Han +6Multimodal ReasoningVisual Reasoning

    University of Chinese Academy of Sciences (UCAS), Beijing, 101480, China. · Tencent CDG, Shenzhen, 518054, China. · Institute of Information Engineering, CAS, Beijing, 100085, China.

  14. E3^3mo-Bench: A Scalable Benchmark for Multimodal Evoked and Expressed Emotion Understanding via Bayesian Pairwise Alignment

    Aug 11, 2026Lancheng Gao, Ziheng Jia, Shengyan Li +4Multimodal Large Language ModelsEvaluation Benchmarks

    Shanghai Jiao Tong University

  15. MameLoshnLM: Yiddish Language Model and Evaluation Benchmark

    Aug 6, 2026Uri Katz, Omer Goldman, Tomasz Limisiewicz +2Text CorporaEvaluation Benchmarks

    Bar-Ilan University · University of Cambridge · University of Washington +1

  16. LiveEvalBench: Toward Open-World Evaluation for Web Generation

    Aug 4, 2026Yiyao Wang, Zhen Wen, Yinghao Tang +5Evaluation BenchmarksDiversity

    State Key Lab of CAD&CG, Zhejiang University · Ant Group · State Key Lab of CAD&CG, Zhejiang University; Laboratory of Art and Archaeology Image (Zhejiang University), Ministry of Education, China

  17. DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

    Aug 4, 2026Boyan Li, Zhuowen Liang, Yupeng Xie +11Data Science AgentsAgentic Benchmarks

    HKUST(GZ) · Tsinghua University

  18. Dynamically Allocating Evaluation Effort for Model Ranking

    Aug 4, 2026Vilém Zouhar, Julia Kreutzer, Alon Lavie +4Model EvaluationEvaluation Benchmarks

    ETH Zurich · Cohere Labs · Carnegie Mellon University +3

  19. MIEScore: Human-Aligned Evaluation for Multi-Source Image Editing

    Aug 3, 2026Zitong Xu, Huiyu Duan, Xinyun Zhang +7Multi-Image Editing BenchmarksImage Editing

    Shanghai Jiao Tong University · Vivo Mobile Communication Co., Ltd · University of Electronic and Science Technology of China

  20. RADAR: Rubric-Aware Dependency and Redundancy Analysis for LLM-as-Judge Evaluation

    Aug 3, 2026Divyansh Singh, Reza Davari, Afra MashhadiLlm-As-A-JudgeGrading

    Microsoft · University of Florida · University of Washington

  21. Scaling Vision-Language Models Is Not Enough to Mitigate Bias

    Jul 30, 2026Ioannis Sarridis, Ioannis Kompatsiaris, Symeon PapadopoulosBiasesEvaluation Benchmarks

    Information Technologies Institute, CERTH Thessaloniki, Greece

  22. ArmnetBench v0.1: Parallel Real-World Evaluation of Manipulation Policies on a Low-Cost Arm Farm

    Jul 27, 2026Praveen Selvaraj, Lorenzo Uttini, Ville KuosmanenEvaluation BenchmarksParallel Rollouts

    Armnet

  23. Webly Supervised Multi-Label Recognition: Evaluation Benchmark and Dual-Branch Multi-Label Contrastive Learning

    Jul 23, 2026Zhihua Xu, Zhijing Yang, Yufeng Yang +1Multi-Label ClassificationContrastive Learning

    aSun Yat-Sen University, Guangzhou, 510006, GuangDong, China · bGuangdong University of Technology, Guangzhou, 510006, GuangDong, China

  24. ALLUDE: A Unified Evaluation System for Configurable Attacks in Differentiable Environments

    Jul 19, 2026Mansi Phute, Alexander Greenhalgh, Matthew Hull +8Evaluation BenchmarksDifferentiable Rendering

    Georgia Institute of Technology, USA · Technological Innovation Institute, UAE

  25. Real-World Evaluation of an AI Agent Drafting Translational Impact Summaries

    Jul 18, 2026Mohammad Arvan, Amber E. Osterholt, Bailee Rue +5Scholar Information DatabaseArtificial Intelligence Agents

    University of Illinois Chicago, Chicago, IL, USA

  26. Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap

    Jul 16, 2026Olivier JeunenVariance ReductionHeterogeneous Treatment Effects

    Independent Researcher Antwerp, Belgium

  27. Continuously Evolving Deepfake Detection: An Architecture and Public-Benchmark Evaluation of a Dynamic Detection System

    Jul 14, 2026Ken Jon Miyachi, Dylan UysDeepfake DetectionDigital Forensics

    BitMind

  28. VQCSim: When Does Compile-Once Statevector Simulation Beat Generic Quantum Frameworks?

    Jul 13, 2026Anton Firc, Martin Perešíni, Vojtěch Mrázek +7Variational Quantum CircuitsHybrid Quantum-Classical Pipeline

    EvoAI Hardware Group, Brno University of Technology, Brno, Czech Republic · eBRAIN Lab, Division of Engineering, New York University Abu Dhabi (NYUAD), Abu Dhabi, UAE · Center for Quantum and Topological Systems (CQTS), NYUAD Research Institute, NYUAD, Abu Dhabi, UAE

  29. A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigation

    Jul 9, 2026Liuyi Wang, Kai Sheng, Zongtao He +8Vision-Language NavigationVisual Perception

    College of Electronic and Information Engineering, Tongji University, Shanghai, China

  30. WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation

    Jul 7, 2026Wei Dong, Tianyu Fu, Zhe Yu +9Web AgentsEvaluation Benchmarks

    Mininglamp Technology · School of Artificial Intelligence, University of Chinese Academy of Sciences · MAIS, Institute of Automation, Chinese Academy of Sciences

  31. MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models

    Jul 2, 2026Yuanzhi Liu, Shousheng Zhao, Bo Zhou +2Multimodal BenchmarksEvaluation Benchmarks

    1Beijing University of Posts and Telecommunications, Beijing, China

  32. Overview of the TalentCLEF 2026: Skill and Job Title Intelligence for Human Capital Management

    Jun 30, 2026Luis Gasco, Hermenegildo Fabregat, Laura García-Sardiña +8JobsSkills

    Avature Machine Learning, Spain · TechWolf, Belgium · NLP & IR Group at UNED, Madrid, Spain

  33. Critical Interval MSE: Toward Reliable Offline Validation for Robot Manipulation Policies

    Jun 29, 2026Haoxu Huang, Tongsam Zheng, Yifan Chen +2Robot PoliciesPolicy Evaluation

    Tsinghua University · Shanghai Qi Zhi Institute · Spirit AI

  34. Scalable Behavior Cloning with Open Data, Training, and Evaluation

    Jun 25, 2026Arthur Allshire, Himanshu Gaurav Singh, Ritvik Singh +15Behavior CloningTeleoperation

    UC Berkeley · MIT · XDOF +2

  35. Theory-Scale Auto-Formalization of Logics for Computer Science

    Jun 25, 2026Yuming Feng, Frederick Pu, One An +5AutoformalizationFormal Verification

    Johns Hopkins University · University of Toronto · University of Pennsylvania +1

  36. What We are Missing in Multimodal LLM Evaluation?

    Jun 24, 2026Po-han Li, Shenghui Chen, Sandeep Chinchali +1Multimodal Large Language ModelsEvaluation Benchmarks

    The University of Texas at Austin, USA

  37. REALM: A Unified Red-Teaming Benchmark for Physical-World VLMs

    Jun 22, 2026Yifei Zhao, Qian Lou, Mengxin ZhengAdversarial EvaluationRed-Teaming

    University of Central Florida

  38. An Extensive Benchmark for Single-round and Multi-round Instruction-based Image Editing

    Jun 14, 2026Yiwei Ma, Ke Ye, Weihuang Lin +4Multi-Image Editing BenchmarksImage Editing

    Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, Fujian, P.R. China. · National University of Singapore, 119077, Singapore.

  39. EIBench: A Simulator-Based Benchmark and Turn-Credit RL for Emotion Management

    Jun 14, 2026Rongzhi Zhu, Xiang Huang, Yuchuan Wu +8User SimulationEmotion Recognition

    State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China · Qwen-Character Team , Alibaba Group

  40. CODA-BENCH: Can Code Agents Handle Data-Intensive Tasks?

    Jun 13, 2026Yuxin Zhang, Ju Fan, Meihao Fan +2Data Science AgentsSynthetic Benchmark

    Renmin University of China

  41. RoboProcessBench: Benchmarking Process-Aware Understanding in Vision-Language Robotic Manipulation

    Jun 11, 2026Dayu Xia, Yue Shi, Yao Mu +7Robotic ManipulationEvaluation Benchmarks

    Shanghai AI Laboratory · Zhejiang University · Shanghai Jiao Tong University +2

  42. OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics

    Jun 8, 2026Mingxian Lin, Shengju Qian, Yuqi Liu +9Agentic BenchmarksWebarena Domains

    1The University of Hong Kong · 2LIGHTSPEED · 3The Chinese University of Hong Kong +1

  43. MMAE: A Massive Multitask Audio Editing Benchmark

    Jun 5, 2026Ziyang Ma, Ruiqi Yan, Ruiyang Xu +35Audio EditingEvaluation Benchmarks

    Shanghai Jiao Tong University · Shanghai Innovation Institute · Nanyang Technological University +5

  44. MacArena: Benchmarking Computer Use Agents on an Online macOS Environment

    Jun 4, 2026Victor Muryn, Maksym Shamrai, Sofiia Mazepa +1Graphical User InterfaceEvaluation Benchmarks

    1MacPaw Research, Kyiv, Ukraine

  45. Enhancing Software Engineering Through Closed-Loop Memory Optimization

    Jun 4, 2026Xuehang Guo, Zora Zhiruo Wang, Qingyun Wang +2Software EngineeringSoftware

    1William & Mary · 2Carnegie Mellon University · 3OpenHands +1

  46. Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation

    Jun 3, 2026Yongjie Wang, Xinyue Zhang, Kunhong Yao +4Agentic BenchmarksEvaluation Benchmarks

    Alibaba-NTU Global e-Sustainability CorpLab (ANGEL) · College of Computing & Data Science · Tongyi Lab, Alibaba Group

  47. What Are We Actually Benchmarking in Robot Manipulation?

    Jun 2, 2026Tianchong Jiang, Xiangshan Tan, Samuel Wheeler +3Robotic ManipulationEvaluation Benchmarks

    Toyota Technological Institute at Chicago, Chicago, IL, USA · Argonne National Laboratory, Lemont, IL, USA · University of Chicago, Chicago, IL, USA

  48. Beyond Ideal Instruction: A Comprehensive Framework for Evaluating LLMs in Realistic Interactions

    Jun 2, 2026Xuan Yang, Hao Xu, Tingfeng Hui +4Large Language Model EvaluationEvaluation Benchmarks

    Department of Data Science, City University of Hong Kong · 3Li Auto Inc. · 2Hong Kong Institute of AI for Science, City University of Hong Kong +1

  49. OmniEEG-Bench: A Standardized Evaluation Benchmark for EEG Foundation Models

    May 30, 2026Ziling Lu, Zongsheng Li, Xinke Shen +11Electroencephalography Foundation ModelsElectroencephalography

    Department of Biomedical Engineering, Southern University of Science and Technology, Shenzhen, China · School of Computer Science and Engineering, The Chinese University of Hong Kong, Shenzhen, China · Omni-Intelligence, Shenzhen, China +1

  50. Pre-Deployment Robustness Stress Testing for CT Segmentation Systems Using Clinically Motivated Multi-Corruption Augmentation

    May 30, 2026CholMin Kanga, Jonghyun Chung, Amanpreet Kaur +2Multi Organ Computed Tomography SegmentationStress Testing

    Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul, Korea · Google Inc., 1600 Amphitheatre Parkway, Mountain View, 94043, California, United States

  51. EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents

    May 27, 2026Yunqi Liu, Tong Niu, Zitong Wang +4Multimodal AgentsAgentic Benchmarks

    Ant Group

  52. MatFormBench: A Benchmarking Evaluation Framework for Target-Driven Materials Formulation

    May 26, 2026Linhan Wu, Chenxi Wang, Chuhan Yang +2Data-Driven Materials DiscoveryGenerative Inverse Design

    DeepVerse

  53. Selecting New Measurement Locations to Diversify Traffic-Pattern Coverage: A Real-World Evaluation for Total Traffic Volume Estimation

    May 25, 2026Masaaki Inoue, Akifumi Okuno, Shintaro FukushimaAccurate Traffic ForecastingTraffic

    1TOYOTA Motor Corporation · Institute of Statistical Mathematics · 3The Graduate University for Advanced Studies, SOKENDAI +1

  54. SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models

    May 25, 2026Khalid Yusuf DahirLarge Language Model SafetyLow-Resource Languages

    Independent researcher

  55. AVBench: Human-Aligned and Automated Evaluation Benchmark for Audio-Video Generative Models

    May 23, 2026Jialiang Yang, Bin Xia, Ruihang Chu +6Audio-Video GenerationEvaluation Benchmarks

    Tsinghua University · The Chinese University of Hong Kong

  56. VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models

    May 20, 2026Alex S. Huang, Jiahui Zhang, Shiqing Tang +1Large-Scale Robot Demonstration DatasetsEvaluation Benchmarks

    Intelligent Robotics and Vision Lab, University of Texas at Dallas · Allen High School

  57. Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges

    May 19, 2026Husnain Amjad, Raja Khurram Shahzad, Aamir Shahzad +1Mathematical ReasoningLLM Reasoning Strategies

    School of Electrical Engineering and Computer Science, National University of Sciences and Technology, Islamabad, Pakistan · Department of Communication, Quality Management and Information Systems, Mid Sweden University, Östersund Campus, Sweden · School of Computing, Data, and Mathematical Sciences, Western Sydney University, Indonesia

  58. Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction

    May 17, 2026Chaoqun He, Mingyang Xiang, Yingjing Xu +5Evaluation BenchmarksOmni

    Tsinghua University · Shanghai Qi Zhi Institute · ModelBest Inc.

  59. Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling

    May 13, 2026Xuehai Bai, Yang Shi, Yi-Fan Zhang +7Multi-Image Editing BenchmarksImage Editing

    HDU · PKU · Kling Team +1

  60. Large Language Models Lack Temporal Awareness of Medical Knowledge

    May 13, 2026Zihan Guan, Qiao Jin, Guangzhi Xiong +6Clinical Reasoning TrainingEvaluation Benchmarks

    University of Virginia · National Institutes of Health · Dana-Farber Cancer Institute +2