Automated Evaluation

Follow this topic Ask about this topic

Momentum

48 papers in the last four weeks, up 12% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 321

All topics

RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions
Jun 2, 2026Zongwei Lv, Zhewen Tan, Yaoming Li +7Computer-Use Agent Benchmarks AI Coding Agents
1Peking University · 2Qiyuan Tech · 2Qiyuan Tech*
WebRISE: Requirement-Induced State Evaluation for MLLM-Generated Web Artifacts
Jun 2, 2026Yuxin Meng, Yuhan Suo, Junjie Wang +9Automated Software Testing Automated Evaluation
Tsinghua University · Huawei Noah’s Ark Lab · East China Normal University +2
Any2Poster: Any-Source Poster Generation Across Modalities and Domains
Jun 1, 2026Amogh Vinaykumar, Aiden Li, Suozhi Huang +1Image Generation Multimodal Generation
Flower Mound High School · University College London · Princeton University
BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning
Jun 1, 2026Shannon Serrao, Soumitra Chatterjee, Dorina Strori +2LLM Agent Evaluation AI Agent Evaluation
Merkle Analytics
RWGBench: Evaluating Scholarly Positioning in Related Work Generation
May 30, 2026Anzhe Xie, Weihang Su, Jiaxin Mao +4LLM Evaluation Automated Evaluation
Tsinghua University Beijing, China · Renmin University of China Beijing, China
How Far Do Auto-Interpretation Labels Generalize: A Controlled Study Across Languages, Scripts, and Rewordings
May 29, 2026Sripad KarneLLM Interpretability Sparse Autoencoders
Columbia University
Refining Word-Based Grammatical Error Annotation for L2 Korean
May 28, 2026Jungyeul Park, Kyungtae Lim, Wonjun Oh +4Grammatical Error Correction Automated Evaluation
Korea Advanced Institute of Science & Technology, South Korea. · The University of British Columbia, Canada · Saint Elizabeth University, USA +1
Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generation
May 28, 2026Haoyue Yang, Zhangxiao Shen, Fan Ding +8Code Generation Evaluation Automated Evaluation
Baidu Inc., Beijing, China
A Novel Evaluation Metric for Unsupervised Learning in AIS-Based Maritime Anomaly Detection: MADQI
May 28, 2026Ismet Gocer, Zakirul Bhuiyan, Raza Hasan +1Unsupervised Anomaly Detection Automated Evaluation
Southampton Solent University School of Technology and Maritime Industries Southampton, United Kingdom
Efficient, Validation-Free Intrinsic Quality Estimation for Large-Scale Face Recognition Datasets
May 28, 2026Zhichao Chen, Yongle Zhao, Kaicheng Yang +3Face Recognition Automated Evaluation
DeepGlint · School of Cyber Science and Technology, University of Science and Technology of China.
Benchmarking AI for low-resource contexts: Thinking beyond leaderboards
May 27, 2026Aakash Pant, Kavya Shah, Apoorv Agnihotri +3Benchmark Design Automated Evaluation
Wadhwani AI Global
Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantification
May 27, 2026Dylan Bouchard, Mohit Singh Chauhan, Zeya Ahmad +1Language Model Calibration LLM Uncertainty Estimation
CVS Health®, Wellesley, MA, USA
Does Capability Transfer to Subjective Behavior -- and Would Our Instruments Tell Us? A Self-Evolving, Trust-by-Construction Evaluation Paradigm
May 27, 2026Yuming, Huang, Yao Liu +3LLM Evaluation Benchmark Validity
Rapheal · Cylingo team
AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applications
May 26, 2026Yifan Sui, Xin Huang, Hongbing Li +14Mobile GUI Automation Computer-Use Agent Benchmarks
1Beijing University of Posts and Telecommunications · 2StepFun · 3Waseda University
MATCHA: Matching Text via Contrastive Semantic Alignment
May 26, 2026Siran Li, Ece Sena Etoglu, Carsten Eickhoff +1LLM Evaluation Semantic Textual Similarity
University of Tübingen
GrowLoop: Self-Evolving Conversation Evaluation Seeded by Human
May 26, 2026Yihang Lin, Yunze Gao, Zeyang Lin +3Benchmark Design Human-in-the-Loop Evaluation
Amap, Alibaba Group · The Chinese University of Hong Kong, Shenzhen
E3: Issue-Level Backtesting for Automated Research Critique
May 26, 2026Yashwardhan Chaudhuri, Sanyam Jain, Paridhi MundraAutomated Evaluation Automated Peer Review
HTMLCure: Turning Browser Experience into State Guided Repair for Interactive HTML
May 26, 2026Jiajun Wu, Jian Yang, Tuney Zheng +4Automated Software Testing Automated Program Repair
1Beihang University · 2IQuest Research · 3Peking University
Workflow Closure Is Not Scientific Closure in Auto-Research Systems
May 25, 2026Shuai Wang, Xinyuan Tian, Pangpang Liu +1Human-in-the-Loop Evaluation Autonomous Scientific Discovery
School of Public Health, Yale University, USA
QUIET: A Multi-Blank Cascaded Story Cloze Benchmark for LLM Creative Generation Capability
May 25, 2026Bo Zou, Chao XuOpen-Ended Generation Creativity Assessment
RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations
May 25, 2026Hanyu Li, Yichi Zhang, Speed Zhu +3AI Coding Agents Software Engineering Benchmarks
Beijing University of Posts and Telecommunications · Tsinghua University · Tencent
A Multi-Agent LLM Framework for Rating the Quality of Surgical Feedback
May 25, 2026Rafal Kocielnik, J. Everett Knudsen, Steven Y. Cen +7Educational Technology Automated Evaluation
1Computing + Mathematical Sciences, California Institute of2026 Technology, 1200 E. California Blvd, Pasadena, 91125, CA, USA. · 3Keck School of Medicine, University of Southern California, 1500 San 25 Pablo Street, Los Angeles, 90033, CA, USA. · Department of Urology, Cedars-Sinai, 8700 Beverly Blvd, Los Angeles, 90048, CA, USA.May
AVBench: Human-Aligned and Automated Evaluation Benchmark for Audio-Video Generative Models
May 23, 2026Jialiang Yang, Bin Xia, Ruihang Chu +6Audio-Video Generation Audio-Visual Alignment
Tsinghua University · The Chinese University of Hong Kong
Towards Evaluation Engineering: An Empirical Study of ML Evaluation Harnesses in the Wild
May 22, 2026Zhimin Zhao, Zehao Wang, Abdul Ali Bangash +2Software Engineering Automated Evaluation
Software Analysis and Intelligence Lab (SAIL), School of Computing, Queen’s University, Canada · Concordia University, Canada · Lahore University of Management Sciences (LUMS), Pakistan
Learning to Evaluate: Cost-Effective Model Evaluation on Unlabeled Data with Meta-Learning
May 22, 2026Trinh Pham, Viet Huynh, Hongzhi Yin +2Automated Evaluation
Griffith University · Edith Cowan University · The University of Queensland
AtelierEval: Agentic Evaluation of Humans & LLMs as Text-to-Image Prompters
May 21, 2026Hanjun Luo, Zhimu Huang, Sylvia Chung +6T2I Generation Agent Evaluation
New York University Abu Dhabi · Zhejiang University · University of Electronic Science and Technology of China +2
Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
May 21, 2026Asaf Yehudai, Lilach Eden, Michal Shmueli-ScheuerLLM Agent Evaluation AI Agent Benchmarks
IIBM Research
The Constraint Tax: Measuring Validity-Correctness Tradeoffs in Structured Outputs for Small Language Models
May 20, 2026Jaideep RayConstrained Decoding Small Language Models
Hack-Verifiable Environments: Towards Evaluating Reward Hacking at Scale
May 20, 2026Amit Roth, Ankur Samanta, Matan Halevy +2Reward Hacking AI Agent Benchmarks
1Tel Aviv University · 2Columbia University · 3Taso Labs
OpenComputer: Verifiable Software Worlds for Computer-Use Agents
May 19, 2026Jinbiao Wei, Qianran Ma, Yilun Zhao +4Computer-Use Agent Benchmarks Computer-Use Agents
Yale NLP Lab · University of Pennsylvania · University of North Carolina at Chapel Hill

Jun 2, 2026, cs.CLJ/K move · Enter open · S save

RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions

Zongwei Lv, Zhewen Tan, Yaoming Li, Yilun Yao+6

1Peking University · 2Qiyuan Tech · 2Qiyuan Tech*

Computer-Use Agent Benchmarks AI Coding Agents Software Engineering Benchmarks AI Agent Benchmarks Automated Evaluation Agent Benchmarks

Open paper AskPDF arXiv

Agent benchmarks should reflect what users actually ask deployed agents to do, yet existing benchmarks often miss key realism properties of real developer-agent sessions. We introduce RealClawBench, a live benchmark framework built from real OpenClaw sessions to capture the distribution, diversity, and real-world difficulty of deployed agent use. Real user requests are challenging to benchmark because they often depend on local execution environments, involve implicit or underspecified intent, and require nontrivial verification. RealClawBench addresses these challenges with two core mechanisms: reconstructed execution environments and deterministic verifiable scorers, which together convert real sessions into reproducible, automatically scored tasks. The resulting release contains 281 executable tasks sampled from a much larger real-session pool while preserving the source distribution, with maximum final-vs-source Jensen-Shannon divergence of 0.0448. Evaluating 14 contemporary models shows that the best system solves only 65.8% of tasks, revealing substantial headroom on realistic developer-agent workloads. By turning real deployed sessions into controlled evaluation instances, RealClawBench provides a practical path toward benchmarks that better measure agent capability in actual use. Code is available at:https://anonymous.4open.science/r/real-claw-bench-582B.

Momentum

Related topics

Latest papers 321