Agent Evaluation

Momentum

23 papers in the last four weeks, up 229% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 97

All topics
CardsList
  1. Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation

    Oct 5, 2026Dishu Yang, Qi Su, Hongbo Qin +1Benchmark AuditingAI Agent Auditing

  2. Continuous Process-Level Evaluation for Evolving Enterprise AI Agent Skills

    Oct 1, 2026Ngoc Phuoc An Vo, Aarya Doshi, Vadim SheininTool-Use EvaluationAI Agent Evaluation

  3. Auditing Web Agent Evaluation on WebArena-Lite: Human Review of Outcomes and Trajectories

    Oct 1, 2026Chengguang Gan, Zimeng He, Yoshihiro Tsujii +3AI Agent AuditingWeb Agent Benchmarks

  4. Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

    Sep 30, 2026Michael Hardy, Ruhana Azam, Anka Reuel +2AI Agent EvaluationAgent Evaluation

  5. Risk-Aware Adaptive Evaluation: Finding High-Impact Failures Under Limited Budgets

    Sep 30, 2026Priyanath Maji, Spandan Ghose ChowdhuryThompson SamplingAI Agent Evaluation

  6. CoSE-E: A Benchmark for Code-switched Speech Evaluation in Enterprise Settings

    Sep 28, 2026Shama Gupta, Hoang H Nguyen, Chelsea Huang +2Code-Switching Speech RecognitionASR Evaluation

  7. Reward Hacking Challenges Oversight of Autonomous Research Agents

    Sep 23, 2026Yue Huang, Zhangchen Xu, Yuchen Ma +12Reward HackingAI Agent Auditing

  8. Skill-based Agentic Evaluation for Real-time Data Science Tasks

    Sep 15, 2026Aniruddha Tamhane, Raghavendra Addanki, Ayushi Aggarwal +4LLM-as-a-JudgeLLM Agent Evaluation

  9. Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers

    Sep 11, 2026Hanhua Hong, Yizhi Li, Luu Gia Huy +3AI Agent EvaluationAgent Evaluation

  10. Qiushi Engine on AstaBench E2E-Bench-Hard

    Sep 8, 2026Wenhao Li, Shuxing Yang, Fujia Chen +13Agent EvaluationAI Agent Benchmarks

  11. APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents

    Sep 7, 2026Jintian Feng, Long Chen, Xiao Yu +7Mobile GUI AutomationGUI Agents

  12. ClaimReceipt: Verifying Evidence Sufficiency and Coverage in Agent Evaluations

    Sep 2, 2026Peiying Zhu, Sidi ChangClaim VerificationData Provenance

  13. DramaChain Bench: An End-to-End Benchmark for Short-Drama Generation

    Sep 1, 2026Haoyuan Shi, Mingtao Chen, Shuo Jiang +12Agent EvaluationTemporal Consistency in Video Generation

  14. Benchmarking LLM Judges for Voice-Agent Evaluation: Reliability, Calibration, and Human Oversight

    Aug 25, 2026Anupam Purwar, Shashank Singh, Kritika SrivastavaVoice Agent EvaluationLLM-as-a-Judge

  15. QuoteBench: How Matched Scores Can Hide Command-Path Failures

    Aug 13, 2026Shangao Li, Yao Zhang, Volker Tresp +1LLM Agent EvaluationAgent Evaluation

  16. DuplexWorld: Can voice agents help you get through the day?

    Aug 11, 2026Aryan Vijay Bhosale, Harshit Rajgarhia, Akhil Pothanapalli +3Voice Agent EvaluationSpoken Dialogue Systems

  17. SkillConsist: Detecting Inconsistencies in Agent Skills via Bidirectional Graph Alignment

    Aug 7, 2026Chaofan Meng, Yuhang Zheng, Yingnan Zhou +1Agent EvaluationAgent Skill Security Auditing

  18. What Is a Skill Worth? Structure-Aware Shapley Valuation of Agent Skills

    Aug 5, 2026Tao Li, Junfeng Liu, Qinghua Zhao +5Agent EvaluationShapley Value Attribution

  19. Agentic Commerce World: An Auditable and Verifiable Environment for Vibe Commerce

    Aug 3, 2026Shicheng Fan, Mingdai Yang, Duohao Wang +9AI Agent EvaluationAgent Evaluation

  20. Hear, Invoke, and Understand: A Skill-Calling Multimodal Agent for Large Audio Language Models

    Aug 3, 2026Yuwen Wang, Tian-Hao Zhang, Minghao Cai +7Tool-Augmented Language Model AgentsAgent Evaluation

  21. ShiJianBench: From Dialogue to Decision for Long-Horizon Evaluation of Investment Advisors

    Aug 2, 2026Jie Gong, Maowei Jiang, Zhiwei Liu +14Financial ServicesLong-Horizon Agent Evaluation

  22. CurveShift: Is Agent Progress Scalar? Separating Level from Shape

    Jul 31, 2026Hanwen Xing, Pengyun Wang, BingXu Meng +8LLM EvaluationAgent Evaluation

  23. MetaRoute-Bench: Evaluating Meta-Decision Policies for Agentic Workflow Routing

    Jul 31, 2026Natan Vidra, Alina Kapanova, Arun Kanhai +1Agentic WorkflowsAgent Evaluation

  24. Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation

    Jul 30, 2026Yuhang Zhu, Mingxuan Du, Benfeng Xu +3LLM EvaluationLarge Language Model-Based Role-Play Simulation

  25. Evidence-Ledger Adjudication for Claim-Evidence Traceability

    Jul 29, 2026Gengyu Chen, Yongjie Yu, Weiling WangClaim VerificationCitation Verification

  26. Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation

    Jul 28, 2026Stefan Krsteski, Charlotte Meyer, Guillaume Allegre +2AI Agent EvaluationAgent Evaluation

  27. Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluation

    Jul 27, 2026Jingkun Luo, Da-Tian PengData ProvenanceBenchmark Auditing

  28. SenWorld: A Digital-Twin Simulation for Generating Context-Rich Evaluation Data

    Jul 22, 2026Zenghui Zhou, Xiaoyang Li, Xiaoxuan Qiao +2Synthetic Data GenerationAgent Evaluation

  29. OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

    Jul 16, 2026Chengyu Shen, Yujie Fu, Gangtao Xin +13Computer-Use Agent BenchmarksAI Agent Evaluation

  30. Can Agentic Trading Systems Pay for Their Own Intelligence?

    Jul 11, 2026Qiqi Duan, Changlun Li, Chen Wang +10LLM Agent EvaluationAgent Evaluation

  31. Reliable and Developer-Aligned Evaluation of Agents for Software Engineering

    Jul 7, 2026Razvan Mihai PopescuSoftware Engineering AgentsLLM Agent Evaluation

  32. Agent Step Value: Auditing Evaluator-Channel Reversals in Black-Box Agent Traces

    Jul 5, 2026Andrew Zhang, Chengzhan LiLLM Agent EvaluationAI Agent Evaluation

  33. LitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review Platform

    Jul 1, 2026Ruotong Zhao, Zhiyu Chen, Xurui Liu +7Human Preference EvaluationLLM-as-a-Judge

  34. Linguistic Firewall: Geometry as Defense in Multi-Agent Systems Routing

    Jun 29, 2026Dvir Alsheich, Adar Peleg, Ben Hagag +3Adversarial Attacks on LLMsAgent Evaluation

  35. Grading the Grader: Lessons from Evaluating an Agentic Data Analysis System

    Jun 23, 2026Tian Zheng, Kai-Tai HsuLLM-as-a-JudgeHuman-in-the-Loop Evaluation

  36. LegalWorld: A Life-Cycle Interactive Environment for Legal Agents

    Jun 17, 2026Songhan Zuo, Shengbin Yue, Tao Chiang +4Agent EvaluationLLM Agents

  37. The Illusion of Multi-Agent Advantage

    Jun 11, 2026Prathyusha Jwalapuram, Hehai Lin, Chuyuan Li +7AI Agent EvaluationAgent Evaluation

  38. Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks

    Jun 10, 2026Mengyu Zheng, Kai Han, Boxun Li +13AI Coding AgentsSoftware Engineering Benchmarks

  39. MedCTA: A Benchmark for Clinical Tool Agents

    Jun 10, 2026Tajamul Ashraf, Hyewon Jeong, Fida Mohammad Thoker +1Agent EvaluationAI Agent Benchmarks

  40. DFAH-Bench: Benchmarking Observable Agent Instability in Financial Decision-Making

    Jun 10, 2026Raffi KhatchadourianAgent EvaluationAI Agent Benchmarks