AI Agent Evaluation

Ask about this topic

Momentum

58 papers in the last four weeks, up 100% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 341

All topics
CardsList

  1. Reward-Adaptive Iterative Discovery: A Case Study on Automated Game Testing for NHL26

    Jul 8, 2026Florian Fuchs, Jessy Gosselin-Grant, Boris Skuin +5Automated Software TestingReinforcement Learning

    Electronic Arts (EA)

  2. Measuring Intelligence Beyond Human Scale

    Jul 8, 2026Jerry Han, Rafael Moschopoulos, Ella Colby +6LLM EvaluationAI Agent Evaluation

    Princeton Superalignment

  3. A Gold-Standard Study of What Makes a Lightweight Game-Playing Agent Strong

    Jul 7, 2026Nima Kelidari, Mohammadsaeed Haghi, Mahdi SalmaniReinforcement LearningGame-Playing Agents

    University of Southern California

  4. An Experimental Design Approach to Evaluating Agentic AI's Autonomous Model Discovery

    Jul 7, 2026Hao He, Xueying Liu, Chris J. Kuhlman +1AI Coding AgentsAI Agent Evaluation

    Department of Statistics, Virginia Tech · Department of Statistical Science, Baylor University · Advanced Research Computing, Virginia Tech +1

  5. AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation

    Jul 7, 2026Andrey Podivilov, Vadim Lomshakov, Sergey Savin +4Computer-Use Agent BenchmarksAI Coding Agents

    1Explyt · 2St. Petersburg Department of the Steklov Institute of Mathematics · 3St. Petersburg State University

  6. Beyond Static Evaluation: Building Simulation Environments for Scalable Agentic Reinforcement Learning

    Jul 7, 2026Akshay Arora, Ishan Nigam, Ashutosh Aggarwal +6Agentic RLAI Agent Evaluation

    Uber AI Solutions San Francisco, CA, USA

  7. Collective Intelligence with Foundation Models

    Jul 6, 2026J. de Curtò, I. de ZarzàAI Agent EvaluationMulti-Agent Collaboration

    Department of Computer Applications in Science & Engineering, BARCELONA Supercomputing Center, 08034 Barcelona, Spain · Escuela Técnica Superior de Ingeniería (ICAI), Universidad Pontificia Comillas, 28015 Madrid, Spain · Human Centered AI, Data & Software, LUXEMBOURG Institute of Science and Technology, L-4362 Esch-sur-Alzette, Luxembourg

  8. Agent Step Value: Auditing Evaluator-Channel Reversals in Black-Box Agent Traces

    Jul 5, 2026Andrew Zhang, Chengzhan LiLLM Agent EvaluationAI Agent Evaluation

    KTH Royal Institute of Technology · University of Electronic Science and Technology of China

  9. Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure

    Jul 5, 2026Guijia Zhang, Harry YangGUI AgentsAI Agent Evaluation

    1Shenzhen University · 2The Hong Kong University of Science and Technology

  10. CAGE-1: Control, Assurance, and Governance Evaluation for Enterprise Agentic AI

    Jul 3, 2026Roopam W. SureAI Agent EvaluationAI Agent Safety

  11. Beyond Forecasting: The Belief-to-Trade Layer in Prediction-Market Agents

    Jul 3, 2026Yishu Wang, Yuxuan Wang, Jiaqi Deng +1AI Agent EvaluationPrediction Markets

    Hong Kong University of Science and Technology · Peking University · The University of Hong Kong +1

  12. TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution

    Jul 2, 2026Jiale Amber Wang, Kaiyuan Wang, Pengyu NieAutomated Software TestingBenchmark Design

    University of Waterloo · 2Google

  13. Accuracy and Cost Claims Do Not Survive Re-Execution in Agentic VideoQA

    Jul 1, 2026Rama AlHamidi, Aseel Mohamed, Rasul Khanbayov +3AI Agent EvaluationVideo QA

    Electrical and Computer Engineering, Texas A&M University, TX, United States · College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar

  14. Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?

    Jul 1, 2026Zhi Chen, Zhensu Sun, Yuling Shi +2Benchmark AuditingBenchmark Design

    Singapore Management University · Singapore, Singapore · Shanghai Jiao Tong University +1

  15. FARS: A Fully Automated Research System Deployed at Scale

    Jun 30, 2026Qiong Tang, Tianxiang Sun, Xiangkun Hu +54AI Agent EvaluationAutonomous Scientific Discovery

  16. SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

    Jun 29, 2026Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal +1Coding AgentsSoftware Engineering Benchmarks

    1Scale AI

  17. SWE-Together: Evaluating Coding Agents in Interactive User Sessions

    Jun 29, 2026Yifan Wu, Zhuokai Zhao, Songlin Li +8Coding AgentsAI Agent Evaluation

  18. SEATauBench: Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages

    Jun 27, 2026My Chiffon Nguyen, Aulia Adila, Saksorn Ruangtanusak +4LLM Agent EvaluationAI Agent Evaluation

    SEACrowd · SCB DataX, SCBX Group · Chulalongkorn University +2

  19. RouterVLA: Turning Smoke Tests into Supervision for Heterogeneous VLA Selection

    Jun 25, 2026Xingyu Ren, Chugang Yi, Ge Ma +1Robot Policy LearningAI Agent Evaluation

    1The Chinese University of Hong Kong · University of Maryland, College Park · 3Tsinghua University

  20. The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators

    Jun 24, 2026Alex Iacob, Andrej Jovanović, William F. Shen +12Evolutionary OptimizationAI Agent Evaluation

    University of Cambridge · Flower Labs · NVIDIA +2

  21. Agentic System as Compressor: Quantifying System Intelligence in Bits

    Jun 24, 2026Zihan Qin, Hongrui ZhangAI Agent EvaluationAutomated Evaluation

    Peking University

  22. Probabilistic Agents in Deterministic Audits: Evaluating Multi-Agent Systems for Automated Audits Based on the German IT-Grundschutz

    Jun 24, 2026Lea Roxanne Muth, Marian MargrafMulti-Agent LLM SystemsAI Agent Evaluation

    Department of Mathematics and Computer Science Freie Universität Berlin Berlin, Germany

  23. Life After Benchmark Saturation: A Case Study of CORE-Bench

    Jun 23, 2026Nitya Nadgir, Sayash Kapoor, Kangheng Liu +11OOD GeneralizationAI Agent Evaluation

    1Independent · 2Princeton University · 3UC Berkeley +1

  24. Heuresis: Search Strategies for Autonomous AI Research Agents Across Quality, Diversity and Novelty

    Jun 23, 2026Antonis Antoniades, Deepak Nathani, Ritam Saha +6Novelty SearchAI Agent Evaluation

    University of California, Santa Barbara · Hexo AI

  25. NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

    Jun 23, 2026Yuru Wang, Lejun Cheng, Yuxin Zuo +14AI for ScienceCoding Agents

    Horizon Research, Frontis.AI Tsinghua University

  26. Litmus: Zero-Label, Code-Driven Metric Specification for Evaluating AI Systems

    Jun 22, 2026Prajjwal Gupta, Prasang Gupta, Vishal Bhutani +4AI Agent EvaluationAI Agent Monitoring

    PricewaterhouseCoopers, U.S.

  27. Position: Correct Answer, Wrong Mechanism -- When AI Scientists Defend General Claims Their Own Data Contradicts

    Jun 22, 2026Steven Young EuligAI Agent ReliabilityReasoning Evaluation

    Department of Physics and Laboratory for Particle Physics and Cosmology (LPPC), Harvard University, Cambridge, MA 02138, USA.

  28. When AUC 0.998 Is Not Enough: A Candidate Evaluation Protocol for Hidden-State Probes of Indirect Prompt Injection in Multimodal Computer-Use Agents

    Jun 22, 2026Yanhang Li, Zhichao Fan, Zexin ZhuangAI Agent EvaluationIndirect Prompt Injection

    Northeastern University Boston, Massachusetts, USA · University of Illinois Urbana-Champaign Urbana, Illinois, USA · Southern Methodist University Dallas, Texas, USA

  29. RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents

    Jun 21, 2026Meher Bhaskar Madiraju, Meher Sai Preetam MadirajuAI Agent ReliabilityCoding Agents

  30. Autonomous Event-Driven Multi-Agent Orchestration for Enterprise AI at Scale

    Jun 18, 2026Harsh Rao Dhanyamraju, Leonidas Raghav, Aaron LeeMulti-Agent OrchestrationAI Agent Evaluation

    Data Scientist, SAP SE · ML Engineer, SAP SE