AI Agent Evaluation

Follow this topic Ask about this topic

Momentum

58 papers in the last four weeks, up 100% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 341

All topics

The DeepSpeak-Agentic Dataset
Jun 2, 2026Sarah Barrington, Maty Bohacek, Hany FaridAI Agent Evaluation Human-AI Interaction
University of California, Berkeley, USA · Stanford University, USA
RobotValues: Evaluating Household Robots When Human Values Conflict
Jun 2, 2026Jongwook Han, Hyeongjin Kim, Yohan JoAI Agent Evaluation VLMs for Robotics
Graduate School of Data Science, Seoul National University
AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents
Jun 1, 2026Yiheng Shu, Bernal Jiménez Gutiérrez, Saisri Padmaja Jonnalagedda +3Continual Learning for LLM Agents AI Agent Evaluation
The Ohio State University · Johns Hopkins University · Intuit AI Research
HLL: Can Agents Cross Humanity's Last Line of Verification?
Jun 1, 2026Xinhao Song, Su Su, Sirui Song +6GUI Agents AI Agent Evaluation
1Shanghai Jiao Tong University · 2Shandong University · 3Tongji University
SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents
Jun 1, 2026Hao Cheng, Changtao Miao, Tianle Song +21AI Agent Evaluation LLM Agent Security
1The Hong Kong University of Science and Technology · 4The Hong Kong University of Science and Technology (Guangzhou) · 2Ant Digital Technologies, Ant Group +8
BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning
Jun 1, 2026Shannon Serrao, Soumitra Chatterjee, Dorina Strori +2LLM Agent Evaluation AI Agent Evaluation
Merkle Analytics
AutoMedBench: Towards Medical AutoResearch with Agentic AI Models
Jun 1, 2026Junqi Liu, Selena Song, Yuhan Wang +12AI Agent Evaluation AI Agent Benchmarks
University of California, Santa Cruz · 2NVIDIA
TraceGraph: Shared Decision Landscapes for Diagnosing and Improving Agent Trajectories
May 29, 2026Junjie Nian, Kang Chen, Ge Zhang +2Agent Reliability Software Engineering Benchmarks
1Fudan University · 2ByteDance · 3Shanghai Innovation Institute
Design Principles for Human-Agent Interaction
May 28, 2026Haiyi Zhu, Canwen Wang, Qing Xiao +1AI Agent Evaluation Human-AI Interaction
Carnegie Mellon University · Pittsburgh, PA 15213
Gram: Assessing sabotage propensities via automated alignment auditing
May 28, 2026David Lindner, Victoria Krakovna, Sebastian FarquharAI Coding Agents AI Agent Auditing
1Google DeepMind
Notation Matters: A Benchmark Study of Token-Optimized Formats in Agentic AI Systems
May 28, 2026Lorenz Kutschka, Bernhard GeigerTool-Augmented Language Model Agents AI Agent Evaluation
Know Center Research GmbH / Sandgasse 34, A-8010 Graz · Signal Processing and Speech Communication, Graz University of Technology / Inffeldgasse 16c, A-8010 Graz · Graz Center for Machine Learning // A-8010 Graz
WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction
May 28, 2026Chengzhi Liu, Yuzhe Yang, Sophia Xiao Pu +14Multimodal Memory LLM Agent Memory
University of California, Santa Barbara · Johns Hopkins University · Carnegie Mellon University +3
Paper Agents, Paper Gains: An Empirical Analysis of DeFi Investment Agents
May 27, 2026Jay Yu, Amy Zhao, Danning SuiFinancial Services AI Agent Evaluation
Pantera Capital · Stanford University · IC3 +1
First head-to-head comparison of agentic AI applied to the analysis of simulated data of the Einstein Telescope
May 27, 2026Gianluca IngugliaAI Agents for Scientific Discovery AI Agent Evaluation
Marietta Blau Institute for Particle Physics, Austrian Academy of Sciences, 1010 Vienna, Austria
AI Research Agents Narrow Scientific Exploration
May 27, 2026Yixuan Tang, Yi YangAI Agents for Scientific Discovery AI Agent Evaluation
1*ISOM, The Hong Kong University of Science and Technology.
EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents
May 27, 2026Yunqi Liu, Tong Niu, Zitong Wang +4AI Agent Evaluation Egocentric Video Understanding
Ant Group
VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions
May 26, 2026Yuxin Chen, Yi Zhang, Zhengzhou Cai +11LLM Agent Memory AI Agent Evaluation
1National University of Singapore · 2Meituan · University of Science and Technology of China +2
Anticipate and Learn: Unleashing Idle-Time Compute in Proactive Agents
May 25, 2026Haoyi Hu, Qirong Lyu, Xianghan Kong +7AI Agent Evaluation AI Agent Benchmarks
1Shanghai Jiao Tong University · 2Tencent
Explore Before You Solve: The Speed--Depth Trade-off in Epistemic Agents for ARC-AGI-3
May 25, 2026Liew Keong HanGame-Playing Agents AI Agent Evaluation
Independent researcher.
Behind EvoMap: Characterizing a Self-Evolving Agent-to-Agent Collaboration Network
May 25, 2026Qiming Ye, Peixain Zhang, Yupeng He +2AI Agent Evaluation Multi-Agent Collaboration
The Hong Kong University of Science and Technology (Guangzhou) China
Insuring Every Action: An Authority Frontier Framework for Runtime Actuarial Control of Autonomous AI Agents
May 25, 2026Hao-Hsuan ChenAI Risk Management AI Agent Evaluation
Department of Risk Management and Insurance National Chengchi University
AION: Next-Generation Tasks and Practical Harness for Time Series
May 24, 2026Tianxiang Zhan, Xiaobao Song, Tong Guan +2Time Series Reasoning Time Series Forecasting
1Griffith University · 2Shenzhen University · 3Zhejiang University
The Open Source Economic Index of AI Adoption and Capability
May 23, 2026Seamus Somerstep, Aritra Guha, Divesh Srivastava +1AI Agent Evaluation AI Agent Benchmarks
University of Michigan · AT&T Chief Data Office · IFM-MBZUAI
AvalancheBench: Evaluating Enterprise Data Agents Through Latent World Recovery
May 22, 2026Darek Kleczek, Fuheng Zhao, Alexander W. Lee +4AI Agent Evaluation AI Agent Benchmarks
Snowflake · Brown University and Snowflake
ProofAgent Harness: Open Infrastructure for Adversarial Evaluation of AI Agents
May 22, 2026Fouad BousetouaneAI Agent Reliability AI Agent Auditing
The University of Chicago, USA
Design and Report Benchmarks for Knowledge Work
May 22, 2026Yining Hua, Hongbin Na, Cyrus Ayubcha +1Benchmark Design AI Agent Evaluation
Harvard University · University of Technology Sydney · Stanford University +1
A measurement substrate for agentic Kubernetes operations: Methodology and a case study in retrieval-compounding falsification
May 21, 2026Joshua Odmark, Gideon Rubin, Deon van der VyverRetrieval-Augmented Generation AI Agent Evaluation
∗Independent. · †LDE. · ‡Cognyx.
Advancing Mathematics Research with AI-Driven Formal Proof Search
May 21, 2026George Tsoukalas, Anton Kovsharov, Sergey Shirobokov +18Automated Theorem Proving AI Agent Evaluation
1Google DeepMind1 · 2Aarhus University · 3Google
Measuring Security Without Fooling Ourselves: Why Benchmarking Agents Is Hard
May 21, 2026Sahar Abdelnabi, Chris Hicks, Konrad Rieck +1AI Agent Evaluation AI Agent Security
ELLIS Institute Tübingen & MPI-IS & Tübingen AI Center, Germany · The Alan Turing Institute, London, UK · BIFOLD & Technische Universität Berlin, Germany +1
Towards Direct Evaluation of Harness Optimizers via Priority Ranking
May 21, 2026Kai Tzu-iunn Ong, Minseok Kang, Dongwook Choi +11AI Agent Evaluation Agent Harness Optimization
Yonsei University · Microsoft Research Asia · Texas A&M University

Jun 2, 2026, cs.AIJ/K move · Enter open · S save

The DeepSpeak-Agentic Dataset

Sarah Barrington, Maty Bohacek, Hany Farid

University of California, Berkeley, USA · Stanford University, USA

AI Agent Evaluation Human-AI Interaction Deepfake Detection Embodied AI

Open paper AskPDF arXiv

We present DeepSpeak-Agentic, a dataset of videos comprising over 37 hours of semi-structured conversations between a human and an embodied AI agent. We use this dataset to evaluate the automatic forensic identification (audio, video, or text) of AI agents, study the nature of human-agent interactions, and provide a benchmark for future advances in the large-language models and AI-generated voices and faces that power embodied AI agents. We also contribute a scalable data-capture system that creates agents, automatically pairs them with human crowd workers, records audiovisual conversations across specified scenarios, and identifies and separates the human and agent in the combined stream.

Momentum

Related topics

Latest papers 341