Agentic Benchmarks

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

13 new papers

A weekly snapshot of new work published in Agentic Benchmarks.

Period ending 2026-09-14

13 new papers

A weekly snapshot of new work published in Agentic Benchmarks.

Period ending 2026-09-07

17 new papers

A weekly snapshot of new work published in Agentic Benchmarks.

Inside this field

Focused directions

Focused topic

Agentic Benchmarks

Within Agentic Benchmarks

253 papersExplore

Focused topic

Agentic Evaluations

Within Agentic Benchmarks

148 papersExplore

Focused topic

Opti-Agent-Bench

Within Agentic Benchmarks

24 papersExplore

414 papers

Latest in Agentic Benchmarks

Browse all topics →

Be Fair! Can Machine Learning Engineering Agents Adhere to Fairness Constraints?
Jun 3, 2026Anna Richter, Julia Stoyanovich, Sebastian SchelterFairness Constraints Agentic Evaluations
Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents
Jun 3, 2026Haoyu Sun, Wenxuan Wang, Mingyang Song +5Large Language Model Planning Agentic Benchmarks
Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety
Jun 3, 2026Catherine Ge-Wang, Tyler Crosse, Benjamin Hadad +3Artificial Intelligence Safety Attack Paper
Can Generalist Agents Automate Data Curation?
Jun 2, 2026Feiyang Kang, Hanze Li, Adam Nguyen +5Data-Curation Agentic Benchmarks
Hedge-Bench: Benchmarking Agents on Hard, Realistic Tasks Pertaining to Financial Reasoning
Jun 2, 2026Eric Cho, Shawn Huang, Alice Lu +1Finance Benchmarks Agentic Benchmarks
Diagnosing Knowledge Gaps in LLM Tool Use: An Agentic Benchmark for Novel API Acquisition
Jun 2, 2026Jinnuo Liu, Yue Peng, Jinhan Niu +1Code Generation Application Programming Interfaces
SAGE: A Quantitative Evaluation of Socialized Evolution in Agent Ecosystems
Jun 2, 2026Linyue Pan, Yaoming Zhu, Lin Qiu +2Self-Evolving Agents Agentic Evaluations
What Benchmarks Don't Measure: The Case for Evaluating Abstention Competence in Autonomous Agents
Jun 1, 2026Victor Ojewale, Suresh VenkatasubramanianAgentic Evaluations Safety Evaluation
Tracking the Behavioral Trajectories of Adapting Agents
Jun 1, 2026Jonah Leshin, Manish Shah, Ian TimmisAgentic Evaluations Trajectory Prediction
IMAC-AgriVLN: Can Agricultural Vision-and-Language Navigation Agents be Aware of Instruction Mistakes?
Jun 1, 2026Xiaobei Zhao, Xingqi Lyu, Xin Chen +1Vision-Language Navigation Agricultural Robots
Monitoring Agentic Systems Before They're Reliable
Jun 1, 2026Marisa Ferrara Boston, Glen Hanson, Effi Georgala +2Agentic Systems Agentic Evaluations
MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulation
Jun 1, 2026Wenhao Wang, Peizhi Niu, Gongyi Zou +9Agentic Benchmarks Model Context Protocol
K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts
Jun 1, 2026Nahyun Lee, Dongkeun Yoon, Guijin Son +12Agentic Benchmarks Korean
SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents
Jun 1, 2026Hao Cheng, Changtao Miao, Tianle Song +21Security Analysis Agentic Evaluations
BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning
Jun 1, 2026Shannon Serrao, Soumitra Chatterjee, Dorina Strori +2Text-To-Sql Agentic Benchmarks
RescueBench: Can Embodied Agents Save Lives in the Wild ?
Jun 1, 2026Kui Wu, Beiyu Guo, Hao Chen +6Embodied Agents Agentic Benchmarks
CRAB-Bench: Evaluating LLM Agents under Complex Task Dependencies and Human-aligned User Simulation
Jun 1, 2026Danqing Wang, Akshay Sivaraman, Lei LiUser Simulation Agentic Benchmarks
From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents
Jun 1, 2026Laksh AdvaniAgentic Benchmarks Falsehood
TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning
May 31, 2026Yaxuan Kong, Qingren Yao, Yuqi Nie +7Agentic Benchmarks Time Series
Momento: Evaluating Persistent Memory and Reasoning with Multi-Session Agentic Conversations
May 30, 2026Adril Putra Merin, David Anugraha, Ayu Purwarianti +1Long-Horizon Agents Agentic Benchmarks
Industrializing Prediction-Powered Inference: The GLIDE Library for Reliable GenAI and Agentic Systems Evaluation
May 29, 2026Grégoire Martinon, Ibrahim Merad, Mohammed RakiAgentic Evaluations Industrial
BlueFin: Benchmarking LLM Agents on Financial Spreadsheets
May 29, 2026Srivatsa Kundurthy, Clara Na, Colton Moraine +6Spreadsheets Finance Domain
EASE Configuration Facilitates A Reproducible Science of LLM Social Simulations
May 28, 2026Sneheel Sarangi, Maximilian Puelma Touzel, Aurélien Bück-Kaeffer +3User Simulation Multi-Agent Simulations
Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams
May 28, 2026Yinsicheng Jiang, Liang Cheng, Yeqi Huang +6Agentic Benchmarks Large Language Model Agents
ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research
May 28, 2026Wanghan Xu, Shuo Li, Tianlin Ye +48Autoresearch Agentic Benchmarks
Redundant or Necessary? A Benchmark for Detecting Redundant Steps in Agent Trajectories
May 28, 2026Minyang Hu, Bo Yang, Zhinuo Zhou +4Agentic Benchmarks Redundancy
Realistic honeypot evaluations for scheming propensity
May 28, 2026Victoria Krakovna, David Lindner, Lewis Ho +2Agentic Evaluations Honeypots
Notation Matters: A Benchmark Study of Token-Optimized Formats in Agentic AI Systems
May 28, 2026Lorenz Kutschka, Bernhard GeigerAgentic Benchmarks
PTCG-Bench: Can LLM Agents Master Pokémon Trading Card Game?
May 28, 2026Dongdong Hua, Yifei Sun, Renhong Huang +3Self-Evolving Agents Agentic Benchmarks
WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction
May 28, 2026Chengzhi Liu, Yuzhe Yang, Sophia Xiao Pu +14Multimodal Memory Multimodal Agents
GTA: Generating Long-Horizon Tasks for Web Agents at Scale
May 28, 2026Tenghao Huang, Kung-Hsiang Huang, Prafulla Kumar Choubey +4Web Agents Agentic Benchmarks
A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks
May 27, 2026Tomer Keren, Nitay Calderon, Asaf Yehudai +3Agentic Benchmarks Task Synthesis
You Live More Than Once: Towards Hierarchical Skill Meta-Evolving
May 27, 2026Xujun Li, Kehan Zheng, Mingyuan Zhao +7Skill Evolution Agentic Benchmarks
UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
May 27, 2026Pengyu Zhu, Lijun Li, Yaxing Lyu +8Agentic Evaluations Large Language Model Evaluation
VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wild
May 27, 2026Xiaohongshu IncAgentic Benchmarks Knowledge Graphs
EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents
May 27, 2026Yunqi Liu, Tong Niu, Zitong Wang +4Multimodal Agents Agentic Benchmarks
Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systems
May 26, 2026Aman Priyanshu, Supriti Vijay, Esha PahwaLarge Language Model Agents Agentic Evaluations
Chain-based Adaptive Reconfiguration Over Lattices for Hallucination Reduction
May 26, 2026Joan Vendrell Gallart, Solmaz Kia, Russell Bent +1Large Language Model Hallucination Hallucination Mitigation
MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research
May 25, 2026Dingbang Wu, Rui Hao, Haiyang Wang +8Footsiesgym Agentic Benchmarks
Claw-Anything: Benchmarking Always-On Personal Assistants with Broader Access to User's Digital World
May 25, 2026Yusong Lin, Xinyuan Liang, Haiyang Wang +8Agentic Benchmarks Artificial Intelligence Assistants
Explore Before You Solve: The Speed--Depth Trade-off in Epistemic Agents for ARC-AGI-3
May 25, 2026Liew Keong HanAgentic Benchmarks Efficient Exploration
CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents
May 25, 2026Bowen Wang, Dunjie Lu, Junli Wang +11Reinforcement Learning With Verifiable Reward Computer-Use Agents
Agent-ToM: Learning to Monitor Autonomous LLM Agents via Theory-of-Mind Reasoning
May 22, 2026Nesreen K. Ahmed, Nima NafisiLarge Language Model Agents Agent Framework
ProofAgent Harness: Open Infrastructure for Adversarial Evaluation of AI Agents
May 22, 2026Fouad BousetouaneAgentic Evaluations Adversarial Evaluation
A measurement substrate for agentic Kubernetes operations: Methodology and a case study in retrieval-compounding falsification
May 21, 2026Joshua Odmark, Gideon Rubin, Deon van der VyverKubernetes Agentic Evaluations
EVE-Agent: Evidence-Verifiable Self-Evolving Agents
May 21, 2026Yamato Arai, Yuma IchikawaSelf-Evolving Agents Agentic Evaluations
AtelierEval: Agentic Evaluation of Humans & LLMs as Text-to-Image Prompters
May 21, 2026Hanjun Luo, Zhimu Huang, Sylvia Chung +6Text-To-Image Agentic Evaluations
Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
May 21, 2026Asaf Yehudai, Lilach Eden, Michal Shmueli-ScheuerAgentic Evaluations Agentic Systems
Measuring Security Without Fooling Ourselves: Why Benchmarking Agents Is Hard
May 21, 2026Sahar Abdelnabi, Chris Hicks, Konrad Rieck +1Security Analysis Agentic Evaluations
SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations
May 21, 2026Shuaiqi Wang, Aadyaa Maddi, Zinan Lin +1Synthetic Data Agentic Evaluations
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
May 21, 2026Zhaoyang Chu, Jiarui Hu, Xingyu Jiang +8Real-World Tasks Agentic Benchmarks
Towards Direct Evaluation of Harness Optimizers via Priority Ranking
May 21, 2026Kai Tzu-iunn Ong, Minseok Kang, Dongwook Choi +9Agentic Optimization Agent Harness
Benchmarking Autonomous Agents against Temporal, Spatial, and Semantic Evasions
May 21, 2026Jianan Ma, Xiaohu Du, Ruixiao Lin +8Evasion Agentic Benchmarks
SGR-Bench: Benchmarking Search Agents on State-Gated Retrieval
May 21, 2026Ningyuan Li, Haiyang Shen, Mugeng Liu +4Search Agents Agentic Benchmarks
What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
May 20, 2026Mahdi Naser Moghadasi, Faezeh GhaderiAgentic Benchmarks Algorithm Auditing
MemGym: a Long-Horizon Memory Environment for LLM Agents
May 20, 2026Wujiang Xu, Yu Wang, Kai Mei +8Agent Memory Agentic Benchmarks
Evaluating Temporal Semantic Caching and Workflow Optimization in Agentic Plan-Execute Pipelines
May 20, 2026Alimurtaza Mustafa Merchant, Krish Veera, Sajal Kumar Goyla +3Execution Efficiency Agentic Workflows
Distribution-Free Uncertainty Quantification for Continuous AI Agent Evaluation
May 19, 2026Yuxuan Gao, Megan Wang, Yi Ling YuAgentic Evaluations Conformal Inference
CutVerse: A Compositional GUI Agents Benchmark for Media Post-Production Editing
May 19, 2026Haobo Hu, Xiangwu Guo, Zhiheng Chen +4Graphical User Interface Agents Multimodal Agents
Toward User Comprehension Supports for LLM Agent Skill Specifications
May 19, 2026Zikai Alex WenAgentic Evaluations Competency Questions

Jun 3, 2026 · cs.LGJ/K move · Enter open · S save

Be Fair! Can Machine Learning Engineering Agents Adhere to Fairness Constraints?

Anna Richter, Julia Stoyanovich, Sebastian Schelter

BIFOLD & TU Berlin · New York University

Machine learning engineering (MLE) agents promise to automate end-to-end ML pipeline development from raw data and natural language instructions, potentially making ML accessible to non-technical domain experts. However, in sensitive and regulated domains, this abstraction creates a responsibility gap: end-users may lack visibility into design choices that affect correctness, robustness, fairness, and regulatory compliance. We argue that existing benchmarks are insufficient to assess whether MLE agents can be safely applied in such settings. We propose desiderata for a responsibility-centered evaluation framework and conduct an exploratory study on melanoma classification, focusing on fairness across skin tones as a responsibility constraint. When evaluating two recent MLE agents, we find that agent-generated pipelines show high variance and consistently underperform manually designed baselines in both predictive quality and fairness, despite fairness-oriented prompts. These preliminary results suggest that further research is needed towards redesigning MLE agents to allow humans to guide the search process and reliably assess the compliance and quality of the generated ML pipelines.

Fairness Constraints Agentic Evaluations Machine Learning

Open paper Ask about it PDF