cs.AINov 19, 2026

AppSci-Bench: Can LLMs Build AI Agents and Make Them Better?

Authors: Sanjari Srivastava, Manpreet Kaur, Cheng Chang, Sofía Ramós Anton, Oumaima Essamadi, Rishu Garg, Alejandro Lavale Jiménez, Marcos Caballero Yus, 

Organizations: Uniphore

Abstract

Large language models (LLMs) are increasingly used not only to power AI agents but also to build them. While many benchmarks assess the quality of LLM- powered agents, few ascertain how good LLMs are at building them. To this end, we introduce AppSci-Bench, where we construct a setting akin to how human Applied Scientists build agents for businesses - through eliciting task requirements from stakeholders, exploring the business's infrastructure, and scientifically improve the target agent by designing experiments and internal benchmarks. AppSci-Bench features seven realistic domains - five new ones (Slides Generation, Order-to-Cash, Compliance Review, Customer Call Insights, and CRM Insights) and two domains recast from tau3-bench (Yao et al., 2024; Shi et al., 2026b)(Airline and Banking) - each with a held-out evaluation suite customized for that task. We evaluate seven frontier models and find AppSci-Bench challenging: the best model, Claude Opus 5.0, only reaches a mean-minimum score of 59.1% across 3 build attempts; other models do significantly worse. Models struggle with eliciting task requirements (only 65% for Opus 5.0), build representative internal evaluations (models saturate their own evaluation 65% of the time but fail to solve the held-out task), presenting challenges for future research. The benchmark code is available here: https://github.com/sanjari-orb/appsci-bench.

Figures & tables

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. EvalBuilder: Can LLMs Build Efficient Benchmarks?

    Oct 14, 2026Huaisheng Zhu, Hangfan Zhang, Teng Xiao +5Benchmark ConstructionAI Agent Benchmarks

  2. SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks

    Oct 23, 2026Tianyun Zhong, Wei Wang, Jialin Li +17Scientific Agent BenchmarksLLM Agent Evaluation

  3. τττ^τ-Bench: An Environment for End-To-End, Realistic Agent Construction

    Sep 4, 2026Quan Shi, Keshav Dhandhania, Karthik Narasimhan +1AI Coding AgentsLLM Agent Evaluation