AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL
Organizations: William & Mary · Salesforce
Abstract
Domain-specific languages (DSLs) are widely used in enterprise software, giving a compact interface to complex tasks, from data access to infrastructure automation. Platforms increasingly extend them to agents, where a developer declares the topics, permitted actions, and guardrails of an agent rather than writing general-purpose code. Large language models (LLMs) write general-purpose code well, but that ability has been measured almost entirely on languages abundant in public pretraining data, and enterprise DSLs are neither abundant nor public. We introduce AgentScript-Eval, an execution-graded benchmark of 100 real tasks for AgentScript, the declarative DSL that Salesforce uses in Agentforce to build conversational agents. Tasks span authoring an agent from a specification, debugging a broken definition, and optimizing a working one. Evaluating five models zero-shot, we find that the regime is hard: the best model reaches only 25.0% pass@3, and every model scores 0% on authoring from scratch. We further find that model scale and price are weak predictors of success -- a 31B open-weight model attains the highest compilation rate of any model tested (48.0%) and matches the strongest frontier model on partial-credit accuracy at the lowest token cost. Adding an agentic compile-and-edit loop is the single largest lever, roughly doubling partial-credit accuracy. Beyond that gain, the wording of domain guidance matters more than its packaging. The same skill pack scores 28% as plain documentation but only 14% as a live plugin. Rewriting one skill inside that plugin then raises it to 37%, our best result, with catalog and tooling untouched. Larger models did not reliably outperform smaller ones; what mattered was the harness surrounding the model and the guidance supplied to it.
Figures & tables
| Configuration | Model | pass@ (%) | Compile (%) | Pass rate (%) | Tokens/task |
|---|---|---|---|---|---|
| Direct generation, zero-shot ( ) | |||||
| Claude Opus 4.8 | 25.0 | 38.0 | 31.3 | 9.9k | |
| GPT-5.5 | 20.0 | 38.0 | 29.1 | 9.5k | |
| Gemma-4 31B IT | 15.0 | 48.0 | 28.1 | 7.0k | |
| GPT-5 mini | 14.0 | 32.0 | 20.9 | 9.2k | |
| Claude Sonnet 5 | 14.0 | 21.0 | 17.4 | 8.2k | |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Defect class | Tasks |
|---|---|
| Reasoning lifecycle (hook ordering, side effects) | 35 |
| Variable declarations (scope, shadowing, missing decls) | 30 |
| DSL syntax (malformed blocks, indentation) | 25 |
| Type serialization (input/output type mismatch) | 23 |
| Config validation | 11 |
| Locale / internationalization | 6 |
| Model | pass@3 (%) | Compile (%) | Pass rate (%) | Tokens/task |
|---|---|---|---|---|
| Claude Opus 4.8 | 21.0 | 60.0 | 30.9 | 14.4k |
| GPT-5.5 | 20.0 | 36.0 | 25.9 | 11.0k |
| Claude Sonnet 5 | 16.0 | 31.0 | 18.4 | 9.8k |
| GPT-5 mini | 15.0 | 24.0 | 19.1 | 10.7k |
| Gemma-4 31B IT | 14.0 | 47.0 | 28.4 | 7.0k |
| Configuration | Model | pass@1 (%) | Compile (%) | Pass rate (%) | Tokens/task |
|---|---|---|---|---|---|
| Baseline (no skills) | Opus 4.8 | 33.0 | 64.0 | 46.5 | 835k |
| Baseline (no skills) | Sonnet 5 | 20.0 | 65.0 | 40.4 | 1,563k |
| Skills v1 (inert docs) | Opus 4.8 | 28.0 | 89.0 | 51.5 | 480k |
| Skills v1 (inert docs) | Sonnet 5 | 27.0 | 79.0 | 43.7 | 1,125k |
| Skills v2 (live plugin) | Opus 4.8 | 14.0 | 86.0 | 43.6 | 389k |
| Skills v2 (live plugin) | Sonnet 5 | 16.0 | 62.0 | 29.4 | 1,123k |
| Configuration | Authoring (%) | Debugging (%) | Optimization (%) |
|---|---|---|---|
| Zero-shot direct | 0.0 | 22.5 | 53.3 |
| Two-shot direct | 10.0 | 15.0 | 40.0 |
| Agentic baseline | 16.7 | 25.0 | 60.0 |
| Skills v1 (inert docs) | 6.7 | 32.5 | 43.3 |
| Skills v2 (live plugin) | 13.3 | 15.0 | 13.3 |
| Skills v3 (rewritten) | 10.0 | 45.0 | 53.3 |
Code Generation
Code generation studies methods for turning natural-language, visual, or formal specifications into executable programs and code edits.
See all 335 papers