EnterpriseBench Corecraft: Training Generalizable Agents on High-Fidelity RL Environments
Organizations: Surge AI
Abstract
We show that training AI agents on high-fidelity reinforcement learning environments produces capabilities that generalize beyond the training distribution. We introduce CoreCraft, the first environment in EnterpriseBench, Surge AI's suite of agentic RL environments. CoreCraft is a fully operational enterprise simulation of a customer support organization, comprising over 2,500 entities across 14 entity types with 23 unique tools, designed to measure whether AI agents can perform the multi-step, domain-specific work that real jobs demand. Frontier models such as GPT-5.2 and Claude Opus 4.6 solve fewer than 30% of tasks when all expert-authored rubric criteria must be satisfied. Using this environment, we train GLM 4.6 with Group Relative Policy Optimization (GRPO) and adaptive clipping. After a single epoch of training, the model improves from 25.37% to 36.76% task pass rate on held-out evaluation tasks. More importantly, these gains transfer to out-of-distribution benchmarks: +4.5% on BFCL Parallel, +7.4% on Tau2-Bench Retail, and +6.8% on Tool Decathlon (Pass@1). We believe three environment properties are consistent with the observed transfer: task-centric world building that optimizes for diverse, challenging tasks; expert-authored rubrics enabling reliable reward computation; and enterprise workflows that reflect realistic professional patterns. Our results suggest that environment quality, diversity, and realism are key factors enabling generalizable agent capabilities.
Figures & tables
| Model | Task Pass Rate (%) |
|---|---|
| Claude Opus 4.6 (Adaptive Thinking + Max Reasoning Effort) | 30.80 |
| GPT-5.2 (High Reasoning Effort) | 29.70 |
| Gemini 3.1 Pro | 27.20 |
| Claude Opus 4.6 (High Reasoning Effort) | 26.20 |
| DeepSeek v3.2 Thinking | 24.10 |
| Claude Opus 4.6 | 22.10 |
| Component | Location | Purpose |
|---|---|---|
| Tasks | tasks/*.json | Prompts and rubrics for training/eval |
| MCP Server | server/ | Handles tool calls from the agent |
| World Data | entities/ | 2,500+ entities across 14 types |
| System Prompts | system-prompts/ | Agent context and policies |
| Tool Definitions | mcp-tools.json | 23 tools with schemas |
| Model | Pass Rate (%) |
|---|---|
| GPT-5.1 High | 36.86 |
| GLM 4.6 (1 Epoch) | 36.76 |
| Claude Opus 4.5 | 33.49 |
| GPT-5 | 30.45 |
| Claude Sonnet 4.5 | 26.44 |
| GLM 4.6 (Baseline) | 25.37 |
| Benchmark | Baseline | Trained | |
|---|---|---|---|
| BFCL Parallel | 91.0% | 95.5% | +4.5% |
| BFCL Simple | 91.5% | 93.5% | +2.0% |
| -Bench Retail | 68.7% | 76.1% | +7.4% |
| Toolathlon | 18.8% | 25.6% | +6.8% |
| Metric | Base | Trained | |
|---|---|---|---|
| Pass@1 | 18.8 2.2% | 25.6 0.6% | +6.8 pp |
| Pass@3 | 29.6% | 35.2% | +5.6 pp |
| Pass 3 | 9.3% | 17.6% | +8.3 pp |
| Category | Tasks | Pass@1 | Pass@3 | Pass 3 |
|---|---|---|---|---|
| Data Analysis / Finance | 15 | 37.8% | 53.3% | 20.0% |
| Web / Forms / Documents | 10 | 36.7% | 50.0% | 30.0% |
| E-Commerce / WooCommerce | 11 | 30.3% | 36.4% | 18.2% |
| Canvas / Education | 15 | 28.9% | 33.3% | 26.7% |
| Git / DevOps / Kubernetes | 9 | 25.9% | 33.3% | 22.2% |
| Travel / Planning | 7 | 23.8% | 42.9% | 14.3% |
LLM Agent Training
Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.
See all 585 papers