EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings

Shiva Krishna Reddy Malay; Shravan Nayak; Jishnu Sethumadhavan Nair; Sagar Davasam; Aman Tiwari; Sathwik Tejaswi Madhusudhan; Sridhar Krishna Nemala; Srinivas Sunkara

cs.AIMar 13, 2026

EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings

Authors: Shiva Krishna Reddy Malay, Shravan Nayak, Jishnu Sethumadhavan Nair, Sagar Davasam, Aman Tiwari, Sathwik Tejaswi Madhusudhan, Sridhar Krishna Nemala, Srinivas Sunkara,

Organizations: ServiceNow Research · Mila - Quebec AI Institute · Université de Montréal

Abstract

Large language models are shifting from passive information providers to active agents intended for complex workflows. However, their deployment as reliable AI workers in enterprise is stalled by benchmarks that fail to capture the intricacies of professional environments, specifically, the need for long-horizon planning amidst persistent state changes and strict access protocols. In this work, we introduce EnterpriseOps-Gym, a benchmark designed to evaluate agentic planning in realistic enterprise settings. Specifically, EnterpriseOps-Gym features a containerized sandbox with 164 database tables and 512 functional tools to mimic real-world search friction. Within this environment, agents are evaluated on 1,150 expert-curated tasks across eight mission-critical verticals (including Customer Service, HR, and IT). Our evaluation of 14 frontier models reveals critical limitations in state-of-the-art models: the top-performing Claude Opus 4.5 achieves only 37.4% success. Further analysis shows that providing oracle human plans improves performance by 14-35 percentage points, pinpointing strategic reasoning as the primary bottleneck. Additionally, agents frequently fail to refuse infeasible tasks (best model achieves 53.9%), leading to unintended and potentially harmful side effects. Our findings underscore that current agents are not yet ready for autonomous enterprise deployment. More broadly, EnterpriseOps-Gym provides a concrete testbed to advance the robustness of agentic planning in professional workflows.

Figures & tables

Figure 1: Performance–cost tradeoff for agentic tool use on EnterpriseOps-Gym We plot task success rate against estimated cost per task for both closed-source and open-source models. Open-source models incur lower cost but achieve consistently lower success rates. While higher-cost models offer modest performance gains, they remain far below reliable task completion.

Figure 2: Overview of EnterpriseOps-Gym : A benchmark for stateful agentic planning and tool use. (Top-left) EnterpriseOps-Gym spans eight enterprise domains and evaluates multi-step agentic planning, policy adherence, state-driven tool calling and cross domain orchestration in a reproducible sandbox. (Top-right) Domain experts create sandox and author realistic single- and cross-domain tasks, execute ground-truth trajectories, and write outcome-based verification logic with multi-stage quality assurance along with a human written oracle plan for completing the task. (Bottom) Given a task and constraints (as system level policies), agents interact with the environment and execute tools. They are evaluated by final-state verifiers that check goal completion, policy compliance, and side effects. In the above example the agent fails to adhere to system policy for linking case knowledge which mandates setting a parameter to “suggested” when a knowledge base is automatically discovered . Furthermore, the agent fails to properly send an email notification due to an unresolved identifier for the given case.

Benchmark	Focus	Num.	Num.	Num.	Avg.	DB	Avg.	Refusal	Human Task	Human
Benchmark	Focus	Domains	Tasks	Tools	Steps	Tables	FK	Ability?	Curation?	Plans?
General Tool Use
API-Bank ( Li et al., 2023 )	Tool-use	8	314	73	3	0	0	✗	✓	✗
ACEBench ( Chen et al., 2025 )	Tool-use	8	2000	4538	2	0	0	✓	✓	✗
$\tau$ -bench ( Yao et al., 2024 )	User Interaction	2	165	28	—	3	0.7	✓	✓	✗
$\tau^{2}$ -bench ( Barres et al., 2025 )	User Interaction	3	279	56	—	9	—	✓	✗	✗

Table 1: Comparison with existing agentic benchmarks. DB Tables reports the number of unique database tables in the environment; Avg. FK measures average foreign keys per table, indicating relational density and the complexity of inter-table dependencies agents must navigate. — denotes values not reported in the original work. ∗ Avg. Steps reflects ideal human-authored execution trajectories; model trajectories may require significantly more steps. Human Plans refer to step-by-step natural language plans written by experts to complete the task. † Parenthetical values indicate the number of unique task templates.

Figure 3: Task distribution across eight EnterpriseOps-Gym domains.

Model	Teams	CSM	Email	ITSM	Calendar	HR	Drive	Hybrid	Infeas.	Avg.
Closed Source Models
Claude Opus 4.5 ( Anthropic, 2025c )	50.0	34.2	51.9	23.8	43.2	32.1	49.5	30.7	50.0	37.4
Gemini-3-Flash ( Gemini Team, 2025 )	47.3	35.0	44.3	28.5	30.5	12.6	49.7	24.2	38.5	31.9
GPT-5.2 (High) ( OpenAI, 2025 )	31.0	34.8	51.0	21.7	38.5	25.0	40.0	22.2	50.0	31.8
Claude Sonnet 4.5 ( Anthropic, 2025c )	51.0	16.7	51.3	17.6	34.6	21.6	52.1	28.1	46.2	30.9
GPT-5 ( OpenAI, 2025 )	26.3	36.4	49.0	18.9	41.3	17.9	34.0	23.5	50.5	29.8

Table 2: Overall task completion performance on EnterpriseOps-Gym . We report the percentage of tasks successfully completed by each model in oracle tool mode, broken down by domain. A task is considered successful only if all outcome verification checks pass.

Figure 4: Performance degrades consistently with planning horizon. Pass@1 accuracy for closed-source (solid) and open-weight (dashed) models across horizon lengths 4–16. Thick lines show the group mean

\pm

1 SE. We observe monotonic degradation of performance for both sets, while open model performance falls more sharply with horizon length.

Figure 5: Impact of thinking budget on performance Histograms show the performance numbers with thinking budget with GPT-OSS-120B model ( OpenAI et al., 2025 ) across domains. The results show that the model with low thinking budget performing poorly with performance steadily increasing with thinking budget.

Model	Plan	CSM		ITSM		HR
Kimi-K2	CP	$19.6$	$\uparrow$ 12.5%	$18.1$	$\uparrow$ 5.9%	$17.2$	$\uparrow$ 9.0%
Kimi-K2	HP	$42.2$	$\uparrow$ 35.1%	$29.1$	$\uparrow$ 16.9%	$34.5$	$\uparrow$ 26.3%
Qwen3-30B	CP	$15.2$	$\uparrow$ 9.8%	$11.7$	$\uparrow$ 5.0%	$17.9$	$\uparrow$ 10.3%
Qwen3-30B	HP	$33.9$	$\uparrow$ 28.5%	$20.9$	$\uparrow$ 14.2%	$33.2$	$\uparrow$ 25.6%
Qwen3-4B	CP	$16.8$	$\uparrow$ 13.0%	$12.2$	$\uparrow$ 6.6%	$19.6$	$\uparrow$ 12.5%
Qwen3-4B	HP	$37.16$	$\uparrow$ 33.4%	$23.3$	$\uparrow$ 17.7%	$36.4$	$\uparrow$ 29.3%

Table 3: Plan-Conditioned Execution Baseline. Comparison of performance with Claude Plans (CP) vs. Human Plans (HP). Green values indicate % improvement over the ReAct baseline from Table 2 .

Figure 6: Impact of agentic orchestration on performance Histograms show the performance numbers with various multi-agentic architectures using Claude-Sonnet-4.5. The baseline is the simple ReAct loop described in Section 4.1 . Planner+Executor architecture first prompts the model for a detailed plan, and performs a ReAct loop conditioned on the plan. Planner+Decompose+Subtask Executor additionally does a task decomposition and calls subagents in a ReAct loop for each subtask. Finally, Oracle Human Plan + Executor performs a ReAct execution loop conditioned on a human written plan.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Distribution of task complexity across domains in EnterpriseOps-Gym. Histograms show the distribution for number of tool-execution steps required to complete tasks in each domain. Red dashed lines indicate the mean steps per domain. The results highlight substantial variation in interaction depth, with CSM and HR exhibiting longer, heavier-tailed execution traces, while Email and Calendar tasks tend to require shorter, more tightly distributed action sequences. Hybrid and ITSM workflows show moderate to high variance.

Model	Teams		CSM		Email		ITSM		Calendar		HR		Drive		Hybrid		Overall
Model	O	VL	O	VL	O	VL	O	VL	O	VL	O	VL	O	VL	O	VL	O	VL
Closed Source Models
Claude Opus 4.5 ( Anthropic, 2025c )	50.0	84.2	34.2	66.9	51.9	72.6	23.9	54.8	43.3	73.0	32.1	67.0	49.5	75.3	30.7	61.7	37.0	67.7
Gemini-3-Flash ( Gemini Team, 2025 )	47.3	81.0	35.0	68.1	44.3	70.0	28.5	58.3	30.5	67.0	12.6	54.9	49.7	76.7	24.2	58.3	31.4	65.0
GPT-5.2 (High) ( OpenAI, 2025 )	31.0	66.3	34.8	67.6	51.0	72.6	21.7	49.9	38.5	71.2	25.0	60.1	40.0	67.9	22.2	54.6	31.3	62.5
Claude Sonnet 4.5 ( Anthropic, 2025c )	51.0	81.9	16.7	55.7	51.3	71.7	17.6	49.9	34.6	70.9	21.6	61.4	52.1	76.0	28.1	61.4	34.1	66.1

Appendix

Table 6: Overall task completion performance on EnterpriseOps-Gym . (Oracle Mode) We report the percentage of tasks successfully completed by each model in oracle tool mode, broken down by domain. A task is considered successful only if all outcome verification checks pass. Results highlight substantial performance degradation on long-horizon and cross-domain (Hybrid) workflows, indicating persistent improvement gap. VL: Verifier Level, O: Overall.

Domain	Task Completion		Integrity Constraints		Policy Compliance
Domain	Samples	Score	Samples	Score	Samples	Score
Teams	100	82.3	20	90.0	8	81.3
CSM	183	58.3	30	70.0	120	45.5
Email	103	70.9	17	76.5	12	79.2
ITSM	176	53.7	30	50.6	70	30.0
Calendar	104	71.5	8	62.5	19	76.1

Appendix

Table 7: Verifier pass rates across domains by category. Models achieve highest scores on Integrity Constraints (system state validity), followed by Task Completion, with Policy Compliance showing the lowest performance. Sample counts vary across categories, with heavier domains such as CSM, ITSM and HR with more emphasis on Integrity Constraints and Policy compliance.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,441 papers

Sep 29, 2026

EnterpriseBench: Benchmarking LLM Agents on Enterprise-Level Strategic Reasoning and Decision-Making

LLM agents are increasingly expected to support enterprise workflows, where tasks often involve missing information, uncertainty, feedback, and long-term trade-offs. However, existing enterprise and financial benchmarks mainly test static capabilities such as information extraction, numerical calculation, domain knowledge, and financial QA, leaving interactive and long-horizon decision-making underexplored. To bridge this gap, we introduce EnterpriseBench, a benchmark that evaluates LLM agents across this spectrum, from static question answering to dynamic decision-making. Specifically, EnterpriseBench reorganizes existing enterprise and financial QA datasets into a unified foundational suite annotated by capability and difficulty, and introduces three professional interactive settings: Consulting, based on management-consulting-style business cases for client problem diagnosis through multi-turn information seeking; the Beer Game, adapted from a classic supply-chain management simulation for inventory control under delayed feedback; and Enterprise Digital Twin, a project-based business simulator for workforce, risk, and project planning. Experiments with nine agent methods under four backbone models show that current agents have not yet achieved stable, comprehensive, and cross-task reliability in enterprise scenarios. These results show that EnterpriseBench provides a practical benchmark for evaluating LLM agents in realistic enterprise strategic reasoning and decision-making.

Min Yang, Yichen Pan, Jinghua Piao +3

Shandong University · Zhongguancun Academy · Beijing Institute of Technology +1

AI-Assisted Decision Making LLM Agent Evaluation Long-Horizon Agent Evaluation AI Agent Benchmarks LLM Decision-Making

Ask PDF

May 9, 2026

Beyond the All-in-One Agent: Benchmarking Role-Specialized Multi-Agent Collaboration in Enterprise Workflows

Large language model (LLM) agents are increasingly expected to operate in enterprise environments, where work is distributed across specialized roles, permission-controlled systems, and cross-departmental procedures. However, existing enterprise benchmarks largely evaluate single agents with broad tool access, while existing multi-agent benchmarks rarely capture realistic enterprise constraints such as role specialization, access control, stateful business systems, and policy-based approvals. We introduce EntCollabBench, a benchmark for evaluating enterprise multi-agent collaboration. EntCollabBench simulates a permission-isolated organization with 11 role-specialized agents across six departments and contains two evaluation subsets: a Workflow subset, where agents collaboratively modify enterprise system states, and an Approval subset, where agents make policy-grounded decisions. Evaluation is based on execution traces, database state verification, and deterministic policy adjudication rather than natural-language response judging. Experiments with representative LLM agents show that current models still struggle with end-to-end enterprise collaboration, especially in delegation, context transfer, parameter grounding, workflow closure, and decision commitment. EntCollabBench provides a reproducible testbed for measuring and improving agent systems intended for realistic organizational environments.

Tao Yu, Hao Wang, Changyu Li +15

BAAI · CASIA · UCAS +2

LLM Agent Evaluation Business Process Automation Multi-Agent Collaboration AI Agent Benchmarks Multi-Agent Systems

Ask PDF

Jun 21, 2026

PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems

LLM agents increasingly operate in large tool ecosystems, where real-world tasks require discovering relevant tools, inferring implicit sub-goals, and adapting to dynamic environments over long horizons. However, existing benchmarks rarely evaluate planning under retrieval-limited tool visibility. To address this gap, we introduce PlanBench-XL, an interactive benchmark of 327 retail tasks over 1,665 tools that tests whether agents can iteratively retrieve usable tools, invoke them to uncover intermediate evidence for subsequent calls toward the final goal. PlanBench-XL further features an optional blocking mechanism that simulates real-world unpredictability through missing, failing, or distracting tool functions, forcing agents to detect disrupted paths and adapt at runtime. Experiments on ten leading LLMs show that massive-tool planning remains challenging: while GPT-5.4 achieves 51.90% accuracy in block-free settings, it collapses to 11.36% under the most severe blocking condition. Further analysis shows that agents are especially vulnerable when failures lack explicit error signals or when recovery requires longer alternative tool-use paths. These results establish PlanBench-XL as a testbed for diagnosing agentic planning failures and highlight the need for robust adaptive planning in long-horizon tasks with large, imperfect tool environments.

Jiayu Liu, Qihan Lin, Cheng Qian +8

University of Illinois Urbana-Champaign

Tool Discovery for LLM Agents Long-Horizon Agent Evaluation AI Agent Benchmarks LLM Agent Planning LLM Agent Reliability

Ask PDF

Abstract

AI Agent Benchmarks

Explore similar work

EnterpriseBench: Benchmarking LLM Agents on Enterprise-Level Strategic Reasoning and Decision-Making

Beyond the All-in-One Agent: Benchmarking Role-Specialized Multi-Agent Collaboration in Enterprise Workflows

PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems