Enterprises increasingly use large language model agents to execute standard operating procedures (SOPs): long, branching, tool-intensive workflows. Yet there is little controlled evidence for choosing among competing agentic architectures. Holding the model, tasks, and tools fixed and varying only the architecture, we compare four designs (a function-calling single agent, a skill-augmented agent, centralized multi-agent orchestration, and a triage swarm; eight arms) across 13 enterprise SOPs, and introduce a skill-definition quality rubric to build matched high and low-quality skill corpora. Three takeaways emerge. (1)There is no universal best architecture; the right choice is SOP-dependent. On a given SOP, task success ranges from under 10% on the hardest to near 100% on the easiest; no architecture is statistically separable in aggregate, and added structure only plausibly pays off on complex, branching SOPs (a directional, not yet significant, trend). (2)Added architecture cost must be justified by a task-success gain. Every coordinated arm (multi-agent or skill-augmented) spends 1.4--4.7× the tokens and 2.6--5.8× the latency of the single agent for gains that are usually small and rarely significant, making the single agent the efficient default; high-quality skills do lift success (to 73.1% from 67.4%) while degraded skills fall below it. (3)~Implementation quality outweighs the single- versus multi-agent choice. A plausible multi-agent implementation loses to the single agent, yet a targeted redesign of it becomes the only arm to significantly beat the baseline (75.6% vs.\ 67.4%).
Figures & tables
Figure 1: Architecture selection as an SOP-specific cost–benefit decision. A candidate should replace the current architecture only when its marginal gain in task success rate is sufficient to justify its total architecture cost. The callouts show measured changes from the function-calling baseline to the descriptive swarm on two illustrative SOPs; the dashed break-even frontier is conceptual and organization-specific, not estimated from these data. This figure is for illustrative purposes only.
Architecture family
Arm
Distinguishing setting
Function calling
Baseline
Single agent; receives the full SOP and all declared tools.
Skill-augmented (CSE)
CSE-high †
High-quality skills; executor exposed only to each skill’s declared tools.
CSE-low
Degraded skills; same tool scoping.
Multi-agent (centralized orchestration)
Orch-v1
Decomposer–worker–synthesizer; each worker sees an isolated SOP slice and tool subset.
Orch-v2
Full SOP and tool set per worker; commit-with-basis contract, one re-dispatch round, output validation.
Triage-guided swarm
Swarm-none
No shared procedure blueprint.
Table 1: Agentic architectures and the eight arms of the shared comparison, all run on a single fixed model ( claude-opus-4-8 ) and pinned code revision. † marks the representative arm used for cross-family statements. Open-Tool Skill Guidance (OSG), a separate skill formulation evaluated on only two SOPs, is described in § 3.2 and is not part of the shared matrix.
SOP
FC
CSE-hi
CSE-lo
Orch-1
Orch-2
Sw-none
Sw-desc
Sw-full
Aircraft
22
44 (3.5,2.6)
49 (2.4,2.4)
52 (1.4,2.0)
46 (4.3,2.7)
46 (2.2,3.4)
48 (2.6,4.1)
48 (2.4,9.3)
Content flag.
29
37 (1.9,1.7)
38 (1.6,2.0)
29 (1.0,1.4)
33 (3.3,2.0)
31 (2.0,3.5)
35 (1.6,3.3)
38 (2.0,3.2)
Customer service
88
81 (2.4,2.9)
74 (2.1,3.2)
74 (1.3,4.0)
90 (5.5,5.2)
94 (1.3,3.9)
94 (1.4,4.1)
94 (1.5,4.2)
Dangerous goods
92
77 (1.8,1.4)
90 (2.4,2.2)
75 (1.8,2.5)
87 (4.4,2.0)
88 (2.3,5.0)
91 (2.3,4.4)
90 (2.7,4.2)
Email intent
87
91 (13.8,5.5)
52 (8.7,4.4)
76 (2.7,3.0)
96 (8.9,4.1)
91 (2.3,4.3)
90 (2.4,4.7)
91 (2.7,4.8)
KYB
39
86 (3.2,2.9)
40 (2.8,2.6)
23 (1.8,2.3)
60 (4.0,2.2)
53 (2.4,3.4)
57 (2.0,3.6)
57 (3.5,10.6)
Table 2: Per-SOP task success rate (TSR, %) for all eight arms, with each coordinated cell annotated TSR (tokens × , latency × ) : the token and wall-clock cost of that arm on that SOP as a multiple of the function-calling baseline (FC) on the same SOP. FC cells show TSR only (the 1.0× reference); bold marks the highest-TSR arm on each SOP (all tied arms bolded). The final row is the macro TSR (mean over the 13 SOPs) ± between-SOP SE. Because multiples are computed per SOP, a column mean differs from the pooled figures in Table 3 .
Arm
TSR
Tokens/task
Latency
(%)
( × base)
( × base)
Baseline (FC)
67.4
24.9k (1.0)
24.0s (1.0)
Orchestration-v1
58.2
35.8k (1.4)
62.4s (2.6)
Swarm-descriptive
70.4
47.3k (1.9)
100.0s (4.2)
Swarm-none
65.4
52.8k (2.1)
104.4s (4.4)
CSE-low
59.8
58.5k (2.4)
62.8s (2.6)
Table 3: Cost versus accuracy, ordered by tokens per task. Multipliers are relative to the baseline. Orchestration-v2 attains the highest accuracy at the highest token cost of any arm: giving every worker the full SOP and tool set plus a re-dispatch round is what makes it both accurate and expensive.
Figure 2: Cost–benefit by architecture: mean token cost ( x ) versus macro TSR ( y ), bubble size proportional to latency; error bars are macro SE ( n=13 ). No arm sits above and to the left of the baseline; Orchestration-v2 reaches the top accuracy at the highest token cost (§ 4.4 ).
Arm
Simple
Complex
ρedge
Orchestration-v2
78.7
72.9
+0.31
CSE-high
75.4
71.1
+0.09
Swarm-full
72.6
68.8
+0.38
Swarm-descriptive
72.8
68.4
+0.36
Swarm-none
67.4
63.7
+0.28
Baseline (FC)
71.8
63.6
n/a
Table 4: Macro TSR (%) on the simple and complex halves (median split on human complexity CH , 6 SOPs each), and the Spearman correlation between each arm’s per-SOP edge over the baseline and CH . Positive ρedge means the arm’s advantage grows with complexity; no ρ is significant at n=13 ( ∣ρ∣>0.56 needed for p<0.05 ).
SOP (# tools)
FC
CSE-hi
CSE-lo
OSG-hi
OSG-lo
KYB (8)
38.9
85.6
40.0
68.9
40.0
Video annot. (26)
34.4
37.6
33.6
57.6
52.8
Table 5: Open-tool (OSG) vs. scoped (CSE) skill execution on the two most tool-heavy SOPs, TSR (%), high/low skill corpora. Hard tool scoping (CSE) wins when the relevant tools are few (KYB) but underperforms open access (OSG) on the 26-tool video_annotation . Two SOPs; exploratory.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Dimension
Max
What it measures
Identity & discoverability
10
Name, description (what the skill does, when to use it, and the expected outcome), and usage scenarios that let a planner discover and select the skill.
Instruction quality
15
Ordered, specifically worded steps with per-step purpose and tool mapping, plus reasoning strategy, termination conditions, and constraints needed to execute the skill correctly.
Contract clarity
10
Preconditions, typed input and output schemas with examples, postconditions, and declared tool dependencies for reliable invocation and composition.
Disambiguation signals
10
Anti-patterns, explicit negative scope (what the skill does not do), and boundary constraints that prevent mis-selection against similar skills.
Governance metadata
5
Domain, complexity, risk level, and tags for cataloguing and lifecycle management.
Total
50
Appendix
Table 6: The five dimensions of the Skill-Definition Quality Rubric and their maximum point allocations.
Dimension
High
Low
Identity & discoverability
10
6
Instruction quality
15
11
Contract clarity
10
9
Disambiguation signals
10
2
Governance metadata
5
5
Total
50
33
Appendix
Table 7: Per-dimension rubric scores for the high- and low-quality versions of assess-patient-risk .
SOP
Base.
CSE-hi
CSE-lo
aircraft_inspection
22.3
43.8
49.1
content_flagging
28.6
36.9
37.5
customer_service
88.5
80.8
74.4
dangerous_goods
91.6
77.0
89.8
email_intent
86.6
90.9
51.6
know_your_business
38.9
85.6
40.0
Appendix
Table 8: Per-SOP TSR (%) for the function-calling baseline and the high- and low-quality skill corpora (CSE-high, CSE-low). The macro row is the mean over the 13 SOPs (equal weight per SOP).
Figure 3: Per-SOP cost of accuracy, baseline ( ∘ ) to CSE-high ( ⋄ ), colored by SOP complexity. Rightward displacement is extra token cost; upward is accuracy gained.
Claude Opus 4.6
Claude Opus 4.7
Claude Opus 4.8
SOP
TSR
Tok.
Lat.
TSR
Tok.
Lat.
TSR
Tok.
Lat.
(%)
(k)
(s)
(%)
(k)
(s)
(%)
(k)
(s)
Aircraft inspection
0.9
22.5
29.1
8.9
23.3
20.0
22.3
24.3
31.1
Content flagging
33.3
19.6
31.9
36.3
21.6
16.8
28.6
21.1
20.8
Customer service
89.1
33.1
21.0
92.3
42.6
14.1
88.5
38.0
23.0
Dangerous goods
93.8
6.3
12.9
90.5
12.9
11.3
91.6
11.4
15.0
Appendix
Table 9: Per-SOP function-calling baseline metrics for three Claude Opus versions. Tok. is mean total tokens per task in thousands; Lat. is mean wall-clock latency per task in seconds. The final row reports the equal-SOP TSR mean ± standard error; token and latency cells are shown as dashes. All values are computed from the common trace set. Each task has one saved outcome at temperature 1, so the table is a model-version sensitivity analysis rather than a stochastic ranking. Latency is descriptive because runs occurred at different times and no service-load metadata was logged.
Enterprise agents must follow long-horizon, conditional, safety-critical standard operating procedures (SOPs). We compile machine-readable SOP constraints into executable pseudo-code and run them with a program-guided (PG) stack machine that pages the active frame while an LLM performs semantic execution. A three-arm SOPBench study across six models separates representation from runtime: compiled text never significantly hurts and gains up to 16.0 points where official prose underperforms. Runtime guidance is capability-gated. Two strong models independently show positive seven-domain PG contrasts (58:19 and 75:31 discordant pairs), whereas weak models are harmed. A full-program cursor ablation (active frame first, complete program retained) recovers much of the strong-model refusal gain; selective visibility adds a smaller improvement. Paired probe and audit measurements track this divide to spontaneous state discipline rather than reconstruction ability. On Bank the three primary arms rise from 70.4 to 86.4 to 92.8, with 100% refusal correctness. Practical guidance: compile first; enable active-frame paging only after a model-level discipline check.
Chenglin Yu, Li Yin, Ying Yu +4
Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University · Department of Data and Systems Engineering, The University of Hong Kong · College of Economics and Management, Zhejiang Normal University +3
Domain-specific languages (DSLs) are widely used in enterprise software, giving a compact interface to complex tasks, from data access to infrastructure automation. Platforms increasingly extend them to agents, where a developer declares the topics, permitted actions, and guardrails of an agent rather than writing general-purpose code. Large language models (LLMs) write general-purpose code well, but that ability has been measured almost entirely on languages abundant in public pretraining data, and enterprise DSLs are neither abundant nor public. We introduce AgentScript-Eval, an execution-graded benchmark of 100 real tasks for AgentScript, the declarative DSL that Salesforce uses in Agentforce to build conversational agents. Tasks span authoring an agent from a specification, debugging a broken definition, and optimizing a working one. Evaluating five models zero-shot, we find that the regime is hard: the best model reaches only 25.0% pass@3, and every model scores 0% on authoring from scratch. We further find that model scale and price are weak predictors of success -- a 31B open-weight model attains the highest compilation rate of any model tested (48.0%) and matches the strongest frontier model on partial-credit accuracy at the lowest token cost. Adding an agentic compile-and-edit loop is the single largest lever, roughly doubling partial-credit accuracy. Beyond that gain, the wording of domain guidance matters more than its packaging. The same skill pack scores 28% as plain documentation but only 14% as a live plugin. Rewriting one skill inside that plugin then raises it to 37%, our best result, with catalog and tooling untouched. Larger models did not reliably outperform smaller ones; what mattered was the harness surrounding the model and the guidance supplied to it.
Enterprise agents increasingly operate inside workspaces: they read heterogeneous files, invoke tools, and deliver business artifacts. We introduce EnterpriseClawBench, an enterprise agent benchmark constructed from proprietary, real-world agent sessions. Starting from a large archive of workplace sessions, the EnterpriseClawBench produces 852 reproducible tasks, each paired with recovered fixtures, rewritten prompts, role classes, skill subclasses, hard rules, and semantic rubrics. Because the sessions contain internal enterprise content, we do not release the benchmark data; instead, our reusable contribution is the construction and evaluation protocol. On EnterpriseClawBench, the best configuration reaches only 0.663 (Codex with GPT-5.5). These results show that enterprise agent evaluation must report harness--model combinations, artifact delivery, visual quality, cost, runtime, and skill-transfer behavior, rather than collapsing performance into a single score. Code: https://github.com/FrontisAI/EnterpriseClawBench