EmailBench: A Benchmark for Evaluating LLM Agents on Enterprise Email and Productivity Tasks

Mukul Singh; Mansi Uniyal; Devin Devlin; Wen Xie; Big Thadawasin; Ritam Dutt; Vivian Lai; Hyeonsu B. Kang

cs.AISep 25, 2026

EmailBench: A Benchmark for Evaluating LLM Agents on Enterprise Email and Productivity Tasks

Authors: Mukul Singh, Mansi Uniyal, Devin Devlin, Wen Xie, Big Thadawasin, Ritam Dutt, Vivian Lai, Hyeonsu B. Kang

Organizations: Microsoft

Abstract

Enterprise email agents must combine information retrieval, structured state changes, temporal reasoning, and multi-step coordination. Recent agent benchmarks include productivity tasks, but few center on typed email workflows in a self-contained environment. We introduce EmailBench, a benchmark of 206 email and productivity scenarios across 16 task categories. The benchmark couples a typed email API specification with provider-neutral naming, a deterministic synthetic Enron-inspired corpus, and a scenario suite whose topic selection was informed by aggregate task-intent telemetry from an interactive prototype. Its hybrid evaluation protocol combines 258 executable static assertions with 211 LLM rubrics. We evaluate eight LM configurations on a fixed single-user corpus. The best-performing configuration passes only 33.5% of scenarios despite 99.7% of its tool calls completing without an observed API failure, with pass rates varying substantially across task categories. This gap shows that valid tool execution is not equivalent to task completion. EmailBench provides a self-contained environment for end-to-end email-agent evaluation, with broader tool coverage, multi-persona testing, and repeated-run evaluation as future work areas.

Figures & tables

Component	Scope
API specification	119 operational methods, 2 evaluation utilities, 36 typed interfaces
Evaluated adapter	46 tools spanning messages, calendar, contacts, todo, folders, filters, settings, and identity
Corpus	65 messages, 15 events, 15 contacts, 10 todo tasks, and linked productivity entities
Scenarios	206 tasks, 16 categories, 8 declared API domains, and three assigned difficulty levels
Grading	258 static assertions and 211 LLM rubrics

Table 1: EmailBench at a glance. The empirical study evaluates a subset of the API specification.

Panel of Figure 1 — Figure 1: Evaluated pipeline. The adapter is narrower than the API specification, and all runs use the same persona corpus.

Category	$n$	Category	$n$
messages-read	39	todo	9
orchestration	17	meeting-prep	9
cross-domain	15	inbox-cleanup	9
messages-write	15	folders	9
multi-domain	12	calendar-triage	8
calendar-read	12	settings	8

Table 2: Distribution of the 206 scenarios. Categories denote task themes and are not a pure read-versus-write partition.

Configuration	Pass	Full	Ret.	Rem.	Static	LLM	Calls	Call ok	Clean	Latency
Sonnet 4.5	69/206	33.5	28.3	48.1	.309	.419	5.68	99.7	94.7	27.06
Opus 4.1	61/206	29.6	25.0	42.6	.306	.419	5.37	99.7	96.6	40.56
GPT-5.4-R	58/206	28.2	24.3	38.9	.230	.372	3.61	96.8	88.8	21.84
GPT-5.5-R	57/206	27.7	25.0	35.2	.300	.407	8.65	94.1	88.3	31.69
GPT-5.4	53/206	25.7	20.4	40.7	.245	.333	2.66	95.4	89.3	17.80
GPT-5.2	47/206	22.8	18.4	35.2	.197	.286	1.86	92.4	58.7	15.84

Table 3: Baseline results from one run per configuration. Pass and Full report the primary 206-scenario suite. Ret. and Rem. are pass percentages on the 152-scenario retained post hoc sensitivity subset and 54-scenario removed slice, respectively. Full, Ret., Rem., call success (Call ok), and clean-execution rate (Clean) are percentages. Static averages the 163 scenarios with static checks, while LLM averages all 206. Calls and latency in seconds are per-scenario means. Clean counts scenarios with at least one call and no observed API failure, using all 206 scenarios as the denominator.

Category	$n$	Son.	5.4-R	GPT-5
filters	10	80.0	40.0	20.0
folders	9	88.9	88.9	44.4
messages-write	15	73.3	46.7	53.3
messages-read	39	35.9	38.5	23.1
calendar-read	12	8.3	25.0	16.7
calendar-write	12	16.7	8.3	8.3

Table 4: Pass rates for representative task categories. The complete eight-configuration table appears in Table 33 .

Domains	Son.	Opus	5.4-R	o3
1 ( $n=122$ )	37.7	33.6	34.4	27.9
2 ( $n=53$ )	34.0	28.3	26.4	20.8
3 ( $n=24$ )	16.7	16.7	8.3	4.2
4 ( $n=7$ )	14.3	14.3	0.0	0.0

Table 5: Pass rates by declared domain count. Cohorts differ in task composition and tool availability.

Scenario	Required workflow and available checks
OB-H312	Aggregate unread percentages by sender; retain rates above 50% and sort descending. Two static assertions and one rubric check the answer, with a known literal-pattern limitation.
OB-N520	Create Projects with Raptor, Dynegy, and FERC subfolders, then route keyword matches using returned IDs. Five static assertions and one rubric check the hierarchy and routing.
OB-H307	Route automated, high-importance, then remaining unread messages in order; exclude Sent Items. Six static assertions and one rubric do not provide equivalent coverage of the full workflow.

Table 6: Concrete task and grading requirements. Complete queries and assertions appear in Appendix L . These cases illustrate coverage differences rather than a sampled error taxonomy.

Configuration	Fail/calls	Fail/none	Pass by 5
Sonnet 4.5	130	7	62/69
Opus 4.1	141	4	45/61
GPT-5.4-R	138	10	47/58
GPT-5.5-R	144	5	38/57
GPT-5.4	140	13	48/53
GPT-5.2	88	71	42/47

Table 7: Failures with and without recorded calls, and eventual passes completed within five iterations. Each row’s two failure counts plus its total passes sum to 206. The final column conditions on the original run’s passing trajectories; it is not a five-iteration rerun.

Figure 2: Retrospective completion of passing trajectories. Values use all 206 scenarios as the denominator. By iteration five, 65–93% of each configuration’s eventual passes have finished. These curves do not estimate performance under a reduced iteration budget.

Figure 3: Observed tradeoff between mean scenario latency and pass rate among the evaluated configurations. Sonnet achieves the highest pass rate, while several GPT-5 configurations offer shorter mean latency at lower task success.

Appendix figures & tables40 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	R	W	Key Ops
messages	7	15	search, send, reply, flag, batch
calendar	7	11	create, RSVP, schedule, reminders
boards	6	9	columns, tasks, assignment
settings	6	9	vacation, labels, working hrs
todo	4	10	lists, tasks, checklists
folders	5	5	create, rename, move, tree

Appendix

Table 8: API domain coverage showing read (R) and write (W) method counts. The 51/68 split excludes the two evaluation utilities. The evaluated adapter is reported separately.

Entity	N	Entity	N
Messages	65	Boards	2
Events	15	Board columns	6
Contacts	15	Board tasks	7
Folders	6	Users	16
Filters	1	Labels	6
Calendars	3	Contact groups	3

Appendix

Table 9: Seed corpus entity counts. Actual message membership is 50 in Inbox and 15 in Sent Items. No message has Archive as its parent folder.

Connection	Description
Email $\leftrightarrow$ Calendar	Meeting attendees (Skilling, Beck, Kitchen) also appear as email senders, and threads reference upcoming meetings
Email $\leftrightarrow$ Contacts	All major senders have contact records with titles, departments, company
Calendar $\leftrightarrow$ Tasks	Todo items reference projects discussed in events (“Raptor” tasks, “FERC” prep)
Conflicts	Two events (FERC Review, Dynegy Review) deliberate overlap for conflict detection
Mixed signals	System accounts (EnronOnline, GasBank) generate noise requiring differential handling

Appendix

Table 10: Cross-entity interconnections enabling multi-domain scenarios.

Panel of Figure 4 — Figure 4: Scenario metadata axes. Category, domain count, and difficulty are separate labels. D3 does not require three or more domains.

Tier	Definition	N	%
D1	Basic assigned tasks	17	8
D2	Intermediate assigned tasks	121	59
D3	Complex assigned tasks	68	33

Appendix

Table 11: Assigned difficulty tiers from scenario definitions. Overall, 92% are D2 or D3. Actual tool-call and domain counts need not follow a fixed threshold.

Category	N	Category	N
messages-read	39	todo	9
orchestration	17	meeting-prep	9
cross-domain	15	inbox-cleanup	9
messages-write	15	folders	9
multi-domain	12	calendar-triage	8
calendar-read	12	settings	8

Appendix

Table 12: Scenario category distribution (206 total).

ID	Evaluated query and grading
OB-H312	D2, messages-read . For each person who emailed me, calculate what percentage of their emails to me are unread. Show only people with $>$ 50% unread rate, sorted by unread percentage descending.
	2 static assertions and 1 LLM rubric.
OB-N520	D3, folders . Create a folder hierarchy: ’Projects’ with subfolders ’Raptor’, ’Dynegy’, and ’FERC’. Then move all emails mentioning each project keyword to the appropriate subfolder.
	5 static assertions and 1 LLM rubric.
OB-H307	D3, inbox-cleanup . Set up an email triage system: (1) Create folders ’Action Required’, ’FYI Only’, and ’Automated’. (2) Move all emails from automated senders (enrononline, legal-tickets, gasbank) to ’Automated’. (3) Move all high-importance emails to ’Action Required’. (4) Move all remaining unread emails to ’FYI Only’. Do NOT move any emails that are already in Sent Items.
	6 static assertions and 1 LLM rubric.

Appendix

Table 13: Three evaluated scenario definitions with recorded traces. OB-H312 analyzes senders, OB-N520 creates a hierarchy, and OB-H307 performs triage. Appendix L details their checks.

Type	What it checks
count	Collection size matches expected value (eq, gte, lte, gt, lt operators)
exists	Predicate returns true on snapshot
not-exists	Predicate returns false on snapshot
property	Strict equality on a computed value
sent-message	Sent message matches subject/to/body
moved-message	Messages moved to target folder

Appendix

Table 14: Nine supported static types. Snapshot checks inspect APISnapshot , while return-contains checks the final answer. Four types occur in the evaluated suite.

Benchmark	Domain	Tasks	API	Typed	Grading	Offline	Multi-D	Corpus
AgentBench	8 envs	1,013	N/A	✗	Functional	✓	✗	Varied
WebArena	Web	812	N/A	✗	Match	✗	✗	Live sites
SWE-bench	Code	2,294	N/A	✗	Unit tests	✓	✗	GitHub
OfficeBench	Office apps	300	N/A	✗	Match/exec.	✗	✓	Synthetic
AppWorld	9 apps	750	457	✓	API state	✓	✓	Synthetic
EnterpriseBench ( Vishwakarma et al., 2025 )	Enterprise	500	N/A	N/A	Task-specific	✓	✓	Synthetic

Appendix

Table 15: Comparison with related benchmarks. “API” denotes a reported method, endpoint, or route count. Counts are not directly equivalent across abstractions. “Multi-D” denotes cross-domain or cross-application scenarios. N/A indicates that the cited paper does not report a directly comparable value.

Configuration label	Internal endpoint ID	Run ID	Start time (UTC)
Claude Sonnet 4.5	dev-anthropic-claude-sonnet-4-5	run-1777853456125	2026-05-04 00:10:56
Claude Opus 4.1	dev-anthropic-claude-opus-4-1	run-1777856235806	2026-05-04 00:57:15
GPT-5.4-reasoning	dev-gpt-54-reasoning	run-1777882982599	2026-05-04 08:23:02
GPT-5.5-reasoning	dev-gpt-55-reasoning	run-1777885083982	2026-05-04 08:58:03
GPT-5.4	dev-gpt-54-chat	run-1777882125342	2026-05-04 08:08:45
GPT-5.2	dev-gpt-52-chat	run-1777881635032	2026-05-04 08:00:35

Appendix

Table 16: Identifiers preserved in the recorded result files. Configuration labels reproduce the experiment metadata. The endpoint-to-public-snapshot mapping is not independently verified.

Model	Pass	Rate	Stat	LLM	Scr	Lat
Claude Sonnet 4.5	69/206	33.5%	.309	.419	.385	27s
Claude Opus 4.1	61/206	29.6%	.306	.419	.383	41s
GPT-5.4-reasoning	58/206	28.2%	.230	.372	.336	22s
GPT-5.5-reasoning	57/206	27.7%	.300	.407	.374	32s
GPT-5.4	53/206	25.7%	.245	.333	.321	18s
GPT-5.2	47/206	22.8%	.197	.286	.269	16s

Appendix

Table 17: Recorded results across 206 scenarios. Stat averages the 163 scenarios with static checks, and LLM averages all 206. Scr is the mean scenario score, with equal weights when both components exist and the available component otherwise. Passing requires score

\geq 0.5

and all critical assertions to pass.

Figure 5: Recorded pass rates with two-sided 95% Wilson score intervals (

n=206

per model). These intervals describe hypothetical scenario sampling, not repeated-run or judge variability. Sonnet has the highest observed rate, 69/206 (33.5%).

Category	Son.	Opus	5.4-R	o3	GPT-5
filters	80	90	40	50	20
folders	89	56	89	44	44
settings	75	75	75	50	62
messages-write	73	67	47	47	53
contacts	17	17	17	17	17
messages-read	36	33	38	23	23

Appendix

Table 18: Per-category pass rates (percent). Some write categories have high observed rates. Board tools are absent from the evaluated adapter.

Panel of Figure 6 — Figure 6: Per-category pass rates for three selected models. Shading groups structured edits, mixed categories, and other workflows. These descriptive groups do not define a read/write partition. Board results reflect an adapter without board tools.

Level	$n$	Son.	5.4-R	o3	GPT-5
D1	17	11.8	5.9	11.8	11.8
D2	121	42.1	37.2	30.6	28.1
D3	68	23.5	17.6	10.3	10.3

Appendix

Table 19: Pass rates (percent) by assigned difficulty. Difficulty labels do not specify an exact number of calls or domains. Unavailable tools can affect low-difficulty tasks.

Domains	Son.	Opus	5.4-R	o3
1 ( $n=122$ )	37.7	33.6	34.4	27.9
2 ( $n=53$ )	34.0	28.3	26.4	20.8
3 ( $n=24$ )	16.7	16.7	8.3	4.2
4 ( $n=7$ )	14.3	14.3	0.0	0.0

Appendix

Table 20: Pass rates (percent) by declared domain count. Cohort sizes sum to 206. Differences combine task composition, access restrictions, and model performance.

Figure 7: Pass rate by declared task-domain count, joined from scenario IDs. Cohort sizes are 122, 53, 24, and 7. Differences are descriptive associations, not an experimentally isolated cost per added domain.

	Son.	Opus	5.4-R	o3
Sonnet 4.5	N/A	0.711	0.530	0.420
Opus 4.1	0.711	N/A	0.506	0.446
GPT-5.4-R	0.530	0.506	N/A	0.507
o3	0.420	0.446	0.507	N/A

Appendix

Table 21: Jaccard similarity of recorded pass sets for the four named models. This selected cohort includes o3 and is not the top four by observed pass rate.

Model	1	2	3	5	7	10	15
Sonnet 4.5	0.0	11.7	22.8	30.1	32.0	32.0	33.5
Opus 4.1	0.0	6.8	15.0	21.8	25.2	26.7	29.6
GPT-5.4-R	0.0	7.8	15.0	22.8	25.2	27.7	28.2
GPT-5.5-R	0.5	8.7	13.6	18.4	21.4	26.2	27.7
GPT-5.4	0.0	14.6	20.4	23.3	25.2	25.2	25.7
GPT-5.2	0.5	11.2	18.0	20.4	21.8	22.3	22.8

Appendix

Table 22: Retrospective completed-pass rates (percent of all 206 scenarios) at iteration

K

. No lower-budget run is simulated. Recorded final outcomes are unchanged.

Figure 8: Per-iteration activity in four recorded runs. Active denotes observed ongoing activity. Engaged denotes at least one call in that iteration. Clean execution denotes engaged trajectories with no observed API failure in that iteration. Cumulative pass counts completed passing runs. All percentages use 206 scenarios. For aborted runs with missing terminal counts, Active is an observed lower bound reconstructed from recorded calls.

Model	1-pass	Later pass	Fail/calls	Fail/no calls
Sonnet 4.5	0	69	130	7
Opus 4.1	0	61	141	4
GPT-5.4-R	0	58	138	10
GPT-5.5-R	1	56	144	5
GPT-5.4	0	53	140	13
GPT-5.2	1	46	88	71

Appendix

Table 23: Trajectory counts partition all 206 outcomes. Iterations use observed activity when a stored count is missing. Failures are split by whether any tool call was recorded. A zero-tool response is not automatically a refusal.

Model	Fail/calls	Repeated	%
Sonnet 4.5	130	72	55.4
Opus 4.1	141	93	66.0
GPT-5.4-R	138	71	51.4
GPT-5.5-R	144	101	70.1
GPT-5.4	140	40	28.6
GPT-5.2	88	21	23.9

Appendix

Table 24: Failed trajectories with adjacent same-tool calls using different arguments. The denominator is failed trajectories with recorded calls. This is a repetition measure, not an error-recovery rate.

Model	Calls/pass	Calls/fail	Iter/pass	Iter/fail
Sonnet 4.5	5.28	5.88	3.61	3.69
Opus 4.1	4.59	5.70	4.75	5.85
GPT-5.4-R	3.00	3.84	4.00	4.80
GPT-5.5-R	7.60	9.05	4.81	4.56
GPT-5.4	2.70	2.64	2.98	3.12
GPT-5.2	3.34	1.42	3.13	2.10

Appendix

Table 25: Mean recorded tool calls and observed iterations, grouped by outcome. The groups differ in task composition. These differences are descriptive, not evidence of a causal effect of extra calls.

Model	1	2	3–4	5–7	8+
Sonnet 4.5	0/7	24/67	32/93	10/26	3/13
Opus 4.1	0/4	14/39	26/72	12/41	9/50
GPT-5.4-R	0/11	16/47	27/74	9/41	6/33
GPT-5.5-R	1/6	17/50	16/53	10/68	13/29
GPT-5.4	0/13	30/91	17/71	5/25	1/6
GPT-5.2	1/72	22/69	17/50	5/10	2/5

Appendix

Table 26: Recorded passes and scenario counts by observed iteration bucket. Small high-iteration cohorts can have unstable rates. These are not rerun results at different budgets.

Panel of Figure 9 — Figure 9: Score distribution for Sonnet and GPT-5 across 206 scenarios. Bins have width 0.1, and the last includes both endpoints [0.9, 1.0]. Shading separates scores below 0.5 from those at or above it. The latter can still fail critical assertions. Both distributions are bimodal, with many scenarios near 0 or above 0.5 and few intermediate cases.

Model	Engage	Clean	Answer	Pass
Sonnet 4.5	96.6	94.7	100.0	33.5
Opus 4.1	98.1	96.6	100.0	29.6
GPT-5.4-R	95.1	88.8	97.1	28.2
GPT-5.5-R	97.1	88.3	99.0	27.7
GPT-5.4	93.7	89.3	100.0	25.7
o3	87.9	79.1	78.6	22.3

Appendix

Table 27: Scenario indicators from recorded runs (percent of 206). Engagement requires a recorded tool call. Clean execution additionally requires no observed API failure. Answer excludes runner error and max-iteration sentinel strings. Pass uses the recorded grade.

Model	Calls	Success %	First %	Failures
Sonnet 4.5	1170	99.7	100.0	4
Opus 4.1	1107	99.7	100.0	3
GPT-5.4-R	743	96.8	100.0	24
GPT-5.5-R	1782	94.1	100.0	106
GPT-5.4	547	95.4	100.0	25
o3	1342	95.8	100.0	57

Appendix

Table 28: Tool-call execution statistics using structured failure evidence. Calls are individual invocations. First-call rates use trajectories with at least one call. Stored grading outcomes are not recomputed.

Model	Failures	Next calls	Success %	Same %
Sonnet 4.5	4	1	100.0	0.0
Opus 4.1	3	0	N/A	N/A
GPT-5.4-R	24	17	52.9	47.1
GPT-5.5-R	106	98	14.3	86.7
GPT-5.4	25	17	5.9	94.1
o3	57	46	30.4	71.7

Appendix

Table 29: Next-call behavior after observed API failures. Denominators exclude terminal failures with no following call. Same-tool continuation and successful next execution do not by themselves establish recovery of the original task.

Figure 10: Signed pass-rate differences (percentage points) for documented complexity proxies. Scenario metadata are joined by ID, and execution-derived features describe observed behavior. Cohort counts and feature definitions accompany the regenerated analysis.

Figure 11: Pass rates grouped by the number of declared task domains: 122, 53, 24, and 7 scenarios. The grouping does not imply a prescribed order of API calls.

Model	Calls	Iterations	Latency (s)
Sonnet 4.5	5.68	3.66	27.06
Opus 4.1	5.37	5.52	40.56
GPT-5.4-R	3.61	4.58	21.84
GPT-5.5-R	8.65	4.63	31.69
GPT-5.4	2.66	3.09	17.80
GPT-5.2	1.86	2.33	15.84

Appendix

Table 30: Per-scenario means from recorded runs. Observed iterations use recorded call indices where a stored iteration count is missing. Exact unrecorded terminal activity remains unknown.

Category	Before	Added	Final
meeting-prep	1	8	9
calendar-triage	2	6	8
settings	2	6	8
folders	3	6	9
inbox-cleanup	3	6	9
calendar-write	7	5	12

Appendix

Table 31: Gap-filling in Batch 4. Categories with fewer than eight scenarios received targeted additions to establish minimum category coverage.

Type	Count	%
return-contains	123	47.7
state-check	101	39.1
exists	32	12.4
sent-message	2	0.8
count	0	0.0
not-exists	0	0.0

Appendix

Table 32: Code-derived distribution of 258 static assertions. Five supported types have no instances, and four types are used.

Tool name	Domain	R/W	Parameters
messages_list	messages	R	top, filter, orderBy, skip, folderId
messages_get	messages	R	id
messages_search	messages	R	query, top
messages_count	messages	R	filter
messages_send	messages	W	subject, body, toRecipients, ccRecipients, importance
messages_createDraft	messages	W	subject, body, toRecipients

Appendix

Table 34: All 46 evaluated tools. R denotes read and W denotes write. Names and parameters are shown as exposed. These abbreviated payloads can differ from the full operational API.

Name	Email	Dept.	Title
Jeff Skilling	[email protected]	Executive	President & COO
Louise Kitchen	[email protected]	Trading	Pres., EnronOnline
Sally Beck	[email protected]	Operations	VP Operations
John Lavorato	[email protected]	Trading	CEO, Enron Americas
Richard Shapiro	[email protected]	Gov. Affairs	VP Gov. Affairs
Steven Kean	[email protected]	Executive	VP & Chief of Staff

Appendix

Table 35: All 15 non-self users in the corpus, spanning executives, analysts, external partners, and system accounts.

Iter	Recorded calls and results (summary)
1	folders_create : Projects, returning folder-5001.
2	Three folders_create calls: Raptor, Dynegy, and FERC, all with parent folder-5001 and returning folder-5002/5003/5004.
3	Three messages_search calls, one per keyword.
4	Three messages_batchMove calls. Results contain 6, 8, and 8 succeeded IDs and empty failed lists.
5	Final answer reports the hierarchy and move counts.

Appendix

Table 36: Recorded OB-N520 success: five iterations, ten calls, and 34.886 s. Static, LLM, and overall scores are all 1.0. The scenario passes.

Iter	Recorded calls and results (summary)
1	Creates Action Required, FYI Only, and Automated folders (folder-5001/5002/5003).
2–5	Retrieves Sent Items, lists messages, searches three automated senders, lists high-importance messages, and attempts an unread filter.
6	Moves three EnronOnline matches in a batch and two other automated messages individually.
7	Moves two further automated messages and six high-importance messages.
8–11	Lists messages, attempts a folder filter, retrieves four folders, and lists folders.
12	Creates another set of the same three named folders.

Appendix

Table 37: Recorded OB-H307 failure: 15 iterations, 29 calls, and 88.868 s. All six critical static checks pass with a static score of 1.0. The critical LLM rubric fails with a score of 0.0, yielding an overall score of 0.50 and a failing outcome.

Level	$n$	Son.	Opus	5.4-R	5.5-R	5.4	5.2	o3	GPT-5
D1	17	11.8	11.8	5.9	11.8	11.8	5.9	11.8	11.8
D2	121	42.1	38.8	37.2	33.1	31.4	30.6	30.6	28.1
D3	68	23.5	17.6	17.6	22.1	19.1	13.2	10.3	10.3

Appendix

Table 38: Pass rates (percent) for all eight configurations by assigned difficulty. These groups differ in tasks and tool availability. Differences do not isolate difficulty as a causal factor.

Attribute	Failed by four (117)	All (206)
Mean assigned difficulty	2.27	2.25
Mean declared domains	1.73	1.59
Mean static assertions	1.57	1.25
Mean LLM assertions	1.01	1.02
D3 fraction	40.2%	33.0%
Calendar domain	45.3%	34.5%

Appendix

Table 39: Scenario metadata for tasks failed by Sonnet, Opus, GPT-5.4-R, and o3. Attributes overlap. This selected four-configuration cohort is not the top four by pass rate.

Model	$\mu$	Med	$\sigma$	Max	1	$\geq 10$	$\geq 15$
Sonnet 4.5	3.66	3	2.67	15	7	9	6
Opus 4.1	5.52	4	3.91	15	4	33	15
GPT-5.4-R	4.58	3	3.19	15	11	14	7
GPT-5.5-R	4.63	4	2.60	13	6	10	0
GPT-5.4	3.09	2	1.88	12	13	4	0
GPT-5.2	2.33	2	1.77	12	72	4	0

Appendix

Table 40: Observed iteration distribution.

\sigma

is the population standard deviation. Counts in the last three columns use all 206 scenarios. Missing stored counts are replaced by observed lower bounds from call indices.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,132 papers

EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions
Jun 22, 2026Jincheng Zhong, Weizhi Wang, Che Jiang +5LLM Agent Evaluation Tool-Augmented Language Model Agents
Horizon Research, Frontis.AI
E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios
Jul 26, 2026Weihuang Zheng, Tianyuan Zou, Eileen Ye +5Computer-Use Agent Benchmarks Tool-Use Evaluation
Hunyuan Team, Tencent · Institute for AI Industry Research, Tsinghua University · School of Computer Science and Engineering, Southeast University
StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows
Aug 18, 2026Liya Zhu, Xin Ma, Tao Liu +35AI Agent Benchmarks LLM Agent Evaluation
ByteDance Seed · Nanjing University · M-A-P

Jun 22, 2026J/K move · Enter open · S save

EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions

Jincheng Zhong, Weizhi Wang, Che Jiang, Kai Tian+4

Horizon Research, Frontis.AI

LLM Agent Evaluation Tool-Augmented Language Model Agents Computer-Use Agent Benchmarks AI Agent Benchmarks Computer-Use Agent Evaluation Enterprise AI Agents+1

Open paper AskPDF arXiv

Enterprise agents increasingly operate inside workspaces: they read heterogeneous files, invoke tools, and deliver business artifacts. We introduce EnterpriseClawBench, an enterprise agent benchmark constructed from proprietary, real-world agent sessions. Starting from a large archive of workplace sessions, the EnterpriseClawBench produces 852 reproducible tasks, each paired with recovered fixtures, rewritten prompts, role classes, skill subclasses, hard rules, and semantic rubrics. Because the sessions contain internal enterprise content, we do not release the benchmark data; instead, our reusable contribution is the construction and evaluation protocol. On EnterpriseClawBench, the best configuration reaches only 0.663 (Codex with GPT-5.5). These results show that enterprise agent evaluation must report harness--model combinations, artifact delivery, visual quality, cost, runtime, and skill-transfer behavior, rather than collapsing performance into a single score. Code: https://github.com/FrontisAI/EnterpriseClawBench

Abstract

LLM Agent Evaluation

Explore similar work

EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions

E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios

StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows