WorkWorlds: An Infrastructure for Evaluating AI Agents on Workplace Tasks

Yining Hua; Levi Lian

cs.AISep 20, 2026

WorkWorlds: An Infrastructure for Evaluating AI Agents on Workplace Tasks

Authors: Yining Hua, Levi Lian

Organizations: Harvard University · Agent Evaluation Science Inc. · Raycaster · Stanford University

Abstract

Many knowledge-work benchmarks are constructed around individual tasks, with the context needed for each task selected together with or after the task has been specified. This design measures performance on workplace-like tasks in an environment assembled for the task. When task specification guides which context is selected, the evaluation can encode task information into the environment and pre-complete part of the information-localization work that workplace performance normally requires. We introduce WorkWorlds, an evaluation infrastructure that separates organizational state from task specification. A world first fixes a revision, date, and employee seat and materializes the organizational state that employee can access; tasks are introduced only afterward. We implement WorkWorlds in a primary synthetic pharmaceutical company with 8 measured tasks across 6 employee seats, and construct additional organizational worlds. Across 192 matched evaluations, task-level curation increased evidence access by 17.6 percentage points, from 72.8% to 90.4%, and criterion pass by 8.7 points, from 68.0% to 76.7%, while pass conditional on evidence access remained nearly unchanged; most of the measured difference occurred before the agent reached sufficient evidence.

Figures & tables

Figure 1: WorkWorlds overview. A revision, date, and employee seat materialize a role-visible projection from a persistent organizational world. The task is applied only after that projection is fixed, after which candidate-visible information is separated from verifier-side grading information and the agent is executed and graded.

Figure 2: Experimental design. Construction validation tests whether a fixed world revision, date, and employee seat reproduce the same workplace. The matched information experiment varies only the information projection while holding the world, task, seat, agent, and grader fixed.

Panel of Figure 3 — Figure 3: Effect of task-level information curation on the frozen 72-criterion panel. The upper row shows criterion pass and sufficient-evidence access, joint access/pass states, and mean search burden per run. The lower row shows criterion pass by task, criterion pass and evidence access by agent, and sensitivity of the pooled pass difference to curator, equal task weighting, and leave-one-task-out analyses. Criterion pass and sufficient-evidence access are GEE-standardized marginal probabilities; pass given access is descriptive. Differences are curated minus full; intervals are 95% task-cluster bootstrap intervals where shown.

Across all 16 curated packets
Measure	Result
Packet size, median (range)	9 (7–11)
Packet size, mean (curator 1 / curator 2)	9.0 / 9.1
Task pairs with different source sets	5/8
Curator source-set Jaccard, mean (range)	0.87 (0.7–1.00)
Initially sufficient criterion–packet pairs	128/144

Table 1: Task-curated packet characteristics. Two domain practitioners independently selected sources from the same employee-visible projection. The upper panel summarizes packet size and agreement across all 16 packets; the lower panel shows retained workspace share for the four MSAT and QC tasks whose seats can each access 775 files.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Task field	Meaning
Assignment	Request presented to the agent.
Employee seat	Role whose employee projection is used.
Date	Point in organizational history at which the assignment occurs.
Starting revision	Frozen organizational version from which the projection is derived.
Available surfaces	Files, records, communication, and tools exposed through that projection.
Permitted changes	State changes the agent is allowed to make.

Appendix

Table A1: Task-level fields added to a pre-existing WorkWorld.

Property	Check	Result
Reproducibility	Clean rebuilds identical	3/3
	Repeated seat materializations identical	18/18
Projection consistency	Same-seat pre-task projections identical	2/2
	Staged and direct projections identical	8/8
Role isolation	Distinct seat fingerprints	6/6
	Authorized files present	Pass

Appendix

Table D1: Construction checks for the frozen eight-task, 72-criterion PharmaCo panel.

Check	Failure exposed	Result
Conformance suite
Seat projection	Different-permission roles receive the same evidence or an incorrect projection hash	4/4
Date projection	Future-record leakage or a date change that does not change the projection	3/3
Verifier sealing	Verifier-only evidence enters the candidate side or required verifier material is missing	2/2
Explicit grading route	Declared deterministic/semantic route does not produce the expected verdict	1/1
Cross-surface agreement	Filesystem and record adapter expose different paths or content	2/2

Appendix

Table D2: Reference-world conformance and seeded isolation checks.

Joint access and pass state, counts
Outcome	Full projection	Task-curated subset
Criterion executions	864	864
Passed / executed	589/864 (68.2%)	664/864 (76.9%)
Accessed / observable	614/842 (72.9%)	767/848 (90.4%)
Unknown access labels	22 (2.5%)	16 (1.9%)
Pass given access	84.2%	82.7%

Appendix

Table E1: Raw matched outcomes for the frozen 72-criterion panel. Access rates use only executions whose access label is observable. Pass given access is descriptive.

Task ( $N$ /condition)	Full	Curated	$\Delta$ pp	Full	Curated	Pooled $\Delta$ pp, task held out
	Criterion pass			Evidence access
Stability update (132)	64.4%	74.2%	+9.8	76.2%	88.5%	+8.5
Overwrap position (132)	68.9%	73.5%	+4.5	76.0%	92.2%	+9.4
Validation defense (108)	78.7%	65.7%	-13.0	78.5%	78.5%	+11.8
Authority response (168)	66.7%	78.0%	+11.3	71.0%	95.8%	+8.0
Records integrity (108)	66.7%	84.3%	+17.6	73.8%	92.4%	+7.4

Appendix

Table E2: Task-level matched results.

N

is criterion executions per condition for that task. Access uses only observable labels; the last column recomputes the pooled pass difference with that task removed.

Agent	Full pass	Curated pass	Pass $\Delta$ pp	Access $\Delta$ pp
Claude Opus 5	229/288 (79.5%)	247/288 (85.8%)	+6.2	+12.8
Grok 4.6	188/288 (65.3%)	225/288 (78.1%)	+12.8	+17.5
GPT-5.6 Sol	172/288 (59.7%)	192/288 (66.7%)	+6.9	+22.3

Appendix

Table E3: Agent-level matched results. Each agent contributes 288 criterion executions per condition.

Benchmark	Structure above task	Role/access structure	Relationship across tasks
TheAgentCompany ( Xu et al., 2025 )	Reusable simulated-company services and data	Company accounts, services, and simulated coworkers	Tasks are instantiated separately from a common company setup; task-dependent services reset per run
EnterpriseOps-Gym ( Malay et al., 2026 )	Reusable enterprise schemas, databases, tools, and policies	Policy-constrained tool access across eight enterprise domains	Expert-written tasks run in separately resettable enterprise instances
APEX-Agents ( Vidgen et al., 2026 )	Data-rich professional project world	Project context; no employee-specific role projection	Multiple tasks are authored from files and tools already present in the same project world
Workspace-Bench ( Tang et al., 2026 )	Large workspace associated with a worker profile	Worker profile defines the occupational context	Tasks require locating, reasoning over, and modifying dependencies within the workspace
EnterpriseBench ( Vishwakarma et al., 2025 )	Enterprise sandbox with organizational metadata	Employee hierarchy, dynamic role-based access control, and persona-specific access	Tasks are generated from organizational metadata and persona-relevant enterprise context
WorkWorlds	Versioned organizational state	Employee seat materialized through declared roles and permissions	Revision, date, and seat fix the pre-task projection across assignments

Appendix

Table G1: Comparison of workplace structures defined above individual evaluation tasks.

Benchmark Design

Benchmark design creates tasks, datasets, metrics, and protocols that make AI capabilities measurable and comparisons credible.

See all 389 papers

Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies
May 5, 2026Zirui Tang, Xuanhe Zhou, Yumou Liu +19Computer-Use Agent Benchmarks AI Agent Evaluation
1Shanghai Jiao Tong University · 2ByteDance · 5Independent Researher +2
Design and Report Benchmarks for Knowledge Work
May 22, 2026Yining Hua, Hongbin Na, Cyrus Ayubcha +1Benchmark Design AI Agent Evaluation
Harvard University · University of Technology Sydney · Stanford University +1
JobBench: Aligning Agent Work With Human Will
May 25, 2026Yuetai Li, Yichen Feng, Zhangchen Xu +21Computer-Use Agent Benchmarks LLM Agent Evaluation
University of Washington · 10Bake AI · University of California, Santa Barbara +10

May 5, 2026J/K move · Enter open · S save

Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies

Zirui Tang, Xuanhe Zhou, Yumou Liu, Linchun Li+18

1Shanghai Jiao Tong University · 2ByteDance · 5Independent Researher+2

Computer-Use Agent Benchmarks AI Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

Workspace learning requires AI agents to identify, reason over, exploit, and update explicit and implicit dependencies among heterogeneous files in a worker's workspace, enabling them to complete both routine and advanced tasks effectively. Despite its importance, existing relevant benchmarks largely evaluate agents on pre-specified or synthesized files with limited real-world dependencies, leaving workspace-level evaluation underexplored. To this end, we introduce Workspace-Bench, a benchmark for evaluating AI agents on Workspace Learning involving Large-Scale File Dependencies. We construct realistic workspaces with 5 worker profiles, 74 file types, 20,476 files (up to 20GB) and curate 388 tasks, each with its own file dependency graph, evaluated across 7,399 total rubrics that require cross-file retrieval, contextual reasoning, and adaptive decision-making. We further provide Workspace-Bench-Lite, a 100-task subset that preserves the benchmark distribution while reducing evaluation costs by about 70%. We evaluate 4 popular agent harnesses and 7 foundation models. Experimental results show that current agents remain far from reliable workspace learning, where the best reaches only about 60%, substantially below the human result of 80.7%, and the average performance across agents is only 43.3%.

Abstract

Benchmark Design

Explore similar work

Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies

Design and Report Benchmarks for Knowledge Work

JobBench: Aligning Agent Work With Human Will