OfficeTown: Generative Organizations for Evolving Agent Intelligence
Authors: Hongchen Wei, Daohong Ying, Zhi Hong, Aidan Wu, Bei Liu, Ruichun Ma, Yifan Yang, Qi Dai, Kai Qiu, Rui Xu, Dongdong Chen, Zhenzhong Chen, Chong Luo
Organizations: Wuhan University · University of California, San Diego · South China University of Technology · Cornell University · Microsoft Research Asia · Microsoft
AI agents increasingly use files and software tools to produce professional deliverables. Such work draws on materials, decisions, and feedback accumulated across assignments. We introduce OfficeTown, a generative simulation framework for professional organizations in which tasks and their context arise from continuing organizational work. From a natural-language description of an organization's purpose and activities, OfficeTown initializes an organization whose members coordinate work, delegate bounded assignments to assistant agents, and review their deliverables. Artifacts, decisions, and unresolved matters persist and shape subsequent work, while responsibilities and permissions determine what each member can see and do. We curate work requests, their context, and review feedback into executable resources for evaluation and learning. We construct OfficeTown Bench with 406 tasks and 4,836 evaluation criteria from 23 simulated organizations across eight professional domains, incorporating domain-expert feedback. Frontier-model evaluations reveal substantial remaining headroom. Supervised fine-tuning and on-policy distillation on OfficeTown data substantially improve Qwen3.5-9B on external professional-work benchmarks, including GDPval. Supplementary evaluations show that the principal--assistant harness achieves higher scores than the single- and multi-agent baselines tested. These results highlight the value of organizational simulation in generating work-grounded data and environments for the continued development of generalist agents.
Figures & tables
Figure 1: OfficeTown models ongoing work in a persistent, permissioned organization. Professionals coordinate requests and review deliverables; assistants execute within authorized workspaces. Private drafts, revisions and unresolved matters carry context into subsequent work. The ten-role Juniper scene and its work links are illustrative. The lower plot shows cumulative materials from one Juniper organization over 181 simulated days (not wall-clock time): 27 initial files and 470 subsequent logical outputs, with revisions and exact copies deduplicated (Appendix A.7 ). These counts track material accumulation, not task quality or success. The right-hand panel shows synthetic inputs and the request for a retained scope-and-routing memo, not a model response.
Figure 2: OfficeTown’s organizational simulation. (a) Organizational objectives, roles, and seed materials establish an organization. (b) Coordination, delegation, execution, and review connect ongoing work through retained materials and follow-up assignments. (c) Tasks drawn from this activity provide fixed inputs and executable environments for training and evaluation. (d) The recorded twelve-role hierarchy of Juniper Financial Modelling; the senior modeller’s computer is highlighted. (e) Filesystem and terminal views used by an assistant. The reconciliation and Juniper examples are separate. Directory names are verified; interfaces and terminal activity are schematic.
Figure 3: OfficeTown Bench composition. (a) Domains, with petal areas and labels indicating task counts. (b) Input-type coverage, including archive contents; types overlap. (c) Input-file counts: 20-file bins starting at 11, with a smoothed frequency curve. All panels use all 406 tasks; an archive counts as one file in (c).
Model
Overall
Assur.
Clin.
Des.
Ent.
Fin.
Ind.
Legal
Soft.
GPT-6 Astra
74.4
63.8
92.9
56.3
72.7
88.1
81.3
92.3
84.4
GPT-5.6 Sol
33.5
38.3
35.7
26.2
45.5
34.3
29.2
46.2
28.9
GPT-5.6 Terra
39.4
48.9
32.1
35.0
43.6
40.3
43.8
38.5
33.3
GPT-5.6 Luna
32.5
48.9
53.6
21.4
41.8
31.3
22.9
30.8
28.9
GPT-5.4
30.5
31.9
39.3
25.2
41.8
32.8
29.2
30.8
20.0
GPT-5.2
21.2
27.7
28.6
13.6
27.3
31.3
14.6
15.4
13.3
Table 2: OfficeTown Bench Pass@1 (%) on 406 tasks, with high reasoning effort. Domain abbreviations follow Figure 4 . 1st 2nd 3rd mark within-column ranks at displayed precision; ties share rank and skip positions. Weighted Scores and scoring definitions appear in Appendices D.1 and C.1 .
Figure 5: Professional requirements beyond aggregate scores. (a) Judged tasks missing at least one applicable critical (P0) requirement. (b) Task-averaged fulfillment of critical and major requirements (P0/P1). (c) Mean task scores grouped by supplied input-file types; groups overlap. Twelve models with bound item-level judgments are shown. Item-level panels exclude unjudged outcomes; file-group scores retain the main table’s declared zeros. Definitions and full values are in Appendix D .
Figure 6: OfficeTown Bench at five recorded turn budgets. All panels use the same 406 tasks. Scores enter only after execution ends; terminal evaluation errors remain zero. (a) High-effort models with the same executor. (b) Low and High effort for DeepSeek-V4-Flash. (c) Execution frameworks for DeepSeek-V4-Flash High. Points show 8, 16, 24, 32, and 48 turns; lines guide the eye. Appendix C.2 specifies counting and execution settings.
Training
GDPval
JobBench-Easy
APEX-Agents
Terminal-Bench 2.1
OfficeTown Bench
Rubric
Weighted
Pass@1
Success
Pass@1
Base
65.75
40.41
9.40
42.31
1.48
SFT
70.30 +4.55
44.87 +4.46
16.16 +6.76
51.22 +8.91
3.45 +1.97
SFT + OPD
73.65 +7.90
51.48 +11.07
19.43 +10.03
63.64 +21.33
4.68 +3.20
Table 3: Learning from OfficeTown: Qwen3.5-9B across training stages (%). Rubric and Weighted denote mean rubric and weighted rubric scores; Success denotes the native binary success rate. Subscripts show percentage-point differences from Base. Protocols: Appendix E.3 .
Harness
PRDBench
GDPevo
Terminal-Bench 2.1
Research-code
DEV
DEBUG
Codex
85.87
89.04
64.51
83.7
40.88
MetaGPT
84.86
86.51
63.44
81.1
41.15
CAMEL
76.74
78.93
63.45
85.1
39.95
OfficeTown Harness
89.67
92.62
74.70
87.4
46.40
Table 4: Harness results on external benchmarks. All harnesses use GPT-5.5 with xhigh reasoning effort. Reported point estimates on a 0–100 scale; higher is better. GDPevo aggregates repeated runs by task group, not best-of-three success. Codex on GDPevo uses the few-shot reference setting. Terminal results use different reported budgets.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: A generated office and one professional in detail. (a) Juniper’s 12 professionals, grouped by department, with examples of retrieved, adapted, and synthetic seed materials. The brief above is an excerpt of the full request, which also includes configuration. (b) Devon Arata’s role, manager, profile, business authority, selected skills, and computer. The blue connector opens the highlighted person’s details; it is not a reporting relationship. Profiles and skills describe the generated configuration; files and saved work records show retained simulation state. The office layout, avatars, and terminal commands are illustrative. R/W denotes read/write access; R/O denotes read-only access, filtered by access-control rules (ACL).
Table 5: Generated properties and recorded settings of the Juniper instance. The simulation period is a configured horizon, not a measure of completed work. These settings are specific to this instance.
Figure 8: Professional coordination in a simulated office. Four consecutive speaking turns from a Juniper meeting about a synthetic customer’s liquidity. Excerpts retain the original wording and order; ellipses mark omitted text. Side objects identify the proposed inputs, model, and review requirements, not completed or validated outputs.
Figure 9: Continuing work through sessions and saved records. (a) In persistent-session mode, the same professional inspects a returned forecast in a review branch and brings its conclusion, actions, and a brief tool record back into coordination. (b) Actions update work records; selected records and personal notes remain available when the context is refreshed under the same identifier on the next simulated day. (c) Compaction is optional; periodic triggering is off by default. The forecast, messages, and records form an illustrative example, not a recorded Juniper interaction.
Figure 10: A bounded file view and persistent local changes. (a) In direct-home mode, an assistant receives a writable copy of its principal’s home and a permission-filtered, read-only company view. Other private homes are not mounted. R/W and R/O denote read/write and read-only access; ACL denotes access-control rules. (b) The runtime checks local changes before committing them; an interrupted commit can restore the preceding files and classification state. The files and changes are illustrative, not execution results. Recovery does not undo messages or other effects already sent to external services.
Figure 11: Work materials accumulated during a six-month Juniper simulation. All dates and days denote simulated calendar time, not wall-clock runtime; the adaptive clock adjusts its rate to work-processing pressure. (a) Cumulative material files: 27 distinct initial resource contents and 470 logical output files, counted at first simulated delivery. Revisions and recorded renames count once; exact copies are merged at first appearance. Navigation READMEs, explicit backups, and trash paths are excluded. Initial resources are grouped separately from subsequent outputs. (b) Selected material excerpts available by each date, redrawn from retained source versions; the final workbook contains synthetic scenario inputs. Counts describe the whole organization, not the illustrated folders.
Professional domain
Tasks
Role titles
Assurance
47
14
Clinical
28
14
Design
103
25
Entertainment
55
17
Finance
67
26
Industrial
48
18
Appendix
Table 6: OfficeTown Bench domains and source role titles. Role titles are deduplicated within each domain; titles can recur across domains. The total uses global deduplication.
Figure 12: Source roles in OfficeTown Bench. Each panel shows the five most frequent delegating role titles and their task counts on a shared scale. Remaining titles and tasks are summarized below each panel. Titles are source job names, not standardized occupations.
File type
Input tasks
Text / markup
406
Documents
318
Tables
280
Structured data
406
Code
250
Images
195
Appendix
Table 7: Input-file type coverage in OfficeTown Bench. Each type is counted once per task across all 406 tasks. Input types include archive contents. Categories overlap.
Figure 13: Riverbend reconciliation case. (a) Initial files, selected columns from the supplied workbook, and a missing-evidence entry from the triage memo. Amounts are synthetic task inputs, not model results. (b) Five condensed checks from the 13-criterion evaluator-only rubric. P0 and P1 denote critical and major importance. Filenames and table headings are shortened; the 35 initial files include task and metadata carriers. Service files are frozen historical projections. Required outputs appear below the input tree; the supplied originals must remain unchanged.
Figure 14: Oakridge fixture-design case. (a) Initial files and the supplied synthetic STEP workpiece, not a proposed fixture. Indented ZIP children are archive members; filenames are shortened. The 39 initial files include task and metadata carriers. The given limits specify the maximum fixture footprint and stack height, not workpiece dimensions. The load and friction coefficient are synthetic inputs. (b) Five condensed checks from the 18-criterion evaluator-only rubric, with the same importance notation as Figure 13 . Required deliverables appear below the input tree.
Model
Overall
Assur.
Clin.
Des.
Ent.
Fin.
Ind.
Legal
Soft.
OpenAI
GPT-6 Astra
88.4
85.8
97.9
79.3
85.3
93.7
91.1
92.3
98.2
GPT-5.6 Sol
68.5
67.7
64.2
66.7
72.4
68.1
73.2
81.7
62.7
GPT-5.6 Terra
76.1
80.0
64.8
79.7
78.0
73.5
75.3
70.6
75.0
GPT-5.6 Luna
75.4
83.6
84.1
71.6
81.4
74.0
71.6
88.4
64.9
GPT-5.4
72.2
73.0
75.7
70.0
80.5
69.8
70.9
59.7
72.2
Appendix
Table 8: OfficeTown Bench Weighted Score (%; higher is better). 1st 2nd 3rd : ranks within each column at displayed precision; ties share rank and skip positions. Models, tasks and high-effort settings match Table 2 . Dashes indicate pending results. Domain abbreviations denote assurance, clinical, design, entertainment, finance, industrial, legal, and software work.
Model
P0
P1
P2
P3
P0 misses
GPT-6 Astra
88.8
88.0
86.7
87.5
20.8
GPT-5.6 Terra
76.5
76.9
82.2
62.5
53.3
GPT-5.6 Luna
76.1
74.5
72.8
50.0
58.3
GPT-5.4
72.2
73.5
82.5
62.5
59.6
GPT-5.2
64.9
64.7
76.0
62.5
67.6
Claude Opus 5.5
70.4
70.6
62.2
50.0
41.4
Appendix
Table 9: Requirement fulfillment by importance, and tasks with a critical miss (%). Fulfillment is task-averaged; higher is better. P0 misses are task-level rates; lower is better. Eligible task counts vary by model: P0 390–405, P1 347–360, P2 49–52, and P3 8. The retained instrument contains 3,156/1,582/90/8 criteria at these tiers before outcome-specific applicability checks. The small P2/P3 populations should not be read as broad capability estimates.
Model
Docs
Tables
Code
Images
CAD
Media
GPT-6 Astra
85.9
87.3
85.4
83.4
90.9
78.8
GPT-5.6 Terra
77.9
76.4
78.9
80.3
68.1
83.0
GPT-5.6 Luna
76.7
75.7
78.4
77.2
71.4
73.7
GPT-5.4
74.6
74.4
74.9
76.0
66.5
68.6
GPT-5.2
65.0
62.7
66.3
65.6
59.2
63.1
Claude Opus 5.5
64.9
67.3
63.0
66.4
59.2
67.1
Appendix
Table 10: Weighted Score by supplied input-file type (%). Group sizes are 318/280/250/195/33/55 tasks, respectively. Groups overlap and retain the main table’s declared zeros. Plain-text and structured inputs occur in every task and would repeat the aggregate; slides have no input coverage, and heterogeneous residual formats are not a single interpretable group.
Figure 15: Pass@1 at five recorded turn budgets. Populations, selected attempts, requested settings and turn counts match Figure 6 . All panels use the same 406 tasks. A final pass contributes only after its recorded execution has ended; all other outcomes contribute zero. Lines guide the eye; no intermediate artifacts are scored.
Item
SFT
OPD after SFT
Initial student
Qwen3.5-9B
SFT-trained Qwen3.5-9B
Training inputs
Complete demonstrations
Requests and initial environments
Training examples
1,982
828
Validation examples
220
None
Supervised tokens in resource
30,704,800
Not applicable
Teacher
DeepSeek-V4-Flash
Qwen3.8-Flash-Next
Appendix
Table 11: Training resources and teachers. Example counts exclude zero-supervision padding rows; the SFT token total describes the complete demonstration resource. Optimization settings are given in Tables 12 and 13 .
Parameter
Setting
Model / training mode
Qwen3.5-9B; full-parameter SFT
Hardware / parallelism
One node, 8× NVIDIA A100 80 GB; FSDP2
Schedule / random seed
2 epochs, 496 steps; seed 42
Global / micro-batch size
8 / 1 per GPU
Optimizer
AdamW; β1=0.9 , β2=0.95
Learning rate
10−6 ; linear decay to 10−7 (minimum ratio 0.1)
Appendix
Table 12: Full-parameter SFT configuration. The 1,982 training and 220 validation demonstrations are distinct from the two and four zero-supervision padding rows. Training uses 248 steps per epoch and 496 steps in total.
Table 13: OPD training configuration. The student is initialized from SFT-trained Qwen3.5-9B. Length limits are in tokens; the final 100-task batch is retained.
Figure 16: JobBench-Easy performance over SFT training. All recorded checkpoints from the same training run and the Base reference are shown. Points are run means and error bars are one sample standard deviation; the 14 individual runs are also displayed. The non-monotonic curve measures training progress, not an ablation of unique training-data size.
Enterprise agents increasingly operate inside workspaces: they read heterogeneous files, invoke tools, and deliver business artifacts. We introduce EnterpriseClawBench, an enterprise agent benchmark constructed from proprietary, real-world agent sessions. Starting from a large archive of workplace sessions, the EnterpriseClawBench produces 852 reproducible tasks, each paired with recovered fixtures, rewritten prompts, role classes, skill subclasses, hard rules, and semantic rubrics. Because the sessions contain internal enterprise content, we do not release the benchmark data; instead, our reusable contribution is the construction and evaluation protocol. On EnterpriseClawBench, the best configuration reaches only 0.663 (Codex with GPT-5.5). These results show that enterprise agent evaluation must report harness--model combinations, artifact delivery, visual quality, cost, runtime, and skill-transfer behavior, rather than collapsing performance into a single score. Code: https://github.com/FrontisAI/EnterpriseClawBench
AI agents are expected to perform professional work across hundreds of occupational domains (from emergency department triage to nuclear reactor safety monitoring to customs import processing), yet existing benchmarks can only evaluate agents in the few domains where public environments exist. We introduce OccuBench, a benchmark covering 100 real-world professional task scenarios across 10 industry categories and 65 specialized domains, enabled by Language Environment Simulators (LESs) that simulate domain-specific environments through LLM-driven tool response generation. Our multi-agent synthesis pipeline automatically produces evaluation instances with guaranteed solvability, calibrated difficulty, and document-grounded diversity. OccuBench evaluates agents along two complementary dimensions: task completion across professional domains and environmental robustness under controlled fault injection (explicit errors, implicit data degradation, and mixed faults). We evaluate 15 frontier models across 8 model families and find that: (1) no single model dominates all industries, as each has a distinct occupational capability profile; (2) implicit faults (truncated data, missing fields) are harder than both explicit errors (timeouts, 500s) and mixed faults, because they lack overt error signals and require the agent to independently detect data degradation; (3) larger models, newer generations, and higher reasoning effort consistently improve performance. GPT-5.2 improves by 27.5 points from minimal to maximum reasoning effort; and (4) strong agents are not necessarily strong environment simulators. Simulator quality is critical for LES-based evaluation reliability. OccuBench provides the first systematic cross-industry evaluation of AI agents on professional occupational tasks.
Xiaomeng Hu, Yinger Zhang, Fei Huang +7
Qwen Team, Alibaba Group · The Chinese University of Hong Kong
The ability of Large Language Model (LLM) agents to complete daily and professional work is receiving increasing attention. Training such agents requires realistic work scenarios. Expert-authored occupational work is costly and slow to produce, while unconstrained synthesis often yields tasks with weak factual grounding or internally inconsistent requirements. To bridge this gap, we introduce WorkGenesis, a framework that constructs executable occupational work from real-world artifacts through two core technical innovations: (1) Evidence-Based Work Construction, which grounds each unit of work in real-world evidence by retrieving public files guided by O*NET occupational knowledge and synthesizing the surrounding context, companion materials, work request, and itemwise rubric around them; and (2) Execution-Guided Consistency Verification, which renders a reference deliverable inside the constructed work, attributes every unsatisfied rubric item to the agent, the task, or the rubric, and uses task and rubric defects as feedback to iteratively repair the work until it passes the audit. Experimental results demonstrate that Fx-Work-35B, trained with simple supervised fine-tuning (SFT) on only 20K units of work synthesized by WorkGenesis, achieves the highest scores among all comparable-scale baselines on the five reported metrics across GDPvalAA-v2, APEX-Agents-AA, and JobBench (31.00 versus 24.79 average score), and even surpasses frontier models such as the 1.6T DeepSeek-V4-Pro-Preview. These results show that WorkGenesis provides scalable training data for working agents.
Xinyu Zhu, Fenyi Liu, Yuzhu Cai +4
Shanghai Jiao Tong University · Endless Frontier · Beihang University