cs.AIOct 16, 2026
SaveOfficeTown: Generative Organizations for Evolving Agent Intelligence
Organizations: Wuhan University · University of California, San Diego · South China University of Technology · Cornell University ·
Abstract
AI agents increasingly use files and software tools to produce professional deliverables. Such work draws on materials, decisions, and feedback accumulated across assignments. We introduce OfficeTown, a generative simulation framework for professional organizations in which tasks and their context arise from continuing organizational work. From a natural-language description of an organization's purpose and activities, OfficeTown initializes an organization whose members coordinate work, delegate bounded assignments to assistant agents, and review their deliverables. Artifacts, decisions, and unresolved matters persist and shape subsequent work, while responsibilities and permissions determine what each member can see and do. We curate work requests, their context, and review feedback into executable resources for evaluation and learning. We construct OfficeTown Bench with 406 tasks and 4,836 evaluation criteria from 23 simulated organizations across eight professional domains, incorporating domain-expert feedback. Frontier-model evaluations reveal substantial remaining headroom. Supervised fine-tuning and on-policy distillation on OfficeTown data substantially improve Qwen3.5-9B on external professional-work benchmarks, including GDPval. Supplementary evaluations show that the principal--assistant harness achieves higher scores than the single- and multi-agent baselines tested. These results highlight the value of organizational simulation in generating work-grounded data and environments for the continued development of generalist agents.
Figures & tables
Figure 1: OfficeTown models ongoing work in a persistent, permissioned organization. Professionals coordinate requests and review deliverables; assistants execute within authorized workspaces. Private drafts, revisions and unresolved matters carry context into subsequent work. The ten-role Juniper scene and its work links are illustrative. The lower plot shows cumulative materials from one Juniper organization over 181 simulated days (not wall-clock time): 27 initial files and 470 subsequent logical outputs, with revisions and exact copies deduplicated (Appendix A.7 ). These counts track material accumulation, not task quality or success. The right-hand panel shows synthetic inputs and the request for a retained scope-and-routing memo, not a model response.
Figure 2: OfficeTown’s organizational simulation. (a) Organizational objectives, roles, and seed materials establish an organization. (b) Coordination, delegation, execution, and review connect ongoing work through retained materials and follow-up assignments. (c) Tasks drawn from this activity provide fixed inputs and executable environments for training and evaluation. (d) The recorded twelve-role hierarchy of Juniper Financial Modelling; the senior modeller’s computer is highlighted. (e) Filesystem and terminal views used by an assistant. The reconciliation and Juniper examples are separate. Directory names are verified; interfaces and terminal activity are schematic.
Figure 3: OfficeTown Bench composition. (a) Domains, with petal areas and labels indicating task counts. (b) Input-type coverage, including archive contents; types overlap. (c) Input-file counts: 20-file bins starting at 11, with a smoothed frequency curve. All panels use all 406 tasks; an archive counts as one file in (c).
| Model | Overall | Assur. | Clin. | Des. | Ent. | Fin. | Ind. | Legal | Soft. |
|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra | 74.4 | 63.8 | 92.9 | 56.3 | 72.7 | 88.1 | 81.3 | 92.3 | 84.4 |
| GPT-5.6 Sol | 33.5 | 38.3 | 35.7 | 26.2 | 45.5 | 34.3 | 29.2 | 46.2 | 28.9 |
| GPT-5.6 Terra | 39.4 | 48.9 | 32.1 | 35.0 | 43.6 | 40.3 | 43.8 | 38.5 | 33.3 |
| GPT-5.6 Luna | 32.5 | 48.9 | 53.6 | 21.4 | 41.8 | 31.3 | 22.9 | 30.8 | 28.9 |
| GPT-5.4 | 30.5 | 31.9 | 39.3 | 25.2 | 41.8 | 32.8 | 29.2 | 30.8 | 20.0 |
| GPT-5.2 | 21.2 | 27.7 | 28.6 | 13.6 | 27.3 | 31.3 | 14.6 | 15.4 | 13.3 |
Table 2: OfficeTown Bench Pass@1 (%) on 406 tasks, with high reasoning effort. Domain abbreviations follow Figure 4 . 1st 2nd 3rd mark within-column ranks at displayed precision; ties share rank and skip positions. Weighted Scores and scoring definitions appear in Appendices D.1 and C.1 .
Figure 5: Professional requirements beyond aggregate scores. (a) Judged tasks missing at least one applicable critical (P0) requirement. (b) Task-averaged fulfillment of critical and major requirements (P0/P1). (c) Mean task scores grouped by supplied input-file types; groups overlap. Twelve models with bound item-level judgments are shown. Item-level panels exclude unjudged outcomes; file-group scores retain the main table’s declared zeros. Definitions and full values are in Appendix D .
Figure 6: OfficeTown Bench at five recorded turn budgets. All panels use the same 406 tasks. Scores enter only after execution ends; terminal evaluation errors remain zero. (a) High-effort models with the same executor. (b) Low and High effort for DeepSeek-V4-Flash. (c) Execution frameworks for DeepSeek-V4-Flash High. Points show 8, 16, 24, 32, and 48 turns; lines guide the eye. Appendix C.2 specifies counting and execution settings.
| Training | GDPval | JobBench-Easy | APEX-Agents | Terminal-Bench 2.1 | OfficeTown Bench |
|---|---|---|---|---|---|
| Rubric | Weighted | Pass@1 | Success | Pass@1 | |
| Base | 65.75 | 40.41 | 9.40 | 42.31 | 1.48 |
| SFT | 70.30 +4.55 | 44.87 +4.46 | 16.16 +6.76 | 51.22 +8.91 | 3.45 +1.97 |
| SFT + OPD | 73.65 +7.90 | 51.48 +11.07 | 19.43 +10.03 | 63.64 +21.33 | 4.68 +3.20 |
Table 3: Learning from OfficeTown: Qwen3.5-9B across training stages (%). Rubric and Weighted denote mean rubric and weighted rubric scores; Success denotes the native binary success rate. Subscripts show percentage-point differences from Base. Protocols: Appendix E.3 .
| Harness | PRDBench | GDPevo | Terminal-Bench 2.1 | Research-code | |
|---|---|---|---|---|---|
| DEV | DEBUG | ||||
| Codex | 85.87 | 89.04 | 64.51 | 83.7 | 40.88 |
| MetaGPT | 84.86 | 86.51 | 63.44 | 81.1 | 41.15 |
| CAMEL | 76.74 | 78.93 | 63.45 | 85.1 | 39.95 |
| OfficeTown Harness | 89.67 | 92.62 | 74.70 | 87.4 | 46.40 |
Table 4: Harness results on external benchmarks. All harnesses use GPT-5.5 with xhigh reasoning effort. Reported point estimates on a 0–100 scale; higher is better. GDPevo aggregates repeated runs by task group, not best-of-three success. Codex on GDPevo uses the few-shot reference setting. Terminal results use different reported budgets.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: A generated office and one professional in detail. (a) Juniper’s 12 professionals, grouped by department, with examples of retrieved, adapted, and synthetic seed materials. The brief above is an excerpt of the full request, which also includes configuration. (b) Devon Arata’s role, manager, profile, business authority, selected skills, and computer. The blue connector opens the highlighted person’s details; it is not a reporting relationship. Profiles and skills describe the generated configuration; files and saved work records show retained simulation state. The office layout, avatars, and terminal commands are illustrative. R/W denotes read/write access; R/O denotes read-only access, filtered by access-control rules (ACL).
| Configuration | Juniper instance |
|---|---|
| Generated organization | 12 professionals; 7 departments; 11 reporting relationships |
| Initial synthetic customers | 3 |
| Staffing target and limit | Target: 12; maximum: 16 |
| Occupational references | O*NET 30.3; ESCO 1.2.1 |
| Recorded generation model | GPT-5.5 |
| Professional and assistant models | GPT-5.5 |
Table 5: Generated properties and recorded settings of the Juniper instance. The simulation period is a configured horizon, not a measure of completed work. These settings are specific to this instance.
Figure 8: Professional coordination in a simulated office. Four consecutive speaking turns from a Juniper meeting about a synthetic customer’s liquidity. Excerpts retain the original wording and order; ellipses mark omitted text. Side objects identify the proposed inputs, model, and review requirements, not completed or validated outputs.
Figure 9: Continuing work through sessions and saved records. (a) In persistent-session mode, the same professional inspects a returned forecast in a review branch and brings its conclusion, actions, and a brief tool record back into coordination. (b) Actions update work records; selected records and personal notes remain available when the context is refreshed under the same identifier on the next simulated day. (c) Compaction is optional; periodic triggering is off by default. The forecast, messages, and records form an illustrative example, not a recorded Juniper interaction.
Figure 10: A bounded file view and persistent local changes. (a) In direct-home mode, an assistant receives a writable copy of its principal’s home and a permission-filtered, read-only company view. Other private homes are not mounted. R/W and R/O denote read/write and read-only access; ACL denotes access-control rules. (b) The runtime checks local changes before committing them; an interrupted commit can restore the preceding files and classification state. The files and changes are illustrative, not execution results. Recovery does not undo messages or other effects already sent to external services.
Figure 11: Work materials accumulated during a six-month Juniper simulation. All dates and days denote simulated calendar time, not wall-clock runtime; the adaptive clock adjusts its rate to work-processing pressure. (a) Cumulative material files: 27 distinct initial resource contents and 470 logical output files, counted at first simulated delivery. Revisions and recorded renames count once; exact copies are merged at first appearance. Navigation READMEs, explicit backups, and trash paths are excluded. Initial resources are grouped separately from subsequent outputs. (b) Selected material excerpts available by each date, redrawn from retained source versions; the final workbook contains synthetic scenario inputs. Counts describe the whole organization, not the illustrated folders.
| Professional domain | Tasks | Role titles |
|---|---|---|
| Assurance | 47 | 14 |
| Clinical | 28 | 14 |
| Design | 103 | 25 |
| Entertainment | 55 | 17 |
| Finance | 67 | 26 |
| Industrial | 48 | 18 |
Table 6: OfficeTown Bench domains and source role titles. Role titles are deduplicated within each domain; titles can recur across domains. The total uses global deduplication.
Figure 12: Source roles in OfficeTown Bench. Each panel shows the five most frequent delegating role titles and their task counts on a shared scale. Remaining titles and tasks are summarized below each panel. Titles are source job names, not standardized occupations.
| File type | Input tasks |
|---|---|
| Text / markup | 406 |
| Documents | 318 |
| Tables | 280 |
| Structured data | 406 |
| Code | 250 |
| Images | 195 |
Table 7: Input-file type coverage in OfficeTown Bench. Each type is counted once per task across all 406 tasks. Input types include archive contents. Categories overlap.
Figure 13: Riverbend reconciliation case. (a) Initial files, selected columns from the supplied workbook, and a missing-evidence entry from the triage memo. Amounts are synthetic task inputs, not model results. (b) Five condensed checks from the 13-criterion evaluator-only rubric. P0 and P1 denote critical and major importance. Filenames and table headings are shortened; the 35 initial files include task and metadata carriers. Service files are frozen historical projections. Required outputs appear below the input tree; the supplied originals must remain unchanged.
Figure 14: Oakridge fixture-design case. (a) Initial files and the supplied synthetic STEP workpiece, not a proposed fixture. Indented ZIP children are archive members; filenames are shortened. The 39 initial files include task and metadata carriers. The given limits specify the maximum fixture footprint and stack height, not workpiece dimensions. The load and friction coefficient are synthetic inputs. (b) Five condensed checks from the 18-criterion evaluator-only rubric, with the same importance notation as Figure 13 . Required deliverables appear below the input tree.
| Model | Overall | Assur. | Clin. | Des. | Ent. | Fin. | Ind. | Legal | Soft. |
|---|---|---|---|---|---|---|---|---|---|
| OpenAI | |||||||||
| GPT-6 Astra | 88.4 | 85.8 | 97.9 | 79.3 | 85.3 | 93.7 | 91.1 | 92.3 | 98.2 |
| GPT-5.6 Sol | 68.5 | 67.7 | 64.2 | 66.7 | 72.4 | 68.1 | 73.2 | 81.7 | 62.7 |
| GPT-5.6 Terra | 76.1 | 80.0 | 64.8 | 79.7 | 78.0 | 73.5 | 75.3 | 70.6 | 75.0 |
| GPT-5.6 Luna | 75.4 | 83.6 | 84.1 | 71.6 | 81.4 | 74.0 | 71.6 | 88.4 | 64.9 |
| GPT-5.4 | 72.2 | 73.0 | 75.7 | 70.0 | 80.5 | 69.8 | 70.9 | 59.7 | 72.2 |
Table 8: OfficeTown Bench Weighted Score (%; higher is better). 1st 2nd 3rd : ranks within each column at displayed precision; ties share rank and skip positions. Models, tasks and high-effort settings match Table 2 . Dashes indicate pending results. Domain abbreviations denote assurance, clinical, design, entertainment, finance, industrial, legal, and software work.
| Model | P0 | P1 | P2 | P3 | P0 misses |
|---|---|---|---|---|---|
| GPT-6 Astra | 88.8 | 88.0 | 86.7 | 87.5 | 20.8 |
| GPT-5.6 Terra | 76.5 | 76.9 | 82.2 | 62.5 | 53.3 |
| GPT-5.6 Luna | 76.1 | 74.5 | 72.8 | 50.0 | 58.3 |
| GPT-5.4 | 72.2 | 73.5 | 82.5 | 62.5 | 59.6 |
| GPT-5.2 | 64.9 | 64.7 | 76.0 | 62.5 | 67.6 |
| Claude Opus 5.5 | 70.4 | 70.6 | 62.2 | 50.0 | 41.4 |
Table 9: Requirement fulfillment by importance, and tasks with a critical miss (%). Fulfillment is task-averaged; higher is better. P0 misses are task-level rates; lower is better. Eligible task counts vary by model: P0 390–405, P1 347–360, P2 49–52, and P3 8. The retained instrument contains 3,156/1,582/90/8 criteria at these tiers before outcome-specific applicability checks. The small P2/P3 populations should not be read as broad capability estimates.
| Model | Docs | Tables | Code | Images | CAD | Media |
|---|---|---|---|---|---|---|
| GPT-6 Astra | 85.9 | 87.3 | 85.4 | 83.4 | 90.9 | 78.8 |
| GPT-5.6 Terra | 77.9 | 76.4 | 78.9 | 80.3 | 68.1 | 83.0 |
| GPT-5.6 Luna | 76.7 | 75.7 | 78.4 | 77.2 | 71.4 | 73.7 |
| GPT-5.4 | 74.6 | 74.4 | 74.9 | 76.0 | 66.5 | 68.6 |
| GPT-5.2 | 65.0 | 62.7 | 66.3 | 65.6 | 59.2 | 63.1 |
| Claude Opus 5.5 | 64.9 | 67.3 | 63.0 | 66.4 | 59.2 | 67.1 |
Table 10: Weighted Score by supplied input-file type (%). Group sizes are 318/280/250/195/33/55 tasks, respectively. Groups overlap and retain the main table’s declared zeros. Plain-text and structured inputs occur in every task and would repeat the aggregate; slides have no input coverage, and heterogeneous residual formats are not a single interpretable group.
Figure 15: Pass@1 at five recorded turn budgets. Populations, selected attempts, requested settings and turn counts match Figure 6 . All panels use the same 406 tasks. A final pass contributes only after its recorded execution has ended; all other outcomes contribute zero. Lines guide the eye; no intermediate artifacts are scored.
| Item | SFT | OPD after SFT |
|---|---|---|
| Initial student | Qwen3.5-9B | SFT-trained Qwen3.5-9B |
| Training inputs | Complete demonstrations | Requests and initial environments |
| Training examples | 1,982 | 828 |
| Validation examples | 220 | None |
| Supervised tokens in resource | 30,704,800 | Not applicable |
| Teacher | DeepSeek-V4-Flash | Qwen3.8-Flash-Next |
Table 11: Training resources and teachers. Example counts exclude zero-supervision padding rows; the SFT token total describes the complete demonstration resource. Optimization settings are given in Tables 12 and 13 .
| Parameter | Setting |
|---|---|
| Model / training mode | Qwen3.5-9B; full-parameter SFT |
| Hardware / parallelism | One node, NVIDIA A100 80 GB; FSDP2 |
| Schedule / random seed | 2 epochs, 496 steps; seed 42 |
| Global / micro-batch size | 8 / 1 per GPU |
| Optimizer | AdamW; , |
| Learning rate | ; linear decay to (minimum ratio 0.1) |
Table 12: Full-parameter SFT configuration. The 1,982 training and 220 validation demonstrations are distinct from the two and four zero-supervision padding rows. Training uses 248 steps per epoch and 496 steps in total.
| Parameter | Setting |
|---|---|
| Hardware | One node, NVIDIA B200 |
| Training schedule | One epoch; 8 updates; seed 42 |
| Learning rate | |
| Batch / mini-batch size | 104 / 104; one optimization pass per update |
| Micro-batch size | 1 per GPU |
| Distillation objective | ; vanilla policy gradient; token-mean aggregation |
Table 13: OPD training configuration. The student is initialized from SFT-trained Qwen3.5-9B. Length limits are in tokens; the final 100-task batch is retained.
Figure 16: JobBench-Easy performance over SFT training. All recorded checkpoints from the same training run and the Base reference are shown. Points are run means and error bars are one sample standard deviation; the 14 individual runs are also displayed. The non-monotonic curve measures training progress, not an ablation of unique training-data size.
More in this topic
LLM Agent Training
Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.
See all 590 papers