OfficePrism: Benchmarking Agents on Single-Domain Operations Across Enterprise Business Functions
Organizations: Alibaba Group, Hangzhou, China · HKUST(GZ), Guangzhou, China · Alibaba International Digital Commerce Group, Beijing, China · Cyberspace Institute of Advanced Technology, Guangzhou University, Guangzhou, China
Abstract
Broad workplace scores can mix competence within one business function with the extra cost of coordinating across functions and systems, making failures hard to localize. OfficePrism evaluates this prerequisite through 420 single-domain enterprise tasks spanning seven business functions under shared, benchmark-defined L1--L3 task-structure levels. Each task runs in a fresh environment combining a shared synthetic organization with one function-specific domain and yped state access. Our pipeline uses grounded records, business rules, and tool schemas to jointly construct a Chinese-language ticket, a hidden executable reference witness, and an expected response, followed by automated filtering and human curation. The same task set is evaluated through full-schema and search-and-dispatch packages. Across 124 model--harness--package configurations, the per-case joint natural-language-and-database pass rate ranges from 22.6% to 76.4%, with a median of 67.9%. L1 is highest in all 124 configurations, and L2 exceeds L3 in 122. Among 50 configurations sharing the same harness and model labels across both packages, search scores higher in 30, ties in one, and scores lower in 19.
Figures & tables
| Primary unit | Scope | Verification | |||||
| Benchmark | App | WS | Ops | Fn | 1-Fn | Build | State |
| Office applications and workspaces | |||||||
| OfficeBench | • | – | – | ✓ | |||
| OdysseyBench | • | – | – | ||||
| Workspace-Bench | • | – | – | – | |||
| Enterprise records and workflows | |||||||
| Harness | Pkg. | Rows | L1 | L2 | L3 | All | Steps |
|---|---|---|---|---|---|---|---|
| ReAct | full | 10 | 85.8 | 75.3 | 59.5 | 69.8 | 12.3 |
| ReAct | search | 14 | 85.7 | 74.7 | 57.4 | 71.2 | 19.5 |
| DeepAgents | full | 14 | 82.8 | 71.8 | 56.6 | 67.9 | 7.3 |
| DeepAgents | search | 14 | 85.7 | 74.3 | 57.4 | 70.5 | 11.8 |
| Claude Code | full | 10 | 85.8 | 72.8 | 56.5 | 69.0 | 8.0 |
| Claude Code | search | 14 | 87.2 | 76.1 | 58.0 | 72.0 | 11.9 |
| Harness | Pkg. | Rows | NL | DB | NL–DB | Strict | Steps |
|---|---|---|---|---|---|---|---|
| ReAct | full | 10 | 78.8 | 83.9 | 69.8 | 55.3 | 12.3 |
| ReAct | search | 14 | 78.1 | 83.6 | 71.2 | 57.6 | 19.5 |
| DeepAgents | full | 14 | 75.3 | 80.3 | 67.9 | 53.6 | 7.3 |
| DeepAgents | search | 14 | 77.0 | 82.3 | 70.5 | 54.4 | 11.8 |
| Claude Code | full | 10 | 75.8 | 80.5 | 69.0 | 56.7 | 8.0 |
| Claude Code | search | 14 | 77.8 | 82.3 | 72.0 | 56.9 | 11.9 |
| Model | Median | Min | Max | Full med. | Search med. |
|---|---|---|---|---|---|
| kimi-k3 | 72.5 | 66.7 | 76.4 | 70.2 | 72.9 |
| glm-5.2 | 71.1 | 67.6 | 74.3 | 70.2 | 71.9 |
| qwen3.7-plus | 68.7 | 59.8 | 72.9 | 67.9 | 70.0 |
| deepseek-v4-pro | 68.1 | 65.5 | 72.9 | 67.6 | 68.1 |
| glm-5.1 | 68.0 | 62.9 | 72.4 | 67.1 | 69.0 |
| qwen3.7-max | 66.5 | 61.4 | 72.6 | 65.5 | 69.0 |
| Harness | Pearson | Spearman | |
|---|---|---|---|
| ReAct | 10 | 0.946 | 0.912 |
| DeepAgents | 10 | 0.617 | 0.664 |
| Claude Code | 10 | 0.797 | 0.637 |
| OpenClaw | 10 | 0.544 | 0.644 |
| Hermes | 10 | 0.823 | 0.855 |
| Step field | Median | Mean | Range |
|---|---|---|---|
| L1 | to | ||
| L2 | to | ||
| L3 | to | ||
| All | to |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Task set | Median | Mean | Wins/ties/losses |
|---|---|---|---|
| 300-task base | pp | pp | 30/3/17 |
| 420-task set | pp | pp | 30/1/19 |
| Judge | Pass 1 | Pass 2 | Pass 3 |
|---|---|---|---|
| qwen3-max | 66.97% | 67.00% | 67.00% |
| qwen3.8-max | 67.24% | 67.26% | 67.25% |
| kimi-k3 | 67.24% | 67.24% | 67.24% |
| Model | Original run | Added run 1 | Added run 2 |
|---|---|---|---|
| kimi-k3 | 74.4% | 73.1% | 72.0% |
| glm-5.2 | 71.3% | 70.2% | 69.1% |
| qwen3.7-max | 65.3% | 65.0% | 65.3% |
| qwen3.6-plus | 61.8% | 61.3% | 62.0% |
| Component | Implementation |
|---|---|
| Scenario inventory | A cached survey is used when available. Otherwise an LLM groups tools and tables, with a full-domain parse fallback. L1–L2 use one group and L3 merges up to three same-domain groups. |
| Grounding packet | The packet contains an active caller, tool names and parameters, at most 300 documentation characters per tool, six shared tables, scenario tables, and at most ten prefix records per included table. |
| Prompt and example | Generation uses at most one level-matched example, preferring same-domain full coverage, then same-domain positive overlap, then cross-function full coverage. |
| Component | Implementation |
|---|---|
| Model calls | Generation and filtering are separate calls that share one model-configuration object. |
| Randomness and attempts | Caller, example, and L3 group choices use a process-wide RNG. UUID4 supplies task IDs. Quota filling has at most three rounds and three attempts per scheduled item. |
| Replay and filter | Replay executes the stored chain sequentially on a copied toolkit. Logic, naturalness, and level-alignment scores must each reach four. |
| Redundancy | Structural collapse uses reference-tool set, caller role, and level. Text similarity uses caller-role–level buckets at 0.85; character trigrams are the default encoder. |
| Task record | OfficePrism-420 stores caller, domain, level, group, tool and table subsets, automated-check fields, a reference tool chain, and an expected answer. |
| Artifact stage | Evidence source | |
|---|---|---|
| Machine-check survivors | 1,135 | Saved single-domain generation records. |
| After redundancy control | 852 | Cleaned single-domain construction files. |
| Human-curated pool | 574 | Aggregate reported by the supplied curation note. |
| OfficePrism-420 | 420 | Checksummed task JSON used by the evaluator and every reported configuration; L1/L2/L3 = 105/140/175. JSON SHA-256 (abbrev.) 7d041782c1ae...7b0096d8 . |
| Level | Ticket | Witness | Response | |
|---|---|---|---|---|
| L1 | 105 | 38.7/35 | 1.50/1 | 129.8/69 |
| L2 | 140 | 91.8/87.5 | 3.34/3 | 146.5/122 |
| L3 | 175 | 209.0/200 | 7.66/7 | 390.1/364 |
| All | 420 | 127.4/103.5 | 4.68/4 | 243.8/163 |
| Layer or function | Overlay state Coll. / records | Active state Coll. / records | Registered functions READ / WRITE / generic |
|---|---|---|---|
| Shared infrastructure | — | 16 / 782 | 27 / 4 / 2 |
| Administration | 42 / 16,344 | 58 / 17,126 | 105 / 47 / 2 |
| Finance | 26 / 4,334 | 42 / 5,116 | 82 / 56 / 2 |
| Human resources | 44 / 11,381 | 60 / 12,163 | 115 / 36 / 2 |
| Information technology | 45 / 6,357 | 61 / 7,139 | 118 / 78 / 2 |
| Legal affairs | 37 / 4,105 | 53 / 4,887 | 110 / 66 / 2 |
| Source and attribute | Value | |
|---|---|---|
| base.employee | ||
| Worker type | 75 | 60 full-time, 5 contractors, 5 interns, 3 campus recruits, and 2 campus interns |
| base.employee | ||
| Employment status | 75 | 63 active, 7 probation, 3 terminated, and 2 on leave |
| base.employee | ||
| Direct manager | 75 | Reference present for 74 employees |
| Model | Pkg. | NL–DB | L1 | L2 | L3 | Success | Strict | Steps |
|---|---|---|---|---|---|---|---|---|
| kimi-k3 | full | 76.4 | 88.6 | 82.1 | 64.6 | 85.5 | 61.4 | 13.1 |
| kimi-k3 | search | 76.2 | 90.5 | 78.6 | 65.7 | 83.3 | 58.3 | 19.0 |
| claude-opus-4-7 | search | 74.0 | 89.5 | 74.3 | 64.6 | 80.5 | 57.9 | 17.3 |
| claude-opus-4-8 | search | 74.0 | 92.4 | 72.9 | 64.0 | 81.0 | 57.9 | 17.2 |
| deepseek-v4-flash | full | 73.6 | 86.7 | 77.1 | 62.9 | 81.4 | 59.3 | 14.8 |
| claude-sonnet-5 | search | 73.1 | 88.6 | 79.3 | 58.9 | 80.2 | 58.3 | 20.5 |
| Model | Pkg. | NL–DB | L1 | L2 | L3 | Success | Strict | Steps |
|---|---|---|---|---|---|---|---|---|
| kimi-k3 | full | 76.0 | 91.4 | 82.9 | 61.1 | 82.1 | 58.8 | 7.5 |
| kimi-k3 | search | 76.0 | 91.4 | 82.1 | 61.7 | 82.4 | 57.4 | 11.6 |
| claude-opus-4-7 | search | 72.6 | 92.4 | 71.4 | 61.7 | 79.0 | 55.5 | 10.3 |
| claude-sonnet-5 | search | 72.6 | 93.3 | 76.4 | 57.1 | 79.8 | 58.6 | 13.0 |
| gpt-5.5 | search | 71.9 | 85.7 | 78.6 | 58.3 | 79.0 | 57.6 | 12.3 |
| claude-opus-4-8 | search | 71.2 | 86.7 | 74.3 | 59.4 | 78.6 | 55.5 | 10.5 |
| Model | Pkg. | NL–DB | L1 | L2 | L3 | Success | Strict | Steps |
|---|---|---|---|---|---|---|---|---|
| glm-5.2 | search | 74.3 | 87.6 | 82.1 | 60.0 | 82.4 | 61.2 | 11.3 |
| kimi-k3 | search | 72.9 | 84.8 | 80.7 | 59.4 | 80.0 | 58.8 | 12.1 |
| qwen3.7-plus | full | 72.9 | 90.5 | 75.0 | 60.6 | 80.2 | 59.5 | 7.6 |
| qwen3.7-max | search | 72.6 | 86.7 | 80.0 | 58.3 | 80.0 | 59.8 | 11.7 |
| qwen3.7-plus | search | 72.6 | 90.5 | 77.1 | 58.3 | 79.3 | 59.0 | 12.2 |
| claude-opus-4-8 | search | 72.4 | 87.6 | 76.4 | 60.0 | 78.3 | 55.7 | 10.2 |
| Model | Pkg. | NL–DB | L1 | L2 | L3 | Success | Strict | Steps |
|---|---|---|---|---|---|---|---|---|
| kimi-k3 | search | 72.1 | 90.5 | 77.1 | 57.1 | 76.9 | 57.1 | 13.2 |
| qwen3.7-max | search | 69.0 | 89.5 | 70.0 | 56.0 | 74.8 | 55.7 | 12.7 |
| glm-5.2 | full | 67.9 | 85.7 | 70.7 | 54.9 | 74.0 | 55.5 | 9.0 |
| glm-5.2 | search | 67.6 | 88.6 | 72.9 | 50.9 | 76.2 | 56.2 | 13.5 |
| kimi-k3 | full | 66.7 | 81.9 | 70.0 | 54.9 | 72.4 | 54.3 | 7.8 |
| deepseek-v4-pro | full | 65.7 | 83.8 | 64.3 | 56.0 | 71.7 | 52.4 | 8.8 |
| Model | Pkg. | NL–DB | L1 | L2 | L3 | Success | Strict | Steps |
|---|---|---|---|---|---|---|---|---|
| glm-5.2 | search | 72.6 | 86.7 | 74.3 | 62.9 | 78.8 | 57.9 | 12.7 |
| kimi-k3 | search | 71.9 | 83.8 | 76.4 | 61.1 | 76.9 | 55.0 | 13.3 |
| glm-5.2 | full | 70.2 | 86.7 | 69.3 | 61.1 | 78.3 | 55.7 | 8.3 |
| claude-opus-4-7 | search | 70.0 | 83.8 | 72.9 | 59.4 | 74.8 | 54.0 | 10.3 |
| claude-opus-4-8 | search | 69.5 | 81.0 | 72.1 | 60.6 | 73.8 | 53.6 | 11.1 |
| kimi-k3 | full | 69.3 | 81.0 | 74.3 | 58.3 | 75.2 | 55.5 | 8.6 |