Organizations: Alibaba Group, Hangzhou, China · HKUST(GZ), Guangzhou, China · Alibaba International Digital Commerce Group, Beijing, China · Cyberspace Institute of Advanced Technology, Guangzhou University, Guangzhou, China
Broad workplace scores can mix competence within one business function with the extra cost of coordinating across functions and systems, making failures hard to localize. OfficePrism evaluates this prerequisite through 420 single-domain enterprise tasks spanning seven business functions under shared, benchmark-defined L1--L3 task-structure levels. Each task runs in a fresh environment combining a shared synthetic organization with one function-specific domain and yped state access. Our pipeline uses grounded records, business rules, and tool schemas to jointly construct a Chinese-language ticket, a hidden executable reference witness, and an expected response, followed by automated filtering and human curation. The same task set is evaluated through full-schema and search-and-dispatch packages. Across 124 model--harness--package configurations, the per-case joint natural-language-and-database pass rate ranges from 22.6% to 76.4%, with a median of 67.9%. L1 is highest in all 124 configurations, and L2 exceeds L3 in 122. Among 50 configurations sharing the same harness and model labels across both packages, search scores higher in 30, ties in one, and scores lower in 19.
Figures & tables
Primary unit
Scope
Verification
Benchmark
App
WS
Ops
Fn
1-Fn
Build
State
Office applications and workspaces
OfficeBench
•
∘
∘
∘
–
–
✓
OdysseyBench
•
∘
∘
∘
–
–
△
Workspace-Bench
∘
•
∘
∘
–
–
–
Enterprise records and workflows
Table 1. Selected benchmarks by organizing unit, single-function task scope, and executable verification.
Figure 1. OfficePrism separates the single-function environment, coupled task synthesis, curation, benchmark selection, and the two evaluation interfaces. A three-panel method overview. Panel A combines a shared organization with exactly one of seven business domains to create a fresh state and typed tool surface. Panel B builds a grounding packet from caller context, rules, tool summaries, selected records, and a task-structure level, then proposes a Chinese ticket, reference witness, and expected response. Panel C applies executable replay, a quality filter, redundancy control, and human review to form a manually curated pool before selecting OfficePrism-420 and presenting it through full-schema or search-and-dispatch interfaces.
Figure 2. Composition and environment scale of OfficePrism-420. The figure reports OfficePrism-420 task counts by business function and difficulty level, reference-witness length ranges and medians, and shared-organization and task-local environment scale.
Figure 3. OfficePrism-420 result panel. (a) All 124 NL–DB Pass rows, with vertically offset full and search bands and open diamonds for unweighted cell medians. (b) Search-minus-full differences; filled points are the common-support 50-configuration panel, open points are four additional DeepAgents comparisons, and diamonds mark common-support medians. Two aligned horizontal dot plots. The left panel shows all 124 NL–DB Pass rows by harness, with full and search rows separated by color, shape, and vertical offset. The right panel shows matched-label score differences on the same five harness rows; points occur on both sides of zero and harness medians differ in sign.
Harness
Pkg.
Rows
L1
L2
L3
All
Steps
ReAct
full
10
85.8
75.3
59.5
69.8
12.3
ReAct
search
14
85.7
74.7
57.4
71.2
19.5
DeepAgents
full
14
82.8
71.8
56.6
67.9
7.3
DeepAgents
search
14
85.7
74.3
57.4
70.5
11.8
Claude Code
full
10
85.8
72.8
56.5
69.0
8.0
Claude Code
search
14
87.2
76.1
58.0
72.0
11.9
Table 2. Median results by harness and interface package.
Harness
Pkg.
Rows
NL
DB
NL–DB
Strict
Steps
ReAct
full
10
78.8
83.9
69.8
55.3
12.3
ReAct
search
14
78.1
83.6
71.2
57.6
19.5
DeepAgents
full
14
75.3
80.3
67.9
53.6
7.3
DeepAgents
search
14
77.0
82.3
70.5
54.4
11.8
Claude Code
full
10
75.8
80.5
69.0
56.7
8.0
Claude Code
search
14
77.8
82.3
72.0
56.9
11.9
Table 3. Median outcome components by harness and interface package. NL, DB, NL–DB, and Strict are percentages; Steps is the recorded interaction length.
Figure 4. Recorded interaction steps by task-structure level across the 124 configurations. Points are configurations, thick bars span the interquartile range, diamonds mark medians, and thin lines show the observed range. Three compact horizontal dot distributions show recorded interaction steps for L1, L2, and L3. Their medians increase from 5.9 to 9.4 and 16.1 steps.
Model
Median
Min
Max
Full med.
Search med.
kimi-k3
72.5
66.7
76.4
70.2
72.9
glm-5.2
71.1
67.6
74.3
70.2
71.9
qwen3.7-plus
68.7
59.8
72.9
67.9
70.0
deepseek-v4-pro
68.1
65.5
72.9
67.6
68.1
glm-5.1
68.0
62.9
72.4
67.1
69.0
qwen3.7-max
66.5
61.4
72.6
65.5
69.0
Table 4. NL–DB Pass across the ten harness–package configurations available for each common-support model label.
Harness
N
Pearson r
Spearman ρ
ReAct
10
0.946
0.912
DeepAgents
10
0.617
0.664
Claude Code
10
0.797
0.637
OpenClaw
10
0.544
0.644
Hermes
10
0.823
0.855
Table 5. Agreement between full and search NL–DB scores over the ten common model labels in each harness.
Step field
Median
Mean
Range
L1
+2.6
+2.85
+1.5 to +5.0
L2
+4.3
+4.62
+2.1 to +8.0
L3
+6.5
+6.95
+2.6 to +11.6
All
+4.7
+5.15
+3.3 to +8.6
Table 6. Search-minus-full recorded steps for the 50 common-support harness–model pairs.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5. Top-30 harness–model combinations by mean full/search NL-DB Pass on the 420-task display set. Each bar end shows the model-family and harness marks followed by the one-decimal score, and the dashed line marks the mean of all 54 combinations. Harness abbreviations: DA (DeepAgents), CC (Claude Code), OC (OpenClaw), HM (Hermes); model names are shown in full. A full-page horizontal bar chart ranks thirty harness and model combinations from highest to lowest mean NL-DB Pass. Bars are colored by harness, each bar end shows a model-family mark, a harness mark, and a one-decimal percentage, and a dashed vertical line marks the overall mean.
Figure 6. Level-reweighted NL-DB Pass by business function, averaged over models within each harness, for the search and full packages. Two radar charts with seven business-function axes compare five harness profiles, the search package on the left and the full package on the right, on a zero to one hundred radial scale.
Figure 7. NL-DB Pass against average steps per task, with bubble area proportional to tokens per task, for the full and search packages. The dotted line connects the labeled Pareto-frontier combinations; dominated combinations are faded. Two bubble panels plot NL-DB Pass against average steps for the full and search packages. Bubble color encodes harness and bubble area encodes tokens per task. In each panel a dotted line connects the labeled Pareto-frontier bubbles, and the other bubbles are drawn faded.
Figure 8. Full and search NL-DB values for the 54 combinations evaluated under both packages, sorted by the search-minus-full difference. Dark dots show full, teal dots show search where it is higher, orange dots where it is lower, and each row is annotated with its signed difference. A two-column dumbbell chart. Each row prints a harness and model label beside a pair of dots connected by a line, a dark full-package dot and a colored search-package dot, teal when search is higher and orange when lower, with the signed difference printed at the right end of the pair.
Figure 9. Tool-pool scaling on the auxiliary 150-task subset. Rows show the ReAct and Claude Code harnesses; columns show NL-DB Pass and tokens per task against the average discoverable tool count. Solid lines are the search package, dashed lines the full package, and bold black lines the package means. Four line panels arranged two by two. The top row shows the ReAct harness and the bottom row the Claude Code harness. Left panels show NL-DB Pass and right panels show tokens per task in thousands, each with four model lines in solid search and dashed full styles plus bold black package means labeled search mean and full mean.
Figure 10. Failure root-cause distributions for the strongest leaderboard combination (Kimi k3 under DeepAgents, search package) on the 420-task display set. The left donut shows the natural-language causes and the right donut the database causes, each labeled with per-cause counts and shares. A failing task can carry both an NL and a DB label, so the two denominators differ. Two donut charts. The left donut shows natural-language root causes and the right donut database root causes for the search package. Each donut shows percentage labels on its wedges, the number of labeled failures in its center, and a legend with per-cause counts below.
Task set
Median
Mean
Wins/ties/losses
300-task base
+0.67 pp
+0.71 pp
30/3/17
420-task set
+1.07 pp
+1.05 pp
30/1/19
Appendix
Table 7. Search-minus-full differences (percentage points) and search wins/ties/losses on the 300-task base and the 420-task set.
Judge
Pass 1
Pass 2
Pass 3
qwen3-max
66.97%
67.00%
67.00%
qwen3.8-max
67.24%
67.26%
67.25%
kimi-k3
67.24%
67.24%
67.24%
Appendix
Table 8. Mean joint NL–DB scores from three judges, three passes each.
Model
Original run
Added run 1
Added run 2
kimi-k3
74.4%
73.1%
72.0%
glm-5.2
71.3%
70.2%
69.1%
qwen3.7-max
65.3%
65.0%
65.3%
qwen3.6-plus
61.8%
61.3%
62.0%
Appendix
Table 9. NL–DB scores of the four models across the three runs.
Component
Implementation
Scenario inventory
A cached survey is used when available. Otherwise an LLM groups tools and tables, with a full-domain parse fallback. L1–L2 use one group and L3 merges up to three same-domain groups.
Grounding packet
The packet contains an active caller, tool names and parameters, at most 300 documentation characters per tool, six shared tables, scenario tables, and at most ten prefix records per included table.
Prompt and example
Generation uses at most one level-matched example, preferring same-domain full coverage, then same-domain positive overlap, then cross-function full coverage.
Appendix
Table 10. Grounding and synthesis configuration.
Component
Implementation
Model calls
Generation and filtering are separate calls that share one model-configuration object.
Randomness and attempts
Caller, example, and L3 group choices use a process-wide RNG. UUID4 supplies task IDs. Quota filling has at most three rounds and three attempts per scheduled item.
Replay and filter
Replay executes the stored chain sequentially on a copied toolkit. Logic, naturalness, and level-alignment scores must each reach four.
Redundancy
Structural collapse uses reference-tool set, caller role, and level. Text similarity uses caller-role–level buckets at 0.85; character trigrams are the default encoder.
Task record
OfficePrism-420 stores caller, domain, level, group, tool and table subsets, automated-check fields, a reference tool chain, and an expected answer.
Appendix
Table 11. Model calls, automated checks, and task records.
Artifact stage
N
Evidence source
Machine-check survivors
1,135
Saved single-domain generation records.
After redundancy control
852
Cleaned single-domain construction files.
Human-curated pool
574
Aggregate reported by the supplied curation note.
OfficePrism-420
420
Checksummed task JSON used by the evaluator and every reported configuration; L1/L2/L3 = 105/140/175. JSON SHA-256 (abbrev.) 7d041782c1ae...7b0096d8 .
Appendix
Table 12. Construction lifecycle and benchmark identity.
Large language model (LLM) agents are increasingly expected to assist users in completing tasks. However, existing benchmarks provide limited support for evaluating whether agents can carry out office-suite workflows at a reasonable cost. We introduce OmegaUse-OfficeVal, a benchmark for evaluating LLM agents on long-horizon office-suite tasks with task-level economic grounding. The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a privacy-preserving process. On average, these tasks require 2.32 hours of human labor to complete. An important feature of the benchmark is that each task is paired with two economic signals: human labor time and task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation. To support stable evaluation, we develop code-based verifiers from fine-grained rubrics. We evaluate several frontier LLMs together with a human baseline. Although all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality. The code and dataset are fully open-sourced, and more information is available on our project website: https://omegause-officeval.github.io.
Jingbo Zhou, Yusai Zhao, Qi Bao +12
Agent Frontier Team, Large Model Frontier Research Department, Baidu Inc., China.
Enterprise agents increasingly operate inside workspaces: they read heterogeneous files, invoke tools, and deliver business artifacts. We introduce EnterpriseClawBench, an enterprise agent benchmark constructed from proprietary, real-world agent sessions. Starting from a large archive of workplace sessions, the EnterpriseClawBench produces 852 reproducible tasks, each paired with recovered fixtures, rewritten prompts, role classes, skill subclasses, hard rules, and semantic rubrics. Because the sessions contain internal enterprise content, we do not release the benchmark data; instead, our reusable contribution is the construction and evaluation protocol. On EnterpriseClawBench, the best configuration reaches only 0.663 (Codex with GPT-5.5). These results show that enterprise agent evaluation must report harness--model combinations, artifact delivery, visual quality, cost, runtime, and skill-transfer behavior, rather than collapsing performance into a single score. Code: https://github.com/FrontisAI/EnterpriseClawBench
We introduce Office Comprehension Bench (OCB), the first public benchmark to jointly evaluate LLM systems on Word, Excel, and PowerPoint comprehension over native file formats (.docx, .xlsx, .pptx) and their variants. OCB consists of two tracks. File Fidelity Q&A tests structural and visual perception of office artifacts - tables, charts, embedded images, formulas, and app-specific elements such as headers, speaker notes, and named ranges. Domain Q&A tests expert-level reasoning grounded in real-world industry documents across 12 professional domains, with queries requiring multi-step analysis and synthesis across documents. Each reference answer is decomposed into atomic, binary-gradable claims, and an ensemble of LLM judges scores responses against each claim independently. Even the strongest frontier system in its default reasoning mode reaches only about 59.3% on Domain Q&A; increasing thinking depth within a tier does not move performance materially, while moving to a higher product tier yields modest gains. We release the dataset, evaluation tooling, judge prompt, and a public leaderboard.
Firoz Shaik, Mateus Picanço Lima Gomes, Tanvir Aumi +17