OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation

Xiaomeng Hu; Yinger Zhang; Fei Huang; Jianhong Tu; Yang Su; Lianghao Deng; Yuxuan Liu; Yantao Liu

cs.CLApr 13, 2026

OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation

Authors: Xiaomeng Hu, Yinger Zhang, Fei Huang, Jianhong Tu, Yang Su, Lianghao Deng, Yuxuan Liu, Yantao Liu,

Organizations: Qwen Team, Alibaba Group · The Chinese University of Hong Kong

Abstract

AI agents are expected to perform professional work across hundreds of occupational domains (from emergency department triage to nuclear reactor safety monitoring to customs import processing), yet existing benchmarks can only evaluate agents in the few domains where public environments exist. We introduce OccuBench, a benchmark covering 100 real-world professional task scenarios across 10 industry categories and 65 specialized domains, enabled by Language Environment Simulators (LESs) that simulate domain-specific environments through LLM-driven tool response generation. Our multi-agent synthesis pipeline automatically produces evaluation instances with guaranteed solvability, calibrated difficulty, and document-grounded diversity. OccuBench evaluates agents along two complementary dimensions: task completion across professional domains and environmental robustness under controlled fault injection (explicit errors, implicit data degradation, and mixed faults). We evaluate 15 frontier models across 8 model families and find that: (1) no single model dominates all industries, as each has a distinct occupational capability profile; (2) implicit faults (truncated data, missing fields) are harder than both explicit errors (timeouts, 500s) and mixed faults, because they lack overt error signals and require the agent to independently detect data degradation; (3) larger models, newer generations, and higher reasoning effort consistently improve performance. GPT-5.2 improves by 27.5 points from minimal to maximum reasoning effort; and (4) strong agents are not necessarily strong environment simulators. Simulator quality is critical for LES-based evaluation reliability. OccuBench provides the first systematic cross-industry evaluation of AI agents on professional occupational tasks.

Figures & tables

Panel of Figure 1 — Figure 1: LES evaluation loop. At each step, the agent issues a tool call $a_{t}$ ; the LES generates an observation $o_{t}$ conditioned on its configuration $c$ and conversation history $H_{t-1}$ . State is maintained implicitly through in-context history. The trajectory is scored by a rubric-based verifier.

Category	#	Representative Scenarios
Business & Enterprise	19	Resume screening, expense auditing, AML review
Technology & IT	16	Linux ops, CI/CD recovery, intrusion response
Industrial & Engineering	12	Production scheduling, mine ventilation
Transportation & Logistics	11	Last-mile delivery, train dispatch
Commerce & Consumer	9	Dynamic pricing, hotel revenue mgmt.
Education & Culture	8	Adaptive curriculum, fact-checking

Table 1: Industry categories and representative scenarios in OccuBench .

Model	Avg	Agri	Biz	Comm	Edu	Hlth	Ind	Pub	Sci	Tech	Trans
GPT-5.2	79.6	84	86	67	77	76	85	84	94	80	72
Gemini 3.1 Pro	72.3	68	73	75	84	62	73	72	81	78	60
Claude Opus 4.6	71.5	74	78	53	75	76	73	68	62	68	77
Qwen 3.5 Plus	69.9	77	70	81	56	81	71	76	69	74	55
DeepSeek V3.2	69.6	65	78	67	66	71	69	72	62	74	64
Claude Opus 4.5	65.2	58	76	56	62	52	65	72	56	68	66

Table 2: E0 completion rate (%) by industry category for all 15 models. All models use thinking mode; for models with adjustable reasoning effort, we set it to high . Bold : best in each category. Models sorted by average score.

Figure 2: Radar chart showing model performance profiles across 10 industry categories (E0). Each model has a distinct shape, indicating different occupational specializations.

Model	E0	E1	E2	E3	Rob.
Gemini 3.1 Pro	72.3	73.3	63.1	65.2	0.87
MiniMax M2.7	53.9	52.9	47.1	46.9	0.87
GPT-5.2	79.6	75.9	70.4	67.0	0.84
GLM-5	62.6	59.4	52.6	47.4	0.76
Claude Opus 4.6	71.5	68.1	53.9	63.9	0.75
DeepSeek V3.2	69.6	59.9	56.0	51.6	0.74

Table 3: Environmental robustness evaluation for 9 flagship models. CR = Completion Rate (%). Rob. = Robustness (

\min(\text{CR}_{E1},\text{CR}_{E2},\text{CR}_{E3})/\text{CR}_{E0}

). Bold : best in each column. Models sorted by Robustness.

Figure 3: Completion rates under clean (E0) and fault-injected (E1–E3) environments. Models are sorted by E0 performance.

Figure 4: Fault parameter ablation under E3 mixed faults. (a) Varying fault count with fixed duration=2. (b) Varying fault duration with fixed count=2.

Figure 5: Large vs. small model variants within each family (E0). Gaps range from 0.3% to 11.0%.

Figure 6: Claude generational progress from v4 to v4.6 (E0). Opus shows consistent improvement (+10.2%); Sonnet shows large initial gains but slight regression from v4.5 to v4.6.

Figure 7: Effect of reasoning effort on agent performance (E0).

	Gemini Flash		Qwen 3.5+		GPT-5.2
Agent	CR	Rk	CR	Rk	CR	Rk
GPT-5.2	79.6	1	74.3	1	42.4	1
Gemini Pro	72.3	2	68.6	2	28.3	4
Opus 4.6	71.5	3	66.2	3	33.5	2
Qwen 3.5+	69.9	4	61.8	6	28.3	4
DeepSeek	69.6	5	65.2	4	29.6	3

Table 4: Cross-simulator evaluation (E0). Each cell shows completion rate (%) and rank among the 8 agents. Rankings are computed independently within each simulator.

Figure 8: Pairwise ranking agreement across simulators. Each cell shows the fraction of the 28 model pairs whose relative ordering is preserved between two simulators.

Figure 12: Average completion rate across 14 models per industry category (E0). Green:

\geq

65%, orange: 60–65%, red:

<

60%. Transportation and Education are the hardest industries.

Claude Opus 4.6 — PASS (6 steps)	DeepSeek V3.2 — FAIL (6 steps)
Step 1–3 (parallel):	Step 1 : get_vehicle_telemetry
query_inventory	$\rightarrow$ battery: 28%, loc: depot_ohare_cargo
$\rightarrow$ MED-602, MED-615 , MED-609, GEN-550	Step 2 : query_inventory
get_vehicle_telemetry	$\rightarrow$ MED-602, MED-615 , MED-609, GEN-550
$\rightarrow$ battery: 28%, loc: depot_ohare_cargo	Step 3 : geocode_delivery_node
geocode_delivery_node	$\rightarrow$ NODE_WALTON_ST_900_NAV

Figure 13: Case study: Last-Mile Delivery Routing. Top: task specification and tool schema. Middle: side-by-side agent trajectories. Bottom: analysis. The key differentiator is whether the agent proactively checks constraints before acting.

Claude Opus 4.6 — PASS (7 steps)	Qwen 3.5 Plus — FAIL (7 steps)
Step 1–3 (parallel profiling):	Step 1–3 (sequential profiling):
get_vertical_profile([0..5])	get_vertical_profile([0.5,2.5,4.5,5.2])
$\rightarrow$ surface 29.5°C, bottom 27.1°C	$\rightarrow$ gradient 2.1°C
$\rightarrow$ bottom DO: 3.5 mg/L (below 4.0)	$\rightarrow$ bottom DO: 4.2 mg/L
get_water_chemistry(5)	get_water_chemistry(5.2)
$\rightarrow$ ammonia: 0.05 mg/L (safe to mix)	$\rightarrow$ ammonia: 0.05 mg/L

Figure 14: Case study 2: Fish Farm Water Quality Control. Both agents achieve the target water parameters, but Qwen 3.5 Plus fails to re-verify bottom chemistry after mixing, a safety-critical omission in aquaculture operations.

Claude Opus 4.6 — PASS (11 steps)	DeepSeek V3.2 — FAIL (12 steps)
Step 1 : get_environment_state	Step 1 : get_environment_state
$\rightarrow$ safety cert: expired , hot work: pending	$\rightarrow$ same initial state
Step 2–4 (pre-work compliance):	Step 2–3 (containment only):
renew_permit (safety_cert) $\checkmark$	manage_containment (activate)
renew_permit (hot_work) $\checkmark$	manage_containment (verify_icra)
manage_containment (activate)

Figure 15: Case study 3: Building Inspection Compliance. Both agents perform similar tool calls, but DeepSeek V3.2 executes them in the wrong order (brazing before permits) and omits valve restoration, which are procedural errors that violate safety protocols.

Claude Opus 4.6 — PASS (12 steps)	Kimi K2.5 — FAIL (2 steps)
Step 1 (parallel):	Step 1 : set_rtpi_visibility (BUS_202)
get_fleet_status , set_rtpi_visibility ,	$\rightarrow$ $\checkmark$ suppressed
check_maintenance_holds (BUS_202),	Step 2 : check_maintenance_holds (BUS_210)
check_maintenance_holds (BUS_210)	$\rightarrow$ 500: Service unavailable
$\rightarrow$ both holds: TimeoutError
“Hold checks timed out. Let me retry.”

Figure 16: Case study 4 (E1): Public Transit Schedule Recovery. Opus persists through 4 explicit errors via retry; Kimi abandons the task after 1 error.

Claude Opus 4.6 — PASS (4 steps)	Kimi K2.5 — FAIL (4 steps)
Step 1 (parallel):	Step 1 : get_current_date
get_property_metadata $\rightarrow$ 15 units	Step 2 : get_property_metadata $\rightarrow$ 15 units
fetch_unit_level_records	Step 3 : fetch_unit_level_records
$\rightarrow$ only 2 units returned (A-01, A-02)	$\rightarrow$ only 2 units returned (A-1, A-2)
get_current_date
“Only 2 of 15 units returned.	Kimi also retries:

Figure 17: Case study 5 (E2): Property Valuation Assessment. Opus detects truncated data (2/15 units) and re-fetches. Kimi assumes truncated data is complete and produces a dangerously wrong financial assessment.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,194 papers

$OneMillion-Bench: How Far are Language Agents from Human Experts?
Mar 9, 2026Yang Liu, Jiaqi Li, Jun Bai +20LLM Agent Evaluation AI Agent Benchmarks
BIGAI · Humanlaya · xbench +1
OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios
Jul 16, 2026Chengyu Shen, Yujie Fu, Gangtao Xin +13Computer-Use Agent Benchmarks AI Agent Evaluation
Huawei Cloud Post-Training Team
JobBench: Aligning Agent Work With Human Will
May 25, 2026Yuetai Li, Yichen Feng, Zhangchen Xu +21Computer-Use Agent Benchmarks LLM Agent Evaluation
University of Washington · 10Bake AI · University of California, Santa Barbara +10

Mar 9, 2026J/K move · Enter open · S save

$OneMillion-Bench: How Far are Language Agents from Human Experts?

Yang Liu, Jiaqi Li, Jun Bai, Qianyu Yang+19

BIGAI · Humanlaya · xbench+1

LLM Agent Evaluation AI Agent Benchmarks AI Agent Reliability

Open paper AskPDF arXiv

As language models (LMs) evolve from chat assistants to long-horizon agents capable of multi-step reasoning and tool use, existing benchmarks remain largely confined to structured or exam-style tasks that fall short of real-world professional demands. To this end, we introduce $OneMillion-Bench ($OMB), a benchmark of 400 expert-curated tasks spanning Law, Finance, Industry, Healthcare, and Natural Science, built to evaluate agents across economically consequential scenarios. Unlike prior work, the benchmark requires retrieving authoritative sources, resolving conflicting evidence, applying domain-specific rules, and making constraint decisions, where correctness depends as much on the reasoning process as the final answer. We adopt a rubric-based evaluation protocol scoring factual accuracy, logical coherence, practical feasibility, and professional compliance, focusing on expert-level problems to ensure meaningful differentiation across agents. Together, $OMB provides a unified testbed for assessing agentic reliability, professional depth, and an indicator of practical readiness in domain-intensive scenarios.

Abstract

LLM Agent Evaluation

Explore similar work

$OneMillion-Bench: How Far are Language Agents from Human Experts?

OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

JobBench: Aligning Agent Work With Human Will