AI agents are increasingly developed and evaluated on benchmarks relevant to human work, yet it remains unclear how representative these benchmarking efforts are of the labor market as a whole. In this work, we systematically study the relationship between agent development efforts and the distribution of real-world human work by mapping benchmark instances to work domains and skills. We first analyze 43 benchmarks and 72,342 tasks, measuring their alignment with human employment and capital allocation across all 1,016 real-world occupations in the U.S. labor market. We reveal substantial mismatches between agent development that tends to be programming-centric, and the categories in which human labor and economic value are concentrated. Within work areas that agents currently target, we further characterize current agent utility by measuring their autonomy levels, providing practical guidance for agent interaction strategies across work scenarios. Building on these findings, we propose three measurable principles for designing benchmarks that better capture socially important and technically challenging forms of work: coverage, realism, and granular evaluation.
Figures & tables
Figure 1: Mapping agent benchmarks to work domains and skills in O*NET.
Category
Benchmark
Size
Coverage (%)
total
sampled
domain
skill
General Digital Work
TheAgentCompany ( Xu et al., 2024 )
175
175
30.4
56.1
GDPval ( Patwardhan et al., 2025 )
220
220
47.8
58.5
Remote Labor Index ( Mazeika et al., 2025 )
9
9
21.7
17.1
WorkArena ( Drouin et al., 2024 )
16,833
300
26.1
22.0
OfficeBench ( Wang et al., 2024b )
300
300
39.1
43.9
Table 1: Agent benchmark sizes and their coverage on the real-world work. GDPval, despite being relatively small, has the highest domain and skill coverage across human work.
Figure 2: Agent benchmarking effort is concentrated in mathematical and engineering domains, despite broad human employment and economic value across other domains.
Figure 3: Agent benchmarking effort emphasizes two granular skills (getting information and working with computers), which together, only cover <5 % human employment, creating a heavily imbalanced distribution across skills. See § A.2 for full skill names.
Figure 4: Number of domains (top) and skills (bottom) an average benchmark example map to.
Figure 5: Exemplar workflow induced from agent low-level action trajectory.
Figure 6: Agent autonomy levels by (left) work domains and skills, and (right) agent frameworks and LM backbones.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Work activities annotation provided by O*NET.
Benchmark
Size
Domain Coverage (%)
Skill Coverage (%)
total
avg
95% CI
avg
95% CI
avg
95% CI
TheAgentCompany
175
175
[175.0,175.0]
62.1
[62.1,62.1]
60.7
[60.7,60.7]
GDPval
220
220
[220.0,220.0]
77.0
[77.0,77.0]
82.7
[82.7,82.7]
Remote Labor Index
9
9
[9.0,9.0]
27.8
[27.8,27.8]
35.7
[35.7,35.7]
WorkArena
300
99.6
[97.2,102.0]
82.5
[81.7,83.4]
69.6
[68.1,71.2]
OfficeBench
300
186.0
[183.5,188.5]
73.4
[72.5,74.3]
82.3
[81.6,82.9]
Appendix
Table 2: Permutation-based sensitivity analysis of the coverage-aware sampling procedure. For each benchmark, we report the distribution (median and 95% confidence interval over 500 permutations) of (i) the number of tasks selected and (ii) Chao1-estimated domain and skill coverage, before stopping.
Label
When to Use
Key Question
All Correct
All assigned locations are appropriate and no relevant locations are missing. The assignment set comprehensively and accurately captures the instruction’s requirements.
Are all assignments appropriate and is the set complete?
All Wrong
Multiple assigned locations are inappropriate and potentially need both addition/removal. The LLM has fundamentally misunderstood the task or assigned completely unrelated locations.
Is every single assignment inappropriate or unrelated?
Missing Locations
All current assignments are correct, but additional relevant locations should be included to fully capture the instruction’s scope.
Are the current assignments right, but incomplete? What’s missing?
Extra Locations
The assignment includes some correct locations but also contains extraneous assignments that are not required by the instruction.
Which specific assignments are inappropriate and should be removed?
Appendix
Table 3: Rubrics for manually validating LM-based mapping results.
Level
Exemplar Task
1
Navigate to the Plus section of Cambridge Dictionary to access the Contents menu for Image Quizzes.
2
Navigating the Google Travel interface menu options.
3
Locate the sparql command and verify rdflib Python module installation.
4
Define functions to load and preprocess training, validation, and test datasets.
5
Authenticate with phone system and retrieve contact relationship information.
6
Retrieve API documentation to understand available endpoints and their functionality.
Appendix
Table 4: Exemplar task instructions at varied task complexities.
Current benchmarks for occupational AI agents are scoped primarily by economic values, telling a replacement story. We introduce JobBench, which evaluates AI agents on the workflows that experts identify as high-priority for delegation, empowering humans based on their needs instead of replacing them with GDP value. JobBench covers 130 agentic tasks across 35 occupations. Each task is packaged as a workspace of heterogeneous reference files, requiring the agent to reason through the cluttered information streams of real professional work. Outputs are graded by a fact-anchored chain of rubrics, averaging 35.6 binary criteria per task. We evaluate 36 models; the strongest, Claude Opus~4.7 under Claude Code, reaches only 45.9 %. We hope JobBench shifts the community's target labour-market effect from replacement to enhancement: building agents that do what humans actually want delegated, not only what is most economically valuable.
Yuetai Li, Yichen Feng, Zhangchen Xu +21
University of Washington · 10Bake AI · University of California, Santa Barbara +10
Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a benchmark designed to evaluate AI agents on long horizon, economically valuable, real world tasks with verifiable outcomes. Developed in collaboration with 250+ industry experts, ALE covers non-physical industries defined with reference to O*NET / SOC 2018 (the U.S. federal occupational taxonomy). It is organized around a task taxonomy with 55 sub fields grouped into 13 industry clusters covering 1K+ tasks. Current results show that the hardest tier remains far from saturated: across mainstream harness and backbone configurations, the average full pass rate is below 1%. ALE is designed as a living benchmark: its task pool grows continuously as new workflows and industries are onboarded. More broadly, ALE is intended not merely as another leaderboard, but as an instrument for closing the gap between benchmark success and GDP relevant impact.
AI agents are reshaping the workspace, leading to drastic change of how humans work. Despite the considerable potential of human-agent collaboration both in preserving human agency and generating economic value, this paradigm remains largely absent from occupational task evaluation, hindered by the difficulty of gathering real human data and accounting for inter-human variability. We introduce CollabSkill, a framework for evaluating human-agent collaboration on real-world occupational tasks. CollabSkill pairs real human workers with AI agents on tasks matched to their occupational background, collecting data that capture the complexity of economically valuable tasks and the usage patterns of real workers. To account for inter-human variability, CollabSkill employs a Bayesian skill rating system to disentangle and quantify the skill contributions of both humans and AI agents. Drawing on over 1,500 prompts from 386 working sessions contributed by 93 human workers, our analysis yields insights on two fronts: on the agent side, rankings on CollabSkill diverge meaningfully from those of existing fully autonomous benchmarks where Codex leads, with Claude Code ranking first; on the human side, CollabSkill reveals that practical experience emerges as the primary driver of collaboration skill, with hands-on collaboration meaningfully shifting workers' AI literacy. Together, we hope CollabSkill enables the community to invest in systematic evaluation of human-agent collaboration and spurs development efforts aimed at building AI agents that genuinely augment human workers.
Yijia Shao, Zora Zhiruo Wang, Neel Ahuja +3
Stanford University · Carnegie Mellon University · University of Rochester +1