AI agents are increasingly developed and evaluated on benchmarks relevant to human work, yet it remains unclear how representative these benchmarking efforts are of the labor market as a whole. In this work, we systematically study the relationship between agent development efforts and the distribution of real-world human work by mapping benchmark instances to work domains and skills. We first analyze 43 benchmarks and 72,342 tasks, measuring their alignment with human employment and capital allocation across all 1,016 real-world occupations in the U.S. labor market. We reveal substantial mismatches between agent development that tends to be programming-centric, and the categories in which human labor and economic value are concentrated. Within work areas that agents currently target, we further characterize current agent utility by measuring their autonomy levels, providing practical guidance for agent interaction strategies across work scenarios. Building on these findings, we propose three measurable principles for designing benchmarks that better capture socially important and technically challenging forms of work: coverage, realism, and granular evaluation.
Figures & tables
Figure 1: Mapping agent benchmarks to work domains and skills in O*NET.
Category
Benchmark
Size
Coverage (%)
total
sampled
domain
skill
General Digital Work
TheAgentCompany ( Xu et al., 2024 )
175
175
30.4
56.1
GDPval ( Patwardhan et al., 2025 )
220
220
47.8
58.5
Remote Labor Index ( Mazeika et al., 2025 )
9
9
21.7
17.1
WorkArena ( Drouin et al., 2024 )
16,833
300
26.1
22.0
OfficeBench ( Wang et al., 2024b )
300
300
39.1
43.9
Table 1: Agent benchmark sizes and their coverage on the real-world work. GDPval, despite being relatively small, has the highest domain and skill coverage across human work.
Figure 2: Agent benchmarking effort is concentrated in mathematical and engineering domains, despite broad human employment and economic value across other domains.
Figure 3: Agent benchmarking effort emphasizes two granular skills (getting information and working with computers), which together, only cover <5 % human employment, creating a heavily imbalanced distribution across skills. See § A.2 for full skill names.
Figure 4: Number of domains (top) and skills (bottom) an average benchmark example map to.
Figure 5: Exemplar workflow induced from agent low-level action trajectory.
Figure 6: Agent autonomy levels by (left) work domains and skills, and (right) agent frameworks and LM backbones.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Work activities annotation provided by O*NET.
Benchmark
Size
Domain Coverage (%)
Skill Coverage (%)
total
avg
95% CI
avg
95% CI
avg
95% CI
TheAgentCompany
175
175
[175.0,175.0]
62.1
[62.1,62.1]
60.7
[60.7,60.7]
GDPval
220
220
[220.0,220.0]
77.0
[77.0,77.0]
82.7
[82.7,82.7]
Remote Labor Index
9
9
[9.0,9.0]
27.8
[27.8,27.8]
35.7
[35.7,35.7]
WorkArena
300
99.6
[97.2,102.0]
82.5
[81.7,83.4]
69.6
[68.1,71.2]
OfficeBench
300
186.0
[183.5,188.5]
73.4
[72.5,74.3]
82.3
[81.6,82.9]
Appendix
Table 2: Permutation-based sensitivity analysis of the coverage-aware sampling procedure. For each benchmark, we report the distribution (median and 95% confidence interval over 500 permutations) of (i) the number of tasks selected and (ii) Chao1-estimated domain and skill coverage, before stopping.
Label
When to Use
Key Question
All Correct
All assigned locations are appropriate and no relevant locations are missing. The assignment set comprehensively and accurately captures the instruction’s requirements.
Are all assignments appropriate and is the set complete?
All Wrong
Multiple assigned locations are inappropriate and potentially need both addition/removal. The LLM has fundamentally misunderstood the task or assigned completely unrelated locations.
Is every single assignment inappropriate or unrelated?
Missing Locations
All current assignments are correct, but additional relevant locations should be included to fully capture the instruction’s scope.
Are the current assignments right, but incomplete? What’s missing?
Extra Locations
The assignment includes some correct locations but also contains extraneous assignments that are not required by the instruction.
Which specific assignments are inappropriate and should be removed?
Appendix
Table 3: Rubrics for manually validating LM-based mapping results.
Level
Exemplar Task
1
Navigate to the Plus section of Cambridge Dictionary to access the Contents menu for Image Quizzes.
2
Navigating the Google Travel interface menu options.
3
Locate the sparql command and verify rdflib Python module installation.
4
Define functions to load and preprocess training, validation, and test datasets.
5
Authenticate with phone system and retrieve contact relationship information.
6
Retrieve API documentation to understand available endpoints and their functionality.
Appendix
Table 4: Exemplar task instructions at varied task complexities.