Organisations deploying AI face a scoping problem: which tasks can be automated, which should remain with humans, and which are best shared between the two. Aggregate benchmark scores provide little insight into where systems will succeed or fail in practice, while human judgements of model capabilities quickly become outdated. We introduce a pipeline that profiles agents and tasks using a shared set of core cognitive capabilities. Cognitive capability profiling infers an agent's capabilities from performance on a benchmark battery annotated for the cognitive demands of each item. Task requirements weighting elicits from domain experts the relative importance of these same capabilities for their work. As both use a common set of cognitive dimensions, they can be updated independently as models and roles change, and combined to estimate AI suitability at the level of a domain, organisation, role, or individual duty. We validate capability recovery on synthetic agents, profile six AI systems, and elicit task requirements from 410 employees across six occupational domains. AI systems differed more across cognitive dimensions than across model families, while workplace activities converged on a shared cognitive core. The resulting scores provide a comparative scoping tool for identifying promising candidates for piloting and areas where current systems are unlikely to be well suited. We discuss extending the framework to profile human workers alongside AI systems, moving from AI suitability towards human-machine task allocation.
Figures & tables
Figure 1: The pipeline for assessing task suitability involves: (1) generating cognitive capability profiles for a selection of agents, (2) weighting the cognitive requirements of target workplace tasks by their relative importance, and (3) mapping capability profiles to relative importance matrices. For simplicity, we have included two candidate AI agents, but commensurable capability profiles can also in principle be generated for humans and for human + AI pairings.
Benchmark
Items
Abstract Narrative Understanding [ 75 ]
303
AGIEval [ 76 ]
1896
BigBenchHard [ 77 ]
1265
BigToM [ 78 ]
686
Cause and Effect [ 75 ]
51
CoQA [ 75 ]
210
Table 1: Benchmarks
Figure 2: Capability clustering. Pearson correlations between the demand profiles of the 16 capabilities retained following reliability screening (Table A2 ), revealing dimensions that placed similar demands on the same items. Correlations were computed from annotations z-scored within each rater and capability, controlling for systematic differences between the two raters. The left-hand dendrogram shows hierarchical agglomerative clustering on the distance 1−r between dimensions. The red dashed line marks the cut point (distance =0.5 ) that yields eight clusters, each merged into a single composite dimension (named on the right). Full names and definitions of each cognitive capability are given in Table A1 ; acronyms and definitions of the resulting composite dimensions are in Table A3 .
> 0
% of active items at level
Dimension
(%)
1
2
3
4
5
IIC
100
23
61
15
1
0
L
100
15
82
3
0
0
SM
99
19
64
9
7
0
APaS
94
54
35
11
0
0
IR
92
50
43
7
0
0
Table 2: Demand-level distribution across the unified battery dimensions.
#
Acronym
Job domain
1
WL
Warehouse or logistics
2
MMR
Manufacture, maintenance, or repair
3
NDP
Numerical, data, or programming
4
AOP
Administration, organisational, or planning
5
CMH
Customer service, marketing, or HR
6
HSC
Hospitality, sales, or client care
Table 3: Job domains
Dimension
τ=0
0.25
0.5
1
2
IIC (100%)
0.61
0.63
0.73
0.83
0.90
L (100%)
0.67
0.81
0.84
0.90
0.86
SM (99%)
0.73
0.84
0.90
0.92
0.91
APS (94%)
0.86
0.87
0.86
0.86
0.91
IR (92%)
0.89
0.85
0.80
0.76
0.72
EM (50%)
0.82
0.83
0.78
0.80
0.79
Table 4: Parameter-recovery r per battery dimension across the soft-min temperature τ .
Free α
Shared α
Fixed α
Recovery shape r
0.92
0.92
0.92
Recovery level r
0.12
0.98
0.98
Effective dims
2.7
6.7
6.7
Within-agent ∣r∣
0.50
0.13
0.13
Worst-dim contraction
0.71
0.81
0.82
Suitability ρ
0.14
0.91
0.91
Table 5: Recovery of capability level and shape, within-agent posterior collinearity, and downstream suitability recovery, under alternative intercept treatments ( τ=1 ).
Figure 3: Capability profiles by model family. Posterior-mean log-capability ( ck=logθk ) across the eight clustered battery dimensions for six LLMs, grouped by developer: Google (Gemini 2.5 Flash, Gemini 3 Flash, Gemini 3.1 Pro; left) and OpenAI (GPT-4o-mini, GPT-5-nano, o4-mini; right). These profiles were produced using soft-min pooling ( τ=1 ) and a shared intercept across systems. Axes: SC = Social Cognition, IR = Instrumental Reasoning, L = Language, SM = Semantic Memory, IIC = Information Integration & Control, APS = Action Planning & Simulation, EM = Episodic Memory, OP = Object Permanence. Spokes show posterior means only; per-dimension uncertainty (95% HDIs) is given in the forest plots (Figure C2 ).
Accuracy
Coverage
cˉ
SD
max∣r∣
SM
59.0%
99%
5.59
0.37
0.27
SC
65.2%
42%
4.08
0.42
0.23
L
58.8%
100%
4.02
0.38
0.40
IIC
58.8%
100%
3.70
0.38
0.33
EM
56.0%
50%
3.23
0.39
0.31
APS
57.3%
94%
1.99
0.19
0.61
Table 6: Benchmark and inference summaries for each capability dimension across the six systems.
Figure 4: Capability importance by work activity across the full sample ( N=410 ). Each cell shows the frequency-adjusted importance of a cognitive capability (columns) for a work activity (rows): the mean number of points allocated from a respondent’s 100-point budget for that activity, averaged across all respondents who performed it, with unselected capabilities treated as zeros. Scores therefore reflect both how frequently a capability is selected and how many points it receives when selected. Rows sum to approximately 93 (from an original 100) across the 16 capabilities retained after the inter-rater reliability screen (Table A2 ). Full names and definitions are given in Table A1 for capabilities, and in Table B1 for work activities.
Figure 5: Task suitability of the six AI systems across the 18 work activities on the log-capability ( c=logθ ) scale (points: posterior means; bars: 95% credible intervals). Suitability is the importance-weighted power mean of ratio-scale capabilities ( θ=ec ), reported as logS (the weighted mean of c at p=0 ). Estimates use the full sample ( n=410 , all domains), with sharpness s=1 , no column normalisation, and empirical weight uncertainty ( κt≈nt ).
Figure 6: Task importance (x; mean rank-derived rating, 1–5) versus mean AI suitability (y; log-capability scale) for Company X ( N=35 ). Suitability is computed as the equally weighted geometric mean across the six profiled AI systems (equivalently, the arithmetic mean on the log-capability scale). Dashed lines show the medians, defining the development-opportunity (top-right), capability-gap (bottom-right), limited-impact (top-left), and low-priority (bottom-left) quadrants. Labels are shown only where a task’s quadrant placement is confident (posterior probability ≥0.75 ); unlabelled points have credible intervals that straddle the suitability median. Data manipulation is also labelled despite straddling the median because of its conspicuous position.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Capability
Definition
Memory Systems
Episodic Memory
EM
Remembering previous events
Semantic Memory
SM
Remembering facts and information
Procedural Memory
ProcM
Remembering how to perform learned tasks or skills
Prospective Memory
ProsM
Remembering to do what you had planned to do
Executive Control
Appendix
Table A1: Core cognitive capabilities
Figure A1: A snippet from the Theory of Mind (ToM) rubric used to annotate benchmark items. The rubric defines how to categorise tasks from level 0 (no ToM capability required) to level 5 (very high ToM capability required), and concrete task examples are provided to help ground the demand levels. The full rubrics for all 18 capabilities are included in the project repository .
Capability
Acronym
ρ
κw
MAE
% ±1
Theory of Mind
ToM
0.81
0.68
0.45
88
Emotion Perception and Empathy
EPaE
0.72
0.72
0.16
97
Spatial Reasoning and Navigation
SRaN
0.72
0.71
0.53
89
Causal Reasoning
CR
0.67
0.59
0.53
94
Procedural Memory
ProcM
0.65
0.65
0.64
82
Object Permanence
OP
0.54
0.51
0.49
85
Appendix
Table A2: Inter-rater reliability of demand annotations between GPT-4o and Gemini 3
Capability
Definition
Constituent abilities
Episodic Memory
EM
Remembering previous events
—
Semantic Memory
SM
Remembering facts and information
—
Object Permanence
OP
Realising that people and objects continue to exist and have impact even if you cannot see them
—
Language
L
Using language to communicate information between you and another person
—
Social Cognition
SC
Understanding and reasoning about emotions, beliefs, desires and social cues in others
Emotion Perception & Empathy (EPaE): Considering and connecting with the feelings and emotions of others; Theory of Mind (ToM): Reasoning about the goals, beliefs and desires of others
Instrumental Reasoning
IR
Reasoning about how objects, actions, and events can bring about desired outcomes
Causal Reasoning (CR): Inferring how outcomes arise from prior actions or events; Functional Perception (FP): Recognising the appropriate roles that objects, people, or information can play in achieving a goal
Appendix
Table A3: Clustered capability dimensions
Work activity
Description
Researching
Learning about latest developments, researching a product or client
Checking
Inspecting products or documents, interviewing staff or reviewing their performance, ensuring compliance with regulations or guidelines
Decision making
Deciding project goals, choosing between different plans or options
Problem solving
Finding ways to overcome roadblocks, fix issues, or resolve conflict
Creative thinking
Coming up with new ideas for products or marketing, innovating
Long-term planning
Considering the overall goals for a project, setting objectives, strategising
Appendix
Table B1: Work activities
Figure B1: The task-capability weighting process. Left: Participants select the five capabilities they consider most essential for performing the target work activity, framed as equipping a robot helper for that role; hovering over any activity reveals a tooltip with concrete examples. Right: Participants distribute 100 ability points across their five selected capabilities, tuning the robot to the configuration they judge optimal – yielding a weighted profile of the most valuable capabilities for that activity. Note : capabilities and examples shown here are illustrative only – the actual questionnaire used the capabilities in Table A1
Companies
Online
Total
Domain
Pre-QC
Post-QC
Pre-QC
Post-QC
Pre-QC
Post-QC
Warehouse or logistics (WL)
2
1
47
36
49
37
Manufacture, maintenance, or repair (MMR)
0
0
48
33
48
33
Numerical, data, or programming (NDP)
18
12
59
50
77
62
Administration, organisational, or planning (AOP)
46
37
125
102
171
139
Customer service, marketing or HR (CMH)
38
24
67
55
105
79
Appendix
Table B2: Participant questionnaire sample.
Group
n
Age (mean ± sd)
% female
Median education
Yrs in role
Yrs in field
AI positivity
Overall
410
40.3±10.7
49.8
Degree
5.7
11.8
57.6
Companies
85
37.7±12.0
65.9
Degree
4.2
8.4
62.1
Online
325
40.9±10.3
45.5
Degree
6.1
12.7
56.4
Appendix
Table B3: Sample demographics, overall and by source.
Cosine
Pearson
Reliability
Reliability
Noise
Disattenuated
[95% CI]
(companies)
(online)
ceiling
r
Per capability
0.846
0.620 [0.508, 0.732]
0.497
0.733
0.603
1.000
Per work activity
0.905
0.876 [0.830, 0.923]
0.788
0.935
0.858
1.000
Appendix
Table B4: Companies vs online agreement: sample validation.
Capabilities
Work activities
Capability
Cosine
Pearson
Work activity
Cosine
Pearson
EM
0.893
0.599
Researching
0.982
0.964
SM
0.875
0.483
Checking
0.889
0.779
ProcM
0.874
0.689
Decision making
0.953
0.805
WM
0.916
0.548
Problem solving
0.950
0.816
CF
0.894
0.383
Creative thinking
0.910
0.694
Appendix
Table B5: Companies vs online agreement per capability and per work activity.
Figure B2: Participant demographic histograms for the full sample after exclusions ( N=410 ).
Figure C1: Fully compensatory capability recovery across dimensions. Each panel plots posterior-mean estimates c^k ( ±1 SD) against ground-truth capability levels ck across synthetic agents ( N=20 ), under a fully compensatory model ( τ=0 ), with the recovery correlation r reported per dimension. Contraction (contr.), a measure of how identifiable each dimension is, is also reported; dimensions with contraction <0.5 are highlighted in red.
Agent
SC
IR
L
SM
IIC
APaS
EM
OP
A1
3.10
2.89
3.51
3.08
2.57
3.29
4.04
3.76
A2
2.44
1.99
2.50
3.03
1.14
2.82
2.00
2.41
A3
2.56
2.75
3.33
3.83
2.90
4.09
2.47
3.28
A4
3.72
3.08
2.41
2.26
2.63
3.18
2.19
2.83
A5
2.87
3.43
3.17
3.28
2.48
2.90
3.63
4.19
A6
1.99
4.21
4.08
3.63
3.21
2.75
4.17
4.57
Appendix
Table C1: Synthetic agent capability profiles used in the recovery analysis
Dimension (coverage)
τ=0
0.25
0.5
1
2
Information Integration & Control (100%)
0.39
0.55
0.68
0.77
0.80
Language (100%)
0.45
0.57
0.69
0.75
0.72
Semantic Memory (99%)
0.45
0.71
0.81
0.84
0.84
Action Planning & Sim. (94%)
0.75
0.74
0.73
0.72
0.71
Object Permanence (36%)
0.80
0.77
0.74
0.71
0.68
Social Cognition (42%)
0.82
0.81
0.80
0.78
0.76
Appendix
Table C2: Posterior contraction per battery dimension across the soft-min temperature τ .
Dimension (coverage)
τ=0
0.25
0.5
1
2
Information Integration & Control (100%)
0.42
0.42
0.40
0.33
0.26
Language (100%)
0.51
0.43
0.38
0.30
0.34
Semantic Memory (99%)
0.45
0.39
0.32
0.29
0.30
Action Planning & Sim. (94%)
0.29
0.29
0.32
0.33
0.25
Instrumental Reasoning (92%)
0.27
0.31
0.35
0.39
0.39
Episodic Memory (50%)
0.34
0.33
0.39
0.34
0.34
Appendix
Table C3: Log-scale recovery error per battery dimension across the soft-min temperature τ .
Free intercept
Shared intercept
Agent
Eff. dims
∣r∣
max∣r∣
cond(R)
worst contr.
Eff. dims
∣r∣
max∣r∣
cond(R)
worst contr.
GPT-4o-mini
3.07
0.42
0.95
98
0.59
5.68
0.16
0.75
12
0.64
o4-mini
2.58
0.50
0.94
180
0.62
5.02
0.22
0.72
23
0.66
Gemini 3 Flash
2.66
0.48
0.98
229
0.60
4.84
0.22
0.86
25
0.70
Gemini 3.1 Pro
3.06
0.42
0.98
224
0.56
5.15
0.20
0.88
25
0.72
GPT-5-nano
2.32
0.53
0.96
214
0.57
4.54
0.24
0.76
24
0.59
Appendix
Table C4: Per-agent within-agent posterior collinearity of the eight log-capabilities ck , with free and shared intercepts.
System
Acc.
N
cˉ
SD
R^max
ESSmin
Gemini 3.1 Pro
73.3%
19,535
3.70
0.30
1.013
363
Gemini 3 Flash
68.4%
19,535
3.38
0.29
1.013
376
o4-mini
61.8%
19,535
2.88
0.29
1.011
377
GPT-4o-mini
48.7%
19,535
2.79
0.32
1.013
362
GPT-5-nano
56.0%
19,535
2.75
0.28
1.013
382
Gemini 2.5 Flash
44.8%
19,535
2.58
0.34
1.013
348
Appendix
Table C5: Per-system summary, aggregated across the eight capability dimensions.
System
IIC
L
SM
APS
IR
EM
SC
OP
Gemini 3.1 Pro
4.90 (0.41)
5.03 (0.41)
6.25 (0.38)
4.25 (0.34)
1.44 (0.12)
2.67 (0.18)
4.79 (0.42)
0.27 (0.13)
Gemini 3 Flash
5.06 (0.44)
4.96 (0.43)
6.12 (0.37)
2.37 (0.15)
1.32 (0.13)
2.93 (0.26)
4.18 (0.40)
0.11 (0.13)
o4-mini
2.92 (0.32)
2.90 (0.27)
6.34 (0.40)
1.63 (0.15)
1.71 (0.15)
3.23 (0.42)
4.42 (0.47)
-0.10 (0.13)
GPT-4o-mini
2.21 (0.21)
4.93 (0.48)
5.73 (0.41)
0.03 (0.12)
2.05 (0.25)
2.98 (0.48)
4.18 (0.48)
0.19 (0.14)
GPT-5-nano
4.19 (0.51)
2.22 (0.18)
6.17 (0.41)
1.34 (0.14)
1.30 (0.15)
4.51 (0.50)
2.59 (0.25)
-0.34 (0.13)
Gemini 2.5 Flash
2.91 (0.39)
4.07 (0.52)
2.95 (0.23)
2.31 (0.25)
-0.51 (0.12)
3.06 (0.47)
4.32 (0.50)
1.58 (0.24)
Appendix
Table C6: Posterior capability estimates for every system × capability.
Figure C2: Per-capability posteriors by system. Each panel is a forest plot of the eight clustered battery dimensions for a single AI system. Dots are posterior means of the log-capability ck=logθk and bars are 95% highest-density intervals, using soft-min pooling ( τ=1 ) and a shared intercept across systems. The dashed vertical line marks c=0 ( θ=1 ). The axis labels for dimensions display coverage (the share of battery items demanding it) in parentheses. Axes: SC = Social Cognition, IR = Instrumental Reasoning, L = Language, SM = Semantic Memory, IIC = Information Integration & Control, APS = Action Planning & Simulation, EM = Episodic Memory, OP = Object Permanence.
Figure C3: Capability importance per work activity, by occupational domain. Panels give the capability-importance matrix (scored as in Figure 4 ) estimated separately within each of the six occupational domains: Warehouse and Logistics (WL), Manufacture, Maintenance, or Repair (MMR), Numerical, Data, or Programming (NDP); Administration, Organisational, or Planning (AOP); Customer Service, Marketing, or HR (CMH); Hospitality, Sales, or Client Care (HSC).
Figure C4: Work-activity importance by occupational domain. Each panel shows one domain ( n = respondents in that domain). Dark dots give the domain’s frequency-adjusted importance for each activity – the mean rank score across all respondents in the domain, scoring 5 points for a respondent’s top-ranked activity down to 1 for the fifth and 0 for the unranked, so that both how often an activity is chosen and how highly it is ranked contribute. Grey dots give the full-sample reference (the equal-weight average across the six domains), identical across panels. Activities are ordered top-to-bottom by that overall reference, and the connecting line marks each domain’s departure from it. Domains: Warehouse and Logistics (WL), Manufacture, Maintenance, or Repair (MMR), Numerical, Data, or Programming (NDP); Administration, Organisational, or Planning (AOP); Customer Service, Marketing, or HR (CMH); Hospitality, Sales, or Client Care (HSC).
Figure C5: Weekly hours per work activity, by occupational domain. Each panel shows one domain ( n = respondents in that domain). Dark dots give the mean hours per week spent on each activity among respondents who perform it ; grey dots give the full-sample reference (the equal-weight average across the six domains), identical across panels. Hours are conditional means among performers (no zero-imputation), so activities not rated in a domain are omitted; activities are ordered by the overall reference. Domains: Warehouse and Logistics (WL), Manufacture, Maintenance, or Repair (MMR), Numerical, Data, or Programming (NDP); Administration, Organisational, or Planning (AOP); Customer Service, Marketing, or HR (CMH); Hospitality, Sales, or Client Care (HSC).
Task
Winner stable
Unique rankings
Invariant pairs
Separated pairs
κ→∞
κt≈nt
Researching
Yes
4
12/15
9/15
6/15
Checking
Yes
5
12/15
9/15
6/15
Decision making
Yes
5
12/15
10/15
6/15
Problem solving
Yes
4
13/15
10/15
8/15
Creative thinking
Yes
3
13/15
9/15
6/15
Appendix
Table C7: Robustness of suitability rankings across compensatory settings.
p
s
−2
−1
−0.5
0
0.5
1
2
1
0.80
0.81
0.82
1.00
0.88
0.85
0.65
2
0.82
0.79
0.79
0.87
0.89
0.81
0.64
4
0.73
0.75
0.75
0.76
0.80
0.79
0.61
8
0.72
0.72
0.70
0.70
0.71
0.65
0.61
Appendix
Table C8: Ranking stability across suitability-policy settings.
Task
WL
MMR
NDP
AOP
CMH
HSC
All
Admin
3.41 [3.17, 3.64]
3.41 [3.19, 3.64]
3.07 [2.84, 3.30]
3.34 [3.11, 3.57]
3.18 [2.95, 3.40]
3.64 [3.41, 3.87]
3.47 [3.24, 3.71]
Researching
3.47 [3.23, 3.71]
3.08 [2.85, 3.30]
3.37 [3.13, 3.61]
3.22 [2.99, 3.44]
3.15 [2.91, 3.38]
3.16 [2.93, 3.39]
3.31 [3.07, 3.55]
Communicating
2.74 [2.51, 2.96]
2.91 [2.68, 3.13]
3.25 [3.03, 3.47]
3.25 [3.02, 3.48]
3.22 [2.99, 3.46]
3.32 [3.08, 3.54]
3.24 [3.01, 3.46]
Listening
3.27 [3.04, 3.51]
3.86 [3.59, 4.11]
3.29 [3.06, 3.52]
3.16 [2.92, 3.38]
3.28 [3.05, 3.50]
2.97 [2.73, 3.20]
3.23 [3.00, 3.46]
Building rapport
3.40 [3.17, 3.63]
3.69 [3.46, 3.92]
3.29 [3.06, 3.53]
3.33 [3.08, 3.56]
2.98 [2.74, 3.22]
3.40 [3.17, 3.64]
3.18 [2.93, 3.42]
Analysing data
3.72 [3.45, 4.00]
2.73 [2.48, 2.96]
3.05 [2.81, 3.28]
3.13 [2.88, 3.37]
2.94 [2.70, 3.18]
3.15 [2.92, 3.38]
3.15 [2.90, 3.39]
Appendix
Table C9: AI suitability per task and work domain
Task
WL
MMR
NDP
AOP
CMH
HSC
All
Researching
4.63 [4.38, 4.87]
4.14 [3.92, 4.37]
4.47 [4.23, 4.71]
4.45 [4.22, 4.67]
4.34 [4.11, 4.58]
4.39 [4.16, 4.62]
4.50 [4.26, 4.74]
Admin
4.14 [3.91, 4.37]
4.51 [4.29, 4.74]
3.94 [3.72, 4.17]
4.39 [4.16, 4.62]
3.94 [3.71, 4.16]
4.67 [4.44, 4.89]
4.41 [4.18, 4.65]
Analysing data
4.97 [4.70, 5.25]
3.98 [3.73, 4.22]
4.37 [4.14, 4.61]
4.33 [4.08, 4.57]
3.90 [3.66, 4.14]
4.42 [4.19, 4.65]
4.36 [4.12, 4.60]
Building rapport
3.72 [3.49, 3.95]
3.69 [3.46, 3.92]
4.32 [4.08, 4.55]
4.36 [4.11, 4.60]
4.34 [4.09, 4.57]
4.74 [4.50, 4.97]
4.33 [4.08, 4.57]
Listening
4.45 [4.22, 4.69]
5.36 [5.09, 5.62]
4.08 [3.85, 4.31]
4.23 [4.00, 4.46]
4.34 [4.11, 4.56]
3.95 [3.71, 4.19]
4.28 [4.05, 4.51]
Communicating
3.56 [3.34, 3.78]
3.98 [3.76, 4.21]
4.30 [4.08, 4.52]
4.31 [4.08, 4.54]
4.28 [4.05, 4.51]
4.29 [4.05, 4.51]
4.27 [4.04, 4.50]
Appendix
Table C10: AI deployment priority score by task and work domain
Figure C6: Task importance (x; mean rank-derived rating, 1–5) vs. AI suitability (y; log-capability scale), pooled across six models for the full participant sample ( N=410 ). Dashed lines are the medians, defining development opportunity (top-right), capability-gap (bottom-right), limited-impact (top-left), and low-priority (bottom-left) quadrants. Labels shown only where a task’s quadrant placement is confident (posterior probability ≥0.75 ); unlabelled points straddle the suitability median.
Figure C7: Task frequency (x; mean hours per week) vs. AI suitability (y; log-capability scale), pooled across six models for the full participant sample ( N=410 ). Dashed lines are the medians, defining development opportunity (top-right), capability-gap (bottom-right), limited-impact (top-left), and low-priority (bottom-left) quadrants. Labels shown only where a task’s quadrant placement is confident (posterior probability ≥0.75 ); unlabelled points straddle the suitability median.
Figure C8: Task frequency (x; mean hours per week) vs. AI suitability (y; log-capability scale) for Company X ( N=35 ), pooled across six models. Dashed lines are the medians, defining development opportunity (top-right), capability-gap (bottom-right), limited-impact (top-left), and low-priority (bottom-left) quadrants. Labels shown only where a task’s quadrant placement is confident (posterior probability ≥0.75 ); unlabelled points straddle the suitability median.
Figure C9: Suitability scores for six AI systems computed after interviewing a senior Customer Service employee within Company X. Suitability is the importance-weighted power mean of ratio-scale capabilities ( θ=ec ), reported as logS , and computed for the role overall and for three core duties within it: (A) informing about rights; (B) helping with applications; and (C) documenting the dialogues.
We work towards measuring both AI adoption and the capability of AI to perform discrete labor tasks across various occupations. To measure adoption, we develop an open-source economic index that uses publicly available user-LLM chat data and ONET tasks to replicate studies produced by frontier AI labs, finding that occupations in the finance, computer science, and arts sectors are those with the highest adoption rates. To measure capabilities, we build a system that generates benchmark scenarios grounded in ONET occupations, tasks, and model-context-protocol (MCP) servers. We test Kimi-k2.5 with an OpenAI agents SDK harness on scenarios across 9 occupations that appear frequently in our index, finding that AI correctly executes high-level workflows but often errs in the granular details (such as specific tool calls used).
Current benchmarks for occupational AI agents are scoped primarily by economic values, telling a replacement story. We introduce JobBench, which evaluates AI agents on the workflows that experts identify as high-priority for delegation, empowering humans based on their needs instead of replacing them with GDP value. JobBench covers 130 agentic tasks across 35 occupations. Each task is packaged as a workspace of heterogeneous reference files, requiring the agent to reason through the cluttered information streams of real professional work. Outputs are graded by a fact-anchored chain of rubrics, averaging 35.6 binary criteria per task. We evaluate 36 models; the strongest, Claude Opus~4.7 under Claude Code, reaches only 45.9 %. We hope JobBench shifts the community's target labour-market effect from replacement to enhancement: building agents that do what humans actually want delegated, not only what is most economically valuable.
Yuetai Li, Yichen Feng, Zhangchen Xu +21
University of Washington · 10Bake AI · University of California, Santa Barbara +10
Some claim that AI agents will free workers from the boring parts of their jobs, yet little is known about how workers themselves identify which tasks should be automated. Prior research focuses on occupations, overlooking that workers experience varying levels of meaning across tasks within the same role. We address this gap with a task-level analysis grounded in Graeber's theory of bullshit jobs. Using ratings from 202 workers on 171 workplace tasks, we (1) validate a five-item scale of perceived bullshitness, (2) show that perceived bullshitness strongly predicts desire for AI delegation, and (3) find that such tasks are also seen as requiring less human oversight. Together, these findings suggest that tasks perceived as bullshit are natural candidates for AI delegation, aligning worker preferences with perceived feasibility.