Using profiles of cognitive capability to assess AI suitability for workplace tasks

Jonathan Prunty; Marko Tešić; Patrick Quinn; José Hernández-Orallo; Lucy Cheke

cs.AIAug 26, 2026

Using profiles of cognitive capability to assess AI suitability for workplace tasks

Authors: Jonathan Prunty, Marko Tešić, Patrick Quinn, José Hernández-Orallo, Lucy Cheke

Organizations: University of Cambridge · Department for Science, Innovation and Technology · Universitat Politècnica de València

Abstract

Organisations deploying AI face a scoping problem: which tasks can be automated, which should remain with humans, and which are best shared between the two. Aggregate benchmark scores provide little insight into where systems will succeed or fail in practice, while human judgements of model capabilities quickly become outdated. We introduce a pipeline that profiles agents and tasks using a shared set of core cognitive capabilities. Cognitive capability profiling infers an agent's capabilities from performance on a benchmark battery annotated for the cognitive demands of each item. Task requirements weighting elicits from domain experts the relative importance of these same capabilities for their work. As both use a common set of cognitive dimensions, they can be updated independently as models and roles change, and combined to estimate AI suitability at the level of a domain, organisation, role, or individual duty. We validate capability recovery on synthetic agents, profile six AI systems, and elicit task requirements from 410 employees across six occupational domains. AI systems differed more across cognitive dimensions than across model families, while workplace activities converged on a shared cognitive core. The resulting scores provide a comparative scoping tool for identifying promising candidates for piloting and areas where current systems are unlikely to be well suited. We discuss extending the framework to profile human workers alongside AI systems, moving from AI suitability towards human-machine task allocation.

Figures & tables

Figure 1: The pipeline for assessing task suitability involves: (1) generating cognitive capability profiles for a selection of agents, (2) weighting the cognitive requirements of target workplace tasks by their relative importance, and (3) mapping capability profiles to relative importance matrices. For simplicity, we have included two candidate AI agents, but commensurable capability profiles can also in principle be generated for humans and for human + AI pairings.

Benchmark	Items
Abstract Narrative Understanding [ 75 ]	303
AGIEval [ 76 ]	1896
BigBenchHard [ 77 ]	1265
BigToM [ 78 ]	686
Cause and Effect [ 75 ]	51
CoQA [ 75 ]	210

Table 1: Benchmarks

Figure 2: Capability clustering. Pearson correlations between the demand profiles of the 16 capabilities retained following reliability screening (Table A2 ), revealing dimensions that placed similar demands on the same items. Correlations were computed from annotations z-scored within each rater and capability, controlling for systematic differences between the two raters. The left-hand dendrogram shows hierarchical agglomerative clustering on the distance $1-r$ between dimensions. The red dashed line marks the cut point (distance $=0.5$ ) that yields eight clusters, each merged into a single composite dimension (named on the right). Full names and definitions of each cognitive capability are given in Table A1 ; acronyms and definitions of the resulting composite dimensions are in Table A3 .

	> 0	% of active items at level
Dimension	(%)	1	2	3	4	5
IIC	100	23	61	15	1	0
L	100	15	82	3	0	0
SM	99	19	64	9	7	0
APaS	94	54	35	11	0	0
IR	92	50	43	7	0	0

Table 2: Demand-level distribution across the unified battery dimensions.

#	Acronym	Job domain
1	WL	Warehouse or logistics
2	MMR	Manufacture, maintenance, or repair
3	NDP	Numerical, data, or programming
4	AOP	Administration, organisational, or planning
5	CMH	Customer service, marketing, or HR
6	HSC	Hospitality, sales, or client care

Table 3: Job domains

Dimension	$\tau{=}0$	0.25	0.5	1	2
IIC (100%)	0.61	0.63	0.73	0.83	0.90
L (100%)	0.67	0.81	0.84	0.90	0.86
SM (99%)	0.73	0.84	0.90	0.92	0.91
APS (94%)	0.86	0.87	0.86	0.86	0.91
IR (92%)	0.89	0.85	0.80	0.76	0.72
EM (50%)	0.82	0.83	0.78	0.80	0.79

Table 4: Parameter-recovery

r

per battery dimension across the soft-min temperature

\tau

	Free $\alpha$	Shared $\alpha$	Fixed $\alpha$
Recovery shape $r$	0.92	0.92	0.92
Recovery level $r$	0.12	0.98	0.98
Effective dims	2.7	6.7	6.7
Within-agent $\|r\|$	0.50	0.13	0.13
Worst-dim contraction	0.71	0.81	0.82
Suitability $\rho$	0.14	0.91	0.91

Table 5: Recovery of capability level and shape, within-agent posterior collinearity, and downstream suitability recovery, under alternative intercept treatments (

\tau=1

Figure 3: Capability profiles by model family. Posterior-mean log-capability ( $c_{k}=\log\theta_{k}$ ) across the eight clustered battery dimensions for six LLMs, grouped by developer: Google (Gemini 2.5 Flash, Gemini 3 Flash, Gemini 3.1 Pro; left) and OpenAI (GPT-4o-mini, GPT-5-nano, o4-mini; right). These profiles were produced using soft-min pooling ( $\tau=1$ ) and a shared intercept across systems. Axes: SC = Social Cognition, IR = Instrumental Reasoning, L = Language, SM = Semantic Memory, IIC = Information Integration & Control, APS = Action Planning & Simulation, EM = Episodic Memory, OP = Object Permanence. Spokes show posterior means only; per-dimension uncertainty (95% HDIs) is given in the forest plots (Figure C2 ).

	Accuracy	Coverage	$\bar{c}$	$SD$	$\max\|r\|$
SM	59.0%	99%	5.59	0.37	0.27
SC	65.2%	42%	4.08	0.42	0.23
L	58.8%	100%	4.02	0.38	0.40
IIC	58.8%	100%	3.70	0.38	0.33
EM	56.0%	50%	3.23	0.39	0.31
APS	57.3%	94%	1.99	0.19	0.61

Table 6: Benchmark and inference summaries for each capability dimension across the six systems.

Figure 4: Capability importance by work activity across the full sample (

N=410

). Each cell shows the frequency-adjusted importance of a cognitive capability (columns) for a work activity (rows): the mean number of points allocated from a respondent’s 100-point budget for that activity, averaged across all respondents who performed it, with unselected capabilities treated as zeros. Scores therefore reflect both how frequently a capability is selected and how many points it receives when selected. Rows sum to approximately

93

(from an original 100) across the 16 capabilities retained after the inter-rater reliability screen (Table A2 ). Full names and definitions are given in Table A1 for capabilities, and in Table B1 for work activities.

Figure 5: Task suitability of the six AI systems across the 18 work activities on the log-capability ( $c=\log\theta$ ) scale (points: posterior means; bars: $95\%$ credible intervals). Suitability is the importance-weighted power mean of ratio-scale capabilities ( $\theta=e^{c}$ ), reported as $\log S$ (the weighted mean of $c$ at $p=0$ ). Estimates use the full sample ( $n=410$ , all domains), with sharpness $s=1$ , no column normalisation, and empirical weight uncertainty ( $\kappa_{t}\approx n_{t}$ ).

Figure 6: Task importance (x; mean rank-derived rating, 1–5) versus mean AI suitability (y; log-capability scale) for Company X ( $N=35$ ). Suitability is computed as the equally weighted geometric mean across the six profiled AI systems (equivalently, the arithmetic mean on the log-capability scale). Dashed lines show the medians, defining the development-opportunity (top-right), capability-gap (bottom-right), limited-impact (top-left), and low-priority (bottom-left) quadrants. Labels are shown only where a task’s quadrant placement is confident (posterior probability $\geq 0.75$ ); unlabelled points have credible intervals that straddle the suitability median. Data manipulation is also labelled despite straddling the median because of its conspicuous position.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Capability		Definition
Memory Systems
Episodic Memory	EM	Remembering previous events
Semantic Memory	SM	Remembering facts and information
Procedural Memory	ProcM	Remembering how to perform learned tasks or skills
Prospective Memory	ProsM	Remembering to do what you had planned to do
Executive Control

Appendix

Table A1: Core cognitive capabilities

Figure A1: A snippet from the Theory of Mind (ToM) rubric used to annotate benchmark items. The rubric defines how to categorise tasks from level 0 (no ToM capability required) to level 5 (very high ToM capability required), and concrete task examples are provided to help ground the demand levels. The full rubrics for all 18 capabilities are included in the project repository .

Capability	Acronym	$\rho$	$\kappa_{w}$	MAE	% ±1
Theory of Mind	ToM	$0.81$	$0.68$	$0.45$	$88$
Emotion Perception and Empathy	EPaE	$0.72$	$0.72$	$0.16$	$97$
Spatial Reasoning and Navigation	SRaN	$0.72$	$0.71$	$0.53$	$89$
Causal Reasoning	CR	$0.67$	$0.59$	$0.53$	$94$
Procedural Memory	ProcM	$0.65$	$0.65$	$0.64$	$82$
Object Permanence	OP	$0.54$	$0.51$	$0.49$	$85$

Appendix

Table A2: Inter-rater reliability of demand annotations between GPT-4o and Gemini 3

Capability		Definition	Constituent abilities
Episodic Memory	EM	Remembering previous events	—
Semantic Memory	SM	Remembering facts and information	—
Object Permanence	OP	Realising that people and objects continue to exist and have impact even if you cannot see them	—
Language	L	Using language to communicate information between you and another person	—
Social Cognition	SC	Understanding and reasoning about emotions, beliefs, desires and social cues in others	Emotion Perception & Empathy (EPaE): Considering and connecting with the feelings and emotions of others; Theory of Mind (ToM): Reasoning about the goals, beliefs and desires of others
Instrumental Reasoning	IR	Reasoning about how objects, actions, and events can bring about desired outcomes	Causal Reasoning (CR): Inferring how outcomes arise from prior actions or events; Functional Perception (FP): Recognising the appropriate roles that objects, people, or information can play in achieving a goal

Appendix

Table A3: Clustered capability dimensions

Work activity	Description
Researching	Learning about latest developments, researching a product or client
Checking	Inspecting products or documents, interviewing staff or reviewing their performance, ensuring compliance with regulations or guidelines
Decision making	Deciding project goals, choosing between different plans or options
Problem solving	Finding ways to overcome roadblocks, fix issues, or resolve conflict
Creative thinking	Coming up with new ideas for products or marketing, innovating
Long-term planning	Considering the overall goals for a project, setting objectives, strategising

Appendix

Table B1: Work activities

Figure B1: The task-capability weighting process. Left: Participants select the five capabilities they consider most essential for performing the target work activity, framed as equipping a robot helper for that role; hovering over any activity reveals a tooltip with concrete examples. Right: Participants distribute 100 ability points across their five selected capabilities, tuning the robot to the configuration they judge optimal – yielding a weighted profile of the most valuable capabilities for that activity. Note : capabilities and examples shown here are illustrative only – the actual questionnaire used the capabilities in Table A1

	Companies		Online		Total
Domain	Pre-QC	Post-QC	Pre-QC	Post-QC	Pre-QC	Post-QC
Warehouse or logistics (WL)	2	1	47	36	49	37
Manufacture, maintenance, or repair (MMR)	0	0	48	33	48	33
Numerical, data, or programming (NDP)	18	12	59	50	77	62
Administration, organisational, or planning (AOP)	46	37	125	102	171	139
Customer service, marketing or HR (CMH)	38	24	67	55	105	79

Appendix

Table B2: Participant questionnaire sample.

Group	$n$	Age (mean $\pm$ sd)	% female	Median education	Yrs in role	Yrs in field	AI positivity
Overall	410	$40.3\pm 10.7$	49.8	Degree	5.7	11.8	57.6
Companies	85	$37.7\pm 12.0$	65.9	Degree	4.2	8.4	62.1
Online	325	$40.9\pm 10.3$	45.5	Degree	6.1	12.7	56.4

Appendix

Table B3: Sample demographics, overall and by source.

	Cosine	Pearson	Reliability	Reliability	Noise	Disattenuated
		[95% CI]	(companies)	(online)	ceiling	$r$
Per capability	0.846	0.620 [0.508, 0.732]	0.497	0.733	0.603	1.000
Per work activity	0.905	0.876 [0.830, 0.923]	0.788	0.935	0.858	1.000

Appendix

Table B4: Companies vs online agreement: sample validation.

Capabilities			Work activities
Capability	Cosine	Pearson	Work activity	Cosine	Pearson
EM	0.893	0.599	Researching	0.982	0.964
SM	0.875	0.483	Checking	0.889	0.779
ProcM	0.874	0.689	Decision making	0.953	0.805
WM	0.916	0.548	Problem solving	0.950	0.816
CF	0.894	0.383	Creative thinking	0.910	0.694

Appendix

Table B5: Companies vs online agreement per capability and per work activity.

Figure B2: Participant demographic histograms for the full sample after exclusions ( $N=410$ ).

Figure C1: Fully compensatory capability recovery across dimensions. Each panel plots posterior-mean estimates $\hat{c}_{k}$ ( $\pm 1$ SD) against ground-truth capability levels $c_{k}$ across synthetic agents ( $N=20$ ), under a fully compensatory model ( $\tau=0$ ), with the recovery correlation $r$ reported per dimension. Contraction (contr.), a measure of how identifiable each dimension is, is also reported; dimensions with contraction $<0.5$ are highlighted in red.

Agent	SC	IR	L	SM	IIC	APaS	EM	OP
A1	3.10	2.89	3.51	3.08	2.57	3.29	4.04	3.76
A2	2.44	1.99	2.50	3.03	1.14	2.82	2.00	2.41
A3	2.56	2.75	3.33	3.83	2.90	4.09	2.47	3.28
A4	3.72	3.08	2.41	2.26	2.63	3.18	2.19	2.83
A5	2.87	3.43	3.17	3.28	2.48	2.90	3.63	4.19
A6	1.99	4.21	4.08	3.63	3.21	2.75	4.17	4.57

Appendix

Table C1: Synthetic agent capability profiles used in the recovery analysis

Dimension (coverage)	$\tau{=}0$	$0.25$	$0.5$	$1$	$2$
Information Integration & Control (100%)	0.39	0.55	0.68	0.77	0.80
Language (100%)	0.45	0.57	0.69	0.75	0.72
Semantic Memory (99%)	0.45	0.71	0.81	0.84	0.84
Action Planning & Sim. (94%)	0.75	0.74	0.73	0.72	0.71
Object Permanence (36%)	0.80	0.77	0.74	0.71	0.68
Social Cognition (42%)	0.82	0.81	0.80	0.78	0.76

Appendix

Table C2: Posterior contraction per battery dimension across the soft-min temperature

\tau

Dimension (coverage)	$\tau{=}0$	0.25	0.5	1	2
Information Integration & Control (100%)	0.42	0.42	0.40	0.33	0.26
Language (100%)	0.51	0.43	0.38	0.30	0.34
Semantic Memory (99%)	0.45	0.39	0.32	0.29	0.30
Action Planning & Sim. (94%)	0.29	0.29	0.32	0.33	0.25
Instrumental Reasoning (92%)	0.27	0.31	0.35	0.39	0.39
Episodic Memory (50%)	0.34	0.33	0.39	0.34	0.34

Appendix

Table C3: Log-scale recovery error per battery dimension across the soft-min temperature

\tau

	Free intercept					Shared intercept
Agent	Eff. dims	$\overline{\|r\|}$	$\max\|r\|$	$\mathrm{cond}(R)$	worst contr.	Eff. dims	$\overline{\|r\|}$	$\max\|r\|$	$\mathrm{cond}(R)$	worst contr.
GPT-4o-mini	3.07	0.42	0.95	98	0.59	5.68	0.16	0.75	12	0.64
o4-mini	2.58	0.50	0.94	180	0.62	5.02	0.22	0.72	23	0.66
Gemini 3 Flash	2.66	0.48	0.98	229	0.60	4.84	0.22	0.86	25	0.70
Gemini 3.1 Pro	3.06	0.42	0.98	224	0.56	5.15	0.20	0.88	25	0.72
GPT-5-nano	2.32	0.53	0.96	214	0.57	4.54	0.24	0.76	24	0.59

Appendix

Table C4: Per-agent within-agent posterior collinearity of the eight log-capabilities

c_{k}

, with free and shared intercepts.

System	Acc.	$N$	$\bar{c}$	$SD$	$\hat{R}_{\max}$	$\mathrm{ESS}_{\min}$
Gemini 3.1 Pro	73.3%	19,535	3.70	0.30	1.013	363
Gemini 3 Flash	68.4%	19,535	3.38	0.29	1.013	376
o4-mini	61.8%	19,535	2.88	0.29	1.011	377
GPT-4o-mini	48.7%	19,535	2.79	0.32	1.013	362
GPT-5-nano	56.0%	19,535	2.75	0.28	1.013	382
Gemini 2.5 Flash	44.8%	19,535	2.58	0.34	1.013	348

Appendix

Table C5: Per-system summary, aggregated across the eight capability dimensions.

System	IIC	L	SM	APS	IR	EM	SC	OP
Gemini 3.1 Pro	4.90 (0.41)	5.03 (0.41)	6.25 (0.38)	4.25 (0.34)	1.44 (0.12)	2.67 (0.18)	4.79 (0.42)	0.27 (0.13)
Gemini 3 Flash	5.06 (0.44)	4.96 (0.43)	6.12 (0.37)	2.37 (0.15)	1.32 (0.13)	2.93 (0.26)	4.18 (0.40)	0.11 (0.13)
o4-mini	2.92 (0.32)	2.90 (0.27)	6.34 (0.40)	1.63 (0.15)	1.71 (0.15)	3.23 (0.42)	4.42 (0.47)	-0.10 (0.13)
GPT-4o-mini	2.21 (0.21)	4.93 (0.48)	5.73 (0.41)	0.03 (0.12)	2.05 (0.25)	2.98 (0.48)	4.18 (0.48)	0.19 (0.14)
GPT-5-nano	4.19 (0.51)	2.22 (0.18)	6.17 (0.41)	1.34 (0.14)	1.30 (0.15)	4.51 (0.50)	2.59 (0.25)	-0.34 (0.13)
Gemini 2.5 Flash	2.91 (0.39)	4.07 (0.52)	2.95 (0.23)	2.31 (0.25)	-0.51 (0.12)	3.06 (0.47)	4.32 (0.50)	1.58 (0.24)

Appendix

Table C6: Posterior capability estimates for every system

\times

capability.

Figure C4: Work-activity importance by occupational domain. Each panel shows one domain ( $n$ = respondents in that domain). Dark dots give the domain’s frequency-adjusted importance for each activity – the mean rank score across all respondents in the domain, scoring 5 points for a respondent’s top-ranked activity down to 1 for the fifth and 0 for the unranked, so that both how often an activity is chosen and how highly it is ranked contribute. Grey dots give the full-sample reference (the equal-weight average across the six domains), identical across panels. Activities are ordered top-to-bottom by that overall reference, and the connecting line marks each domain’s departure from it. Domains: Warehouse and Logistics (WL), Manufacture, Maintenance, or Repair (MMR), Numerical, Data, or Programming (NDP); Administration, Organisational, or Planning (AOP); Customer Service, Marketing, or HR (CMH); Hospitality, Sales, or Client Care (HSC).

Figure C5: Weekly hours per work activity, by occupational domain. Each panel shows one domain ( $n$ = respondents in that domain). Dark dots give the mean hours per week spent on each activity among respondents who perform it ; grey dots give the full-sample reference (the equal-weight average across the six domains), identical across panels. Hours are conditional means among performers (no zero-imputation), so activities not rated in a domain are omitted; activities are ordered by the overall reference. Domains: Warehouse and Logistics (WL), Manufacture, Maintenance, or Repair (MMR), Numerical, Data, or Programming (NDP); Administration, Organisational, or Planning (AOP); Customer Service, Marketing, or HR (CMH); Hospitality, Sales, or Client Care (HSC).

Task	Winner stable	Unique rankings	Invariant pairs	Separated pairs
				$\kappa\!\to\!\infty$	$\kappa_{t}\!\approx\!n_{t}$
Researching	Yes	4	12/15	9/15	6/15
Checking	Yes	5	12/15	9/15	6/15
Decision making	Yes	5	12/15	10/15	6/15
Problem solving	Yes	4	13/15	10/15	8/15
Creative thinking	Yes	3	13/15	9/15	6/15

Appendix

Table C7: Robustness of suitability rankings across compensatory settings.

	$p$
$s$	$-2$	$-1$	$-0.5$	$0$	$0.5$	$1$	$2$
1	0.80	0.81	0.82	1.00	0.88	0.85	0.65
2	0.82	0.79	0.79	0.87	0.89	0.81	0.64
4	0.73	0.75	0.75	0.76	0.80	0.79	0.61
8	0.72	0.72	0.70	0.70	0.71	0.65	0.61

Appendix

Table C8: Ranking stability across suitability-policy settings.

Task	WL	MMR	NDP	AOP	CMH	HSC	All
Admin	3.41 [3.17, 3.64]	3.41 [3.19, 3.64]	3.07 [2.84, 3.30]	3.34 [3.11, 3.57]	3.18 [2.95, 3.40]	3.64 [3.41, 3.87]	3.47 [3.24, 3.71]
Researching	3.47 [3.23, 3.71]	3.08 [2.85, 3.30]	3.37 [3.13, 3.61]	3.22 [2.99, 3.44]	3.15 [2.91, 3.38]	3.16 [2.93, 3.39]	3.31 [3.07, 3.55]
Communicating	2.74 [2.51, 2.96]	2.91 [2.68, 3.13]	3.25 [3.03, 3.47]	3.25 [3.02, 3.48]	3.22 [2.99, 3.46]	3.32 [3.08, 3.54]	3.24 [3.01, 3.46]
Listening	3.27 [3.04, 3.51]	3.86 [3.59, 4.11]	3.29 [3.06, 3.52]	3.16 [2.92, 3.38]	3.28 [3.05, 3.50]	2.97 [2.73, 3.20]	3.23 [3.00, 3.46]
Building rapport	3.40 [3.17, 3.63]	3.69 [3.46, 3.92]	3.29 [3.06, 3.53]	3.33 [3.08, 3.56]	2.98 [2.74, 3.22]	3.40 [3.17, 3.64]	3.18 [2.93, 3.42]
Analysing data	3.72 [3.45, 4.00]	2.73 [2.48, 2.96]	3.05 [2.81, 3.28]	3.13 [2.88, 3.37]	2.94 [2.70, 3.18]	3.15 [2.92, 3.38]	3.15 [2.90, 3.39]

Appendix

Table C9: AI suitability per task and work domain

Task	WL	MMR	NDP	AOP	CMH	HSC	All
Researching	4.63 [4.38, 4.87]	4.14 [3.92, 4.37]	4.47 [4.23, 4.71]	4.45 [4.22, 4.67]	4.34 [4.11, 4.58]	4.39 [4.16, 4.62]	4.50 [4.26, 4.74]
Admin	4.14 [3.91, 4.37]	4.51 [4.29, 4.74]	3.94 [3.72, 4.17]	4.39 [4.16, 4.62]	3.94 [3.71, 4.16]	4.67 [4.44, 4.89]	4.41 [4.18, 4.65]
Analysing data	4.97 [4.70, 5.25]	3.98 [3.73, 4.22]	4.37 [4.14, 4.61]	4.33 [4.08, 4.57]	3.90 [3.66, 4.14]	4.42 [4.19, 4.65]	4.36 [4.12, 4.60]
Building rapport	3.72 [3.49, 3.95]	3.69 [3.46, 3.92]	4.32 [4.08, 4.55]	4.36 [4.11, 4.60]	4.34 [4.09, 4.57]	4.74 [4.50, 4.97]	4.33 [4.08, 4.57]
Listening	4.45 [4.22, 4.69]	5.36 [5.09, 5.62]	4.08 [3.85, 4.31]	4.23 [4.00, 4.46]	4.34 [4.11, 4.56]	3.95 [3.71, 4.19]	4.28 [4.05, 4.51]
Communicating	3.56 [3.34, 3.78]	3.98 [3.76, 4.21]	4.30 [4.08, 4.52]	4.31 [4.08, 4.54]	4.28 [4.05, 4.51]	4.29 [4.05, 4.51]	4.27 [4.04, 4.50]

Appendix

Table C10: AI deployment priority score by task and work domain

Figure C6: Task importance (x; mean rank-derived rating, 1–5) vs. AI suitability (y; log-capability scale), pooled across six models for the full participant sample ( $N=410$ ). Dashed lines are the medians, defining development opportunity (top-right), capability-gap (bottom-right), limited-impact (top-left), and low-priority (bottom-left) quadrants. Labels shown only where a task’s quadrant placement is confident (posterior probability $\geq 0.75$ ); unlabelled points straddle the suitability median.

Figure C7: Task frequency (x; mean hours per week) vs. AI suitability (y; log-capability scale), pooled across six models for the full participant sample ( $N=410$ ). Dashed lines are the medians, defining development opportunity (top-right), capability-gap (bottom-right), limited-impact (top-left), and low-priority (bottom-left) quadrants. Labels shown only where a task’s quadrant placement is confident (posterior probability $\geq 0.75$ ); unlabelled points straddle the suitability median.

Figure C8: Task frequency (x; mean hours per week) vs. AI suitability (y; log-capability scale) for Company X ( $N=35$ ), pooled across six models. Dashed lines are the medians, defining development opportunity (top-right), capability-gap (bottom-right), limited-impact (top-left), and low-priority (bottom-left) quadrants. Labels shown only where a task’s quadrant placement is confident (posterior probability $\geq 0.75$ ); unlabelled points straddle the suitability median.

Figure C9: Suitability scores for six AI systems computed after interviewing a senior Customer Service employee within Company X. Suitability is the importance-weighted power mean of ratio-scale capabilities ( $\theta=e^{c}$ ), reported as $\log S$ , and computed for the role overall and for three core duties within it: (A) informing about rights; (B) helping with applications; and (C) documenting the dialogues.

AI and Labor

AI and labor studies how artificial intelligence changes jobs, wages, skills, hiring, and the organization and quality of work.

See all 95 papers

The Open Source Economic Index of AI Adoption and Capability
May 23, 2026Seamus Somerstep, Aritra Guha, Divesh Srivastava +1AI Agent Evaluation AI Agent Benchmarks
University of Michigan · AT&T Chief Data Office · IFM-MBZUAI
JobBench: Aligning Agent Work With Human Will
May 25, 2026Yuetai Li, Yichen Feng, Zhangchen Xu +21Computer-Use Agent Benchmarks LLM Agent Evaluation
University of Washington · 10Bake AI · University of California, Santa Barbara +10
Will AI Agents Free Us From Meaningless Work? A Human-Centered Analysis
May 15, 2026Davide Ghia, Jaspreet Ranjit, Tania Cerquitelli +1Human-Centered AI Human Oversight of AI
Politecnico di Torino · Italy · University of Southern California +3

May 23, 2026J/K move · Enter open · S save

The Open Source Economic Index of AI Adoption and Capability

Seamus Somerstep, Aritra Guha, Divesh Srivastava, Yuekai Sun

University of Michigan · AT&T Chief Data Office · IFM-MBZUAI

AI Agent Evaluation AI Agent Benchmarks AI and Labor

Open paper AskPDF arXiv

We work towards measuring both AI adoption and the capability of AI to perform discrete labor tasks across various occupations. To measure adoption, we develop an open-source economic index that uses publicly available user-LLM chat data and ONET tasks to replicate studies produced by frontier AI labs, finding that occupations in the finance, computer science, and arts sectors are those with the highest adoption rates. To measure capabilities, we build a system that generates benchmark scenarios grounded in ONET occupations, tasks, and model-context-protocol (MCP) servers. We test Kimi-k2.5 with an OpenAI agents SDK harness on scenarios across 9 occupations that appear frequently in our index, finding that AI correctly executes high-level workflows but often errs in the granular details (such as specific tool calls used).

Abstract

AI and Labor

Explore similar work

The Open Source Economic Index of AI Adoption and Capability

JobBench: Aligning Agent Work With Human Will

Will AI Agents Free Us From Meaningless Work? A Human-Centered Analysis