As Large Language Models reshape the global labor market, policymakers and workers need empirical data on which occupational skills may be most susceptible to automation. We present the Skill Automation Feasibility Index (SAFI), benchmarking four frontier LLMs -- LLaMA 3.3 70B, Mistral Large, Qwen 2.5 72B, and Gemini 2.5 Flash -- across 263 text-based tasks spanning all 35 skills in the U.S. Department of Labor's O*NET taxonomy (1,052 total model calls, 0% failure rate). Cross-referencing with real-world AI adoption data from the Anthropic Economic Index (756 occupations, 17,998 tasks), we propose an AI Impact Matrix -- an interpretive framework that positions skills along four quadrants: High Displacement Risk, Upskilling Required, AI-Augmented, and Lower Displacement Risk. Key findings: (1) Mathematics (SAFI: 73.2) and Programming (71.8) receive the highest automation feasibility scores; Active Listening (42.2) and Reading Comprehension (45.5) receive the lowest; (2) a "capability-demand inversion" where skills most demanded in AI-exposed jobs are those LLMs perform least well at in our benchmark; (3) 78.7% of observed AI interactions are augmentation, not automation; (4) all four models converge to similar skill profiles (3.6-point spread), suggesting that text-based automation feasibility may be more skill-dependent than model-dependent. SAFI measures LLM performance on text-based representations of skills, not full occupational execution. All data, code, and model responses are open-sourced.
Figures & tables
Figure 1: Average skill importance by category across all 1,016 O*NET occupations. Process and Content skills are universally important; Technical skills are more specialized.
Figure 2: Distribution of AI exposure across 756 occupations from the Anthropic Economic Index. Most occupations have minimal AI exposure; a long tail of 67 occupations exceeds 0.30.
Figure 3: The 25 most AI-exposed occupations. Computing, data, and customer-facing roles dominate.
Figure 4: SAFI ranking across all 35 O*NET skills. Mathematics and Programming receive the highest text-based automation feasibility scores. Active Listening and Reading Comprehension score lowest. Color indicates skill category; zone annotations indicate relative automation feasibility.
Category
SAFI
σ
Skills
Technical Skills
62.1
0.99
11
Systems Skills
60.0
0.67
3
Complex Problem Solving
59.2
0.53
1
Resource Management
59.2
0.54
4
Process Skills
57.9
1.05
4
Social Skills
57.3
0.95
6
Table 1: SAFI scores by skill category. Technical Skills score highest; Content Skills score lowest in text-based evaluation.
Figure 5: Complete SAFI heatmap: 35 skills × 4 models. Color strip on right indicates skill category. Note the gradient from green (lower SAFI, top) to red (higher SAFI, bottom).
Figure 6: SAFI scores by model across skill categories. All four models show similar profiles, with Technical Skills consistently highest and Content Skills lowest.
Figure 7: SAFI scores by difficulty level and model. All models show higher scores on harder tasks—a scoring artifact of longer, more structured responses rather than genuinely better problem-solving. Mistral Large leads at every difficulty level.
Figure 8: Correlation between skill importance and real-world AI exposure across 744 occupations. Programming (+0.455) is most concentrated in AI-exposed jobs; Operation and Control ( − 0.456) is most concentrated in occupations with low AI exposure. The split between cognitive and physical skills is notable.
Figure 9: SAFI score vs. real-world AI exposure correlation ( r=−0.196 , p=0.26 , n=35 ). The negative trend, while not statistically significant at α=0.05 , is consistent with a capability-demand inversion: skills most important in AI-exposed jobs tend to receive lower SAFI scores.
Figure 10: Automation vs. augmentation patterns from 3,364 task interactions. Nearly four in five AI interactions are collaborative, not replacement-oriented.
Figure 11: SAFI radar profiles for each model across seven skill categories. Mistral Large shows the most balanced capability; all four models converge on similar shapes despite diverse origins.
Figure 12: The AI Impact Matrix: SAFI score × real-world AI exposure correlation for all 35 O*NET skills. Each dot represents one skill; color indicates category. Programming sits in the High Displacement Risk quadrant; Content skills cluster in the AI-Augmented quadrant; physical Technical skills fall in the Upskilling Window.
Skill documents, structured natural-language instructions that guide Large Language Model (LLM) agents, are critical to modern agent frameworks, yet LLMs struggle to write skills that actually work. On SkillsBench, human-authored skills improve pass rates by 16.2 percentage points, while LLM-authored skills provide no measurable gain. We introduce SkillAxe, a fully unsupervised framework that enables LLMs to iteratively diagnose and refine their own skills. SkillAxe decomposes skill quality into four interpretable dimensions (quality impact, trigger precision, instruction compliance with fault attribution, and solution-path coverage), producing structured improvement briefs that require no ground-truth labels, test suites, or environment rewards. On SkillsBench, SkillAxe improves pass rates by 28% relative over unimproved LLM skills and closes 47--67% of the gap to human-authored skills. We validate the approach as a continuous improvement engine in the wild on SpreadsheetBench, where a SkillAxe-built skill library learns from past agent trajectories and raises pass rate from 16.0% to 52.0% using only 22 skills.
This paper establishes the theoretical and practical foundations for using Large Language Models (LLMs) as measurement instruments for latent economic variables -- specifically variables that describe the cognitive content of occupational tasks at a level of granularity not achievable with existing survey instruments. I formalize four conditions under which LLM-generated scores constitute valid instruments: semantic exogeneity, construct relevance, monotonicity, and model invariance. I then apply this framework to the Augmented Human Capital Index (AHC_o), constructed from 18,796 O*NET task statements scored by Claude Haiku 4.5, and validated against six existing AI exposure indices. The index shows strong convergent validity (r = 0.85 with Eloundou GPT-gamma, r = 0.79 with Felten AIOE) and discriminant validity. Principal component analysis confirms that AI-related occupational measures span two distinct dimensions -- augmentation and substitution. Inter-rater reliability across two LLM models (n = 3,666 paired scores) yields Pearson r = 0.76 and Krippendorff's alpha = 0.71. Prompt sensitivity analysis across four alternative framings shows that task-level rankings are robust. Obviously Related Instrumental Variables (ORIV) estimation recovers coefficients 25% larger than OLS, consistent with classical measurement error attenuation. The methodology generalizes beyond labor economics to any domain where semantic content must be quantified at scale.
Cristian Espinal Maya
Department of Economics, Universidad EAFIT, Medellín, Colombia.
We work towards measuring both AI adoption and the capability of AI to perform discrete labor tasks across various occupations. To measure adoption, we develop an open-source economic index that uses publicly available user-LLM chat data and ONET tasks to replicate studies produced by frontier AI labs, finding that occupations in the finance, computer science, and arts sectors are those with the highest adoption rates. To measure capabilities, we build a system that generates benchmark scenarios grounded in ONET occupations, tasks, and model-context-protocol (MCP) servers. We test Kimi-k2.5 with an OpenAI agents SDK harness on scenarios across 9 occupations that appear frequently in our index, finding that AI correctly executes high-level workflows but often errs in the granular details (such as specific tool calls used).