Crashing Waves vs. Rising Tides: Findings on AI Automation from Thousands of Worker Evaluations of Labor Market Tasks

Matthias Mertens; Adam Kuzee; Brittany S. Harris; Harry Lyu; Wensu Li; Jonathan Rosenfeld; Meiri Anto; Martin Fleming

cs.AIApr 1, 2026

Crashing Waves vs. Rising Tides: Findings on AI Automation from Thousands of Worker Evaluations of Labor Market Tasks

Authors: Matthias Mertens, Adam Kuzee, Brittany S. Harris, Harry Lyu, Wensu Li, Jonathan Rosenfeld, Meiri Anto, Martin Fleming,

Organizations: MIT FutureTech

Abstract

We characterize AI automation as a continuum between crashing waves, in which capabilities jump abruptly across narrow task sets, and rising tides, in which capabilities improve continuously and broadly. Using evidence from more than 6,000 text-based, LLM-addressable tasks derived from the U.S. Department of Labor's O*NET taxonomy and over 60,000 evaluations by experienced workers, we find little evidence of crashing waves (contrary to existing views). Instead, rising tides are the primary form of AI progress. AI performance is high and improving rapidly across many tasks. In 2024-Q2, models completed text-based tasks that take humans about 1.5 hours to complete with roughly 60% success, rising above 70% by 2025-Q3. If recent trends in AI capability growth persist, frontier LLMs will be able to complete most text-based tasks at minimally sufficient quality with 88%-97% success by 2030.

Figures & tables

Panel of Figure 1 — Figure 1: Crashing Waves vs Rising Tides in AI Automation

Panel of Figure 2 — Figure 2: Survey Coverage and Task Duration Distribution

Score	Evaluation	Implication & interpretation
1–3	Not useful	The output requires substantial or complete rework. The LLM does not meaningfully substitute for human task performance.
4–6	Useful with edits	The output assists the worker but requires human revision. This represents task augmentation rather than autonomous task completion.
7	Minimally sufficient without edits	The output can be used as delivered and meets the minimum requirements of the task. This is our baseline measure of potential task automation.
8	Average-worker quality without edits	The output can be used as delivered and is comparable in quality to the work of an average human worker.
9	Superior quality without edits	The output can be used as delivered and exceeds the quality of an average human worker.

Table 1: LLM Response Quality Scores

Figure 3: Task Instance Automation by Required Task Completion Time

Panel of Table 2 — Table 2: Task Automatability And Task Duration: by Job Family

Panel of Figure 4 — Figure 4: Task Automation and Task Length: Model Size and Vintages

Panel of Figure 5 — Figure 5: Task Duration and Success Rate Thresholds over Time (Score $\geq$ 7)

Figure 6: Predicted AI Success Rates Over Time

Panel of Figure 7 — Figure 7: Task Filtering Process

	Mean	St.Dev.	P10	P50	P90
	(1)	(2)	(3)	(4)	(5)
Score	6.92	1.94	4	7	9
Manager Acceptance (0/1; 1 if score $\geq$ 7)	0.60	0.49	0	1	1
Time required to complete task instance (hours)	10.61	26.20	0.5	2.75	25
Observations: 60,845

Table 3: Summary Statistics

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 11 — Figure A.1 : Task Duration and Success Rate Thresholds over Time, Logistic Function vs. Complementary Log-Log.

Figure A.2 : Task Automation and Task Length: Different Specifications (Threshold Score $\geq 7$ ).

	Dependent variable: $\mathbbm{1}\{\text{acceptance}\geq 7\}$
	(1)	(2)	(3)
Intercept	0.759***	0.801***	0.636***
	(0.048)	(0.054)	(0.058)
$\log_{10}(\text{Time to Complete})$	-0.226***	-0.275***	-0.242***
	(0.020)	(0.023)	(0.025)
Large ( $>$ 100B)	0.535***		0.480***

Appendix

Table A.1 : Baseline Regressions with Size and Model Publication Time Effects (Acceptance

\geq

	Dependent variable: $\mathbbm{1}\{\text{acceptance}\geq 8\}$
	(1)	(2)	(3)
Intercept	0.333***	0.349***	0.201***
	(0.048)	(0.054)	(0.058)
$\log_{10}(\text{Time to Complete})$	-0.218***	-0.264***	-0.234***
	(0.020)	(0.023)	(0.025)
Large ( $>$ 100B)	0.475***		0.417***

Appendix

Table A.2 : Baseline Regressions with Size and Model Publication Time Effects (Acceptance

\geq

	Dependent variable: $\mathbbm{1}\{\text{acceptance}=9\}$
	(1)	(2)	(3)
Intercept	-0.727***	-0.779***	-0.903***
	(0.054)	(0.064)	(0.068)
$\log_{10}(\text{Time to Complete})$	-0.175***	-0.224***	-0.195***
	(0.023)	(0.028)	(0.030)
Large ( $>$ 100B)	0.406***		0.334***

Appendix

Table A.3 : Baseline Regressions with Size and Model Publication Time Effects (Acceptance

=9

)

	Dependent variable: $\mathbbm{1}\{\text{acceptance}\geq 7\}$
	Additive release date	Additive and interacted release date
	(1)	(2)
Intercept	0.438***	0.563***
	(0.064)	(0.156)
$\log_{10}(\text{Time to Complete})$	-0.308***	-0.363***
	(0.021)	(0.066)

Appendix

Table A.4 : Extended Specifications of the Baseline Model (logistic regressions).

	Dependent variable: Mean success rate
	10 bins	25 bins	50 bins	100 bins
	(1)	(2)	(3)	(4)
Intercept	0.6326***	0.6271***	0.6240***	0.6183***
	(0.0090)	(0.0057)	(0.0060)	(0.0095)
$\log_{10}(\text{Mean Duration})$	-0.0639***	-0.0584***	-0.0550***	-0.0505***
	(0.0069)	(0.0043)	(0.0047)	(0.0074)

Appendix

Table A.5 : OLS-Regression of Average Success Rates by Task-Duration Bins (Equal-Width Log-Spaced Bins) on Log Task Duration

	Dependent variable: $\mathbbm{1}\{\text{acceptance}\geq 7\}$
	All respondents	Failed 0 checks	Failed 1 check
	(1)	(2)	(3)
Intercept	1.017***	1.004***	1.039***
	(0.040)	(0.051)	(0.063)
$\log_{10}(\text{Time to Complete})$	-0.269***	-0.267***	-0.272***
	(0.017)	(0.022)	(0.026)

Appendix

Table A.6 : Baseline Model by Number of Failed Attention Checks (logistic regressions).

	Dependent variable: $\mathbbm{1}\{\text{acceptance}\geq 7\}$
	All respondents	Failed 0 checks	Failed 1 check
	(1)	(2)	(3)
Intercept	0.274***	0.200***	0.385***
	(0.047)	(0.061)	(0.074)
$\log_{10}(\text{Time to Complete})$	-0.273***	-0.272***	-0.275***
	(0.017)	(0.022)	(0.027)

Appendix

Table A.7 : Baseline Model by Number of Failed Attention Checks (logistic regressions).

	Dependent variable: $\mathbbm{1}\{\text{acceptance}\geq 7\}$
	All respondents	Failed 0 checks	Failed 1 check
	(1)	(2)	(3)
$\log_{10}(\text{Time to Complete})$	-0.242***	-0.239***	-0.244***
	(0.017)	(0.022)	(0.027)
Age (years)	0.006***	0.006***	0.005***
	(0.001)	(0.001)	(0.002)

Appendix

Table A.8 : Baseline Model with Demographic Controls, by Number of Failed Attention Checks (logistic regressions).

Panel of Figure 21 — Figure A.3 : OLS-Regression of Average Success Rates by Task-Duration Bins (Equal-Width Log-Spaced Bins) on Log Task Duration

Figure A.4 : Old vs New models by Job Family

Panel of Figure 23 — Figure A.5 : Individual Frontier LLM Success-Duration Relationships

Panel of Figure 25 — Figure A.7 : Task Duration and Success Rate Thresholds over Time (threshold score $\geq$ 8)

Panel of Figure 27 — Figure A.9 : Task Duration and Success Rate Thresholds over Time (alternative frontier definition)

Panel of Figure 28 — Figure A.10 : Time Duration Increase Based on Success-Duration curve slopes

Panel of Figure 29 — Figure A.11 : Predicted AI Success Rates Over Time, Alternative Success Thresholds

Panel of Figure 30 — Figure A.12 : Predicted AI Success Rates Over Time, Logistic Function vs. Complementary Log-Log

Figure A.13 : Assessing Measurement Error and Attenuation Bias

	Dependent variable: $\mathbbm{1}\{\text{acceptance}\geq 7\}$
	Baseline (pooled)	Non- randomized	Randomized	Adjusted (pooled)
	(1)	(2)	(3)	(4)
$(\text{Intercept})$	1.017***	1.175***	0.971***	0.973***
	(0.040)	(0.081)	(0.045)	(0.040)
$\log_{10}(\text{Time to Complete})$	-0.269***	-0.361***	-0.243***	-0.263***
	(0.017)	(0.035)	(0.019)	(0.017)

Appendix

Table B.1 : The Impact of Randomizing the Order of Displayed LLM Responses

	(1)	(2)	(3)
$\log_{10}(\text{Time to Complete})$	-0.269***	-0.270***	-0.262***
	(0.017)	(0.017)	(0.015)
Observations	60,845	60,470	60,845
Controls	None	None	None

Appendix

Table C.1 : The Impact of Measuring Task Length Pre vs. Post Evaluation

Figure E.1 : Bad response example 1

Figure E.2 : Good response example 1

(a) Bad response example 2

(a) Bad response example 2 (continued)

(a) Good response example 2

(a) Good response example 2 (continued)

Figure E.8 : Bad response example 3

(a) Good response example 3

(a) Good response example 3 (continued)

Large ( $>$ 100B)	Small ( $\leq$ 100B)
GPT-5	GPT-5-mini
GPT-5-Thinking	GPT-5-nano
Claude-Opus-4.1	GPT-OSS-20B-A3.6B
GPT-OSS-120B-A5.1B	Gemini-2.5-Flash-Lite
Gemini-2.5-Pro	Qwen-3-32B
Claude-Sonnet-4	Qwen-3-14B

Appendix

Table G.1 : Large vs Small

New (2025)	Old (2023–24)
GPT-5-nano	DeepSeek-V3
GPT-5-mini	GPT-4o
GPT-5	Claude-Haiku-3.5
GPT-5-Thinking	Gemini-1.5-Pro
GPT-OSS-20B-A3.6B	Llama-3.1-405B-Instruct
GPT-OSS-120B-A5.1B	Llama-3.1-8B-Instruct

Appendix

Table G.2 : New vs Old

Quarter	Frontier models (alternative/more stringent definition in bold )
Q2 2024	Qwen-2-72B-Instruct , Claude-Opus-3, GPT-4
Q3 2024	Gemini-1.5-Pro , Llama-3.1-405B-Instruct , GPT-4o-mini , Qwen-2-72B-Instruct , Claude-Opus-3
Q4 2024	DeepSeek-V3 , GPT-4o , GPT-4o-mini , Gemini-1.5-Pro , Llama-3.1-405B-Instruct
Q1 2025	Claude-Sonnet-3.7 , DeepSeek-R1 , DeepSeek-V3 , GPT-4o , Gemini-1.5-Pro
Q2 2025	Claude-Opus-4.1 , Claude-Sonnet-4 , Claude-Sonnet-3.7 , Gemini-2.5-Pro , Qwen-3-235B , o3 , DeepSeek-R1 , DeepSeek-V3 , Gemini-2.5-Flash, o4-mini
Q3 2025	GPT-5 , GPT-5-Thinking , o3 , Claude-Opus-4.1 , Claude-Sonnet-4 , Claude-Sonnet-3.7 , Gemini-2.5-Pro , GPT-5-mini, o4-mini, Gemini-2.5-Flash

Appendix

Table G.3 : Frontier models by quarter

AI and Labor

AI and labor studies how artificial intelligence changes jobs, wages, skills, hiring, and the organization and quality of work.

See all 94 papers

CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks
Aug 19, 2026Pattaraphon Kenny Wongchamcharoen, Kris Gulati, Min Min Fong +1LLM Agent Evaluation Human-AI Collaboration
Data Innovation and AI Lab (DIAL) at the Haas School of Business, UC Berkeley · UC Berkeley · UC Berkeley and NBER
The AI Skills Shift: Mapping Skill Obsolescence, Emergence, and Transition Pathways in the LLM Era
Apr 8, 2026Rudra Jadhav, Janhavi DanveAI and Labor LLM Evaluation
Department of Computer Science, Savitribai Phule Pune University, Pune, India
Artificial Effort
Apr 17, 2026Federico Belotti, Stefano Coniglio, Antonio Cosma +1LLM Evaluation
University of Bergamo, Italy.

Aug 19, 2026J/K move · Enter open · S save

CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks

Pattaraphon Kenny Wongchamcharoen, Kris Gulati, Min Min Fong, Abhishek Nagaraj

Data Innovation and AI Lab (DIAL) at the Haas School of Business, UC Berkeley · UC Berkeley · UC Berkeley and NBER

LLM Agent Evaluation Human-AI Collaboration Human-AI Complementarity AI and Labor Multi-LLM Collaboration

Open paper AskPDF arXiv

Most LLM benchmarks rank models on their ability to automate work tasks. In practice, however, models are often used to assist other (human or LLM) agents. The question that drives model selection is therefore not only which model produces the best output, but which model most improves the work of another (weaker) agent. We introduce a unified framework that evaluates the capability of models to automate and augment another agent's performance. Across seven economically grounded real-world tasks, an assistant model writes assistance text for a standardized lower-capacity worker model, which produces the deliverable. In automation mode, the assistant produces the output directly. Outputs are scored through blind pairwise comparisons by an LLM judge panel with task-specific rubrics, replicated across ten runs. Rankings across the two regimes are only modestly correlated, and the automation winner loses augmentation on five of seven tasks. Assistance is not reliably positive. The unaided worker outranks every assisted condition on three tasks, and only one model's guidance beats no guidance on average. These results suggest that automation ability is an incomplete proxy for assistance quality, motivating benchmarks that evaluate models according to the roles they play in human-AI and multi-agent systems.

Abstract

AI and Labor

Explore similar work

CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks

The AI Skills Shift: Mapping Skill Obsolescence, Emergence, and Transition Pathways in the LLM Era

Artificial Effort