Iterated creation, grading, and revision of data science projects by language models
Abstract
We evaluate the extent to which agentic AI models can complete an end-to-end data science project, and how this improves with feedback. A 'writer' model with specific tools creates a complete data science project including downloading, cleaning, and analyzing a dataset, and writing a short paper about it. The project is executed and a 'grader' model evaluates it. The writer can then revise its submission, and this repeats for a total of three rounds. We consider seven models from five developers: Anthropic, OpenAI, Google, DeepSeek, and Z.AI. Each model is both a writer and a grader, so there are 49 writer-grader pairs and we run each pair three times. Of the 147 initial submissions, 107 run and render, and the average score increases from 59 per cent to 76 per cent after the final round of feedback. By writer the extent of the increase ranges from 3 percentage points for Anthropic's Opus 5.5 to 46 percentage points for Anthropic's Haiku 4.5, and by grader it ranges from 13 percentage points for Google's Gemini 3.8 Flash to 32 percentage points for Anthropic's Haiku 4.5. When a model grades its own writing the score tends to be about 9 percentage points higher than the average of what the other models give it. About half of the difference that we see from round to round is within the grader's own noise. Feedback generally results in the highest marks, but even just having the chance to revise, or having the additional budget, in terms of time or tool calls, without any feedback, results in considerable increases for some of the models but not for others. We find that how much such a loop helps depends on which model plays which role, but weaker models generally do not improve stronger models.
Figures & tables
| Submissions that execute | First submission | How the session ended | |||||||
| Writer | First round | Final round | Any round | Minutes | Tool calls | Writer ended | Tool limit | Time limit | No tools |
| Opus 5.5 | 21 of 21 | 21 of 21 | 63 of 63 | 31.1 | 43 | 12 | 1 | 0 | 8 |
| Haiku 4.5 | 18 of 21 | 18 of 21 | 49 of 63 | 9.7 | 58 | 15 | 6 | 0 | 0 |
| GPT-6 Sol | 21 of 21 | 21 of 21 | 63 of 63 | 17.3 | 56 | 21 | 0 | 0 | 0 |
| GPT-6 Luna | 18 of 21 | 19 of 21 | 54 of 63 | 12.7 | 58 | 3 | 17 | 1 | 0 |
| DeepSeek V4.1 Flash | 11 of 21 | 20 of 21 | 47 of 63 | 17.5 | 58 | 1 | 18 | 2 | 0 |
| Dataset | Opus 5.5 | Haiku 4.5 | GPT-6 Sol | GPT-6 Luna | DeepSeek V4.1 Flash | Gemini 3.8 Flash | GLM-5.3 Flash | All writers |
| World Bank indicators | 1 | 4 | 4 | 6 | 1 | 0 | 0 | 16 |
| Toronto shelter occupancy | 6 | 0 | 0 | 0 | 0 | 7 | 0 | 13 |
| USGS earthquake catalogue | 0 | 0 | 10 | 0 | 1 | 0 | 1 | 12 |
| CO2 and energy by country | 0 | 0 | 4 | 5 | 1 | 0 | 1 | 11 |
| NHANES health surveys | 0 | 0 | 0 | 0 | 8 | 0 | 2 | 10 |
| Toronto apartment evaluations | 2 | 0 | 0 | 0 | 0 | 8 | 0 | 10 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Runs with feedback | |||||||||||
| Model | Generation | Revision | Grading | Without feedback | One session | Every grader | Same grader again | Planted faults | Student projects | Route check | Total |
| Opus 5.5 | 46.34 | 21.72 | 20.61 | 28.41 | 23.17 | 146.97 | 13.40 | 7.08 | 26.77 | 4.09 | 338.55 |
| Haiku 4.5 | 12.00 | 14.73 | 3.91 | 12.20 | 3.13 | 25.47 | 2.72 | 1.28 | 4.90 | 1.15 | 81.49 |
| GPT-6 Sol | 17.89 | 12.07 | 13.92 | 23.20 | 6.43 | 51.31 | 8.32 | 1.32 | 10.00 | 1.18 | 145.64 |
| GPT-6 Luna | 0.91 | 0.93 | 0.59 | 1.25 | 0.53 | 2.75 | 0.35 | 0.08 | 0.54 | 0.07 | 7.99 |
| DeepSeek V4.1 Flash | 1.30 | 1.42 | 0.55 | 1.27 | 0.59 | 4.03 | 0.21 | 0.13 | 0.87 | 0.00 | 10.37 |
| Writer | Round 1 | Round 2 | Round 3 |
| Opus 5.5 | 2.30 | 0.72 | 0.55 |
| Haiku 4.5 | 0.64 | 0.40 | 0.43 |
| GPT-6 Sol | 0.92 | 0.55 | 0.52 |
| GPT-6 Luna | 0.11 | 0.10 | 0.12 |
| DeepSeek V4.1 Flash | 0.16 | 0.14 | 0.13 |
| Gemini 3.8 Flash | 0.53 | 0.37 | 0.36 |
| Grader | Route | Mean difference | Mean absolute difference |
| OpenRouter | -1.8 | 2.2 | |
| Opus 5.5 | Direct | -1.1 | 1.9 |
| OpenRouter | -2.4 | 6.1 | |
| Haiku 4.5 | Direct | -0.2 | 4.3 |
| OpenRouter | +0.7 | 3.7 | |
| GPT-6 Sol | Direct | +0.3 | 3.2 |
| Purpose | Number | Packages |
| Reading and wrangling data | 28 | arrow, data.table, digest, dplyr, dtplyr, forcats, fs, glue, here, janitor, jsonlite, lubridate, magrittr, naniar, purrr, readr, scales, skimr, slider, stringi, stringr, tibble, tidyr, tidytext, tidyverse, tsibble, vctrs, zoo |
| Data access | 28 | AER, babynames, cansim, countrycode, curl, DBI, duckdb, eurostat, fivethirtyeight, gapminder, haven, httr, httr2, Lahman, nycflights13, OECD, opendatatoronto, palmerpenguins, pdftools, readxl, RSQLite, rvest, tidytuesdayR, wbstats, WDI, wooldridge, writexl, xml2 |
| Models | 36 | boot, broom, car, did, effectsize, emmeans, estimatr, fable, feasts, fixest, forecast, ggeffects, glmnet, insight, lme4, lmtest, marginaleffects, MASS, matrixStats, mgcv, mice, modelr, nlme, nnet, parameters, performance, plm, quantreg, ranger, sandwich, splines2, srvyr, survey, survival, tidymodels, xgboost |
| Bayesian models | 11 | bayesplot, brms, broom.mixed, coda, ggdist, loo, posterior, rstan, rstanarm, shinystan, tidybayes |
| Tables and figures | 40 | chromote, corrplot, cowplot, DT, ggalluvial, GGally, ggbeeswarm, ggcorrplot, ggplot2, ggpubr, ggrepel, ggridges, ggtext, ggthemes, gridExtra, gt, gtsummary, kableExtra, knitr, latex2exp, leaflet, mapproj, maps, modelsummary, patchwork, plotly, quarto, RColorBrewer, rmarkdown, rnaturalearth, rnaturalearthdata, see, sf, sjPlot, stargazer, tinytable, treemapify, viridis, webshot2, wesanderson |
| Testing and workflow | 17 | assertr, bench, checkmate, furrr, future, lintr, pak, pointblank, progressr, remotes, renv, reticulate, styler, testthat, tictoc, usethis, validate |
| First round | After three rounds | Whole allowance at once | ||||||
| Writer | Score | Without feedback | With feedback | Score | Tool calls (of 140) | Minutes (of 100) | Execute | Writer ended |
| Opus 5.5 | 93 | 93 | 94 | 95 | 96 | 59 | 3 of 3 | 1 of 3 |
| Haiku 4.5 | 4 | 8 | 45 | 0 | 58 | 8 | 3 of 3 | 3 of 3 |
| GPT-6 Sol | 90 | 92 | 93 | 90 | 73 | 15 | 3 of 3 | 3 of 3 |
| GPT-6 Luna | 77 | 78 | 87 | 85 | 139 | 17 | 3 of 3 | 3 of 3 |
| DeepSeek V4.1 Flash | 63 | 81 | 86 | 85 | 139 | 49 | 3 of 3 | 3 of 3 |
| Grader | Projects | Mean absolute error | Bias | Correlation |
| Opus 5.5 | 56 | 41.4 | -41.2 | 0.74 |
| Haiku 4.5 | 50 | 14.0 | -6.3 | 0.46 |
| GPT-6 Sol | 56 | 48.1 | -48.1 | 0.61 |
| GPT-6 Luna | 56 | 41.0 | -40.9 | 0.70 |
| DeepSeek V4.1 Flash | 56 | 22.3 | -21.3 | 0.71 |
| Gemini 3.8 Flash | 56 | 13.9 | -10.2 | 0.69 |
| Fault | Opus 5.5 | Haiku 4.5 | GPT-6 Sol | GPT-6 Luna | DeepSeek V4.1 Flash | Gemini 3.8 Flash | GLM-5.3 Flash | All graders |
| Unchanged | 82 | 100 | 92 | 95 | 98 | 100 | 98 | 95 |
| Data generated, described as downloaded | 0 | 100 | 0 | 0 | 0 | 0 | 0 | 14 |
| Latitude adjustment dropped from the model | 77 | 100 | 77 | 84 | 98 | 97 | 97 | 90 |
| Headline result typed in, wrong | 82 | 100 | 87 | 89 | 95 | 100 | 95 | 92 |
| Tests that assert nothing | 80 | 100 | 88 | 92 | 93 | 98 | 96 | 92 |
| Wrong data source named | 83 | 100 | 90 | 91 | 94 | 100 | 91 | 93 |
Data Analysis Agents
Data analysis agents use language models and tools to explore datasets, run analyses, and produce answers or reports grounded in evidence.
See all 120 papers