Higher-Scoring Coding Models Did Not Get More Work Done
Organizations: Harvard University, Boston, MA, USA
Abstract
New coding models advertise themselves on score improvements of coding benchmarks. Yet no study has tested if a model that scores higher on these benchmarks gets more real work done. On June 12, 2026, a U.S. export control directive banned access to Claude Fable 5 for 19 days, which forced users to switch to other models. During this time, most former Fable 5 users switched to Opus 4.8 (11 points lower on SWE-Bench Pro compared to Fable 5). We treat this suspension as a natural experiment, where the people and projects stayed the same and only the model changed, so we could isolate the effect of Fable 5 on user productivity. We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension. We used 8,908 developers who stayed on Opus 4.8 all the time as the control group. We also repeated the comparison across five model upgrades before and after the suspension in 2026 (1,208 developers, 15,442 pull requests). Fitting a difference-in-differences model with developer and date fixed effects, we found that having access to Fable 5 increased the 7-day merge rate of pull requests by only 1.4 percentage points [-1.4; +4.3], which was not statistically significant. Similarly, no significant increases of merge rates were found with the upgrades (weighted mean: -0.9 percentage points [-1.9; +0.2]); all five estimates were negative (95% intervals included 0, did not differ from each other). We also did not observe a significant increase in the number of pull requests merged per week (+2.4% [-4.9%; +10.2%]).
Figures & tables
| Suspension (P0) | Upgrades (week before) | |||||
|---|---|---|---|---|---|---|
| Treated | Control | Std. diff. | Switched | Did not switch | Std. diff. | |
| Developers with PRs | 244 | 315 | 1,208 | 944 | ||
| PRs | 21,377 | 30,630 | 7,829 | 4,502 | ||
| PRs per developer, median (IQR) | 34 (10–88) | 35 (10–89) | −0.06 | 2 (1–6) | 2 (1–5) | +0.12 |
| Merged within 7 days (%) | 92.7 | 91.5 | +0.05 | 95.0 | 94.2 | +0.03 |
| Merged within one hour (%) | 73.9 | 71.5 | +0.05 | 77.8 | 77.4 | +0.01 |
| Outcome | Suspension analysis | Upgrade analysis |
|---|---|---|
| Main outcomes | ||
| Merged within 7 days (percentage points) | +1.4 (−1.4, +4.3) | −0.9 (−1.9, +0.2) |
| Weekly merged PRs (%) | not estimated b | +2.4 (−4.9, +10.2) a |
| Secondary outcomes | ||
| Merged within 1 hour (percentage points) | 0.0 (−5.8, +5.8) | +0.8 (−2.1, +3.8) a |
| Closed without merge (percentage points) | −0.5 (−3.0, +2.1) | +0.5 (−1.3, +2.3) a |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Check | Reversal contrast | P0 placebo |
|---|---|---|
| Main estimate (equal weight per PR, merged within 7 days) | +1.4 (−1.4, +4.3) | −2.6 (−6.2, +1.1) |
| Weighting | ||
| Equal weight per developer | +3.7 (−5.5, +12.9) | −2.0 (−13.7, +9.8) |
| At most 20 PRs per developer | +3.6 (−1.1, +8.3) | |
| At most 50 PRs per developer | +3.4 (−0.7, +7.5) | |
| Developer means: suspension (n = 84) | +3.5 (−2.5, +9.7) |
| Month | Sonnet 4.5 | Opus 4.5 | Opus 4.6 | Sonnet 4.6 |
|---|---|---|---|---|
| Apr 2026 | 458 | 95 | 6,547 | 2,620 |
| Mar 2026 | 305 | 185 | 9,724 | 2,548 |
| Feb 2026 | 1,648 | 782 | 4,639 | 514 |
| Jan 2026 | 2,330 | 2,348 | 13 | 4 |
| Dec 2025 | 1,987 | 654 | 32 | 3 |
| Nov 2025 | 52 | 5 | 31 | 0 |
| Item | Setting |
|---|---|
| Source | GitHub REST search API (search/issues), 100 results per page, at most 30 requests per minute |
| Collection date | November 2, 2026 (UTC) for the main collections, with later checks dated in their rows |
| Model-line query | is:pr “Co-Authored-By: Claude <model>” in:body created:<window> |
| Developer histories | is:pr author:<login> “Co-Authored-By: Claude” in:body created:2026-05-26..2026-07-16 |
| Windows | any window above 1,000 matches split into shorter windows (down to 30 minutes), so no window was truncated |
| Model-line check | regular expression for “Co-Authored-By: Claude”, a family name (Opus, Sonnet, Haiku, Fable or Mythos) and a version; bodies naming two models dropped |
| Event | Release and models | PRs, older / newer model | Developers who switched (with outside-author PRs) | PRs of developers who switched | Change, all PRs | Interaction | Pool |
|---|---|---|---|---|---|---|---|
| 1 | Feb 5, Opus 4.5 to 4.6 | 677 / 1,112 | 54 (10) | 443 | +1.7 (−5.2, +8.5) | +11.8 (−0.8, +24.5) | six only |
| 2 | Feb 17, Sonnet 4.5 to 4.6 | 578 / 270 | 9 (2) | 104 | +5.1 (−51.3, +61.4) | too few to estimate | none |
| 3 | Apr 16, Opus 4.6 to 4.7 | 2,473 / 2,795 | 115 (26) | 1,516 | −4.1 (−9.7, +1.6) | −6.8 (−29.6, +16.0) | five and six |
| 4 | May 28, Opus 4.7 to 4.8 | 4,869 / 3,513 | 113 (24) | 3,482 | −0.1 (−1.9, +1.6) | −2.9 (−8.1, +2.4) | five and six |
| 5 | Jun 30, Sonnet 4.6 to 5 | 3,676 / 4,037 | 193 (8) | 1,440 | +1.8 (−0.6, +4.3) | +6.7 (−9.8, +23.2) | none |
| 6 | Jul 24, Opus 4.8 to 5 | 7,425 / 6,489 | 404 (47) | 5,477 | −0.8 (−2.3, +0.7) | +3.1 (−3.9, +10.1) | five and six |
| Event | Comparison model | Developers who did not switch (PRs) | Their change | Change for developers who switched | Calendar-controlled change |
|---|---|---|---|---|---|
| 1 | Sonnet 4.5 | 17 (129) | −14.0 (−31.4, +3.4) | +1.7 (−5.2, +8.5) | +15.5 (−0.8, +31.7) |
| 3 | Sonnet 4.6 | 27 (260) | −8.5 (−18.8, +1.9) | −4.1 (−9.7, +1.6) | +5.4 (−4.5, +15.4) |
| 4 | Sonnet 4.6 | 27 (286) | +0.3 (−16.3, +16.8) | −0.1 (−1.9, +1.6) | −1.0 (−17.1, +15.1) |
| 5 | Opus 4.8 | 418 (4,991) | −0.3 (−1.9, +1.3) | +1.8 (−0.6, +4.3) | +1.9 (−0.9, +4.7) |
| 6 | Sonnet 5 | 256 (2,213) | −0.1 (−2.3, +2.0) | −0.8 (−2.3, +0.7) | −0.5 (−3.0, +2.0) |
| 7 | Sonnet 5 | 210 (1,911) | +0.2 (−1.9, +2.3) | −1.7 (−5.9, +2.6) | −2.4 (−7.5, +2.7) |
| Treated | Control | Weighted control | Std. diff. | Std. diff. (weighted) | |
|---|---|---|---|---|---|
| Developers with PRs in P0 | 244 | 315 | 312.6 (effective) | ||
| PRs in P0 | 21,377 | 30,630 | 30,630 | ||
| PRs per developer, median (IQR) | 34(10–88) | 35(10–89) | 32(10–84) | −0.06 | −0.02 |
| Merged within 7 days (%) | 92.7 | 91.5 | 92.7 | +0.05 | 0.00 |
| Merged within one hour (%) | 73.9 | 71.5 | 72.8 | +0.05 | +0.03 |
| Closed without merge (%) | 6.0 | 6.8 | 5.6 | −0.03 | +0.02 |
AI-Assisted Software Development
AI-assisted software development uses language models and coding agents to help people design, write, test, review, and maintain software.
See all 116 papers