cs.LGNov 26, 2025

On the Origin of Algorithmic Progress in AI: Evidence from Language Model Pre-Training

Authors: Hans Gundlach, Alex Fogelson, Jayson Lynch, Ana Trisovic, Jonathan Rosenfeld, Anmol Sandhu, Neil Thompson

Organizations: MIT FutureTech, CSAIL · Olin College

Abstract

Algorithms have been estimated to increase AI pretraining FLOP efficiency by a factor of 22,000 between 2012 and 2023 (Ho et al., 2024). Running ablation experiments on key innovations from this time period, we are able to account for less than 10x of these gains at small compute scales. Surveying the broader literature, we estimate that additional innovations not included in our ablations also account for less than 10x, yielding less than 100x efficiency gains total in low-compute regimes. This leads us to conduct scaling experiments, which provide evidence that the efficiency gap can be explained by algorithms with scale-dependent efficiency improvements. In particular, we conduct scaling experiments between LSTMs and Transformers, finding evidence for exponent differences in their compute-optimal scaling law while finding little scaling difference for many other innovations. These experiments demonstrate that, contrary to standard assumptions, an algorithm's efficiency gains are tied to compute scale. Using experimental scaling extrapolations and literature estimates, we can account for the majority of efficiency gains over the same time period. Our results indicate that algorithmic progress at smaller training compute levels has been far slower than previously assumed, and that measures of algorithmic efficiency are strongly reference-dependent.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Bridging Compute- and Data-Optimal Pretraining

    Jul 28, 2026Tian Qin, Kimia Hamidieh, David Alvarez-MelisLanguage Model PretrainingLanguage Model Scaling Laws

  2. Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

    Feb 17, 2026Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis +1Language Model Performance PredictionLanguage Model Scaling Laws