cs.LGNov 9, 2026

An Analytical Model of Compute-limited Multistage Training Pipelines

Authors: Nishil Patel, Jin Hwa Lee, Basile Confavreux, Andrew Saxe

Organizations: Gatsby Computational Neuroscience Unit, University College London · Sainsbury Wellcome Centre, University College London

Abstract

Modern foundation models are trained through multiple stages, yet how these stages interact and how compute should be allocated across them remain poorly understood. We introduce an analytically tractable high-dimensional model combining unsupervised pretraining with a linear autoencoder, supervised fine-tuning (SFT) with noisy stepwise labels, and reinforcement learning (RL) with sparse episodic rewards. Using random matrix theory and an operator-projection closure, we derive approximate low-dimensional learning dynamics that closely match simulations. We show that finite pretraining data imposes a representation-dependent performance limit, with Bayes classification error scaling as the inverse square root of the pretraining sample ratio, and observe the same exponent in asymptotic SFT and RL errors. Better pretraining also reduces sensitivity to noisy supervision. SFT initialization shortens the initial RL plateau induced by sparse rewards, an effect we also observe in OLMo-2 1B on GSM8K. Longer SFT warm starts additionally improve asymptotic RL performance. Optimizing compute across all three stages reveals that the optimal allocation depends on supervision quality and total budget, with a non-monotonic dependence of the RL compute share on budget. Finally, a complementary compositional model shows how RL selectively strengthens task-relevant pretrained skills while largely preserving unused skills. Together, these results identify mechanisms linking representation quality, feedback structure, and compute allocation in multistage training.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Training Foundation Models with a Single Algorithm

    Oct 29, 2026Yuda Song, Alvin Zhang, Fahim Tajwar +7LLM TrainingRL for Language Models

  2. Learning to Fine-tune Foundation Models under Resource Limitations

    Jul 12, 2026Thomas Tsouparopoulos, Iordanis KoutsopoulosFine-TuningLLM Fine-Tuning

  3. Sequential Pretraining Favors Large Models

    Oct 7, 2026Mohnish Harwani, Yujia ZhengLanguage Model PretrainingContinued Pretraining