cs.LGNov 8, 2026

FRESCO: A Novel Consistency Control for Asynchronous Pipeline Parallel Training

Authors: Dongyeop Lee, Namhoon Lee

Organizations: POSTECH

Abstract

Asynchronous pipeline parallelism can accelerate distributed model training, but suffers from instability caused by inconsistent gradient computations. We identify a fundamental source of this inconsistency: error signals are coupled to forward-time features, not just parameter states. This means that parameters may evolve as long as they preserve forward-equivalence. Leveraging this insight, we introduce FRESCO, which frames asynchronous updates as a constrained optimization problem: it finds the minimal update modification that acts trivially on the activation subspace. We demonstrate that FRESCO consistently outperforms representative asynchronous baselines in LLM pre-training, with the gains most pronounced under deep pipelines and high inconsistency levels: at pipeline depth 32, it reaches the same loss as the strongest baseline with 43% fewer iterations and 30% less wall-clock time, while requiring 8~921×\times less memory overhead. Moving beyond the current limitations, FRESCO establishes a new foundation for high-utilization pipeline parallel training, providing a scalable path for large-model development on diverse, resource-constrained systems.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistency

    Jun 5, 2026Itay Elam, Eliron Rahimi, Avi Mendelson +1Neural Network Training DynamicsPipeline Parallelism

  2. One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining

    Jun 29, 2026Philip Zmushko, Egor Petrov, Nursultan Abdullaev +2Error FeedbackMuon Optimizer

  3. AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training

    May 28, 2026Ling Chen, Houming Wu, Wenjie YuPipeline ParallelismLLM Training