cs.LGOct 2, 2026

Zephon: Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelines

Authors: Maximilian Böther, Josh Wills, Ties Robroek, Sonnet Xu, Paul Burstein, Daniel Zayas, Cody Blakeney, Siddharth Joshi, 

Organizations: DatologyAI & ETH Zürich · ETH Zürich · Stanford University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Deterministic data loading is important for foundation model development: model researchers need confidence that differences they observe across costly ablations are caused by the parameter they changed rather than non-determinism in the training data sequence. The data loader must provide elastic determinism, i.e., a deterministic sequence of global training data batches despite changes to the GPU topology across runs (e.g., due to GPU scarcity), frequent checkpoint-resume cycles, and different data processing execution backends. Achieving this is difficult because modern foundation model data pipelines tokenize, pack, and mix samples online, introducing stateful n-to-m transformations that break sample indexing. Existing data loaders largely assume indexable 1-to-1 pipelines, and the common workaround of offline materialization is expensive and, for some modalities such as video, infeasible. We present Zephon, a data loader for foundation models that supports online, stateful pipelines while providing elastic determinism and efficient resumption from checkpoints. It partitions the global stream into topology-independent lanes, serializes ordering decisions while parallelizing stateless work on interchangeable backends, and checkpoints only bounded in-flight state so recovery cost does not grow with training progress. We evaluate Zephon on text and vision-language workloads and show that it achieves competitive throughput while providing a combination of guarantees that no existing loader offers for online, stateful pipelines.

Figures & tables

Figure 1. The effects of elastic determinism: a 1 B text model is trained on 16 GPUs, checkpointed at step 150, and resumed on 8, 32, and 64 GPUs while the original run continues uninterrupted, causing loss variance between the runs when not using Zephon . The barcode below each plot shows the per-step batch SHA-256 hash of the 8-GPU continuation: blue shows matching batches while red shows differing batches. TODO!
Figure 2. A non-indexable VLM preprocessing pipeline. Operators vary in cardinality (below the operator) and statefulness (blue solid/yellow dotted). Arrow color/pattern indicates the source, thick purple arrows are packed samples. TODO!
Figure 3. Zephon ’s high-level architecture. The WorkSource produces chunks for each lane (colored lines), which are converted into samples by the Fetch operator. Each lane holds an independent chunk stream and independent operator states. TODO!
Figure 4. Zephon ’s runtime architecture. The engine feeds chunks into a pipeline compiled into stages, each assigned a runner. Within each stage, a single-threaded accumulator dispatches microbatches to parallel stateless workers. TODO!
Figure 5. Per-chunk completion bitmaps for in-flight chunks. Each cell is one base offset; it closes when its last-child contributor is delivered. Non-monotone operators reorder samples across chunks, so chunks close out of order, and a chunk is evicted only once its bitmap is full.
Figure 6. Fetch-and-batch throughput for text (top) and VLM (bottom) data with Zephon ’s process ( Z-Proc ) and thread ( Z-Thread ) runners. On text data, Zephon is sufficiently fast for stall-free training; on VLM data, it outperforms all baselines.
Figure 8. Score spread (max - min) across GPU counts (8, 16, 32, 64) for 20 B-token training. The dashed line is the reference level above which differences are meaningful. TODO!
Figure 9. VLM training loss across 8–64 GPUs.
More in this topic

ML Reproducibility

Machine learning reproducibility studies whether model results and scientific claims can be independently checked and reliably repeated.

See all 138 papers

Explore similar work

CardsList

  1. BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Training

    May 11, 2026Ting Sun, Junjie Zhang, Xiao Yan +7Fault-Tolerant Distributed TrainingDistributed Training

    Lionrock AI Lab, China Merchants Group, Hong Kong, China · Wuhan University, Wuhan, China · The Chinese University of Hong Kong, Shenzhen, China

  2. Online Dynamic Batching with Formal Guarantees for LLM Training

    Jun 18, 2026Dian Li, Zekun Wang, Yaoru Wang +1LLM Fine-TuningDistributed Training

    Tencent

  3. Attention Once Is All You Need: Efficient Streaming Inference with Stateful Transformers

    May 13, 2026Victor NorgrenTransformer InferenceLLM Inference

    LayerScale, Inc.