RPEG: Training Reasoning Language Models Scalably and Cost-Effectively
Organizations: Department of Computer Science, ETH Zurich
Abstract
Reinforcement learning (RL) is now central to training Reasoning Language Models (RLMs), but RLM pipelines are difficult to scale and costly to run because each RL step couples multiple large models, heterogeneous stages, cross-step state, and complex asynchronous dependencies. Existing systems typically optimize only parts of this design space. We address this complexity by jointly optimizing the pipeline structure as well as its execution using a novel data-centric graph intermediate representation called RLM Pipeline Execution Graph (RPEG). RPEG exposes structural choices such as bounded asynchrony and actor-critic sharing together with execution optimizations such as stage fusion, fine-grained scheduling, and multi-dimensional model parallelism. It is coupled with a comprehensive analytical performance model for RLM training, jointly capturing compute, communication, memory, and hierarchical rack-, node-, and GPU-level effects. Across systems with up to 256 GH200 GPUs, our distributed runtime engine implementing RPEG improves performance by up to 2x over ReaL and reduces step latency by up to 2.48x over verl, while increasing hardware utilization by up to 30%. Overall, RPEG provides a graph-driven foundation for next-generation training pipelines for frontier reasoning LLMs.
Figures & tables
| Models & their architecture and workload | |
| Set of all models; a trainable subset of models ( ). | |
| Batch size (input prompts per step); generated rollouts per prompt. | |
| Average prompt length; average rollout length. | |
| Number of bytes per scalar in working precision ( for BF16). | |
| Hardware & its architecture | |
| Device space; . | |
| Primitive | Time | Application |
| AllReduce | TP partial-result aggregation | |
| AllGather | ZeRO-2 weight gather | |
| ReduceScatter | ZeRO-2 gradient reduce | |
| AlltoAll | Reshard nodes | |
| P2P | PP send/recv, Comm nodes |
| Optimized RPEG component | Runtime action |
| and its typed edges | Instantiate step-indexed model operations, data dependencies, and explicit transfer nodes. |
| Placement | Bind the invocation’s worker ranks to its selected physical GPUs. |
| Parallelism | Form DP replica groups, PP layer stages, and TP tensor shards/collectives in Megatron. |
| Microbatch count | Split each DP replica’s local batch into the selected number of microbatches and issue the corresponding model operations. |
| Publication edges and | Resolve each consumer’s required parameter version and install its corresponding shards. |
| Cross-layout dependencies | Route tensor fragments between producer and consumer layouts through Comm / Reshard operations. |
| Ablation | 16 GPU, 13B | 32 GPU, 34B | 64 GPU, 70B | Avg. |
| (PFLOP/s) | (PFLOP/s) | (PFLOP/s) | ||
| Full system ( ) | 0.233 | 0.656 | 1.339 | – |
| A1: gen-inf overlap | 0.203 (-12.8%) | 0.569 (-13.2%) | 1.318 (-1.5%) | -9.2% |
| A2: async execution | 0.189 (-18.8%) | 0.497 (-24.3%) | 1.270 (-5.2%) | -16.1% |
| A3: mb tuning | 0.238 (+2.1%) | 0.672 (+2.4%) | 1.105 (-17.5%) | -4.3% |
| A7: polish phase | 0.233 (+0.0%) | 0.647 (-1.5%) | 1.269 (-5.2%) | -2.2% |
| Nodes | AC structure | Baseline time [seconds/step] | RPEG time [seconds/step] | Speedup |
| 2 | Shared | 11.678 | 8.026–8.158 | 1.432–1.455 |
| 4 | Shared | 14.311 | 10.726–10.960 | 1.306–1.334 |
| 4 | Separate | 16.752 | 11.392–12.436 | 1.347–1.471 |
| 8 | Shared | 12.822 | 9.129–9.167 | 1.399–1.404 |
| 8 | Separate | 16.770 | 11.837–12.780 | 1.312–1.417 |
| 16 | Shared | 13.189 | 10.479–10.552 | 1.250–1.259 |
| Framework | Cross-iteration asynchrony | Intra-iteration overlap | Sub-invocation execution | Execution plan search | Pipeline structure search | Performance + memory model | Plan verification | Resource-eff. objective |
| ReaL [ 31 ] | (indep. ops) | (place+3D+mb) | (profiled) | |||||
| HybridFlow/ verl [ 45 ] | ( verl runtime) | (place+3D) | (simulation) | |||||
| RLHFuse [ 63 ] | (fusion) | (sample+mb) | (fusion plans) | (fusion) | (simulation) | |||
| AReaL [ 22 ] | (rollout/train overlap) | (samples) | (resource split) | (version control) | ||||
| AsyncFlow [ 24 ] | (streaming pipeline) | (pipeline) | (load balance) | (staleness checks) | ||||
| Laminar [ 44 ] | (samples) | (repacking) |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Vertex | Role | Model | Description |
| generation | Autoregressive rollout: produces completions and log-probabilities . | ||
| inference | Reference forward pass: produces baseline log-probabilities . | ||
| inference | Reward scoring: produces per-sequence rewards . | ||
| inference | Value estimation: produces per-token values . | ||
| training | Actor policy-gradient update. | ||
| training | Critic value-loss update. |
| Container | Description | ||
| Weight containers (carried by publication edges) | |||
| Actor parameters. | |||
| Critic parameters. | |||
| Reference parameters (frozen). | |||
| Reward parameters (frozen). | |||
| Data containers (carried by data edges) | |||
| Vertex | ||
| GPUs | Topology | A/C/R (B) | Evals/600 s | ||
| 64 | 13/13/7 | 1024 | 128/384 | 220K | |
| 128 | 34/13/13 | 1024 | 128/384 | 200K | |
| 256 | 70/34/13 | 1024 | 128/384 | 160K | |
| 512 | 70/70/34 | 2048 | 128/384 | 105K | |
| 1024 | 175/70/34 | 2048 | 128/384 | 75K |
| Scenario | GPUs | Topology | HBM/GPU | A/C/R (B) | |
| Mem-tight | 16 | 50 GB | 13/13/13 | 128;128/896 | |
| Wide | 128 | 102 GB | 7/7/7 | 1024;128/384 | |
| Hetero | 32 | 102 GB | 34/7/7 | 512;128/128 | |
| Deep | 64 | 102 GB | 7/70/70 | 512;128/128 |
| Role | Parallelism | Primitive | Count | Message bytes |
| Generation prefill | TP | AllReduce | ||
| Generation decode | TP | AllReduce | ||
| Inference | TP | AllReduce | ||
| Training | TP | AllReduce | ||
| Inference | PP | P2P | ||
| Training | PP | P2P |
LLM Training
Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.
See all 376 papers