ReFlow: Replay-Driven Decoupling for Efficient RL Post-Training

Yukun Tian; Junwei Su; Haofeng Xu; Lansong Diao; Zhengping Qian; Weibo Zhao; Wenchao Zhou; Ruiting Zhou

cs.LGOct 15, 2026

ReFlow: Replay-Driven Decoupling for Efficient RL Post-Training

Authors: Yukun Tian, Junwei Su, Haofeng Xu, Lansong Diao, Zhengping Qian, Weibo Zhao, Wenchao Zhou, Ruiting Zhou,

Organizations: The University of Hong Kong · Alibaba Group · University of Science and Technology of China · Southeast University

Abstract

Reinforcement learning (RL) has been a critical paradigm for improving large language model (LLM) reasoning and agentic capabilities, but its efficiency remains a major bottleneck. Existing RL systems commonly rely on asynchronous execution to overlap rollout generation with policy optimization for enhanced efficiency. However, this approach faces inherent limitations: mild policy staleness offers only limited training efficiency gains, whereas aggressive asynchrony can degrade model quality and yields diminishing efficiency gains. The dynamic nature of LLM RL and multi-turn agentic workloads further challenges RL system design, as resource demands shift over time, making static resource allocation and generation strategy inefficient. This paper presents ReFlow, a system-algorithm co-designed RL framework for efficient LLM post-training. Its key insight is that, unlike existing RL frameworks which consume each trajectory only once, experience replay of recent trajectories enables effective sample reuse, thereby amortizing the high cost of rollout generation without incurring excessive policy staleness. To fully exploit this opportunity, ReFlow adopts a decoupled and adaptive architecture with three key designs: a dual replay data cycle supports efficient experience reuse while balancing training throughput and optimization stability; an elastic resource pool enables dynamic resource placement to accommodate evolving RL workloads; an adaptive scheduler coordinates generation strategies and resource allocation to improve resource utilization and boost generation throughput. Experiments across representative single-turn reasoning and multi-turn agentic RL tasks show that ReFlow improves training throughput by up to $2.06\times$ over state-of-the-art synchronous and asynchronous RL systems, while maintaining comparable or better model performance.

Figures & tables

Figure 1: Typical LLM RL workflow: (a) synchronous RL; (b) asynchronous RL. $W$ denotes different versions of weights.

Panel of Figure 2 — Figure 2: GPU idle ratio (idle time / step time) under different resource allocations for Qwen3-4B training on 8 GPUs.

Panel of Figure 3 — Figure 3: Training model performance under different staleness settings when training Qwen3-1.7B on 8 GPUs, where staleness denotes the number of training iterations between two consecutive parameter synchronizations.

Figure 4: Diverse and evolving generation characteristics under different training configurations, demonstrating the highly dynamic workload pattern of LLM RL.

Panel of Figure 5 — Figure 5: Qwen3-4B on DAPO-MATH-17K. Naive experience replay slows convergence and degrades performance.

Symbol	Description
$\sigma$	Maximum fraction of replay data in a training batch.
$\tau$	Loss weight controlling the relative contribution of replayed trajectories.
$\delta$	Maximum data policy lag allowed for training.

Table 1: Hyperparameters in dual replay data cycle.

Figure 8: Cold-start overhead. (a) Total overhead under Qwen3 models. (b) Breakdown for Qwen3-32B on 128 GPUs.

Figure 9: Workflow of ReFlow’s resource rebalance. We use TP=2 for the rollout worker and FSDP group size=4 for the trainer as an example.

\lambda_{\mathrm{r}}\leftarrow\frac{\Delta N_{\mathrm{prod}}/\Delta t}{1-i_{\mathrm{rollout}}+\epsilon},\hskip 18.49988pt\lambda_{\mathrm{d}}\leftarrow\frac{\Delta N_{\mathrm{cons}}/\Delta t}{1-i_{\mathrm{train}}+\epsilon}.

Algorithm 1 Rollout/Train Resource Scheduler

Panel of Figure 10 — Figure 10: System throughput under different RL workloads.

Panel of Figure 11 — Figure 11: The training performance on Retool agentic tasks for Qwen3-8B.

Panel of Figure 12 — Figure 12: Training reward with respect to step.

Panel of Figure 13 — Figure 13: Training reward with respect to wall clock time.

Figure 14: Trainer and rollout worker idle ratio comparison.

Rollout : Train Setting	Direction	Cost (s)
32 GPUs (16:16 $\rightarrow$ 24:8)	train to rollout	5.72
32 GPUs (24:8 $\rightarrow$ 16:16)	rollout to train	9.20

Table 2: Dynamic resource placement overhead in ReFlow.

Setting	Average Generation
	Throughput (token/s)
w/o adaptive scheduling	1585.1 (+0.0%)
w/ adaptive scheduling	1749.8 (+10.4%)

Table 3: Effect of adaptive generation scheduling in ReFlow.

RL Post-Training

Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.

See all 411 papers

AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training
Jul 2, 2025Zhenyu Han, Ansheng You, Haibo Wang +16RL for Language Models Asynchronous RL
Huawei · Individual Researcher
Libra: Efficient Resource Management for Agentic RL Post-Training
Jun 2, 2026Kaiwen Chen, Xin Tan, Jingzong Li +6Agentic RL Reinforcement Learning
The Chinese University of Hong Kong · The Hang Seng University of Hong Kong · Independent Researcher
DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training
Apr 29, 2026Tianhao Hu, Xiangcheng Liu, Youshao Xiao +15RL for Language Models Asynchronous RL
Meituan, China

Jul 2, 2025J/K move · Enter open · S save

AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training

Zhenyu Han, Ansheng You, Haibo Wang, Kui Luo+15

Huawei · Individual Researcher

RL for Language Models Asynchronous RL Reinforcement Learning Distributed RL RL Post-Training

Open paper AskPDF arXiv

Reinforcement learning (RL) has become a pivotal technology in the post-training phase of large language models (LLMs). Traditional task-collocated RL frameworks suffer from significant scalability bottlenecks, while task-separated RL frameworks face challenges in managing complex dataflows and resolving resource idling. Furthermore, most existing frameworks are tightly coupled with LLM training or inference engines, making them difficult to support custom-designed engines. To address these challenges, we propose AsyncFlow, an asynchronous streaming RL framework tailored for efficient post-training. Specifically, we introduce a distributed data storage and transfer module that provides panoramic data management and fine-grained scheduling capabilities in a fully streamed manner. This architecture inherently enables automated pipeline overlapping among RL tasks and dynamic load-balancing. Moreover, we propose an asynchronous producer-consumer workflow, which is engineered to minimize computational idleness by strategically deferring the parameter update process within staleness thresholds. Finally, the core capabilities of AsyncFlow are architecturally decoupled from underlying training and inference engines and encapsulated by service-oriented user interfaces, offering a modular and customizable user experience. Extensive experiments demonstrate an average throughput of 1.59x compared to the state-of-the-art baseline. The architecture presented in this work provides actionable insights for designing next-generation RL training systems.

Abstract

RL Post-Training

Explore similar work

AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training

Libra: Efficient Resource Management for Agentic RL Post-Training

DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training