Conduit: An Experience Data Plane for Distributed Reinforcement Learning

Sitong Zhang; Tuo Shi; Mario Di Francesco; Zeke Wang; Bo Zhao

cs.DCSep 21, 2026

Conduit: An Experience Data Plane for Distributed Reinforcement Learning

Authors: Sitong Zhang, Tuo Shi, Mario Di Francesco, Zeke Wang, Bo Zhao

Organizations: Aalto University · Shenzhen University of Advanced Technology · Zhejiang University

Abstract

Distributed reinforcement learning (RL) scales training by parallelizing actors and learners around an Experience Buffer. As RL workloads grow, however, the buffer becomes more than a replay queue: it is the storage substrate of a large-capacity, latency-critical experience path that every iteration traverses to move, transform, sample, and batch experiences before learner updates can begin. Existing RL systems embed this path inside framework control flow or expose it as a request-driven buffer service, leaving experience placement fixed and experience-path work difficult to schedule independently as a runtime-level optimization target. We present Conduit, a framework-agnostic runtime that exposes RL experience management as an explicit systems optimization problem. At its core is the Experience Data Plane (EDP), a runtime abstraction that separates RL experience-handling semantics from framework-specific execution logic by exposing experience ingestion, experience placement, and experience delivery as explicit control points. Built on EDP, Conduit introduces capacity-constrained, bandwidth-aware placement, which distributes experience state across CPU/GPU memory tiers and nodes under heterogeneous interconnect and device-memory constraints, and latency-aware scheduling, which controls when experience-path handling runs to reduce exposed experience-path latency while preserving RL semantics. Integrated with RLlib without changing its framework execution logic, Conduit reduces exposed experience-path latency by up to 97% and end-to-end iteration latency by up to 38%, scales to 1,024 GPUs, and preserves convergence.

Figures & tables

Figure 1. Experience-path latency is a major component of iteration time in distributed RL: 50.2% in off-policy Deep Q-Network (DQN) and 26.2% in on-policy proximal policy optimization (PPO) across representative workloads on an 8

\times

A100 cluster.

Figure 2. Experience-path handling drives latency through transfer and processing operations. Existing systems embed this handling in their actor/learner workflow; Conduit instead abstracts it into an explicit Experience Data Plane .

Figure 3. Distributed deep reinforcement learning architecture.

Category	System	Decoupling	Placement	Scheduling
Built-in Experience Buffers	RLlib ( Liang et al., 2021 )	Embedded module	Fixed (CPU)	Reactive (inline in actor/learner loops)
	MSRL ( Zhu et al., 2023 )		Fixed (CPU/GPU)	Reactive (inline in actor/learner loops)
	SRL ( Mei et al., 2023 )		Fixed (pinned CPU)	Reactive (fixed one-step overlap)
Experience Buffer Services	Reverb ( Cassirer et al., 2021 )	Process-decoupled service	Fixed (CPU)	Reactive (request-driven)
Experience Buffer Services	Gear ( Wang et al., 2023 )	Process-decoupled service	Fixed (pinned CPU & GPU - driven access)	Reactive (request-driven)
Experience Management Layer	Conduit	Multi-dimensional Decoupling (system, algorithm, hardware)	Adaptive (capacity-constrained, bandwidth-aware)	Proactive (schedulable execution timing & processing granularity)

Table 1. Comparison of existing experience buffer implementations for distributed RL.

Panel of Figure 4 — Figure 4. Non-uniform intra-node transfers on an 8-GPU AMD MI250X node. (a) interconnect topology; (b) GPU $\leftrightarrow$ GPU latency variation for a 5 GB transfer, making placement performance-critical.

Figure 5. Operator dependencies ( D1 – D4 are batches): (a) sequential, (b) overlapped via timing , (c) overlapped and batched via joint timing and granularity .

Figure 6. Larger operator batches reduce per-transition processing cost for both ingestion and delivery.

Figure 7. Conduit workflow: the Experience Data Plane exposes three control points— experience ingestion , placement , and delivery —around the Experience Buffer.

Figure 8. Conduit integration interface: frameworks register ingestion/delivery logic via buffer hooks; Conduit controls placement and scheduling.

Panel of Figure 9 — Figure 9. Decoupling ingestion (delivery) from actor (learner) execution. Actors push rollouts asynchronously, while the ingestion operator pulls data at configurable granularity, enabling batching and pipelining (left). The delivery operator prepares training-ready batches ahead of time, freeing learners from synchronous data-preparation waits (right).

Figure 10. Ingestion /delivery-path modes: overlap

\times

granularity across iterations (D1–D4) and mini-batches (D1 1 –D1 4 ).

Figure 11. Conduit ’s per-iteration latency reduction over RLlib : exposed experience-path (left), end-to-end (right). Workloads: MC =MountainCar, CP =MA-CartPole, MW =Meta-World, OXE =Open X-Embodiment

Figure 12. Placement ablation: Conduit vs. four placements ( Reverb =CPU pageable, Gear =CPU pinned with zero-copy access, single GPU, shared GPU). Missing points denote OOM.

Figure 13. Online migration as transition dimension

d

grows, vs. Greedy (single GPU) and Conservative (all-GPU) baselines. Top: iteration latency (log scale); bottom: cumulative latency.

Figure 14. Impact of granularity

g

under varying batch sizes (left) and transition dimensions (middle) for

g

g_{\text{ing}}

g_{\text{del}}

; the right panel shows the impact of execution timing

overlap

under varying transition dimensions.

Figure 15. Auto vs. manual over the joint placement

\times

overlap

\times

granularity space. Gray:

32

manual configs; red star: Conduit ’s auto pick.

Panel of Figure 16 — Figure 16. Strong scaling of Conduit with GPU count on an NVIDIA A100 cluster and an AMD MI250X supercomputer (end-to-end iteration time, log scale).

Panel of Figure 17 — Figure 17. Convergence analysis for Conduit on RLlib.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	End-to-end		Exp-path
Workload	SRL	+ Conduit	SRL	+ Conduit
DQN–Qbert	275.8	191.3 ( $\downarrow$ 31%)	106.6	18.6 ( $\downarrow$ 83%)
PPO–Meta-World	235.9	193.6 ( $\downarrow$ 18%)	63.5	23.2 ( $\downarrow$ 63%)

Appendix

Table 2. Conduit on SRL : per-iteration end-to-end and exposed experience-path latency (ms).

#	Verl stage	Role
1	generate_sequences (rollout)	actor (upstream)
2	old / ref log-prob	EDP	ingestion
3	reward, KL penalty
4	compute_advantage
5	dispatch batch to update workers		delivery
6	update_actor	learner (downstream)

Appendix

Table 3. Conduit mapped onto Verl ’s per-iteration experience path (GRPO): post-generation processing (log-prob, reward, advantage) is ingestion ; the hand-off of the training batch to the update workers is delivery .

	Best $T_{\text{iter}}$ (ms)
$d$	Measured	Predicted	Top- $8$ ranking match
$128$	$17.1$	$20.1$	✓
$256$	$18.3$	$20.8$	✓
$512$	$25.4$	$29.9$	✓
$1{,}000$	$26.4$	$33.9$	✓
$2{,}000$	$30.0$	$35.8$	✓

Appendix

Table 4. Cost-model accuracy per workload dimension

d

of Fig. 15 (

32

configurations each). For every dimension, the predicted ordering of the top-

8

configurations exactly matches the measured ordering.

Distributed RL

Distributed reinforcement learning coordinates agents, learners, and compute to scale policy training across networks and machines.

See all 20 papers

Aug 11, 2026

TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling

Reinforcement learning (RL) for large language models is moving toward multi-turn agentic workloads, where rollout tasks repeatedly pause for external environments, resume with growing contexts, and finish at highly variable times. In this setting, RL training goodput, measured by training throughput, matters more than raw GPU occupancy: GPU waiting and repeated prefill recomputation are pure overhead. We present TideRL, a readiness-aware elastic RL system with Continuous Task Batching, Resource-Aware Ref-Actor Pipelining, and Elastic Resource Scaling. CTB preserves useful rollout state,

\textrm{RA}^2\textrm{P}

selects between decoupled streaming and colocated aggregation from the ready backlog and arrival interval, and ERS moves ranks between rollout and training using the same readiness signals. Across text-only and multi-modal agentic workloads, TideRL improves RL training goodput by up to 5.6

\times

over synchronous baselines and over 33% over asynchronous baselines, while reaching similar task performance. It also improves KV cache hit rate by 1.58

\times

, reduces per-step training time by up to 44.3%, and cuts total waiting time by up to 77.6%.

Yanyu Ren, Xizheng Wang, Xiao Liu +8

Tsinghua University · Zhongguancun Laboratory

Agentic RL Asynchronous RL Reinforcement Learning RL for LLM Agents

Ask PDF

Apr 10, 2026

TensorHub: Scalable and Elastic Weight Transfer for LLM RL Training

Modern LLM reinforcement learning (RL) workloads require a high-performance weight transfer system to scale training across heterogeneous compute resources. However, efficiently transferring terabyte-scale model weights across thousands of GPUs remains challenging because the system must accommodate clusters that dynamically scale up and down while keeping coordination, data movement, and storage overhead low. We introduce Reference-Oriented Storage (ROS), a new storage abstraction for RL weight transfer that exploits highly replicated model weights in place. ROS presents the illusion that certain versions of the model weights are stored and can be fetched on demand. Underneath, ROS does not physically store any copies of the weights; instead, it tracks the workers that hold these weights on GPUs for inference. Upon request, ROS directly uses them to serve reads. We build TensorHub, a production-quality system that instantiates the ROS idea with topology-aware transfer, model-parallel consistency, and fault tolerance. Evaluation shows that TensorHub saturates RDMA bandwidth and adapts to three distinct rollout workloads with minimal engineering effort. Specifically, TensorHub reduces total GPU stall time by up to 6.7x for standalone rollouts, accelerates weight updates for elastic rollouts by up to 4.8x, and cuts cross-datacenter rollout stall time by up to 19x. TensorHub has been deployed in ByteDance production to support cutting-edge RL training.

Chenhao Ye, Huaizheng Zhang, Mingcong Han +11

ByteDance Seed · University of Wisconsin–Madison

RL for Language Models Communication-Efficient Distributed Training GPU-Accelerated RL

Ask PDF

Apr 26, 2026

JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Training

We present JigsawRL, a cost-efficient framework that explores Pipeline Multiplexing as a new dimension of RL parallelism. JigsawRL decomposes each pipeline into a Sub-Stage Graph that exposes the intra-stage and inter-worker imbalance hidden by stage-level systems. On this abstraction, JigsawRL resolves multiplexing interference through dynamic resource allocation, eliminates fragmented utilization by migrating long-tail rollouts across workers, and formulates their coordination as a graph scheduling problem solved with a look-ahead heuristic. On 4-64 H100/A100 GPUs across different agentic RL pipelines and models, JigsawRL achieves up to 1.85x throughput over Verl on synchronous RL, 1.54x over StreamRL and AReaL on asynchronous RL, and supports heterogeneous pipelines with moderate latency trade-off.

Zhengding Hu, Hehua Ouyang, Chang Chen +6

University of California, San Diego

RL Post-Training Pipeline Parallelism RL for LLM Agents

Ask PDF

Abstract

Distributed RL

Explore similar work

TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling

TensorHub: Scalable and Elastic Weight Transfer for LLM RL Training

JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Training