Conduit: An Experience Data Plane for Distributed Reinforcement Learning

Sitong Zhang; Tuo Shi; Mario Di Francesco; Zeke Wang; Bo Zhao

cs.DCSep 21, 2026

Conduit: An Experience Data Plane for Distributed Reinforcement Learning

Authors: Sitong Zhang, Tuo Shi, Mario Di Francesco, Zeke Wang, Bo Zhao

Organizations: Aalto University · Shenzhen University of Advanced Technology · Zhejiang University

Abstract

Distributed reinforcement learning (RL) scales training by parallelizing actors and learners around an Experience Buffer. As RL workloads grow, however, the buffer becomes more than a replay queue: it is the storage substrate of a large-capacity, latency-critical experience path that every iteration traverses to move, transform, sample, and batch experiences before learner updates can begin. Existing RL systems embed this path inside framework control flow or expose it as a request-driven buffer service, leaving experience placement fixed and experience-path work difficult to schedule independently as a runtime-level optimization target. We present Conduit, a framework-agnostic runtime that exposes RL experience management as an explicit systems optimization problem. At its core is the Experience Data Plane (EDP), a runtime abstraction that separates RL experience-handling semantics from framework-specific execution logic by exposing experience ingestion, experience placement, and experience delivery as explicit control points. Built on EDP, Conduit introduces capacity-constrained, bandwidth-aware placement, which distributes experience state across CPU/GPU memory tiers and nodes under heterogeneous interconnect and device-memory constraints, and latency-aware scheduling, which controls when experience-path handling runs to reduce exposed experience-path latency while preserving RL semantics. Integrated with RLlib without changing its framework execution logic, Conduit reduces exposed experience-path latency by up to 97% and end-to-end iteration latency by up to 38%, scales to 1,024 GPUs, and preserves convergence.

Figures & tables

Figure 1. Experience-path latency is a major component of iteration time in distributed RL: 50.2% in off-policy Deep Q-Network (DQN) and 26.2% in on-policy proximal policy optimization (PPO) across representative workloads on an 8

\times

A100 cluster.

Figure 2. Experience-path handling drives latency through transfer and processing operations. Existing systems embed this handling in their actor/learner workflow; Conduit instead abstracts it into an explicit Experience Data Plane .

Figure 3. Distributed deep reinforcement learning architecture.

Category	System	Decoupling	Placement	Scheduling
Built-in Experience Buffers	RLlib ( Liang et al., 2021 )	Embedded module	Fixed (CPU)	Reactive (inline in actor/learner loops)
	MSRL ( Zhu et al., 2023 )		Fixed (CPU/GPU)	Reactive (inline in actor/learner loops)
	SRL ( Mei et al., 2023 )		Fixed (pinned CPU)	Reactive (fixed one-step overlap)
Experience Buffer Services	Reverb ( Cassirer et al., 2021 )	Process-decoupled service	Fixed (CPU)	Reactive (request-driven)
Experience Buffer Services	Gear ( Wang et al., 2023 )	Process-decoupled service	Fixed (pinned CPU & GPU - driven access)	Reactive (request-driven)
Experience Management Layer	Conduit	Multi-dimensional Decoupling (system, algorithm, hardware)	Adaptive (capacity-constrained, bandwidth-aware)	Proactive (schedulable execution timing & processing granularity)

Table 1. Comparison of existing experience buffer implementations for distributed RL.

Panel of Figure 4 — Figure 4. Non-uniform intra-node transfers on an 8-GPU AMD MI250X node. (a) interconnect topology; (b) GPU $\leftrightarrow$ GPU latency variation for a 5 GB transfer, making placement performance-critical.

Figure 5. Operator dependencies ( D1 – D4 are batches): (a) sequential, (b) overlapped via timing , (c) overlapped and batched via joint timing and granularity .

Figure 6. Larger operator batches reduce per-transition processing cost for both ingestion and delivery.

Figure 7. Conduit workflow: the Experience Data Plane exposes three control points— experience ingestion , placement , and delivery —around the Experience Buffer.

Figure 8. Conduit integration interface: frameworks register ingestion/delivery logic via buffer hooks; Conduit controls placement and scheduling.

Panel of Figure 9 — Figure 9. Decoupling ingestion (delivery) from actor (learner) execution. Actors push rollouts asynchronously, while the ingestion operator pulls data at configurable granularity, enabling batching and pipelining (left). The delivery operator prepares training-ready batches ahead of time, freeing learners from synchronous data-preparation waits (right).

Figure 10. Ingestion /delivery-path modes: overlap

\times

granularity across iterations (D1–D4) and mini-batches (D1 1 –D1 4 ).

Figure 11. Conduit ’s per-iteration latency reduction over RLlib : exposed experience-path (left), end-to-end (right). Workloads: MC =MountainCar, CP =MA-CartPole, MW =Meta-World, OXE =Open X-Embodiment

Figure 12. Placement ablation: Conduit vs. four placements ( Reverb =CPU pageable, Gear =CPU pinned with zero-copy access, single GPU, shared GPU). Missing points denote OOM.

Figure 13. Online migration as transition dimension

d

grows, vs. Greedy (single GPU) and Conservative (all-GPU) baselines. Top: iteration latency (log scale); bottom: cumulative latency.

Figure 14. Impact of granularity

g

under varying batch sizes (left) and transition dimensions (middle) for

g

g_{\text{ing}}

g_{\text{del}}

; the right panel shows the impact of execution timing

overlap

under varying transition dimensions.

Figure 15. Auto vs. manual over the joint placement

\times

overlap

\times

granularity space. Gray:

32

manual configs; red star: Conduit ’s auto pick.

Panel of Figure 16 — Figure 16. Strong scaling of Conduit with GPU count on an NVIDIA A100 cluster and an AMD MI250X supercomputer (end-to-end iteration time, log scale).

Panel of Figure 17 — Figure 17. Convergence analysis for Conduit on RLlib.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	End-to-end		Exp-path
Workload	SRL	+ Conduit	SRL	+ Conduit
DQN–Qbert	275.8	191.3 ( $\downarrow$ 31%)	106.6	18.6 ( $\downarrow$ 83%)
PPO–Meta-World	235.9	193.6 ( $\downarrow$ 18%)	63.5	23.2 ( $\downarrow$ 63%)

Appendix

Table 2. Conduit on SRL : per-iteration end-to-end and exposed experience-path latency (ms).

#	Verl stage	Role
1	generate_sequences (rollout)	actor (upstream)
2	old / ref log-prob	EDP	ingestion
3	reward, KL penalty
4	compute_advantage
5	dispatch batch to update workers		delivery
6	update_actor	learner (downstream)

Appendix

Table 3. Conduit mapped onto Verl ’s per-iteration experience path (GRPO): post-generation processing (log-prob, reward, advantage) is ingestion ; the hand-off of the training batch to the update workers is delivery .

	Best $T_{\text{iter}}$ (ms)
$d$	Measured	Predicted	Top- $8$ ranking match
$128$	$17.1$	$20.1$	✓
$256$	$18.3$	$20.8$	✓
$512$	$25.4$	$29.9$	✓
$1{,}000$	$26.4$	$33.9$	✓
$2{,}000$	$30.0$	$35.8$	✓

Appendix

Table 4. Cost-model accuracy per workload dimension

d

of Fig. 15 (

32

configurations each). For every dimension, the predicted ordering of the top-

8

configurations exactly matches the measured ordering.

Distributed RL

Distributed reinforcement learning coordinates agents, learners, and compute to scale policy training across networks and machines.

See all 20 papers

TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling
Aug 11, 2026Yanyu Ren, Xizheng Wang, Xiao Liu +8Agentic RL Asynchronous RL
Tsinghua University · Zhongguancun Laboratory
TensorHub: Scalable and Elastic Weight Transfer for LLM RL Training
Apr 10, 2026Chenhao Ye, Huaizheng Zhang, Mingcong Han +11RL for Language Models Communication-Efficient Distributed Training
ByteDance Seed · University of Wisconsin–Madison
JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Training
Apr 26, 2026Zhengding Hu, Hehua Ouyang, Chang Chen +6RL Post-Training Pipeline Parallelism
University of California, San Diego

Aug 11, 2026J/K move · Enter open · S save

TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling

Yanyu Ren, Xizheng Wang, Xiao Liu, Bowen Lv+7

Tsinghua University · Zhongguancun Laboratory

Agentic RL Asynchronous RL Reinforcement Learning RL for LLM Agents

Open paper AskPDF arXiv

Reinforcement learning (RL) for large language models is moving toward multi-turn agentic workloads, where rollout tasks repeatedly pause for external environments, resume with growing contexts, and finish at highly variable times. In this setting, RL training goodput, measured by training throughput, matters more than raw GPU occupancy: GPU waiting and repeated prefill recomputation are pure overhead. We present TideRL, a readiness-aware elastic RL system with Continuous Task Batching, Resource-Aware Ref-Actor Pipelining, and Elastic Resource Scaling. CTB preserves useful rollout state,

\textrm{RA}^2\textrm{P}

selects between decoupled streaming and colocated aggregation from the ready backlog and arrival interval, and ERS moves ranks between rollout and training using the same readiness signals. Across text-only and multi-modal agentic workloads, TideRL improves RL training goodput by up to 5.6

\times

over synchronous baselines and over 33% over asynchronous baselines, while reaching similar task performance. It also improves KV cache hit rate by 1.58

\times

, reduces per-step training time by up to 44.3%, and cuts total waiting time by up to 77.6%.

Abstract

Distributed RL

Explore similar work

TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling

TensorHub: Scalable and Elastic Weight Transfer for LLM RL Training

JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Training