GzDRL: Reproducible and Scalable Deep Reinforcement Learning with Gazebo

Amal Dev Haridevan; Junjie Kang; Jinjun Shan

cs.ROSep 3, 2026

GzDRL: Reproducible and Scalable Deep Reinforcement Learning with Gazebo

Authors: Amal Dev Haridevan, Junjie Kang, Jinjun Shan

Abstract

We present GzDRL, a novel single-process reinforcement learning (RL) framework for Gazebo that overcomes longstanding bottlenecks in scalable, reproducible robotics experimentation. Unlike conventional middleware-based RL-Gazebo integrations that suffer from nondeterminism and irreproducibility, GzDRL introduces a systematic, middleware-free environment-stepping mechanism that directly synchronizes agent actions and physics updates. This design enables deterministic, high-throughput data collection, efficient vectorization, and reproducible RL training and evaluation. Comprehensive benchmarks demonstrate that GzDRL achieves the highest workstation throughput among the evaluated frameworks while remaining competitive with GPU-accelerated simulators on laptop hardware, and maintains precise agent-environment synchronization, multi-agent scalability, and experiment-level reproducibility. We further validate sim-to-real transfer by deploying learned policies directly onto a physical quadrotor, without fine-tuning. Our results establish GzDRL as an accessible and reproducible platform for advancing RL in robotics and automation.

Figures & tables

Fig. 1: Representative GzDRL scenarios, ranging from drone racing to cooperative payload transport.

Fig. 2: Various control hierarchies supported by GzDRL

	Simulation				Robotics Integration		RL Interface
Framework	Physics engine	Rendering engine	Runtime rand.	Sync. stepping	Native ROS	Sensors	Language	Agent support
RotorS [ 15 ]	DART, Bullet, ODE	OGRE 1.0	—	—	✓	IMU, RGB-D	C ++	—
AirSim [ 17 ]	PhysX	Unreal	—	—	Part.	IMU, RGB-D, S	C ++ , Py.	Single, Multi
Flightmare [ 16 ]	Flexible	Unity	✓	✓	—	IMU, RGB-D, S	C ++ , Py.	Single
PyBullet-Drones [ 20 ]	Bullet	OpenGL	—	✓	—	IMU, RGB-D, S	Python	Single, Multi
OmniDrones [ 19 ]	PhysX	Omniverse RTX	✓	✓	Part.	IMU, RGB-D, S, F, C	Python	Single, Multi

TABLE I: Comparison of multirotor simulation frameworks. ✓ denotes native/full support, — no support, and Part. support through an external bridge.

Fig. 3: GzDRL system architecture (top). An EnvPool-derived scheduler dispatches environment-indexed actions to $N_{T}$ threads, which write an observation block after $N_{B}$ completions. Each environment contains a DRLServer to interface with Gazebo for performing resets and executing actions. Internal execution pipeline from action processing to observation generation (bottom). The processing scripts can be provided by the user.

Fig. 4: Transition synchronization metrics over 100 trials: (a) Latency between action and physics update; (b) Observation delay; (c) Intervening physics steps. GzDRL achieves deterministic, minimal-latency transitions, while ROS 2 incurs variable delays.

Fig. 5: Cross-run trajectory reproducibility: (a) Position spread versus action index; (b) Trajectory RMSE distribution over 100 trials. GzDRL maintains coincident trajectories; ROS 2 introduces cumulative dispersion.

Fig. 6: Training reproducibility on the GzDRL hover task. Five independent runs with the same RL seed (left) produce coincident learning curves, while five different seeds (right) yield substantial dispersion. Each checkpoint is evaluated on the same fixed episode set; shaded region denotes $\pm 1$ standard deviation.

Configuration	Final return	CV (%)	Mean ckpt. $\boldsymbol{\sigma}$	Identical ckpts. (%)
Same seed	$18.43\pm 0.00$	0.00	0.00	100
Different seeds	$18.08\pm 1.27$	7.05	1.32	0

TABLE II: Training reproducibility on the GzDRL hover task over five process-isolated runs per configuration. Final return is reported as mean

\pm

standard deviation across runs; all checkpoints are evaluated on the same fixed deterministic episode set.

Panel of Figure 9 — Fig. 7: Backend scalability on workstation (top) and laptop (bottom): (a) environment throughput, (b) CPU utilization, (c) GPU utilization. Markers indicate trial means; error bars show 95% confidence intervals.

	Workstation				Laptop
Architecture	Peak rate	$N_{E}^{\star}$	CPU (%)	GPU (%)	Peak rate	$N_{E}^{\star}$	CPU (%)	GPU (%)
GzDRL (GazeboPool)	$78.58\mathbin{\pm}2.08$	128	78.1	0.0	$43.38\mathbin{\pm}0.56$	32	82.7	0.0
OmniDrones	$21.15\mathbin{\pm}0.22$	128	30.7	18.8	$35.47\mathbin{\pm}0.20$	128	9.6	28.5
Aerial Gym	$21.02\mathbin{\pm}0.27$	128	3.7	20.1	$44.56\mathbin{\pm}0.42$	128	9.0	37.7
PyBullet-Drones	$21.95\mathbin{\pm}1.25$	16	16.3	0.0	$26.92\mathbin{\pm}1.30$	16	49.0	0.0
GzDRL (AsyncDRLServerPool)	$19.10\mathbin{\pm}1.79$	32	20.6	0.0	$24.59\mathbin{\pm}1.37$	32	45.8	0.0

TABLE III: Backend scalability. Peak rate is the maximum mean environment throughput over the tested

N_{E}

values and is reported as mean

\mathbin{\pm}

95% Student’s

t

confidence interval.

N_{E}^{\star}

, CPU, and GPU utilization are evaluated at the same operating point. Rates are in

10^{3}

environment steps/s.

Platform	Architecture	$N_{A}=2$	$N_{A}=5$	$N_{A}=10$	$N_{A}=20$
Workstation	GzDRL (GazeboPool)	35.52	17.82	8.10	3.18
	OmniDrones	4.24	4.19	4.15	4.10
	Aerial Gym	7.15	7.07	6.62	5.53
	PyBullet-Drones	13.89	10.23	6.65	2.69
Laptop	GzDRL (GazeboPool)	32.95	11.54	5.01	2.16
	OmniDrones	7.11	6.94	6.89	6.77

TABLE IV: Multi-agent throughput. Entries are trial-mean environment throughput in

10^{3}

steps/s; uncertainty is reported in the corresponding scalability figure.

Panel of Figure 12 — Fig. 8: Multi-agent scalability on workstation (top) and laptop (bottom): (a) environment throughput, (b) CPU utilization, (c) GPU utilization as the number of interacting UAVs per environment increases.

Fig. 9: Hardware trajectory tracking for nine reference trials. Each column: one flight; rows: three orthogonal projections. Dashed: commanded reference; solid: measured trajectory; color: instantaneous position error. The same frozen policy is used for all trials.

Fig. 10: Learning curves for payload-transport (left) and trajectory-tracking (right) tasks.

Fig. 11: Inverted-pendulum training (left) and evaluation under randomized dynamics (right).

Metric	Hardware result
Mean Euclidean error [cm]	$20.2\pm 3.4$
3D RMS error [cm]	$26.2\pm 7.4$
$\mathrm{RMSE}_{x}$ [cm]	$15.7\pm 6.4$
$\mathrm{RMSE}_{y}$ [cm]	$20.0\pm 5.9$
$\mathrm{RMSE}_{z}$ [cm]	$2.9\pm 0.4$

TABLE V: Sim-to-real trajectory tracking on QDrone2 across nine reference trajectories. Values are mean

\pm

95% confidence intervals.

Explore similar work

GPUSimBench: Towards Scalable and Reliable GPU-Accelerated Simulators in Embodied AI
Jul 6, 2026Huzhenyu Zhang, Shenghai Yuan, Wenrui Yan +4GPU Acceleration Robotics Simulation
Shanghai AI Laboratory, China · MIRAI, Russia · Nanyang Technological University, Singapore +2
A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control
Oct 8, 2026Octi Zhang, Mateo Guaman Castro, Patrick Yin +4Robotic RL Data-Efficient Robot Learning
University of Washington · NVIDIA
Learning Reach-Avoid Task with Reinforcement Learning: Vectorized Simulation and Benchmark
Jul 17, 2026Jonas Weihing, Shahram EivaziRobotics Simulation Robotic RL
*Tübingen university

Jul 6, 2026, cs.ROJ/K move · Enter open · S save

GPUSimBench: Towards Scalable and Reliable GPU-Accelerated Simulators in Embodied AI

Huzhenyu Zhang, Shenghai Yuan, Wenrui Yan, Li Ma+3

Shanghai AI Laboratory, China · MIRAI, Russia · Nanyang Technological University, Singapore+2

GPU Acceleration Robotics Simulation ML Reproducibility Physics-Based Simulation

Open paper AskPDF arXiv

Data-driven embodied AI is rapidly transitioning into a paradigm that scales training through massively parallel simulation, where GPU-accelerated simulators serve as the foundational data infrastructure. However, as computational throughput scales, the underlying trade-offs between parallel efficiency, physical fidelity, and execution determinism remain largely unexamined, hindering the development of reliable robot learning. In this paper, we expose the hidden limits of mainstream GPU-based robotic simulators (e.g., Isaac Lab, Genesis) by introducing GPUSimBench, which focuses on scalability, physical consistency, and computational determinism. First, GPUSimBench establishes a physical grounding evaluation with a controlled inclined-plane task, quantifying the distributional alignment between simulated dynamics and their real-world counterparts. Second, we benchmark parallel scalability by measuring throughput and memory footprints across scaling environment counts. Crucially, beyond standard performance metrics, we unveil and quantify the inherent non-determinism introduced by GPU-batched execution, characterized by significant run-to-run and inter-environment variability even under identical initial conditions. Finally, we identify four empirical regimes of stochasticity within current simulator stacks, highlighting that unbounded scaling can compromise reproducibility without explicit constraints.

Abstract

Explore similar work

GPUSimBench: Towards Scalable and Reliable GPU-Accelerated Simulators in Embodied AI

A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control

Learning Reach-Avoid Task with Reinforcement Learning: Vectorized Simulation and Benchmark