RobustRLlib: A Unified Library and Benchmark for Robust Reinforcement Learning Algorithms

Xi Chen; Fengqi Hu; Zeyu Shen; Jiaying Xu; Shangding Gu; Pan Xu; Anqi Liu; Laixi Shi

cs.LGOct 23, 2026

RobustRLlib: A Unified Library and Benchmark for Robust Reinforcement Learning Algorithms

Authors: Xi Chen, Fengqi Hu, Zeyu Shen, Jiaying Xu, Shangding Gu, Pan Xu, Anqi Liu, Laixi Shi

Organizations: Department of Computer Science, Johns Hopkins University, Baltimore, MD 21218, USA. · Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD 21218, USA. · University of California San Diego, La Jolla, CA 92093, USA. · Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, Berkeley, CA 94720, USA. ·

Abstract

Robust reinforcement learning (RL) is critical for reliable real-world deployment, where policies may fail under diverse shifts and disruptions. Existing robust RL algorithms offer a rich set of mechanisms and principles, yet two major barriers limit their broader potential: limited algorithm reusability and an unclear empirical capability boundary. We introduce RobustRLlib, an algorithm-centric library and benchmark that addresses both challenges. It integrates 17 representative robust online, offline, and safe RL algorithms under a unified interface for substantially improved reusability, together with an enhanced evaluation platform spanning diverse real-world shifts and advanced humanoid and vision-language-action tasks. Systematic evaluation over 273 perturbation configurations provides an up-to-date capability map of current robust RL algorithms, revealing where they remain effective and where substantial robustness gaps persist. As an open-source, user-friendly platform, RobustRLlib offers a reproducible and extensible codebase for applying existing methods to new problems and developing new robust RL algorithms.

Figures & tables

Problem	Algorithm feature	Method	Target	Base	Shifted-env	Adversarial	Learned
		Method	shift	algorithm	rollout	network	model
Robust Offline	Learner-centric	RFQI	Dynamic	FQI
		RORL	Observation	SAC
		ATLA	Observation	IQL		✓
		RSC-IQL	Semantics	IQL			✓
	Data-centric	RAMBO	Dynamic	SAC		✓	✓

Table 1: Structural comparison of the robust methods in the library. Online methods are grouped as learner-centric or environment-centric, offline methods as learner-centric or data-centric; generative model is a data-centric subfamily, and robust safe methods form a separate family. Shifted-env rollout marks rollouts collected from a deliberately perturbed physical environment. Adversarial network marks a learned adversary optimized against the policy. Learned model marks a trained next-state predictor used by the method. Target shift records the intended shift source. A checkmark denotes a present feature; a blank denotes its absence.

Panel of Figure 2 — Figure 2: Shift sources covered in RobustRLlib . (a) Shifts of the MDP elements: dynamic, observation, action, and reward. (b) Latency between the policy’s and the executed time stamp. (c) Semantic shift in appearance and goal displacement.

Figure 3: Performance evaluation of 20 robust RL algorithms in the RobustRLlib library under unified perturbation configurations. Every colored bar is one of the fourteen robust methods, keyed by algorithm feature and online/offline setting. Scores average the perturbation configurations within each shift factor, factors within each task, and then tasks with equal weight. The aggregation covers 20 algorithms across 220 distinct perturbation configurations.

Figure 4: Robust RL gains across shift levels and algorithm features. Left : Aggregated performance of standard and robust RL algorithms across diverse environment shift levels Q1–Q4 (Refer to Appendix B.2 for more details). Right : performance gains over respective base algorithms (dark) and the average of standard RL (light) across diverse algorithm designs. See Appendix G.1 for details.

Figure 5: Evaluation under diverse individual shift scenarios. The figure aggregates 14 robust and 2 standard RL methods across 44 distinct perturbation configurations. Semantic binding is evaluated only online, on the goal-conditioned causal door task. See Appendix G.4 for full results.

Figure 8: Performance evaluation on Isaac Lab and VLA tasks. Left: each point is one algorithm, plotting task performance against success rate on two tasks (G1: survival under action delay; Drawer: episodes within the joint-limit cost budget). See Appendices B.2 and G.3 for the detailed setting and results. Right: performance retention of a VLA policy as eight shift factors moves from Q1 to Q4. (Q1-Q4 definitions see Appendix B.2 ). The diamond marks each factor’s operating point measured on the real platform.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark		Shifts				Task expansion
	Robust algorithms	MDP	Latency	Semantic	Compound
RLlib ( Liang et al., 2018 )	0	0/4	✗	✗	✗	✓
RRLS ( Zouitine et al., 2024 )	4	2/4	✗	✗	❍	✓
ODRL ( Lyu et al., 2024 )	0	1/4	✗	✗	✗	✗
Robust-Gymnasium ( Gu et al., 2025 )	4	4/4	✗	✓	✓	✓
RWRL Suite ( Dulac-Arnold et al., 2020 )	0	3/4	✓	✗	✓	❍

Appendix

Table 2: Comparison with RL libraries and robustness benchmarks. Robust algorithms counts robust and robust safe RL algorithms evaluated by each benchmark. A checkmark denotes explicit, evaluated support; ❍ denotes partial support; ✗ indicates that the capability is absent or not demonstrated in the cited release. MDP counts, out of four, the MDP components on which shift is evaluated: observations, actions, transitions, and reward/cost signals. Compound requires multiple shift sources to be active jointly. Task expansion indicates an extensible interface for carrying the benchmark to additional tasks.

Source	Operator	Intervention	Magnitude and resampling
Stochastic
Observation, action, reward/cost	gauss , uniform	Add componentwise signal noise	Standard deviation or sampling range; per step
Observation	relative	Scale noise by each feature’s reference spread, $\sigma_{i}=c\hat{\sigma}_{i}$	Relative scale $c$ ; per step
Observation	bias	Draw and hold a sensor offset	Offset range; per episode
Dynamics	gauss , uniform , loguniform	Sample physical parameters or their multipliers	Parameter range; per reset
Latency	buffer , interp	Delay a buffered command or interpolate timestamped observations	Control steps or milliseconds; configured refresh

Appendix

Table 3: Shift operator index. The first three families construct interventions; non-stationary and composition determine how interventions evolve and combine. Each row gives the compatible source, mode, intervention, and magnitude or resampling rule.

Environment	Shift factor	Unit	$Q1$	$Q2$	$Q3$	$Q4$
Online
Hopper-v5	Gravity	$\times$	0.9, 1.1	0.8, 1.2	0.7, 1.3	0.6, 1.4
	Morphology	$\times$	0.9, 1.1	0.8, 1.2	0.7, 1.3	0.6, 1.4
	Actuator gear	$\times$	0.9, 1.1	0.8, 1.2	0.7, 1.3	0.6, 1.4
HalfCheetah-v5	Gravity	$\times$	0.9, 1.1	0.8, 1.2	0.7, 1.3	0.6, 1.4
	Morphology	$\times$	0.9, 1.1	0.8, 1.2	0.7, 1.3	0.6, 1.4

Appendix

Table 4: The 220 perturbation configurations and the shift levels. Each row is one shift factor on one environment; the four cells list every perturbation configuration evaluated on that factor, grouped by shift levels.

Q

indexes displacement from nominal. Adversarial-action is discrete attack modes with no scalar distance from nominal.

Shift source	Perturbation	Mode	Setting
Observation	obs_rel_0.10	stochastic	Gaussian at $0.10\,\sigma_{i}$ , $\sigma_{i}$ the dimension’s own std
	obs_uni_0.10	stochastic	$\mathcal{U}[-0.10,\,0.10]$
	obs_gauss_0.10	stochastic	$\sigma=0.10$
	mad_0.02	adversarial	MAD gradient attack, $\epsilon=0.02$
	mad_0.05	adversarial	MAD gradient attack, $\epsilon=0.05$
Action	act_uni_0.10	stochastic	$\mathcal{U}[-0.10,\,0.10]$ on the clipped action

Appendix

Table 5: 44 individual shift perturbation configuration. The 22 fixed sets on Hopper-v5 are instantiated separately for online and offline evaluation, composing 44 perturbation configurations used to compute the scores in Section 4.2 , and the In-training perturbation means the shift acts during learning, the evaluation is nominal. A control step is

8

ms ( frame_skip

4\times 2

ms).

Shift source	Shift factor	Resampling rule	Setting
Hopper-v5 $\leftarrow$ OmniH2O ( He et al., 2024 )
Dynamics	Foot/link friction	log-uniform, per episode	$\times[0.38,1.06]$
Dynamics	Link mass	log-uniform, per episode	$\times[0.81,1.17]$
Dynamics	Actuator gear	log-uniform, per episode	$\times[0.84,1.14]$
Dynamics	Torso push	onset phase per episode, sign per push	$119$ N, every $625$ steps, $10$ steps
Action	Torque noise	Gaussian, per step, pre-clip	$\sigma=0.06$

Appendix

Table 6: Reference-derived compound shifts scenario configuration. Every factor of a profile is applied jointly in one mode; this sheet lists the reference system’s own factor settings.

Task	Metric	$R_{\min}$	$R_{\max}$	Source
Hopper-v5	episodic return	$-20.27$	$3234.3$	D4RL v2 random / expert
HalfCheetah-v5	episodic return	$-280.18$	$12135.0$	D4RL v2 random / expert
Walker2d-v5	episodic return	$1.63$	$4592.3$	D4RL v2 random / expert
AdroitHandDoor-v1	episodic return	$-56.51$	$2880.57$	D4RL v2 random / expert
PointMaze_Open-v3	goals per 300 steps	$0.0$	$3.28$	no goal/ IQL nominal
LunarLanderContinuous-v3	episodic return	$-213.9$	$136.9$	random policy / IQL nominal

Appendix

Table 7: Normalization anchors

(R_{\min},R_{\max})

. The pairs entering equation 2 , fixed per environment and shared by every algorithm, dataset quality, and perturbation configurations. Source names the construction behind each pair: D4RL v2 where a published pair exists, and otherwise a random-policy floor with either a task-defined or a reference-learner ceiling.

Shift factor	Training setting	Evaluation setting
Contact friction	$\mathcal{U}[0.2,1.1]$	$0.1$
Link mass	$\times\,\mathcal{U}[0.7,1.3]$	$\times\,1.5$
Base CoM, $x$ and $y$	$\pm 0.1$ m	$+0.15$ m along $x$
Torque noise	$\pm 10\%$ of the effort limit	$20\%$
Action delay	$1$ – $3$ policy steps ( $20$ – $60$ ms)	$4$ steps ( $80$ ms)
Actuator $K_{p}$ , $K_{d}$	$\times\,\mathcal{U}[0.75,1.25]$	boundary only, $0.75$ and $1.25$

Appendix

Table 8: G1: training and evaluation configuration.

Shift factor	Training setting	Evaluation setting
Robot friction	static and dynamic $\mathcal{U}[0.8,1.25]$	both $0.4$ ; both $1.8$
Handle friction	static $\mathcal{U}[1.0,1.25]$ , dynamic $\mathcal{U}[1.25,1.5]$	both $0.5$ ; both $2.0$
Cabinet mass	$\times\,\mathcal{U}[0.8,1.2]$	$\times\,0.6$ ; $\times\,1.4$
Arm $K_{p}$	$\times\,\mathcal{U}[0.9,1.1]$	$\times\,0.75$ ; $\times\,1.25$
Arm $K_{d}$	$\times\,\mathcal{U}[0.9,1.1]$	$\times\,0.75$ ; $\times\,1.25$
Drawer damping	$\times\,\mathcal{U}[0.5,1.5]$	$\times\,0.25$ ; $\times\,2.0$

Appendix

Table 9: Drawer: training and evaluation configuration.

Figure 9: VLA evaluation grid. Each row is one shift factor, coloured by its shift source, with the nominal value on the left and the setting of each level above its circle. Q1–Q4 order the levels from nearest to farthest from nominal; the three dynamics shift have three levels. The diamond marks the operating point measured on the real robot: it sits at Q2, so Q1 is milder than the hardware and Q3–Q4 go beyond it.

Task	Observation	Action (rate)	Horizon	Task type
Standard control
Hopper-v5	11-D state	3-D	1,000	Locomotion
Walker2d-v5	17-D state	6-D	1,000	Locomotion
HalfCheetah-v5	17-D state	6-D	1,000	Locomotion
AdroitHandDoor-v1	39-D state	28-D	200	Manipulation
FetchReachDense-v3	13-D proprioception and relative goal	4-D	50	Manipulation

Appendix

Table 10: Benchmark tasks. Observation and Action specify the default policy interface; control rates are shown in parentheses where reported. Numeric horizons are maximum environment steps per episode, while Demo follows the length of the recorded demonstration used for VLA evaluation.

Task	Dataset	Source or collection
Hopper, Walker2d, HalfCheetah	Minari medium-v0	Collected in Gymnasium v5 environments
AdroitHandDoor	D4RL/door/expert-v2	Minari port of D4RL trajectories
PointMaze_Open	D4RL/pointmaze/open-v3	Minari port of D4RL trajectories
LunarLanderContinuous	Collected for this benchmark	Deterministic SAC checkpoints trained for $1.5$ million steps; selected medium corpus of $1{,}741$ episodes

Appendix

Table 11: Offline datasets. Each dataset contains approximately one million transitions.

Panel of Figure 10 — Figure 10: Isaac Lab tasks. Left : Humanoid G1 locomotion task; right : Franka drawer manipulation task.

Figure 11: VLA task. Bimanual insert-mouse-battery demonstrations of the RSS 2026 post-training challenge ( Zhang et al., 2026 ) .

Figure 12: Online training curves.

Task	Method	Env. steps	Wall-clock time (h)
Walker2d-v5	PPO	1M	$0.40\pm 0.15$
	PPO	2M	$0.80\pm 0.29$
	SAC	1M	$1.60\pm 0.03$

Appendix

Table 12: Walker2d training wall-clock time on H100 GPUs. Values are mean

\pm

sample standard deviation over three training seeds. Wall-clock time is measured from the start of training to the corresponding checkpoint save.

Budget rule	Methods	Steps
SAC-based	SAC, DR-SAC, RSC-SAC	1M
PPO/TRPO-based	PPO, RARL-PPO, RARL-TRPO	2M
Official ATLA configuration	ATLA, ATLA-SA	$\sim$ 5M

Appendix

Table 13: Online training budget summary.

Algorithm	Feature	Base algorithm	Budget
Online
SAC	Standard	—	1M steps
PPO	Standard	—	2M steps
ATLA	Learner-centric	PPO	$2441{\times}2048$ steps
ATLA-SA	Learner-centric	PPO (LSTM)	$2441{\times}2048$ steps
RSC-SAC	Learner-centric	SAC	1M steps

Appendix

Table 14: Training budget. Budgets are environment steps for online methods and policy/critic gradient updates for offline methods. Training budget alignment explanation is in Section E.2 .

Task	PPO, DR-PPO, RAMU-PPO	ATLA-PPO	SPiDR-PPO	$B$
G1	$1500\times 4096\times 24$	$750\times 2\times 4096\times 24$	$900\times 512\times 320$	$147{,}456{,}000$
Drawer	$3200\times 512\times 96$	$1600\times 2\times 512\times 96$	$3200\times 512\times 96$	$157{,}286{,}400$

Appendix

Table 15: Training budgets for Isaac Lab tasks. Iterations are chosen so that the retained primary training interactions

B

match within each task.

Algorithm feature	Paired gain	Pooled gain	$n$ paired
Learner-centric (Online)	$+5.5$ $[+2.8,+8.3]$	$+9.5$ $[+7.2,+11.8]$	3
Environment-centric	$+6.6$ $[+3.8,+9.6]$	$+9.6$ $[+7.3,+11.8]$	2
Learner-centric (Offline)	$+3.8$ $[+0.8,+6.4]$	$+2.9$ $[+1.3,+4.4]$	1
Data-centric	$-0.1$ $[-2.7,+2.1]$	$+7.6$ $[+4.2,+11.1]$	1
Generative model	$+2.5$ $[-0.3,+4.9]$	$+9.8$ $[+7.9,+11.7]$	3

Appendix

Table 41: The two columns of the algorithm feature panel, with intervals. Point estimates and

95\%

seed-level stratified bootstrap intervals from

10{,}000

replicates. Paired subtracts each member’s own base algorithm; pooled subtracts the mean of all six Standard RL references.

n

is the number of members that have a matched base learner and therefore enter the paired column.

Figure 13: On-policy and off-policy performance aggregated by task.

Figure 14: Isaac Lab results. Each panel repeats Figure 8 on one factor: safety success rate against performance, one point per algorithm, bars the s.d. over three training seeds.

Figure 15: Supplementary online results of Section 4.1 .

Figure 16: Supplementary online results of Section 4.1 .

Figure 17: Supplementary offline results of Section 4.1 .

Figure 18: Supplementary offline results of Section 4.1 .

Figure 19: Supplementary online results of Section 4.2 .

Figure 20: Supplementary offline results of Section 4.2 .

Figure 21: Supplementary results of Section 4.2 .

Robust RL

Robust reinforcement learning develops agents that maintain performance and safety under uncertainty, disturbances, attacks, and distribution shifts.

See all 228 papers

Jun 29, 2026

RoAd-RL: A Unified Library and Benchmark for Robust Adversarial Reinforcement Learning

Deep Reinforcement Learning (DRL) has achieved significant success in robotics and autonomous systems, yet remains vulnerable to adversarial perturbations that can severely degrade performance. Research in adversarial reinforcement learning is often limited by fragmented implementations, inconsistent evaluation protocols, and poor reproducibility. To address these challenges, we present RoAd-RL, an open-source benchmarking framework that provides unified abstractions for policies, attacks, defenses, and robustness metrics, together with reproducible evaluation pipelines and seamless integration with Stable-Baselines3 and Gymnasium. We evaluate DQN, PPO, and SAC agents in LunarLander and Highway-v0 under 192 attack-defense configurations. Results reveal substantial variations in robustness across environments and show that some commonly used defenses can be more detrimental than the attacks they aim to mitigate, while temporal smoothing consistently achieves strong performance. RoAd-RL establishes a standardized benchmark for adversarial reinforcement learning research and is publicly available at https://pypi.org/project/road-rl.

Adithya Mohan, Daniel Kriegl, Torsten Schön

AImotion Bavaria · Technische Hochschule Ingolstadt · Ingolstadt, Germany

RL Benchmarks Reinforcement Learning Adversarial Robustness Adversarial MARL Robust RL

Ask PDF

Oct 17, 2026

LIBERO-Pro2.0: A Benchmark for Evaluating the Robustness of Embodied Foundation Models

LIBERO has emerged as a widely adopted benchmark for evaluating robotic manipulation policies. However, strong performance on LIBERO does not necessarily indicate robust embodied capabilities, as models may exploit benchmark-specific regularities rather than generalize to diverse operating conditions. In this work, we introduce LIBERO-Pro2.0, a comprehensive benchmark for systematically evaluating the robustness of embodied foundation models under distribution shifts. LIBERO-Pro2.0 encompasses 22 static perturbations and 20 runtime interventions across six dimensions: Condition, Environment, Observation, Execution, Robot State Configuration, and Language, yielding 1,648 validated shifted task instances. To support robustness-oriented learning, we further collect 39,390 training trajectories through model-generated and human demonstrations. All shifted task instances undergo six rounds of iterative human review and correction to ensure physical plausibility and task validity. We evaluate 17 representative robot policies spanning vision-language-action models, world-action models, and robustness-oriented policies. Our results reveal substantial robustness gaps across all evaluated paradigms: under runtime distribution shifts, the evaluated models exhibit an average performance drop of 26.5 percentage points, with individual drops reaching 32.5 points. These findings highlight the gap between nominal benchmark performance and robustness under distribution shifts, underscoring the need for systematic robustness evaluation of embodied foundation models.

Xueyang Zhou, Yangming Xu, Qianjiang Li +10

Huazhong University of Science and Technology · Tsinghua University · University of Toronto +2

Distribution Shift Robustness Embodied Agent Benchmarks Robot Manipulation Benchmarks Robustness of VLA Models Robot Manipulation Datasets

Ask PDF

Aug 4, 2026

Robust General Utility for Reinforcement Learning

Reinforcement learning (RL) with general utility extends classic RL by optimizing an arbitrary utility functional of the policy-induced occupancy measure, thereby enabling a broader range of applications. However, previous work on general utility RL typically assumes the evaluation utility is fixed and correctly specified. In practice, the utility used at deployment can deviate from the training one, creating a robustness gap that prior work does not address. Motivated by this, we propose robust general-utility RL, a minimax learning framework that trains policies against utility misspecification within a prescribed uncertainty set. Our framework strictly generalizes standard general-utility RL while also providing a unified view of many existing RL frameworks, including reward-robust RL and constrained RL, through appropriate choices of the utility uncertainty set. We further develop provably convergent stochastic algorithms for two regimes. For concave utilities, we develop a projected stochastic gradient descent-ascent method and establish stationarity guarantees. For the more challenging nonconcave regime, we propose a stochastic prox-extragradient algorithm that mitigates ill-posed behavior induced by nonconcavity, with convergence guarantees to approximate first-order stationarity. Experiments on LLM safety alignment and exploration maximization tasks further corroborate the convergence behavior consistent with our theory.

Zixuan Liu, Fangzheng Wu, Brian Summa +1

Department of Computer Science, Tulane University, New Orleans, LA, 70118, USA

Stochastic Optimization Reinforcement Learning Robust RL

Ask PDF

Abstract

Robust RL

Explore similar work

RoAd-RL: A Unified Library and Benchmark for Robust Adversarial Reinforcement Learning

LIBERO-Pro2.0: A Benchmark for Evaluating the Robustness of Embodied Foundation Models

Robust General Utility for Reinforcement Learning