Fuz-RL: A Fuzzy-Guided Robust Framework for Safe Reinforcement Learning under Uncertainty

Xu Wan; Chao Yang; Cheng Yang; Jie Song; Mingyang Sun

cs.LGFeb 24, 2026

Fuz-RL: A Fuzzy-Guided Robust Framework for Safe Reinforcement Learning under Uncertainty

Authors: Xu Wan, Chao Yang, Cheng Yang, Jie Song, Mingyang Sun

Organizations: Zhejiang University · Alibaba DAMO Academy · Peking University

Abstract

Safe Reinforcement Learning (RL) is crucial for achieving high performance while ensuring safety in real-world applications. However, the complex interplay of multiple uncertainty sources in real environments poses significant challenges for interpretable risk assessment and robust decision-making. To address these challenges, we propose Fuz-RL, a fuzzy measure-guided robust framework for safe RL. Specifically, our framework develops a novel fuzzy Bellman operator for estimating robust value functions using Choquet integrals. Theoretically, we prove that solving the Fuz-RL problem (in Constrained Markov Decision Process (CMDP) form) is equivalent to solving distributionally robust safe RL problems (in robust CMDP form), effectively avoiding min-max optimization. Empirical analyses on safe-control-gym and safety-gymnasium scenarios demonstrate that Fuz-RL effectively integrates with existing safe RL baselines in a model-free manner, significantly improving both safety and control performance under various types of uncertainties in observation, action, and dynamics.

Figures & tables

Figure 1: Training Dynamics of PPOLag and Fuz-PPOLag under multi-source uncertainty on Safety-Gymnasium tasks. The perturbation intensity during training is set to $\varepsilon=0.5$ .

Figure 2: Test Comparison of PPOLag and Fuz-PPOLag under multi-source uncertainty setting over 5 Episodes and 5 seeds on Safety-Gymnasium tasks. The cost_limit is set to 0.1.

Tasks	Methods	Observation Uncertainty		Action Uncertainty		Dynamics Uncertainty
Tasks	Methods	AvgRet $\uparrow$	AvgRisk $\downarrow$	AvgRet $\uparrow$	AvgRisk $\downarrow$	AvgRet $\uparrow$	AvgRisk $\downarrow$
Cartpole Stab	PPOL	45 ± 28	0.40 ± 0.14	99 ± 19	0.27 ± 0.14	87 ± 16	0.27 ± 0.09
	CUP	32 ± 26	0.29 ± 0.11	42 ± 18	0.30 ± 0.13	50 ± 14	0.30 ± 0.13
	CPPO	41 ± 19	0.47 ± 0.10	76 ± 19	0.36 ± 0.14	77 ± 14	0.31 ± 0.11
	RAMU	35 ± 22	0.49 ± 0.16	86 ± 10	0.28 ± 0.07	86 ± 13	0.20 ± 0.01
	Fuz-PPOL	47 ± 25 $\uparrow$ 2	0.34 ± 0.11 $\downarrow$ 0.06	102 ± 17 $\uparrow$ 3	0.24 ± 0.12 $\downarrow$ 0.03	93 ± 13 $\uparrow$ 6	0.22 ± 0.08 $\downarrow$ 0.05

Table 1: Detailed evaluation of Safe RL, Fuz-RL, and RAMU on Safe-Control-Gym tasks with observation, action, dynamics uncertainty. Each value is reported as mean ± standard deviation for 10 episodes and 10 seeds. We shadow the highest AvgRet and lowest AvgRisk for each task.

Figure 3: Ablation study of the uncertainty level $K$ .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Schematics, state and input vectors of the cart-pole, and the 1D and 2D quadrotor environments in safe-control-gym.

	CartPole-Stab	CartPole-Track	QuadRotor-Stab	QuadRotor-Track
rollout length	150	150	250	250
training epoch	500	500	1000	1000
batch size	64	64	64	128
cost limit	1	1	10	10
uncertainty level $K$	10	10	15	15
optimization step	40	40	80	80

Appendix

Table 2: Hyperparameter Settings of Fuz-RL Training and Testing

Uncertainty Object	Type	Config	System
Uncertainty Object	Type	Config	CartPole	QuadRotor
Observation	white noise	std:[-0.1, 0.1]	$(x,\dot{x})$	$(x,\dot{x})$
			$(\theta,\dot{\theta})$	$(z,\dot{z})$
				$(\theta,\dot{\theta})$
Dynamics	white noise	std:[-0.1, 0.1]	pole length	quadrotor mass
Dynamics	white noise	std:[-0.1, 0.1]	pole mass	quadrotor inertia

Appendix

Table 3: The observation, dynamics and action uncertainty settings of Safe-Control-Gym tasks

Figure 5: (a) Hierarchical relationship of safety sets, (b) Cost space and (c) Reward space trajectory comparisons, with dashed lines indicating safety boundaries.

Figure 6: Training curve comparison in double integrator environment, with shaded regions indicating standard deviation across 5 random seeds.

Case	Method	Observation Noise	Action Noise	Dynamic Noise
IEEE-39 Bus	PPOL	-5456.30 / 0.17	-6357.81 / 0.16	-7471.96 / 0.52
IEEE-39 Bus	Fuz-PPOL	-4822.03 / 0.14	-5789.19 / 0.13	-7363.20 / 0.47

Appendix

Table 4: Performance on IEEE 39-Bus Frequency Control (AvgRet / AvgRisk)

Figure 7: Average episodic rewards and average episodic risk of three safe RL and Fuz-RL under various uncertainty settings in Cartpole Stab task.

Figure 8: Average episodic rewards and average episodic risk of three safe RL and Fuz-RL under various uncertainty settings in Cartpole Track task.

Figure 9: Average episodic rewards and average episodic risk of three safe RL and Fuz-RL under various uncertainty settings in Quadrotor Stab task.

Figure 10: Average episodic rewards and average episodic risk of three safe RL and Fuz-RL under various uncertainty settings in Quadrotor Track task.

Figure 11: Average episodic reward (top) and average episodic risk (bottom) of Fuz-RL and RAMU in different scales’ observation uncertainty settings. The horizontal axis represents the uncertainty level.

Figure 12: Average episodic reward (top) and average episodic risk (bottom) of Fuz-RL and RAMU in different scales’ action uncertainty settings. The horizontal axis represents the uncertainty level.

Figure 13: Average episodic reward (top) and average episodic risk (bottom) of Fuz-RL and RAMU in different scales’ dynamics uncertainty settings. The horizontal axis represents the uncertainty level.

Figure 14: Average episodic reward (top) and average episodic risk (bottom) of Fuz-RL and RAMU in different scales’ observation, dynamics, and action uncertainty settings. The horizontal axis represents the uncertainty level.

Safe RL

Safe reinforcement learning studies how agents can learn and act while respecting safety constraints and limiting harmful outcomes.

See all 213 papers

Sampling-Based Safe Reinforcement Learning
May 19, 2026Luca Vignola, Bruce D. Lee, Manish Prajapat +4Robot Safety Safe RL
ETH Zurich
Evaluating Fuzz Testing for Reinforcement Learning Agents
Jul 27, 2026Zhibin Kang, Hanmo You, Dong Wang +2Safe RL Robust RL
College of Intelligence and Computing, Tianjin University, Tianjin, China
Provably Safe, Yet Scalable Reinforcement Learning
Jun 12, 2026Kai S. Yun, Zeyang Li, Navid AzizanRL Control Safety-Critical Control
MIT

May 19, 2026J/K move · Enter open · S save

Sampling-Based Safe Reinforcement Learning

Luca Vignola, Bruce D. Lee, Manish Prajapat, Manuel Wendl+3

ETH Zurich

Robot Safety Safe RL Sample-Efficient RL Model-Based RL

Open paper AskPDF arXiv

Safe exploration remains a fundamental challenge in reinforcement learning (RL), limiting the deployment of RL agents in the real world. We propose Sampling-Based Safe Reinforcement Learning (SBSRL), a model-based RL algorithm that maintains safety throughout the learning process by enforcing constraints jointly across a finite set of dynamics samples. This formulation approximates an intractable worst-case optimization over uncertain dynamics and enables practical safety guarantees in continuous domains. We further introduce an exploration strategy based on constraining epistemic uncertainty, eliminating the need for explicit exploration bonuses. Under regularity conditions, we derive high-probability guarantees of safety throughout learning and a finite-time sample complexity bound for recovering a near-optimal policy. Empirically, SBSRL achieves safe and efficient exploration both in simulation and in real robotic hardware, and readily extends to practical deep-ensemble implementations that scale to high-dimensional continuous control problems.

Abstract

Safe RL

Explore similar work

Sampling-Based Safe Reinforcement Learning

Evaluating Fuzz Testing for Reinforcement Learning Agents

Provably Safe, Yet Scalable Reinforcement Learning