Continual robot navigation requires reinforcement learning policies to continuously adapt to new environments while retaining previously acquired knowledge. Meanwhile, explainable reinforcement learning aims to learn meaningful behavioral representations through policy interaction. However, these objectives are inherently conflicting, as continual learning emphasizes policy stability whereas explainable behavior learning requires adaptive exploration. Existing methods rarely address both simultaneously. To resolve this issue, this paper proposes a Fuzzy Explainable Continual Hierarchical Reinforcement Learning (FECHRL) method. A two-stage behavior cloning strategy is introduced to balance rapid policy adaptation and explainable behavior evolution by progressively transitioning from strong imitation guidance to autonomous reinforcement learning. Furthermore, a fuzzy critic based on a Takagi--Sugeno fuzzy inference system is proposed to alleviate value extrapolation errors during the transition between imitation-guided learning and autonomous exploration, thereby improving continual training stability. Experimental results on sequential multi-scenario navigation tasks demonstrate that the proposed method effectively unifies continual learning capability and policy explainability within a single reinforcement learning framework, achieving stable knowledge retention and meaningful behavioral explanation.
Figures & tables
Fig. 1: The overall framework of the proposed methodology.
Fig. 2: Simulation environment
Fig. 3: Real environment
Items
Value
optimizer π / gψ / Qη / Qξ
Adam lr:0.0003/0.0003/0.001/0.01
γ / λ / τc
0.99/1/0.05/ 0.1 /0.1
Buffer_size/ Btask / Bmem
100000/256/64
Episodes
300(Env1) 1000(Env2-Env4) 500(Env5)
max_steps
200(Env1),350(Env2-Env4),500(Env5)
TABLE I: PARAMETER SETTINGS
Fig. 4: Implementation framework
Fig. 5: Training success rates of different RL methods
Method
Train after
MT1
MT2
MT3
MT4
FTk
FTTk
KGDDPG
T1
0.91 ±0.01
0.91 ±0.01
0.90 ±0.00
0.75 ±0.01
-
-
T2
0.09 ±0.02
0.79 ±0.05
0.61 ±0.05
0.43 ±0.01
0.000 ±0.026
0.084 ±0.017
T3
0.01 ±0.00
0.46 ±0.05
0.78 ±0.05
0.42 ±0.04
0.110 ±0.034
0.162 ±0.021
T4
0.02 ±0.01
0.03 ±0.02
0.14 ±0.01
0.80 ±0.07
0.296 ±0.040
0.131 ±0.127
KGEWC
T1
0.75 ±0.06
0.77 ±0.01
0.73 ±0.01
0.55 ±0.02
-
-
TABLE II: Performance after training each task across different methods.
Fig. 6: Training success rates of different switching
Method
Train after
MT1
MT2
MT3
MT4
FTk
FTTk
Progressive
T1
0.97 ±0.01
1.00 ±0.00
1.00 ±0.00
1.00 ±0.00
-
-
T2
0.03 ±0.01
0.91 ±0.01
0.84 ±0.02
0.83 ±0.01
-0.030 ±0.008
0.033 ±0.005
T3
0.02 ±0.01
0.76 ±0.01
0.92 ±0.01
0.93 ±0.02
0.020 ±0.014
0.212 ±0.005
T4
0.00 ±0.00
0.27 ±0.03
0.33 ±0.03
0.89 ±0.02
0.013 ±0.005
0.208 ±0.010
Soft switching
T1
0.97 ±0.01
1.00 ±0.00
1.00 ±0.00
1.00 ±0.00
-
-
TABLE III: Mean performance after training each task for Progressive and Soft switching.
Fig. 7: Comparison of different critics
Fig. 8: Comparison of different critics
Fig. 9: Comparison of Behavior Modes under Different Clone Switching
Fig. 10: Status, reward weight, and mode for steps(0)
Fig. 11: Status, reward weight, and mode for steps(22)
Fig. 12: Status, reward weight, and mode for steps(61)
Fig. 13: Status, reward weight, and mode for steps(70)
Fig. 14: Rule activation heatmap
Axes
Target
Exp1
x
1.20
1.40
1.60
2.00
2.50
y
0.00
0.00
0.00
0.00
0.00
Exp2
x
1.90
2.40
1.70
1.00
0.20
y
-0.50
0.25
0.90
0.56
0.00
TABLE IV: TARGET FOR EXPERIMENT 1 AND 2
Fig. 15: Recorded trajectories and corresponding real image
Fig. 16: Mode and reward weight of Exp1 targert1
Fig. 17: Recorded trajectories and corresponding real image
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Deep Reinforcement Learning (DRL) agents achieve remarkable performance in continuous control but remain opaque, hindering deployment in safety-critical domains. Existing explainability methods either provide only local insights (SHAP, LIME) or employ over-simplified surrogates failing to capture continuous dynamics (decision trees). This work proposes a Hierarchical Takagi-Sugeno-Kang (TSK) Fuzzy Classifier System (FCS) distilling neural policies into human-readable IF-THEN rules through K-Means clustering for state partitioning and Ridge Regression for local action inference. Three quantifiable metrics are introduced: Fuzzy Rule Activation Density (FRAD) measuring explanation focus, Fuzzy Set Coverage (FSC) validating vocabulary completeness, and Action Space Granularity (ASG) assessing control mode diversity. Dynamic Time Warping (DTW) validates temporal behavioral fidelity. Empirical evaluation on Lunar Lander(Continuous) shows the Triangular membership function variant achieves 81.48% ± 0.43% fidelity, outperforming Decision Trees by 21 percentage points. The framework exhibits statistically superior interpretability (FRAD = 0.814 vs. 0.723 for Gaussian, p<0.001) with low MSE (0.0053) and DTW distance (1.05). Extracted rules such as ``IF lander drifting left at high altitude THEN apply upward thrust with rightward correction'' enable human verification, establishing a pathway toward trustworthy autonomous systems.
Sanup S. Araballi, Simon Khan, Chilukuri K. Mohan
Dept. of EECS, Syracuse University, NY 13244, USA · Air Force Research Laboratory, Rome, NY 13441, USA
Reinforcement learning (RL) often struggles in real-world tasks with high-dimensional state spaces and long horizons, where sparse or fixed rewards severely slow down exploration and cause agents to get trapped in local optima. This paper presents a fuzzy logic based reward shaping method that integrates human intuition into RL reward design. By encoding expert knowledge into adaptive and interpreable terms, fuzzy rules promote stable learning and reduce sensitivity to hyperparameters. The proposed method leverages these properties to adapt reward contributions based on the agent state, enabling smoother transitions between fast motion and precise control in challenging navigation tasks. Extensive simulation results on autonomous drone racing benchmarks show stable learning behavior and consistent task performance across scenarios of increasing difficulty. The proposed method achieves faster convergence and reduced performance variability across training seeds in more challenging environments, with success rates improving by up to approximately 5 percent compared to non fuzzy reward formulations.
Hürkan Şahin, Van Huyen Dang, Erdi Sayar +2
Automatic Control Group (RAT), Paderborn University, 33098 Paderborn, Germany
A key challenge in modern robotics is to adapt to changing environments, a challenge that is exacerbated when simulations cannot encompass every possible real-world configuration, and therefore Reinforcement Learning (RL) in the physical world becomes necessary. Continual Reinforcement Learning provides the tools to address this challenge; however, both the frameworks and the methods remain underexplored. Autonomous Racing and in particular the RoboRacer competition provide a testing ground for such methods, as learning to drive on a new track-floor combination with the least amount of new experience naturally frames a continual learning problem. This work tries to address this gap by proposing a continual RL framework based on Continual Backpropagation that is able, with only real-world data, to train a generalistic policy on a set of tracks and then fine- tune it within 15 minutes to outperform classical controllers. Furthermore, a comparison method based on offline RL is proposed, and a simulation analysis of the plasticity properties of the methods is conducted.
Joel Siegert, Edoardo Ghignone, Michele Magno
Center for Project-Based Learning, D-ITET, ETH Zurich