Integrating AIRL-Based Reward Estimation into a Control Method Using Fuzzy Inference and Policy Gradient

Seiji Ishihara

cs.LGOct 10, 2026

Integrating AIRL-Based Reward Estimation into a Control Method Using Fuzzy Inference and Policy Gradient

Authors: Seiji Ishihara

Organizations: School of Science and Engineering, Tokyo Denki University, Saitama, Japan.

Abstract

A method integrating policy gradient with fuzzy inference has been proposed. This method, hereafter referred to as the base model, has high flexibility with respect to the configuration of fuzzy inference rules and fuzzy representation. It can also automatically adjust the rule weights and the membership functions in the antecedent and consequent parts using the same framework for learning. In methods that integrate reinforcement learning with fuzzy inference, however, the reward function must be designed in advance. It is therefore useful to add learning-based reward estimation to the base model. This paper proposes a fuzzy control method that integrates a reward estimation mechanism based on the adversarial inverse reinforcement learning (AIRL) framework into the base model. The generative adversarial nets (GAN) framework adopted in AIRL, however, tends to suffer from unstable learning. To address this, the proposed method treats the policy of the base model as the generator and also uses the set of membership functions in the policy as the cost function of the discriminator. This configuration allows the discriminator update for reward estimation and the generator update for policy estimation to be performed simultaneously on the parameters shared by both, without estimating a value function. In simulations of an illustrative vehicle speed control task, the proposed method acquired policies with high generalization performance through stable and efficient learning.

Figures & tables

Figure 1: Configuration of the proposed method

Figure 2: Configuration of AIRL-BMP

	Avg. task	Successful tasks				Avg. no. of failed tasks
	success rate	min.	max.	avg.	SD	collision	out of range	delay
Proposed	0.981	565	572	569.25	2.34	10.50	0.00	0.25
AIRL-BMP	0.721	285	539	418.45	74.59	19.45	44.15	97.95
Base model \tnote a	0.983	519	580	570.05	12.16	6.90	3.05 \tnote b

Table 1: Control results for the 580 training problems (task success rate, number of successful tasks, and breakdown of failed tasks) over the 20 sets of learned parameters

	Training epochs
	min.	max.	avg.	SD
Proposed	35	78	48.80	10.51
AIRL-BMP	327	1350	746.65	257.83

Table 2: Number of training epochs over the 20 training runs

Figure 3: Average reward per time step for all fuzzy representations at each training epoch. a Proposed method, b AIRL-BMP

	Time to settle		Settling delay		Exits from range		SD after settling
	avg.	SD	avg.	SD	avg.	SD	pooled value
Proposed	22.83	1.84	5.71	1.28	0.48	0.13	0.28
AIRL-BMP	47.91	6.65	29.51	10.22	1.35	0.41	0.96
Demonstration	14.60	—	2.68	—	0.21	—	0.35

Table 3: Trajectory characteristics for the successful training problems

Figure 4: Trajectories of the inter-vehicle distance with the proposed method, AIRL-BMP, and the demonstration. a Problem T1, b Problem T2

	Avg. task	Successful tasks				Avg. no. of failed tasks
	success rate	min.	max.	avg.	SD	collision	out of range	delay
Proposed	1.000	722	722	722.00	0.00	0.00	0.00	0.00
AIRL-BMP	0.924	586	722	667.15	47.05	0.00	12.65	42.20
Base model \tnote a	0.994	637	722	717.65	18.99	0.00	4.35 \tnote b

Table 4: Control results for the 722 evaluation problems (task success rate, number of successful tasks, and breakdown of failed tasks) over the 20 sets of learned parameters

	Time to settle		Settling delay		Exits from range		SD after settling
	avg.	SD	avg.	SD	avg.	SD	pooled value
Proposed	17.51	2.06	4.56	0.47	0.28	0.02	0.81
AIRL-BMP	31.26	3.22	11.61	3.66	0.45	0.19	2.20

Table 5: Trajectory characteristics for the successful evaluation problems

Figure 5: Trajectories of the inter-vehicle distance with the proposed method and AIRL-BMP. a Problem E1, b Problem E2

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

demonstration $\geq$ sample		demonstration $<$ sample
$x_{\textrm{d}}$	$x_{\textrm{v}}$	$x_{\textrm{d}}$	$x_{\textrm{v}}$
$\dfrac{\partial v_{\alpha^{\textrm{long}}_{1}}}{\partial\alpha^{\textrm{long}}_{1}}$	$\dfrac{\partial v_{\alpha^{\textrm{fast}}_{2}}}{\partial\alpha^{\textrm{fast}}_{2}}$	$-\dfrac{\partial v_{\alpha^{\textrm{long}}_{1}}}{\partial\alpha^{\textrm{long}}_{1}}$	$-\dfrac{\partial v_{\alpha^{\textrm{fast}}_{2}}}{\partial\alpha^{\textrm{fast}}_{2}}$
$-\dfrac{\partial v_{\alpha^{\textrm{short}}_{1}}}{\partial\alpha^{\textrm{short}}_{1}}$	$-\dfrac{\partial v_{\alpha^{\textrm{slow}}_{2}}}{\partial\alpha^{\textrm{slow}}_{2}}$	$\dfrac{\partial v_{\alpha^{\textrm{short}}_{1}}}{\partial\alpha^{\textrm{short}}_{1}}$	$\dfrac{\partial v_{\alpha^{\textrm{slow}}_{2}}}{\partial\alpha^{\textrm{slow}}_{2}}$

Appendix

Table 1: Gradient of

|v_{\alpha^{i}_{j}}|

in each case

$y_{1}$	$-5$	$-4$	$-3$	$-2$	$-1$	$0$	$1$	$2$	$3$	$4$	$5$
$B_{1}^{\textrm{high-ac.}}$	0	0	0	0	0	0	0.2	0.3	0.6	0.8	0.9
$B_{1}^{\textrm{low-ac.}}$	0	0	0	0	0	1	0.8	0.6	0.5	0.4	0.2
$B_{1}^{\textrm{high-br.}}$	0.9	0.7	0.5	0.3	0.2	0	0	0	0	0	0
$B_{1}^{\textrm{low-br.}}$	0.1	0.2	0.4	0.6	0.7	1	0	0	0	0	0

Appendix

Table D1: Node values defining the consequent membership functions for the accelerator and brake inputs

RL Control

Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.

See all 274 papers

Fuzzy Logic Theory-based Adaptive Reward Shaping for Robust Reinforcement Learning (FARS)
Apr 17, 2026Hürkan Şahin, Van Huyen Dang, Erdi Sayar +2Reward Shaping Autonomous Drone Racing
Automatic Control Group (RAT), Paderborn University, 33098 Paderborn, Germany
Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matching
Sep 30, 2026Yang Chen, Yitan Zhang, Michael Witbrock +1Reinforcement Learning
Shanghai Artificial Intelligence Laboratory · University of Auckland
Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates
May 10, 2026Anish Diwan, Davide Tateo, Christopher E. Mower +3Reinforcement Learning Policy Optimization
Technical University of Darmstadt · Robotics Institute Germany (RIG) · Lund University +3

Apr 17, 2026J/K move · Enter open · S save

Fuzzy Logic Theory-based Adaptive Reward Shaping for Robust Reinforcement Learning (FARS)

Hürkan Şahin, Van Huyen Dang, Erdi Sayar, Alper Yegenoglu+1

Automatic Control Group (RAT), Paderborn University, 33098 Paderborn, Germany

Reward Shaping Autonomous Drone Racing Robotic RL Fuzzy Logic Robust RL Reward Design for RL

Open paper AskPDF arXiv

Reinforcement learning (RL) often struggles in real-world tasks with high-dimensional state spaces and long horizons, where sparse or fixed rewards severely slow down exploration and cause agents to get trapped in local optima. This paper presents a fuzzy logic based reward shaping method that integrates human intuition into RL reward design. By encoding expert knowledge into adaptive and interpreable terms, fuzzy rules promote stable learning and reduce sensitivity to hyperparameters. The proposed method leverages these properties to adapt reward contributions based on the agent state, enabling smoother transitions between fast motion and precise control in challenging navigation tasks. Extensive simulation results on autonomous drone racing benchmarks show stable learning behavior and consistent task performance across scenarios of increasing difficulty. The proposed method achieves faster convergence and reduced performance variability across training seeds in more challenging environments, with success rates improving by up to approximately 5 percent compared to non fuzzy reward formulations.

Abstract

RL Control

Explore similar work

Fuzzy Logic Theory-based Adaptive Reward Shaping for Robust Reinforcement Learning (FARS)

Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matching

Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates