Boosting deep Reinforcement Learning using pretraining with Logical Options

Zihan Ye; Phil Chau; Raban Emunds; Jannis Blüml; Cedric Derstroff; Quentin Delfosse; Oleg Arenz; Kristian Kersting

cs.AIMar 6, 2026

Boosting deep Reinforcement Learning using pretraining with Logical Options

Authors: Zihan Ye, Phil Chau, Raban Emunds, Jannis Blüml, Cedric Derstroff, Quentin Delfosse, Oleg Arenz, Kristian Kersting

Organizations: CS Department, Technical University of Darmstadt, Germany · AIML Group · Hessian AI · IAS Group ·

Abstract

Deep reinforcement learning agents are often misaligned, as they over-exploit early reward signals. Recently, several symbolic approaches have addressed these challenges by encoding sparse objectives along with aligned plans. However, purely symbolic architectures are complex to scale and difficult to apply to continuous settings. Hence, we propose a hybrid approach, inspired by humans' ability to acquire new skills. We use a two-stage framework that injects symbolic structure into neural-based reinforcement learning agents without sacrificing the expressivity of deep policies. Our method, called Hybrid Hierarchical RL (H^2RL), introduces a logical option-based pretraining strategy to steer the learning policy away from short-term reward loops and toward goal-directed behavior while allowing the final policy to be refined via standard environment interaction. Empirically, we show that this approach consistently improves long-horizon decision-making and yields agents that outperform strong neural, symbolic, and neuro-symbolic baselines.

Figures & tables

Figure 1: Deep reinforcement learning policies are often misaligned , exemplified on neural PPO agents. Although the oxygen is running low in Seaquest (left) and the goal in Kangaroo (right) is to go up, PPO agent fails to choose the optimal actions (in green). Instead, they focus on immediate rewards, e.g., keep attacking enemies.

Figure 2: Overview of the framework. Through logic-informed pretraining, H 2 RL embeds logic priors directly into neural policies, thereby addressing the deep policy misalignment issue. H 2 RL provides a two-stage training paradigm. In the first stage, the deep policy is jointly trained with the logic manager and the gating module (referred to as deep policy pretraining). In the second stage, the deep policy is further trained through direct interaction with the environment (referred to as deep policy post-training). See Sec. 3 for details.

Figure 3: Leveraging logic-informed pretraining, H 2 RL with its variants (bolded), outperforms baselines on challenging ALE tasks ( Seaquest , Kangaroo , and DonkeyKong ) with long-horizon dependencies and reward traps. Although DQN and PPO achieve high returns in Kangaroo , their learned policies remain misaligned; see Sec. 4 (RQ1 and RQ3) for details. Episodic returns are averaged over 12 environments (with 200 runs per environment). Results are presented on a log scale to normalize for the disparate reward magnitudes across games.

	H 2 PPO +	PPO	H 2 DQN +	DQN	H 2 C51 +	C51
Seaquest	1802 $\pm$ 400	3247 $\pm$ 881	1966 $\pm$ 903	3259 $\pm$ 1001	2774 $\pm$ 1367	4381 $\pm$ 1988
Kangaroo	2754 $\pm$ 2626	14592 $\pm$ 491	114665 $\pm$ 16932	14822 $\pm$ 1810	8193 $\pm$ 5928	13854 $\pm$ 1021
DonkeyKong	87780 $\pm$ 32786	4536 $\pm$ 296	6887 $\pm$ 1765	3205 $\pm$ 988	183645 $\pm$ 107378	3393 $\pm$ 721

Table 1: H 2 RL serves as a universal pretraining substrate for on-policy and off-policy deep RL methods. Shown are the means of the episodic return (averaged over 12 environments, 200 runs per environment) with the std of the pretrained and base policy. Although DQN and PPO achieve high returns in Kangaroo , their learned policies remain misaligned; Details see Sec. 4 (RQ3).

	PPO	DQN	C51	hReason	H 2 PPO	H 2 DQN +	H 2 C51 +
Floor 2	0% $\pm$ 0%	0% $\pm$ 0%	0% $\pm$ 0%	100% $\pm$ 0%	100% $\pm$ 0%	100% $\pm$ 0%	100% $\pm$ 0%
Floor 3	0% $\pm$ 0%	0% $\pm$ 0%	0% $\pm$ 0%	30% $\pm$ 20%	60% $\pm$ 10%	100% $\pm$ 0%	100% $\pm$ 0%
Floor 4	0% $\pm$ 0%	0% $\pm$ 0%	0% $\pm$ 0%	30% $\pm$ 10%	50% $\pm$ 10%	100% $\pm$ 0%	100% $\pm$ 0%

Table 2: H 2 RL mitigates policy misalignment via logic-informed pretraining for on-policy and off-policy deep RL methods. The table reports success rates for reaching different floors in Kangaroo for agents pre-trained using H 2 RL, its base variant, and the logic manager (hReason). Success rates averaged over 5 seeds, with 6 runs per seed.

	H 2 PPO	PPO	hPPO	hReason	exPPO
Seaquest	2812 $\pm$ 1477	3247 $\pm$ 881	1906 $\pm$ 628	1281 $\pm$ 963	1848 $\pm$ 34
Kanga./Succ.	5351 $\pm$ 4132 / 0.6 $\pm$ 0.1	14592 $\pm$ 491 / 0	10601 $\pm$ 914 / 0	2238 $\pm$ 2140 / 0.3 $\pm$ 0.2	14247 $\pm$ 1085 / 0
DonkeyKong	33657 $\pm$ 14578	4536 $\pm$ 296	418 $\pm$ 139	905 $\pm$ 1335	4268 $\pm$ 249

Table 3: The synergy between the logic and neural modules is crucial to H 2 RL’s improved performance. As an ablation study, we compare H 2 PPO (the default H 2 RL) with PPO (neural-only), hPPO (neural manager), hReason (logic manager), and exPPO (PPO with extended inputs) on Seaquest , Kangaroo , and DonkeyKong . The table reports the average episodic returns and success rates for reaching the third floor in Kangaroo . For details, see Sec. 4 (RQ3).

Figure 4: H 2 RL effectively leverages logic reasoning in continuous action spaces and improves deep agents’ performance. We compare H 2 RL with methods applicable to continuous action space on the Kangaroo and DonkeyKong tasks in CALE, where H 2 RL consistently outperforms these baselines. Details see Sec. 4 : RQ4.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Layer	Configuration
Convolutional	Conv2d(4, 32, 8, stride=4)
	ReLU
Convolutional	Conv2d(32, 64, 4, stride=2)
	ReLU
Convolutional	Conv2d(64, 64, 3, stride=1)
	ReLU

Appendix

Table 4: Meta-Controller Architecture

Layer	Configuration
Convolutional	Conv2d(5, 32, 8, stride=4)
	ReLU
Convolutional	Conv2d(32, 64, 4, stride=2)
	ReLU
Convolutional	Conv2d(64, 64, 3, stride=1)
	ReLU

Appendix

Table 5: Controller Network Architecture

Layer	Configuration
Convolutional	Conv2d(4, 32, 8, stride=4)
	ReLU
Convolutional	Conv2d(32, 64, 4, stride=2)
	ReLU
Convolutional	Conv2d(64, 64, 3, stride=1)
	ReLU

Appendix

Table 6: Visual Encoder Architecture (Pixel Stream)

Layer	Configuration
Fully Connected	Linear(4, 128)
	ReLU
Fully Connected	Linear(128, 64)
	ReLU
	Flatten
Fully Connected	Linear(64 $\cdot$ 4 $\cdot$ $N$ , 32)

Appendix

Table 7: Object Encoder Architecture (Symbolic Stream)

Layer	Configuration
Concatenation	Concat( $y^{\text{pixel}}\!\in\!\mathbb{R}^{512}$ , $y^{\text{symb.}}\!\in\!\mathbb{R}^{32}$ )
Fully Connected	Linear(544, 32)
	ReLU

Appendix

Table 8: Fusion Trunk Architecture (Pixel + Symbolic Fusion)

Parameter	Value
Gamma	0.99
Learning rate	0.00025
Blending entropy coefficient	0.01
Entropy coefficient	0.01
Clipping coefficient	0.1
Maximum gradient norm	0.5

Appendix

Table 9: Hyperparameters used when training the models.

Layer	Configuration
Convolutional	Conv2d(4, 32, 8, stride=4)
	ReLU
Convolutional	Conv2d(32, 64, 8, stride=2)
	ReLU
Convolutional	Conv2d(64, 64, 3, stride=1)
	ReLU

Appendix

Table 10: Pixel State Backbone Architecture

Layer	Configuration
Fully Connected	Linear( $N_{in}$ , 512)
	ReLU
Fully Connected	Linear(512, 120)
	ReLU
Fully Connected	Linear(120, $N_{out}$ )

Appendix

Table 11: Object-Centric Backbone Architecture

	H 2 RL	PPO	hPPO	hReason
Kangaroo (cont.)	84665 $\pm$ 49767	1785 $\pm$ 72	19854 $\pm$ 18586	557 $\pm$ 167
DonkeyKong (cont.)	10818 $\pm$ 7431	3836 $\pm$ 530	991.0 $\pm$ 446	542 $\pm$ 975

Appendix

Table 12: H 2 RL outperforms other compared baselines in the Continuous Atari learning environment (CALE). Reward comparison of H 2 RL against other baseline methods which can work in the continuous action space versions of Kangaroo and DonkeyKong.

	H 2 RL ++	H 2 RL +	H 2 RL	PPO	NUDGE
Seaquest	4759 $\pm$ 1004	1802 $\pm$ 400	2812 $\pm$ 1477	3247 $\pm$ 881	63 $\pm$ 18
Kangaroo	131842 $\pm$ 1221	2754 $\pm$ 2626	5148 $\pm$ 3950	14592 $\pm$ 491	404 $\pm$ 230
DonkeyKong	216793 $\pm$ 125655	87780 $\pm$ 32786	33690 $\pm$ 14565	4536 $\pm$ 296	21 $\pm$ 43

	BlendRL	Option-critic	hDQN	hPPO	hReason
Seaquest	117 $\pm$ 62	38 $\pm$ 19	112 ∗	1906 $\pm$ 628	1186 $\pm$ 930

Appendix

Table 13: H 2 RL outperforms other compared baselines in the classical Atari Learning environment. Although PPO and hPPO achieve relatively high scores in Kangaroo, their improvements stem from policy misalignment. We evaluate H 2 RL against other methods in Kangaroo, Seaquest, and DonkeyKong. ∗ Due to instability during evaluation, we report the maximum performance achieved by hDQN during training as a representative metric.

Environment	Modification	Description
Seaquest	random_start	randomize the starting position of the player
	random_surfacing	set the player to a random position when reaching the surface
	random_lane	set the starting position of the player in one of the 4 lanes where enemies and divers spawn
	disable_underwater_enemies	prevent all underwater enemies from appearing in the game
	disable_enemies	prevent all enemies from appearing in the game
	unlimited_collected_diver	always set the amount of collected divers to 1

Appendix

Table 14: A list of modifications for each environment.

Environment	Option	Applied modifications
Seaquest	get_air, deliver_diver	random_start
		disable_underwater_enemies
		unlimited_collected_diver
		empty_divers_automatically
		random_surfacing
		enable_surface_enemies

Appendix

Table 15: A list of modifications used when training options in each environment.

Environment	Option	Reward function
Seaquest	get_air, deliver_diver	+1.5 when player is at the surface
	get_diver	+1 when player collects a diver
	shoot_enemy/wait_in_line	+1 when the player shoots an enemy
		-10 when the player moves up or down
Kangaroo	ascend	+20 when player reaches certain positions
	deal_with_enemies	+1 when the player obtains a score from punching an enemy

Appendix

Table 16: A list of reward functions used when training options in each environment.

Neuro-Symbolic RL

See all 13 papers

Sample-efficient Neuro-symbolic Proximal Policy Optimization
Apr 28, 2026Simone Murari, Celeste Veronese, Daniele MeliReinforcement Learning Proximal Policy Optimization
Dept. of Computer Science, University of Verona, Italy
Neurosymbolic Reasoning with Incremental Knowledge for Sample Efficient Hierarchical Reinforcement Learning
Aug 4, 2026Subrat Prasad Panda, Blaise Genest, Arvind EaswaranReinforcement Learning Symbolic Planning
CCDS, NTU Singapore · IPAL, CNRS, France
HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents
Feb 18, 2026Jiangweizhi Peng, Yuanxin Liu, Ruida Zhou +4RL for LLM Agents Hierarchical RL
University of Minnesota · Northwestern University · Amazon AGI +2

Apr 28, 2026J/K move · Enter open · S save

Sample-efficient Neuro-symbolic Proximal Policy Optimization

Simone Murari, Celeste Veronese, Daniele Meli

Dept. of Computer Science, University of Verona, Italy

Reinforcement Learning Proximal Policy Optimization Sparse-Reward RL Policy Learning Sample-Efficient RL Neuro-Symbolic RL

Open paper AskPDF arXiv

Deep Reinforcement Learning (DRL) algorithms often require a large amount of data and struggle in sparse-reward domains with long planning horizons and multiple sub-goals. In this paper, we propose a neuro-symbolic extension of Proximal Policy Optimization (PPO) that transfers partial logical policy specifications learned in easier instances to guide learning in more challenging settings. We introduce two integrations of symbolic guidance: (i) H-PPO-Product, which biases the action distribution at sampling time, and (ii) H-PPO-SymLoss, which augments the PPO loss with a symbolic regularization term. We evaluate our methods on three benchmarks (OfficeWorld, WaterWorld, and DoorKey), showing consistently faster learning and higher return at convergence than PPO and a Reward Machine baseline, also under imperfect symbolic knowledge.

Abstract

Neuro-Symbolic RL

Explore similar work

Sample-efficient Neuro-symbolic Proximal Policy Optimization

Neurosymbolic Reasoning with Incremental Knowledge for Sample Efficient Hierarchical Reinforcement Learning

HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents