Managing Action Preconditions in Neuro-Symbolic RL: Three Placement Strategies for Embodied Agents

Norbert Oswald; Fabian Deuser; Thomas Bräunl

cs.LGSep 13, 2026

Managing Action Preconditions in Neuro-Symbolic RL: Three Placement Strategies for Embodied Agents

Authors: Norbert Oswald, Fabian Deuser, Thomas Bräunl

Organizations: Institute of Distributed Intelligent Systems University of the Bundeswehr Munich Munich, Germany · School of Engineering The University of Western Australia Perth, Australia

Abstract

Humans carry behaviour knowledge of how to act in familiar situations into every new task rather than relearning it from scratch. There is no reason a Reinforcement Learning (RL) agent shouldn't do the same: known behaviour patterns need not be learned, only applied. Neuro-symbolic RL bridges prior knowledge and RL by injecting symbolic knowledge alongside a learned policy. The point at which this knowledge is integrated is critical: a poor choice can produce, for instance, hallucinated preconditions, which surface as safety and reliability problems in agents acting in changing environments. We formalise this behavioural knowledge as a precondition Bayesian network (BN) over the agent's structural actions - the actions whose legality depends on preconditions, such as picking up a key, grasping a block, toggling a door, or dropping an object. The BN restricts when these actions may fire, and we inject it into the RL loop at three placements: (1) a symbolic verifier, consulted only at inference, that fires a structural action once its preconditions hold; (2) a symbolic enforcer, active during both training and inference, that governs structural-action use throughout learning; and (3) a symbolic learner, which folds the knowledge into the network and learns the restriction and use of structural actions itself. To test the three variants we run experiments on two benchmarks with opposite regimes: one built on long, ordered planning chains, the other on continuous manipulation. We compare against strong baselines on solution quality, sample efficiency, and traceability. The payoff is substantial. On MiniGrid, all three placements improve the solution quality over the PPO+RND baseline, the symbolic enforcer leading at $98.2\%$ against the baseline's $88.8\%$ . On Fetch, $\dots$

Figures & tables

Panel of Figure 1 — Figure 1: Precondition BN of FetchPickAndPlace describe active subgoals and when GRASP fires.

Panel of Figure 2 — Figure 2: Three symbolic placement strategies shown as closed RL loops. $\pi_{\theta}$ is any RL policy (PPO, SAC, GNN, …) wrapped as a black box with subgoal g*; SL additionally attaches a head $P_{\psi}$ .

Benchmark	$\mathcal{A}_{\text{nav}}$	$\|\mathcal{A}_{\text{struct}}\|$	#Predicates	Baseline
MiniGrid	$\{$ left, right, forward $\}$	5: $\{$ pickup key/ball, toggle box/door, drop ball $\}$	$9$	PPO+RND
Fetch	$\Delta xyz$	1: $\{$ grasp $\}$	$4$	SAC $+$ HER

Table 1: Benchmark configuration summary.

Method	Env-steps	Success ( $\%$ )	Norm. return
PPO + RND (baseline)	$10$ M	$88.8\pm 6.1$	$0.83$
SV: PPO + RND	$10$ M	$90.0\pm 8.4$	$0.84$
SE: PPO + RND	$10$ M	$98.2\pm 0.3$	$0.92$
SL: PPO + RND	$10$ M	$93.8\pm 7.3$	$0.88$

Table 2: ObstructedMaze (left) and results on placement variants with two

\pi_{\theta}

compared to baseline.

Figure 3: Fetch sample efficiency (a) and cross-benchmark summary (b)

Panel of Figure 4 — Figure 4: Astoria street network, graph, and precondition BN [ 9 ] .

Neuro-Symbolic RL

See all 13 papers

Neurosymbolic Reasoning with Incremental Knowledge for Sample Efficient Hierarchical Reinforcement Learning
Aug 4, 2026Subrat Prasad Panda, Blaise Genest, Arvind EaswaranReinforcement Learning Symbolic Planning
CCDS, NTU Singapore · IPAL, CNRS, France
Sample-efficient Neuro-symbolic Proximal Policy Optimization
Apr 28, 2026Simone Murari, Celeste Veronese, Daniele MeliReinforcement Learning Proximal Policy Optimization
Dept. of Computer Science, University of Verona, Italy
Neuro-Symbolic Injection of LTLf Constraints in Autoregressive Reinforcement Learning Policies
Jun 6, 2026Ashkan Ansarifard, Matteo Mancanelli, Elena Umili +1Reinforcement Learning Transformer-Based RL
Sapienza University of Rome · University of Rome La Sapienza, Rome, Italy

Aug 4, 2026J/K move · Enter open · S save

Neurosymbolic Reasoning with Incremental Knowledge for Sample Efficient Hierarchical Reinforcement Learning

Subrat Prasad Panda, Blaise Genest, Arvind Easwaran

CCDS, NTU Singapore · IPAL, CNRS, France

Reinforcement Learning Symbolic Planning Hierarchical RL Neuro-Symbolic Reasoning Sparse-Reward RL Sample-Efficient RL+1

Open paper AskPDF arXiv

(Flat) Reinforcement Learning (RL) agents face significant challenges in environments with sparse rewards that require long-horizon reasoning. A compelling approach to improve sample efficiency is to incorporate knowledge into learning and decision-making. In standard Hierarchical RL (HRL), knowledge is encoded in a fixed, non-updatable form, such as architectural choices, and remains unchanged throughout learning. With fixed HRL, reasoning with incremental knowledge learned during exploration is impractical before sufficient environmental knowledge is acquired, leading to poor sample efficiency. In this work, we propose neurosymbolic HRL with {\em Incremental Knowledge (InK)}: symbolic high-level components perform {\em symbolic planning} (e.g. using

D^*

) on an updatable representation of current InK, while low-level goal-conditioned neural modules learn motion primitives through experience using reward shaping. Experiments on navigation tasks demonstrate that incorporating InK substantially improves sample efficiency. Additionally, to perform {\em optimal} symbolic planning given {\em prior} knowledge about the world, we develop Belief World Tree Search. The code is available at https://github.com/CPS-research-group/ink_bwts.

Abstract

Neuro-Symbolic RL

Explore similar work

Neurosymbolic Reasoning with Incremental Knowledge for Sample Efficient Hierarchical Reinforcement Learning

Sample-efficient Neuro-symbolic Proximal Policy Optimization

Neuro-Symbolic Injection of LTLf Constraints in Autoregressive Reinforcement Learning Policies