Managing Action Preconditions in Neuro-Symbolic RL: Three Placement Strategies for Embodied Agents
Authors: Norbert Oswald, Fabian Deuser, Thomas Bräunl
Organizations: Institute of Distributed Intelligent Systems University of the Bundeswehr Munich Munich, Germany · School of Engineering The University of Western Australia Perth, Australia
Humans carry behaviour knowledge of how to act in familiar situations into every new task rather than relearning it from scratch. There is no reason a Reinforcement Learning (RL) agent shouldn't do the same: known behaviour patterns need not be learned, only applied. Neuro-symbolic RL bridges prior knowledge and RL by injecting symbolic knowledge alongside a learned policy. The point at which this knowledge is integrated is critical: a poor choice can produce, for instance, hallucinated preconditions, which surface as safety and reliability problems in agents acting in changing environments. We formalise this behavioural knowledge as a precondition Bayesian network (BN) over the agent's structural actions - the actions whose legality depends on preconditions, such as picking up a key, grasping a block, toggling a door, or dropping an object. The BN restricts when these actions may fire, and we inject it into the RL loop at three placements: (1) a symbolic verifier, consulted only at inference, that fires a structural action once its preconditions hold; (2) a symbolic enforcer, active during both training and inference, that governs structural-action use throughout learning; and (3) a symbolic learner, which folds the knowledge into the network and learns the restriction and use of structural actions itself. To test the three variants we run experiments on two benchmarks with opposite regimes: one built on long, ordered planning chains, the other on continuous manipulation. We compare against strong baselines on solution quality, sample efficiency, and traceability. The payoff is substantial. On MiniGrid, all three placements improve the solution quality over the PPO+RND baseline, the symbolic enforcer leading at 98.2% against the baseline's 88.8%. On Fetch, …
Figures & tables
Figure 1: Precondition BN of FetchPickAndPlace describe active subgoals and when GRASP fires.
Figure 2: Three symbolic placement strategies shown as closed RL loops. πθ is any RL policy (PPO, SAC, GNN, …) wrapped as a black box with subgoal g*; SL additionally attaches a head Pψ .
Benchmark
Anav
∣Astruct∣
#Predicates
Baseline
MiniGrid
{ left, right, forward }
5: { pickup key/ball, toggle box/door, drop ball }
9
PPO+RND
Fetch
Δxyz
1: { grasp }
4
SAC + HER
Table 1: Benchmark configuration summary.
Method
Env-steps
Success ( % )
Norm. return
PPO + RND (baseline)
10 M
88.8±6.1
0.83
SV: PPO + RND
10 M
90.0±8.4
0.84
SE: PPO + RND
10 M
98.2±0.3
0.92
SL: PPO + RND
10 M
93.8±7.3
0.88
Table 2: ObstructedMaze (left) and results on placement variants with two πθ compared to baseline.
Figure 3: Fetch sample efficiency (a) and cross-benchmark summary (b)
Figure 4: Astoria street network, graph, and precondition BN [ 9 ] .
(Flat) Reinforcement Learning (RL) agents face significant challenges in environments with sparse rewards that require long-horizon reasoning. A compelling approach to improve sample efficiency is to incorporate knowledge into learning and decision-making. In standard Hierarchical RL (HRL), knowledge is encoded in a fixed, non-updatable form, such as architectural choices, and remains unchanged throughout learning. With fixed HRL, reasoning with incremental knowledge learned during exploration is impractical before sufficient environmental knowledge is acquired, leading to poor sample efficiency. In this work, we propose neurosymbolic HRL with {\em Incremental Knowledge (InK)}: symbolic high-level components perform {\em symbolic planning} (e.g. using D∗) on an updatable representation of current InK, while low-level goal-conditioned neural modules learn motion primitives through experience using reward shaping. Experiments on navigation tasks demonstrate that incorporating InK substantially improves sample efficiency. Additionally, to perform {\em optimal} symbolic planning given {\em prior} knowledge about the world, we develop Belief World Tree Search. The code is available at https://github.com/CPS-research-group/ink_bwts.
Deep Reinforcement Learning (DRL) algorithms often require a large amount of data and struggle in sparse-reward domains with long planning horizons and multiple sub-goals. In this paper, we propose a neuro-symbolic extension of Proximal Policy Optimization (PPO) that transfers partial logical policy specifications learned in easier instances to guide learning in more challenging settings. We introduce two integrations of symbolic guidance: (i) H-PPO-Product, which biases the action distribution at sampling time, and (ii) H-PPO-SymLoss, which augments the PPO loss with a symbolic regularization term. We evaluate our methods on three benchmarks (OfficeWorld, WaterWorld, and DoorKey), showing consistently faster learning and higher return at convergence than PPO and a Reward Machine baseline, also under imperfect symbolic knowledge.
Simone Murari, Celeste Veronese, Daniele Meli
Dept. of Computer Science, University of Verona, Italy
In this work we study offline reinforcement learning (RL) under temporally extended task constraints expressed in Linear Temporal Logic over finite traces (LTLf). Recently, transformer-based approaches such as Trajectory Transformers and Decision Transformers have been adopted to address RL as a sequence modeling problem. However, these methods optimize purely for reward and do not account for high-level temporal requirements. Here, we introduce a neurosymbolic framework that injects LTLf background knowledge into such transformer-based RL policies. Our approach compiles LTLf formulas into deterministic finite automata (DFAs) and integrates them into the learning process through a differentiable representation and a logic-based loss function. In particular, we derive differentiable satisfaction signals from DFA progression and use them as a regularization term during training. The resulting method is architecture-agnostic across different models. We evaluate the proposed framework on navigation environments with specification suites covering combinations of safety and reachability temporal properties. Experimental results show that incorporating background knowledge not only improves constraint satisfaction, but also maintains competitive return compared to vanilla baselines.
Ashkan Ansarifard, Matteo Mancanelli, Elena Umili +1
Sapienza University of Rome · University of Rome La Sapienza, Rome, Italy