Interpretable Meta-Policies for Neural Decision Making
Authors: Raban Emunds, Jannis Blüml, Quentin Delfosse, Kristian Kersting
Organizations: Department of Computer Science, Technical University Darmstadt, Germany · Hessian Center for Artificial Intelligence (hessian.AI) · Centre for Cognitive Science, Darmstadt · German Research Center for Artificial Intelligence (DFKI)
Deep Reinforcement Learning remains difficult to interpret and align due to the black-box nature of neural networks. While (neuro-)symbolic approaches improve transparency by incorporating symbolic reasoning, applying them directly to low-level actions results in overly complex, unreadable policies. We introduce NEXUS, a hierarchical RL framework that resolves this tension by combining neural skills with symbolic meta-policies. By shifting transparent reasoning to high-level skills rather than atomic actions, NEXUS drastically reduces symbolic complexity without sacrificing environment performance. Object-centric observations enable high-level reasoning and the design of reward functions and meta-policies by interaction with LLMs. We demonstrate that NEXUS agents learn disentangled skills, are interpretable regarding high-level goals, are better aligned to goals described in the game manual, less prone to reward hacking and more robust to game simplifications than deep RL. Finally, we isolate and evaluate the impact of different meta-policy architectures (neural, symbolic, neuro-symbolic) on environmental return and aligned task performance.
Figures & tables
Figure 1: NEXUS provides high-level interpretability through symbolic reasoning and abstraction. Neural network-based policies are non-modular black box models. While symbolic approaches are fully transparent by leveraging logical rules or decision trees, they quickly become excessively complex if applied to low-level actions. NEXUS abstracts low-level actions into skills and provides transparent high-level decisions.
Figure 2: NEXUS with a neuro-symbolic meta-policy. First, the symbolic state is extracted from the observation. Then, the LLM-generated rules filter potential relevant skills, which the trained meta-policy Q-function selects between. Finally, the selected neural skill outputs the low-level action. Using the same experience, the meta-policy’s Q-function is trained to optimize the environmental reward, while each skill is trained using its own LLM-generated reward, yielding disentangled skills with interpretable objectives.
Figure 3: NEXUS learns disentangled skills from off-policy data. In Seaquest (top), the baseline methods mainly focus on shooting enemies, while the NEXUS approaches acquire the target skills more evenly. Similarly, in Kangaroo (bottom), the NEXUS approaches learn ’Move Up’ and ’Collect Fruits’ reliably, while the baselines focus mostly on ’Handle Threats’. Aggregated over 6 seeds, ±1 standard deviation.
Figure 4: NEXUS succesfully learns LLM-proposed Crafter skills. Compared to the baseline, NEXUS approaches often converge faster on the proposed skills. Aggregated over 6 seeds, ±1 standard deviation.
Figure 5: NEXUS (nesy) restricts skill selection through interpretable rules but remains flexible in ambiguous scenarios. Left: enemy_closeby and diver_available both hold; the meta-policy prefers fighting, likely as the diver is near the enemy. Center: with the diver more accessible, rescuing is preferred. Right: with all divers collected and an enemy nearby, surfacing has the higher estimated return.
Figure 6
Policy
Rule Size and Length Lakkaraju et al. (2016)
ADPL Wu et al. (2017)
NUDGE ( )
16 rules, 46 predicates
23.56 evaluations
NEXUS ( )
4 rules, 4 predicates
3 evaluations
NEXUS ( )
2 rules, 3 predicates
2.50 evaluations
Table 1: Quantitative interpretability metrics of the symbolic decision logic, comparing NUDGE (action level) and NEXUS (skill level) on Seaquest. NEXUS metrics cover the meta-policy only; low-level skills are neural.
Figure 6: NEXUS is competitive. In Seaquest, NEXUS outperforms the baselines on both game reward and rescuing divers. In Kangaroo, NEXUS approaches progress farther. Aggregated over 6 seeds, ±1 standard deviation.
Figure 7: NEXUS remains practical on simplified games. While baseline performance on aligned metrics remains low, neuro-symbolic and fully symbolic NEXUS improve on modified Seaquest (left). Modified kangaroo (middle) reduces performance of symbolic NEXUS, while neuro-symbolic NEXUS remains effective. Crafter performance (right) remains consistent or improves for all NEXUS versions. Aggregated over 6 seeds, ±1 standard deviation.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Existing approaches are not interpretable. Neural RL agents are black-boxes and learn shortcuts. Hierarchical RL remains a black box and typically produces entangled skills. Symbolic agents are transparent, but grow massively with policy complexity.
Figure 9: The evaluation environments: Crafter, Seaquest and Kangaroo.
Parameter
PQN(-based)
PPO
Total Timesteps
5×107
5×107
Num Environments
1024
128
Num Steps per Update
32
32
Learning Rate
2.5×10−4
2.5×10−4
Max Grad Norm
10
0.5
Discount Factor ( γ )
0.99
0.99
Appendix
Table 2: Atari Hyperparameters
Parameter
PQN(-based)
Total Timesteps
1×109
Num Environments
512
Num Steps per Update
128
Learning Rate
3.0×10−4
Max Grad Norm
0.5
Discount Factor ( γ )
0.99
Appendix
Table 3: Crafter Hyperparameters
Figure 10: Reward-shaping is not sufficient. Replacing NEXUS’s neuro-symbolic meta-policy with LLM-designed or statically weighted reward shaping leads to severe performance degradation across both environments. While baselines successfully acquire isolated skills (e.g., surface in Seaquest), they fail to learn all skills equally well or to coordinate them—demonstrating that NEXUS’s skill disentanglement is essential for achieving high overall performance.
Figure 11: Skill activation dynamics and performance impact of the neuro-symbolic meta-policy in Seaquest and Kangaroo . Replacing the meta-policy with random skill choices significantly degrades performance, demonstrating its critical role in maximizing returns. Default skills ( fight_enemies in Seaquest , navigate in Kangaroo ) dominate usage, while specialized skills trigger selectively based on context: collect_divers (20% total / 30% eligible) and upward (7% total / 70% eligible) in Seaquest , versus collect_fruits (13% total / 46% eligible) and handle_enemies (9% total / 40% eligible) in Kangaroo .
Figure 12: NEXUS successfully learns skills in Pong (top), Breakout (middle), and Freeway (bottom).
Figure 13: NEXUS achieves performance comparable to baseline methods across the three evaluated environments.
Figure 14: Comparison to other neuro-symbolic RL methods.
Figure 15: Comparison to other neuro-symbolic RL methods on simplified games.
Algorithm
Seaquest
Kangaroo
Default
Divers Collected
Simplification
Divers Collected
Default
Level Completion (%)
Simplification
Level Completion (%)
PPO
915.3 ± 101.1
(6.7 ± 0.1)
0.0 ± 0.0
(4.7 ± 0.6)
550.0 ± 353.6
(33.3 ± 0.0)
150.0 ± 212.1
(33.3 ± 0.0)
PQN
1361.3 ± 294.6
(4.8 ± 2.8)
0.0 ± 0.0
(4.0 ± 1.5)
949.7 ± 279.9
(47.3 ± 0.4)
338.9 ± 209.1
(40.5 ± 13.6)
NEXUS (neural)
2100.7 ± 614.5
(9.2 ± 4.2)
0.0 ± 0.0
(6.7 ± 6.8)
1889.0 ± 1770.6
(70.6 ± 28.8)
551.0 ± 513.2
(58.9 ± 30.9)
NEXUS (symbolic)
2229.1 ± 172.9
(12.4 ± 0.7)
9416.7 ± 4363.2
(41.0 ± 13.1)
1030.3 ± 553.3
(83.8 ± 2.6)
448.7 ± 147.4
(52.0 ± 9.4)
NEXUS (nesy)
2185.6 ± 1000.1
(10.8 ± 2.9)
2483.3 ± 1607.7
(15.3 ± 9.0)
909.4 ± 519.1
(85.6 ± 12.8)
674.8 ± 115.8
(72.1 ± 16.9)
Appendix
Table 4: Performance Comparison on Seaquest and Kangaroo
Figure 16: Noisy evaluation under 10% misdetection and noise.
Figure 17: Noisy training and evaluation under 10% misdetection and noise.
Figure 18: Subsequent prompts (chat-function) to generate a skill-list and corresponding reward functions (top) and meta-policy function (bottom) for the game Kangaroo.
Figure 19: Generated reward functions for Kangaroo by GPT4-O
Figure 20: Generated fixed meta-policy function for Kangaroo by GPT4-O
Figure 21: Generated reward functions for Seaquest by GPT4-O
Figure 22: Generated fixed meta-policy for Seaquest by GPT4-O
Figure 23: Prompt used to generate a skill-list and meta-policy function for Crafter.
Figure 24: Generated list of essential skills and a rule-based policy for solving Crafter by GPT4-O. The policy prioritizes survival, then combat, followed by resource gathering and crafting, and defaults to exploration otherwise.
Env.
Component
Change from LLM output to final implementation
Type
Seaquest
Skill set
4 skills → 3; avoid obstacles dropped (death penalty moved into fight )
S
rcollect
diver proximity count → direct divers_collected counter
F
rfight
enemy count decrease (+0.5) → score increase (+1), life lost ( −1 )
same conditions; thresholds set (enemy < 40px, oxygen < 10), priority reordered
F+S
Kangaroo
Skill set
4 skills → 3; goal reaching merged into navigate
S
Appendix
Table 5: Human edits to LLM-generated components (GPT-4o, App. F ). F: framework adaptation (JAX, state fields); S: semantic change; M: written manually.
Deep reinforcement learning agents are often misaligned, as they over-exploit early reward signals. Recently, several symbolic approaches have addressed these challenges by encoding sparse objectives along with aligned plans. However, purely symbolic architectures are complex to scale and difficult to apply to continuous settings. Hence, we propose a hybrid approach, inspired by humans' ability to acquire new skills. We use a two-stage framework that injects symbolic structure into neural-based reinforcement learning agents without sacrificing the expressivity of deep policies. Our method, called Hybrid Hierarchical RL (H^2RL), introduces a logical option-based pretraining strategy to steer the learning policy away from short-term reward loops and toward goal-directed behavior while allowing the final policy to be refined via standard environment interaction. Empirically, we show that this approach consistently improves long-horizon decision-making and yields agents that outperform strong neural, symbolic, and neuro-symbolic baselines.
Zihan Ye, Phil Chau, Raban Emunds +5
CS Department, Technical University of Darmstadt, Germany · AIML Group · Hessian AI +2
Explainability remains a key issue in reinforcement learning (RL). Distilling an interpretable policy from an agent trained in a complex environment is particularly challenging when the action space is continuous. We introduce ORCAID, a novel method for extracting interpretable rule-based policies from RL agents operating in mixed continuous-discrete environments with continuous action spaces. Our main contribution is an efficient oblique decision tree training algorithm that partitions the state space by hyperplanes and fits local linear models. The key idea lies in a three-stage split search: efficient random initialization, local refinement, and backward elimination. Finally, adjacent leaves are merged to yield a concise set of interpretable rules describing a given deep RL policy. We evaluate ORCAID across multiple RL environments, demonstrating that the extracted rule-based policies maintain strong performance with a low number of parameters and can even be used to improve the performance of the original deep RL policy.
Ignacio D. Lopez-Miguel, Ezio Bartocci, Thomas Eiter +1
Neuro-symbolic Reinforcement Learning (NeSy-RL) combines symbolic reasoning with gradient-based optimization to achieve interpretable and generalizable policies. Relational concepts, such as "left of" or "close by", serve as foundational building blocks that structure how agents perceive and act. However, conventional approaches require human experts to manually define these concepts, limiting adaptability since concept semantics vary across environments. We propose GRAIL (Grounding Relational Agents through Interactive Learning), a framework that autonomously grounds relational concepts through environmental interaction. GRAIL leverages large language models (LLMs) to provide generic concept representations as weak supervision, then refines them to capture environment-specific semantics. This approach addresses both sparse reward signals and concept misalignment prevalent in underdetermined environments. Experiments on the Atari games Kangaroo, Seaquest, and Skiing demonstrate that GRAIL matches or outperforms agents with manually crafted concepts in simplified settings, and reveals informative trade-offs between reward maximization and high-level goal completion in the full environment.
Hikaru Shindo, Henri Rößler, Quentin Delfosse +1
Technical University of Darmstadt · Google Intrinsic · German Research Center for Artificial Intelligence (DFKI) +1