Interpretable Meta-Policies for Neural Decision Making

Raban Emunds; Jannis Blüml; Quentin Delfosse; Kristian Kersting

cs.LGOct 27, 2026

Interpretable Meta-Policies for Neural Decision Making

Authors: Raban Emunds, Jannis Blüml, Quentin Delfosse, Kristian Kersting

Organizations: Department of Computer Science, Technical University Darmstadt, Germany · Hessian Center for Artificial Intelligence (hessian.AI) · Centre for Cognitive Science, Darmstadt · German Research Center for Artificial Intelligence (DFKI)

Abstract

Deep Reinforcement Learning remains difficult to interpret and align due to the black-box nature of neural networks. While (neuro-)symbolic approaches improve transparency by incorporating symbolic reasoning, applying them directly to low-level actions results in overly complex, unreadable policies. We introduce NEXUS, a hierarchical RL framework that resolves this tension by combining neural skills with symbolic meta-policies. By shifting transparent reasoning to high-level skills rather than atomic actions, NEXUS drastically reduces symbolic complexity without sacrificing environment performance. Object-centric observations enable high-level reasoning and the design of reward functions and meta-policies by interaction with LLMs. We demonstrate that NEXUS agents learn disentangled skills, are interpretable regarding high-level goals, are better aligned to goals described in the game manual, less prone to reward hacking and more robust to game simplifications than deep RL. Finally, we isolate and evaluate the impact of different meta-policy architectures (neural, symbolic, neuro-symbolic) on environmental return and aligned task performance.

Figures & tables

Figure 1: NEXUS provides high-level interpretability through symbolic reasoning and abstraction. Neural network-based policies are non-modular black box models. While symbolic approaches are fully transparent by leveraging logical rules or decision trees, they quickly become excessively complex if applied to low-level actions. NEXUS abstracts low-level actions into skills and provides transparent high-level decisions.

Figure 2: NEXUS with a neuro-symbolic meta-policy. First, the symbolic state is extracted from the observation. Then, the LLM-generated rules filter potential relevant skills, which the trained meta-policy Q-function selects between. Finally, the selected neural skill outputs the low-level action. Using the same experience, the meta-policy’s Q-function is trained to optimize the environmental reward, while each skill is trained using its own LLM-generated reward, yielding disentangled skills with interpretable objectives.

Figure 3: NEXUS learns disentangled skills from off-policy data. In Seaquest (top), the baseline methods mainly focus on shooting enemies, while the NEXUS approaches acquire the target skills more evenly. Similarly, in Kangaroo (bottom), the NEXUS approaches learn ’Move Up’ and ’Collect Fruits’ reliably, while the baselines focus mostly on ’Handle Threats’. Aggregated over 6 seeds,

\pm 1

standard deviation.

Figure 4: NEXUS succesfully learns LLM-proposed Crafter skills. Compared to the baseline, NEXUS approaches often converge faster on the proposed skills. Aggregated over 6 seeds,

\pm 1

standard deviation.

Figure 5: NEXUS (nesy) restricts skill selection through interpretable rules but remains flexible in ambiguous scenarios. Left: enemy_closeby and diver_available both hold; the meta-policy prefers fighting, likely as the diver is near the enemy. Center: with the diver more accessible, rescuing is preferred. Right: with all divers collected and an enemy nearby, surfacing has the higher estimated return.

Figure 6

Policy	Rule Size and Length Lakkaraju et al. (2016)	ADPL Wu et al. (2017)
NUDGE ( )	16 rules, 46 predicates	23.56 evaluations
NEXUS ( )	4 rules, 4 predicates	3 evaluations
NEXUS ( )	2 rules, 3 predicates	2.50 evaluations

Table 1: Quantitative interpretability metrics of the symbolic decision logic, comparing NUDGE (action level) and NEXUS (skill level) on Seaquest. NEXUS metrics cover the meta-policy only; low-level skills are neural.

Figure 6: NEXUS is competitive. In Seaquest, NEXUS outperforms the baselines on both game reward and rescuing divers. In Kangaroo, NEXUS approaches progress farther. Aggregated over 6 seeds,

\pm 1

standard deviation.

Figure 7: NEXUS remains practical on simplified games. While baseline performance on aligned metrics remains low, neuro-symbolic and fully symbolic NEXUS improve on modified Seaquest (left). Modified kangaroo (middle) reduces performance of symbolic NEXUS, while neuro-symbolic NEXUS remains effective. Crafter performance (right) remains consistent or improves for all NEXUS versions. Aggregated over 6 seeds,

\pm 1

standard deviation.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Existing approaches are not interpretable. Neural RL agents are black-boxes and learn shortcuts. Hierarchical RL remains a black box and typically produces entangled skills. Symbolic agents are transparent, but grow massively with policy complexity.

\phi_{\text{meta}}\leftarrow\phi_{\text{meta}}+\frac{\alpha_{t}}{2b}\nabla_{\phi_{\text{meta}}}\sum_{(j,\tau)\in B}\left(R_{\lambda,\tau}^{j}-Q_{\phi_{\text{meta}}}(s_{\tau}^{j})\right)^{2}

Appendix

Algorithm 2 Hierarchical PQN

Panel of Figure 9 — Figure 9: The evaluation environments: Crafter, Seaquest and Kangaroo.

Parameter	PQN(-based)	PPO
Total Timesteps	$5\times 10^{7}$	$5\times 10^{7}$
Num Environments	1024	128
Num Steps per Update	32	32
Learning Rate	$2.5\times 10^{-4}$	$2.5\times 10^{-4}$
Max Grad Norm	10	0.5
Discount Factor ( $\gamma$ )	0.99	0.99

Appendix

Table 2: Atari Hyperparameters

Parameter	PQN(-based)
Total Timesteps	$1\times 10^{9}$
Num Environments	512
Num Steps per Update	128
Learning Rate	$3.0\times 10^{-4}$
Max Grad Norm	0.5
Discount Factor ( $\gamma$ )	0.99

Appendix

Table 3: Crafter Hyperparameters

Figure 10: Reward-shaping is not sufficient. Replacing NEXUS’s neuro-symbolic meta-policy with LLM-designed or statically weighted reward shaping leads to severe performance degradation across both environments. While baselines successfully acquire isolated skills (e.g., surface in Seaquest), they fail to learn all skills equally well or to coordinate them—demonstrating that NEXUS’s skill disentanglement is essential for achieving high overall performance.

Figure 11: Skill activation dynamics and performance impact of the neuro-symbolic meta-policy in Seaquest and Kangaroo . Replacing the meta-policy with random skill choices significantly degrades performance, demonstrating its critical role in maximizing returns. Default skills ( fight_enemies in Seaquest , navigate in Kangaroo ) dominate usage, while specialized skills trigger selectively based on context: collect_divers (20% total / 30% eligible) and upward (7% total / 70% eligible) in Seaquest , versus collect_fruits (13% total / 46% eligible) and handle_enemies (9% total / 40% eligible) in Kangaroo .

Figure 12: NEXUS successfully learns skills in Pong (top), Breakout (middle), and Freeway (bottom).

Figure 13: NEXUS achieves performance comparable to baseline methods across the three evaluated environments.

Figure 14: Comparison to other neuro-symbolic RL methods.

Figure 15: Comparison to other neuro-symbolic RL methods on simplified games.

Algorithm	Seaquest				Kangaroo
Algorithm	Default	Divers Collected	Simplification	Divers Collected	Default	Level Completion (%)	Simplification	Level Completion (%)
PPO	915.3 $\pm$ 101.1	(6.7 $\pm$ 0.1)	0.0 $\pm$ 0.0	(4.7 $\pm$ 0.6)	550.0 $\pm$ 353.6	(33.3 $\pm$ 0.0)	150.0 $\pm$ 212.1	(33.3 $\pm$ 0.0)
PQN	1361.3 $\pm$ 294.6	(4.8 $\pm$ 2.8)	0.0 $\pm$ 0.0	(4.0 $\pm$ 1.5)	949.7 $\pm$ 279.9	(47.3 $\pm$ 0.4)	338.9 $\pm$ 209.1	(40.5 $\pm$ 13.6)
NEXUS (neural)	2100.7 $\pm$ 614.5	(9.2 $\pm$ 4.2)	0.0 $\pm$ 0.0	(6.7 $\pm$ 6.8)	1889.0 $\pm$ 1770.6	(70.6 $\pm$ 28.8)	551.0 $\pm$ 513.2	(58.9 $\pm$ 30.9)
NEXUS (symbolic)	2229.1 $\pm$ 172.9	(12.4 $\pm$ 0.7)	9416.7 $\pm$ 4363.2	(41.0 $\pm$ 13.1)	1030.3 $\pm$ 553.3	(83.8 $\pm$ 2.6)	448.7 $\pm$ 147.4	(52.0 $\pm$ 9.4)
NEXUS (nesy)	2185.6 $\pm$ 1000.1	(10.8 $\pm$ 2.9)	2483.3 $\pm$ 1607.7	(15.3 $\pm$ 9.0)	909.4 $\pm$ 519.1	(85.6 $\pm$ 12.8)	674.8 $\pm$ 115.8	(72.1 $\pm$ 16.9)

Appendix

Table 4: Performance Comparison on Seaquest and Kangaroo

Figure 16: Noisy evaluation under 10% misdetection and noise.

Figure 17: Noisy training and evaluation under 10% misdetection and noise.

Panel of Figure 18 — Figure 18: Subsequent prompts (chat-function) to generate a skill-list and corresponding reward functions (top) and meta-policy function (bottom) for the game Kangaroo.

Panel of Figure 19 — Figure 19: Generated reward functions for Kangaroo by GPT4-O

Panel of Figure 20 — Figure 20: Generated fixed meta-policy function for Kangaroo by GPT4-O

Panel of Figure 21 — Figure 21: Generated reward functions for Seaquest by GPT4-O

Panel of Figure 22 — Figure 22: Generated fixed meta-policy for Seaquest by GPT4-O

Panel of Figure 23 — Figure 23: Prompt used to generate a skill-list and meta-policy function for Crafter.

Panel of Figure 24 — Figure 24: Generated list of essential skills and a rule-based policy for solving Crafter by GPT4-O. The policy prioritizes survival, then combat, followed by resource gathering and crafting, and defaults to exploration otherwise.

Env.	Component	Change from LLM output to final implementation	Type
Seaquest	Skill set	4 skills $\rightarrow$ 3; avoid obstacles dropped (death penalty moved into fight )	S
	$r_{\text{collect}}$	diver proximity count $\rightarrow$ direct divers_collected counter	F
	$r_{\text{fight}}$	enemy count decrease (+0.5) $\rightarrow$ score increase (+1), life lost ( $-1$ )	S
	$r_{\text{surface}}$	surfacing with 6 divers / low oxygen $\rightarrow$ oxygen refill (+0.1)	S
	Meta-policy	same conditions; thresholds set (enemy $<$ 40px, oxygen $<$ 10), priority reordered	F+S
Kangaroo	Skill set	4 skills $\rightarrow$ 3; goal reaching merged into navigate	S

Appendix

Table 5: Human edits to LLM-generated components (GPT-4o, App. F ). F: framework adaptation (JAX, state fields); S: semantic change; M: written manually.

Interpretability in RL

Interpretability in reinforcement learning makes learned policies, decisions, and internal representations understandable and auditable.

See all 71 papers

Boosting deep Reinforcement Learning using pretraining with Logical Options
Mar 6, 2026Zihan Ye, Phil Chau, Raban Emunds +5Neuro-Symbolic RL Reinforcement Learning
CS Department, Technical University of Darmstadt, Germany · AIML Group · Hessian AI +2
ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies
Jul 8, 2026Ignacio D. Lopez-Miguel, Ezio Bartocci, Thomas Eiter +1Reinforcement Learning Interpretable ML
TU Wien Vienna, Austria
GRAIL: Autonomous Concept Grounding for Neuro-Symbolic Reinforcement Learning
Apr 18, 2026Hikaru Shindo, Henri Rößler, Quentin Delfosse +1Reinforcement Learning LLM-Guided RL
Technical University of Darmstadt · Google Intrinsic · German Research Center for Artificial Intelligence (DFKI) +1

Mar 6, 2026J/K move · Enter open · S save

Boosting deep Reinforcement Learning using pretraining with Logical Options

Zihan Ye, Phil Chau, Raban Emunds, Jannis Blüml+4

CS Department, Technical University of Darmstadt, Germany · AIML Group · Hessian AI+2

Neuro-Symbolic RL Reinforcement Learning Hierarchical RL

Open paper AskPDF arXiv

Deep reinforcement learning agents are often misaligned, as they over-exploit early reward signals. Recently, several symbolic approaches have addressed these challenges by encoding sparse objectives along with aligned plans. However, purely symbolic architectures are complex to scale and difficult to apply to continuous settings. Hence, we propose a hybrid approach, inspired by humans' ability to acquire new skills. We use a two-stage framework that injects symbolic structure into neural-based reinforcement learning agents without sacrificing the expressivity of deep policies. Our method, called Hybrid Hierarchical RL (H^2RL), introduces a logical option-based pretraining strategy to steer the learning policy away from short-term reward loops and toward goal-directed behavior while allowing the final policy to be refined via standard environment interaction. Empirically, we show that this approach consistently improves long-horizon decision-making and yields agents that outperform strong neural, symbolic, and neuro-symbolic baselines.

Abstract

Interpretability in RL

Explore similar work

Boosting deep Reinforcement Learning using pretraining with Logical Options

ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies

GRAIL: Autonomous Concept Grounding for Neuro-Symbolic Reinforcement Learning