Prism: Policy Reuse via Interpretable Strategy Mapping in Reinforcement Learning

Thomas Pravetz

cs.LGMar 4, 2026

Prism: Policy Reuse via Interpretable Strategy Mapping in Reinforcement Learning

Authors: Thomas Pravetz

Abstract

We present PRISM (Policy Reuse via Interpretable Strategy Mapping), a framework that grounds reinforcement learning agents' decisions in discrete, causally validated concepts and uses those concepts as a zero-shot transfer interface between agents trained with different algorithms. PRISM clusters each agent's encoder features into $K$ concepts via K-means. Causal intervention establishes that these concepts directly drive - not merely correlate with - agent behavior: overriding concept assignments changes the selected action in 69.4% of interventions ( $p = 8.6 \times 10^{-86}$ , 2500 interventions). Concept importance and usage frequency are dissociated: the most-used concept (C47, 33.0% frequency) causes only a 9.4% win-rate drop when ablated, while ablating C16 (15.4% frequency) collapses win rate from 100% to 51.8%. Because concepts causally encode strategy, aligning them via optimal bipartite matching transfers strategic knowledge zero-shot. On Go~7 $\times$ 7 with three independently trained agents, concept transfer achieves 69.5% $\pm$ 3.2% and 76.4% $\pm$ 3.4% win rate against a standard engine across the two successful transfer pairs (10 seeds), compared to 3.5% for a random agent and 9.2% without alignment. Transfer succeeds when the source policy is strong; geometric alignment quality predicts nothing ( $R^2 \approx 0$ ). The framework is scoped to domains where strategic state is naturally discrete: the identical pipeline on Atari Breakout yields bottleneck policies at random-agent performance, confirming that the Go results reflect a structural property of the domain.

Figures & tables

Source	Target	Align Sim	Win Rate	$p$
BC	DQN	0.045	76.4% $\pm$ 3.4%	$<0.001$
PPO	DQN	0.021	69.5% $\pm$ 3.2%	$<0.001$
DQN	PPO	0.021	49.8% $\pm$ 2.6% †	0.82
DAgger	PPO	0.045	41.5% $\pm$ 6.0% †	0.002 ‡
DQN	BC	0.045	38.7% $\pm$ 4.9% †	0.0001 ‡
PPO	BC	—	0.0% (degenerate)	—

Table 1: Zero-shot transfer on Go 7

\times

7, 10 seeds

\times

100 games.

p

is a one-sample

t

-test against the 50% null; random agent achieves 3.5%. Daggers ( † ) mark results not significantly above 50%.

Method	Type	Win Rate	$p$ vs. Hungarian
Greedy NN †	1:1 greedy	97.2% $\pm$ 1.6%	$<$ 0.001 ∗
Hungarian (PRISM)	1:1 optimal	68.8% $\pm$ 4.7%	—
Procrustes	1:1 rotated	51.6% $\pm$ 6.4%	0.003 ∗
Random permutation	1:1 random	48.4% $\pm$ 33.7%	0.29
Identity (no alignment)	1:1 trivial	9.2% $\pm$ 3.0%	$<$ 0.001 ∗

Table 2: Alignment method comparison for PPO

\to

DQN, 5 seeds

\times

50 games. Win rates

\pm

std. Pairwise Welch’s

t

-test vs. Hungarian;

{}^{*}p<0.05

. † Greedy NN is degenerate: all source concepts map to one target concept, producing a fixed-action policy (mean game length 48.9 moves vs. 27.5 for Hungarian).

Method	Zero-Shot WR	Requires Training?
Prism (10 seeds avg.)	69.5% $\pm$ 3.2%	No
From scratch (gen 0)	0%	—
From scratch (gen 10, vs. GnuGo)	$\approx$ 52%	Yes (200K steps)
From scratch (gen 49 final)	72% (noisy, single seed)	Yes (1M steps)
Random mapping (3 seeds)	57%, 95%, 0%	No

Table 3: Transfer method comparison, PPO

\to

DQN on Go 7

\times

Figure 1: Fine-tuning after zero-shot transfer. Both conditions run REINFORCE on the DQN concept bottleneck; they differ only in initialization. The transferred policy crosses 60% at generation 5 (50K steps); the from-scratch policy reaches 27% at generation 40 (400K steps) without crossing the threshold. Single seed; treat magnitude as indicative.

Agent	Bottleneck	Baseline
PPO	99.1% $\pm$ 0.7%	98.5% $\pm$ 1.5%
DQN	64.1% $\pm$ 5.4%	54.5% $\pm$ 12.8%

Table 4: Bottleneck vs. unconstrained baseline, mean

\pm

std win rate aggregated across all GnuGo evaluation runs. GnuGo level has no consistent effect on 7

\times

7; results are pooled.

Figure 2: Concept ablation: frequency vs. win-rate drop when ablated. C16 (15.4% frequency) causes a 48.2 pp drop; C47 (33.0% frequency, the most-used concept) causes only a 9.4 pp drop. Three concepts improve win rate when removed (negative drop), suggesting they encode suboptimal patterns.

Concept	Frequency	Ablated WR	WR Drop
C16	15.4%	51.8%	48.2%
C47	33.0%	90.6%	9.4%
C29	6.3%	90.6%	9.4%
C6	4.6%	92.8%	7.2%

Table 5: Top concept ablation results. Baseline win rate = 100% vs. GnuGo. Ablation replaces concept actions with uniform random legal moves (500 games per concept).

Figure 3: Win rate vs.

K

(number of concepts) at 300K training steps against GnuGo L3. Transfer performance peaks at

K{=}32

(76%); direct bottleneck performance is higher at

K{=}8

and

K{=}64

. The paper uses

K{=}64

based on direct performance at full training (3.4M steps), where larger

K

captures finer-grained structure. Results at 300K steps are not directly comparable to Table 1 (full curriculum).

Zero-Shot Policy Transfer

See all 14 papers

MAGIK: Mapping to Analogous Goals via Imagination-enabled Knowledge Transfer
Jun 2, 2025Ajsal Shereef Palattuparambil, Thommen George Karimpanal, Santu RanaReinforcement Learning Transfer Learning
Applied Artificial Intelligence Initiative, Deakin University · School of IT, Deakin University
Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL
Jul 31, 2026Ruiming Liang, Yi Zhong, Yizhen Yuan +6Multi-Objective Reinforcement Learning RL for Language Models
1Fundation Model Research Center, CASIA · School of Artificial Intelligence, UCAS · Institute for AI Industry Research (AIR), Tsinghua University +1
Selecting Decision-Relevant Concepts in Reinforcement Learning
Apr 6, 2026Naveen Raman, Stephanie Milani, Fei FangReinforcement Learning Interpretability in RL
Carnegie Mellon University · New York University

Jun 2, 2025J/K move · Enter open · S save

MAGIK: Mapping to Analogous Goals via Imagination-enabled Knowledge Transfer

Ajsal Shereef Palattuparambil, Thommen George Karimpanal, Santu Rana

Applied Artificial Intelligence Initiative, Deakin University · School of IT, Deakin University

Reinforcement Learning Transfer Learning Zero-Shot RL Zero-Shot Policy Transfer

Open paper AskPDF arXiv

Humans excel at analogical reasoning - applying knowledge from one task to a related one with minimal relearning. In contrast, reinforcement learning (RL) agents typically require extensive retraining even when new tasks share structural similarities with previously learned ones. In this work, we propose MAGIK, a novel framework that enables RL agents to transfer knowledge to analogous tasks without interacting with the target environment. Our approach leverages an imagination mechanism to map entities in the target task to their analogues in the source domain, allowing the agent to reuse its original policy. Experiments on custom MiniGrid and MuJoCo tasks show that MAGIK achieves effective zero-shot transfer using only a small number of human-labelled examples. We compare our approach to related baselines and highlight how it offers a novel and effective mechanism for knowledge transfer via imagination-based analogy mapping.

Abstract

Zero-Shot Policy Transfer

Explore similar work

MAGIK: Mapping to Analogous Goals via Imagination-enabled Knowledge Transfer

Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL

Selecting Decision-Relevant Concepts in Reinforcement Learning