Prism: Policy Reuse via Interpretable Strategy Mapping in Reinforcement Learning

Thomas Pravetz

cs.LGMar 4, 2026

Prism: Policy Reuse via Interpretable Strategy Mapping in Reinforcement Learning

Authors: Thomas Pravetz

Abstract

We present PRISM (Policy Reuse via Interpretable Strategy Mapping), a framework that grounds reinforcement learning agents' decisions in discrete, causally validated concepts and uses those concepts as a zero-shot transfer interface between agents trained with different algorithms. PRISM clusters each agent's encoder features into $K$ concepts via K-means. Causal intervention establishes that these concepts directly drive - not merely correlate with - agent behavior: overriding concept assignments changes the selected action in 69.4% of interventions ( $p = 8.6 \times 10^{-86}$ , 2500 interventions). Concept importance and usage frequency are dissociated: the most-used concept (C47, 33.0% frequency) causes only a 9.4% win-rate drop when ablated, while ablating C16 (15.4% frequency) collapses win rate from 100% to 51.8%. Because concepts causally encode strategy, aligning them via optimal bipartite matching transfers strategic knowledge zero-shot. On Go~7 $\times$ 7 with three independently trained agents, concept transfer achieves 69.5% $\pm$ 3.2% and 76.4% $\pm$ 3.4% win rate against a standard engine across the two successful transfer pairs (10 seeds), compared to 3.5% for a random agent and 9.2% without alignment. Transfer succeeds when the source policy is strong; geometric alignment quality predicts nothing ( $R^2 \approx 0$ ). The framework is scoped to domains where strategic state is naturally discrete: the identical pipeline on Atari Breakout yields bottleneck policies at random-agent performance, confirming that the Go results reflect a structural property of the domain.

Figures & tables

Source	Target	Align Sim	Win Rate	$p$
BC	DQN	0.045	76.4% $\pm$ 3.4%	$<0.001$
PPO	DQN	0.021	69.5% $\pm$ 3.2%	$<0.001$
DQN	PPO	0.021	49.8% $\pm$ 2.6% †	0.82
DAgger	PPO	0.045	41.5% $\pm$ 6.0% †	0.002 ‡
DQN	BC	0.045	38.7% $\pm$ 4.9% †	0.0001 ‡
PPO	BC	—	0.0% (degenerate)	—

Table 1: Zero-shot transfer on Go 7

\times

7, 10 seeds

\times

100 games.

p

is a one-sample

t

-test against the 50% null; random agent achieves 3.5%. Daggers ( † ) mark results not significantly above 50%.

Method	Type	Win Rate	$p$ vs. Hungarian
Greedy NN †	1:1 greedy	97.2% $\pm$ 1.6%	$<$ 0.001 ∗
Hungarian (PRISM)	1:1 optimal	68.8% $\pm$ 4.7%	—
Procrustes	1:1 rotated	51.6% $\pm$ 6.4%	0.003 ∗
Random permutation	1:1 random	48.4% $\pm$ 33.7%	0.29
Identity (no alignment)	1:1 trivial	9.2% $\pm$ 3.0%	$<$ 0.001 ∗

Table 2: Alignment method comparison for PPO

\to

DQN, 5 seeds

\times

50 games. Win rates

\pm

std. Pairwise Welch’s

t

-test vs. Hungarian;

{}^{*}p<0.05

. † Greedy NN is degenerate: all source concepts map to one target concept, producing a fixed-action policy (mean game length 48.9 moves vs. 27.5 for Hungarian).

Method	Zero-Shot WR	Requires Training?
Prism (10 seeds avg.)	69.5% $\pm$ 3.2%	No
From scratch (gen 0)	0%	—
From scratch (gen 10, vs. GnuGo)	$\approx$ 52%	Yes (200K steps)
From scratch (gen 49 final)	72% (noisy, single seed)	Yes (1M steps)
Random mapping (3 seeds)	57%, 95%, 0%	No

Table 3: Transfer method comparison, PPO

\to

DQN on Go 7

\times

Figure 1: Fine-tuning after zero-shot transfer. Both conditions run REINFORCE on the DQN concept bottleneck; they differ only in initialization. The transferred policy crosses 60% at generation 5 (50K steps); the from-scratch policy reaches 27% at generation 40 (400K steps) without crossing the threshold. Single seed; treat magnitude as indicative.

Agent	Bottleneck	Baseline
PPO	99.1% $\pm$ 0.7%	98.5% $\pm$ 1.5%
DQN	64.1% $\pm$ 5.4%	54.5% $\pm$ 12.8%

Table 4: Bottleneck vs. unconstrained baseline, mean

\pm

std win rate aggregated across all GnuGo evaluation runs. GnuGo level has no consistent effect on 7

\times

7; results are pooled.

Figure 2: Concept ablation: frequency vs. win-rate drop when ablated. C16 (15.4% frequency) causes a 48.2 pp drop; C47 (33.0% frequency, the most-used concept) causes only a 9.4 pp drop. Three concepts improve win rate when removed (negative drop), suggesting they encode suboptimal patterns.

Concept	Frequency	Ablated WR	WR Drop
C16	15.4%	51.8%	48.2%
C47	33.0%	90.6%	9.4%
C29	6.3%	90.6%	9.4%
C6	4.6%	92.8%	7.2%

Table 5: Top concept ablation results. Baseline win rate = 100% vs. GnuGo. Ablation replaces concept actions with uniform random legal moves (500 games per concept).

Figure 3: Win rate vs.

K

(number of concepts) at 300K training steps against GnuGo L3. Transfer performance peaks at

K{=}32

(76%); direct bottleneck performance is higher at

K{=}8

and

K{=}64

. The paper uses

K{=}64

based on direct performance at full training (3.4M steps), where larger

K

captures finer-grained structure. Results at 300K steps are not directly comparable to Table 1 (full curriculum).

Zero-Shot Policy Transfer

See all 14 papers

Jun 2, 2025

MAGIK: Mapping to Analogous Goals via Imagination-enabled Knowledge Transfer

Humans excel at analogical reasoning - applying knowledge from one task to a related one with minimal relearning. In contrast, reinforcement learning (RL) agents typically require extensive retraining even when new tasks share structural similarities with previously learned ones. In this work, we propose MAGIK, a novel framework that enables RL agents to transfer knowledge to analogous tasks without interacting with the target environment. Our approach leverages an imagination mechanism to map entities in the target task to their analogues in the source domain, allowing the agent to reuse its original policy. Experiments on custom MiniGrid and MuJoCo tasks show that MAGIK achieves effective zero-shot transfer using only a small number of human-labelled examples. We compare our approach to related baselines and highlight how it offers a novel and effective mechanism for knowledge transfer via imagination-based analogy mapping.

Ajsal Shereef Palattuparambil, Thommen George Karimpanal, Santu Rana

Applied Artificial Intelligence Initiative, Deakin University · School of IT, Deakin University

Reinforcement Learning Transfer Learning Zero-Shot RL Zero-Shot Policy Transfer

Ask PDF

Jul 31, 2026

Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL

Modern large language models (LLMs) are expected not just to answer correctly, but to adapt their behavior to different human values and use cases. As a result, multi-reward reinforcement learning (RL) has become an increasingly important problem for LLMs, where each reward captures a different aspect of desired behavior. However, optimizing with multiple rewards suffers from a more severe alignment tax issue, where different optimization objectives can trade off or even conflict with each other, leading to unstable and inefficient post-training. In this work, we propose PRISM, a new multi-reward RL framework built upon the idea of policy-space decomposition and composition. Instead of compositing different rewards, PRISM optimizes a set of standalone positive policies and a global negative policy. This alleviates the potential conflict during multi-reward policy optimization, while enabling controllability during inference by flexible policy composition. Experiments on scientific reasoning, tool-use reasoning, and helpfulness-safety alignment show that PRISM consistently outperforms existing multi-reward RL baselines, with extra controllability for inference-time preference control.

Ruiming Liang, Yi Zhong, Yizhen Yuan +6

1Fundation Model Research Center, CASIA · School of Artificial Intelligence, UCAS · Institute for AI Industry Research (AIR), Tsinghua University +1

Multi-Objective Reinforcement Learning RL for Language Models Policy Learning

Ask PDF

Apr 6, 2026

Selecting Decision-Relevant Concepts in Reinforcement Learning

Training interpretable concept-based policies requires practitioners to manually select which human-understandable concepts an agent should reason with when making sequential decisions. This selection demands domain expertise, is time-consuming and costly, scales poorly with the number of candidates, and provides no performance guarantees. To overcome this limitation, we propose the first algorithms for principled automatic concept selection in sequential decision-making. Our key insight is that concept selection can be viewed through the lens of state abstraction: intuitively, a concept is decision-relevant if removing it would cause the agent to confuse states that require different actions. As a result, agents should rely on decision-relevant concepts; states with the same concept representation should share the same optimal action, which preserves the optimal decision structure of the original state space. This perspective leads to the Decision-Relevant Selection (DRS) algorithm, which selects a subset of concepts from a candidate set, along with performance bounds relating the selected concepts to the performance of the resulting policy. Empirically, DRS automatically recovers manually curated concept sets while matching or exceeding their performance, and improves the effectiveness of test-time concept interventions across reinforcement learning benchmarks and real-world healthcare environments.

Naveen Raman, Stephanie Milani, Fei Fang

Carnegie Mellon University · New York University

Reinforcement Learning Interpretability in RL

Ask PDF

Abstract

Zero-Shot Policy Transfer

Explore similar work

MAGIK: Mapping to Analogous Goals via Imagination-enabled Knowledge Transfer

Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL

Selecting Decision-Relevant Concepts in Reinforcement Learning