Reinforcement Learning (RL) has demonstrated its ability to solve complex decision-making problems in a variety of domains, by optimizing reward signals obtained through interaction with an environment. However, many real-world scenarios involve multiple, potentially conflicting objectives that cannot be easily represented by a single scalar reward. Multi-Objective Reinforcement Learning (MORL) addresses this limitation by enabling agents to optimize several objectives simultaneously, explicitly reasoning about trade-offs between them. However, the ``black box" nature of the RL models makes the decision process behind chosen objective trade-offs unclear. Current Explainable Reinforcement Learning (XRL) methods are typically designed for single scalar rewards and do not account for explanations with respect to distinct objectives or user preferences. To address this gap, in this paper we propose TREX, a Trajectory based Explainability framework to explain Multi-objective Reinforcement Learning policies, based on trajectory attribution. TREX generates trajectories directly from the learned expert policy, across different user preferences and clusters them into semantically meaningful temporal segments. We quantify the influence of these behavioural segments on the Pareto trade-off by training complementary policies that exclude specific clusters, measuring the resulting relative deviation on the observed rewards and actions compared to the original expert policy. Experiments on multi-objective MuJoCo environments - HalfCheetah, Ant and Swimmer, demonstrate the framework's ability to isolate and quantify the specific behavioural patterns.
Figures & tables
Figure 1: Overview of the Preference-Level Analysis (PLA) pipeline: (1) sampling trajectories from an expert agent, (2) encoding sub-trajectories into latent space embeddings, (3) clustering trajectories into semantically distinct behaviours, (4) training original policy alongside complementary policies that iteratively exclude specific cluster trajectories, and (5) conducting attribution analysis to quantify the behavioural influence of each cluster.
Figure 2: MuJoCo multi-objective environments
preference
policy
R1
R2
ΔR1
ΔR2
ΔR(c)
RAS(c)
(0.25,0.75)
expert
1033.403
2417.319
-
-
-
original
917.385
2419.520
-
-
-
c0
769.243
2440.969
0.161
-0.009
0.162
0.170
c1
780.596
2435.989
0.149
-0.007
0.149
0.156
c2
917.432
2414.269
0.000
0.002
0.002
0.002
(0.5,0.5)
expert
1975.156
2237.225
-
-
-
Table 1: Preference Level analysis results for MO-HalfCheetah environment - Best overall return deviation ΔR and Reward attribution score for each preference is highlighted in bold.
preference
policy
R1
R2
ΔR1
ΔR2
ΔR(c)
RAS(c)
(0.25,0.75)
expert
1007.504
2427.701
-
-
-
-
original
912.711
2385.175
-
-
-
-
c0
777.709
938.847
0.148
0.606
0.624
0.418
c1
837.784
2307.194
0.082
0.033
0.088
0.004
c2
915.246
2406.504
-0.003
-0.009
0.009
0.006
(0.5,0.5)
expert
2066.393
1940.264
-
-
-
-
Table 2: Preference Level analysis results for MO-Ant environment. Best overall return deviation ΔR and Reward attribution score for each preference is highlighted in bold.
preference
policy
R1
R2
ΔR1
ΔR2
ΔR(c)
RAS(c)
(0.25,0.75)
expert
51.324
149.375
-
-
-
-
original
69.763
147.884
-
-
-
-
c0
73.300
147.842
-0.051
0.000
0.051
0.013
c1
85.556
146.778
-0.226
0.007
0.227
0.062
c2
59.052
147.240
0.154
0.004
0.154
0.035
c3
50.273
148.138
0.279
-0.002
0.279
0.071
Table 3: Preference Level analysis results for MO-Swimmer environment. Best overall return deviation ΔR and Reward attribution score for each preference is highlighted in bold.
Figure 3: MO-HalfCheetah trajectories for the clusters with (0.5, 0.5) user-preference
Figure 4: MO-Ant trajectories for the clusters with (0.5, 0.5) user-preference
Figure 5: MO-Ant Cluster 2 trajectory with (0.25, 0.75) user-preference
A major challenge of Reinforcement Learning is that agents often learn undesired behaviors that seem to defy the reward structure they were given. Explainable Reinforcement Learning (XRL) methods can answer queries such as "explain this specific action", "explain this specific trajectory", and "explain the entire policy". However, XRL lacks a formal definition for behavior as a pattern of actions across many episodes. We provide such a definition, and use it to enable a new query: "Explain this behavior". We present Behavior-Explainable Reinforcement Learning (BXRL), a new problem formulation that treats behaviors as first-class objects. BXRL defines a behavior measure as any function m:Π→R, allowing users to precisely express the pattern of actions that they find interesting and measure how strongly the policy exhibits it. We define contrastive behaviors that reduce the question "why does the agent prefer a to a′?" to "why is m(π) high?" which can be explored with differentiation. We do not implement an explainability method; we instead analyze three existing methods and propose how they could be adapted to explain behavior. We present a port of the HighwayEnv driving environment to JAX, which provides an interface for defining, measuring, and differentiating behaviors with respect to the model parameters.
Ram Rachum, Yotam Amitai, Yonatan Nakar +2
University of California, Berkeley · Tufts University · Independent Researcher +1
Real-world decision-making often requires optimizing multiple competing objectives simultaneously. In reinforcement learning (RL), this is typically addressed by combining reward signals into a single scalar objective via a scalarization function, which can be fragile: small changes in the weights can induce drastically different policies. Multi-objective reinforcement learning (MORL) instead produces sets of policies that explicitly represent trade-offs between objectives. However, these policies are typically presented to the decision maker only through their value vectors, which can obscure substantial behavioral variation: policies that induce distinct trajectories may appear indistinguishable when evaluated solely by expected returns. We propose an exploratory diagnostic workflow that automatically highlights behavioral variation along the Pareto front that objective values alone do not reveal, providing both quantitative and visual tools to support policy inspection. We validate our approach on simple grid examples and scale it to continuous control benchmarks, demonstrating that it remains effective as problem complexity increases.
Antonio Mone, Zuzanna Osika, Florian Felten +4
Delft University of Technology, the Netherlands · ETH Zurich, Switzerland & herve.review, Belgium · ETH Zurich, Switzerland
Many sequential decision-making tasks involve optimizing multiple conflicting objectives, requiring policies that adapt to different user preferences. In multi-objective reinforcement learning (MORL), one widely studied approach} addresses this by training a single policy network conditioned on preference-weighted rewards. In this paper, we explore a novel algorithmic perspective: leveraging reward-free reinforcement learning (RFRL) for MORL. While RFRL has historically been studied independently of MORL, it learns optimal policies for any possible reward function, making it a natural fit for MORL's challenge of handling unknown user preferences. We propose using the RFRL's training objective as an auxiliary task to enhance MORL, enabling more effective knowledge sharing beyond the multi-objective reward function given at training time. To this end, we adapt a state-of-the-art RFRL algorithm to the MORL setting and introduce a preference-guided exploration strategy that focuses learning on relevant parts of the environment. Through extensive experiments and ablation studies, we demonstrate that our approach significantly outperforms the state-of-the-art MORL methods across diverse MO-Gymnasium tasks, achieving superior performance and data efficiency. This work provides the first systematic adaptation of RFRL to MORL, demonstrating its potential as a scalable and empirically effective solution to multi-objective policy learning.
Ying-Tu Chen, Wei Hung, Bing-Shu Wu +2
National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Massachusetts Institute of Technology