TREX: Trajectory Explanations for Multi-Objective Reinforcement Learning

Dilina Rajapakse; Juan C. Rosero; Ivana Dusparic

cs.LGMar 23, 2026

TREX: Trajectory Explanations for Multi-Objective Reinforcement Learning

Authors: Dilina Rajapakse, Juan C. Rosero, Ivana Dusparic

Organizations: Trinity College Dublin, Ireland

Abstract

Reinforcement Learning (RL) has demonstrated its ability to solve complex decision-making problems in a variety of domains, by optimizing reward signals obtained through interaction with an environment. However, many real-world scenarios involve multiple, potentially conflicting objectives that cannot be easily represented by a single scalar reward. Multi-Objective Reinforcement Learning (MORL) addresses this limitation by enabling agents to optimize several objectives simultaneously, explicitly reasoning about trade-offs between them. However, the ``black box" nature of the RL models makes the decision process behind chosen objective trade-offs unclear. Current Explainable Reinforcement Learning (XRL) methods are typically designed for single scalar rewards and do not account for explanations with respect to distinct objectives or user preferences. To address this gap, in this paper we propose TREX, a Trajectory based Explainability framework to explain Multi-objective Reinforcement Learning policies, based on trajectory attribution. TREX generates trajectories directly from the learned expert policy, across different user preferences and clusters them into semantically meaningful temporal segments. We quantify the influence of these behavioural segments on the Pareto trade-off by training complementary policies that exclude specific clusters, measuring the resulting relative deviation on the observed rewards and actions compared to the original expert policy. Experiments on multi-objective MuJoCo environments - HalfCheetah, Ant and Swimmer, demonstrate the framework's ability to isolate and quantify the specific behavioural patterns.

Figures & tables

Figure 1: Overview of the Preference-Level Analysis (PLA) pipeline: (1) sampling trajectories from an expert agent, (2) encoding sub-trajectories into latent space embeddings, (3) clustering trajectories into semantically distinct behaviours, (4) training original policy alongside complementary policies that iteratively exclude specific cluster trajectories, and (5) conducting attribution analysis to quantify the behavioural influence of each cluster.

Panel of Figure 2 — Figure 2: MuJoCo multi-objective environments

preference	policy	$R^{1}$	$R^{2}$	$\Delta R^{1}$	$\Delta R^{2}$	$\Delta R\left(c\right)$	RAS(c)
(0.25,0.75)	expert	1033.403	2417.319	-	-	-
	original	917.385	2419.520	-	-	-
	c0	769.243	2440.969	0.161	-0.009	0.162	0.170
	c1	780.596	2435.989	0.149	-0.007	0.149	0.156
	c2	917.432	2414.269	0.000	0.002	0.002	0.002
(0.5,0.5)	expert	1975.156	2237.225	-	-	-

Table 1: Preference Level analysis results for MO-HalfCheetah environment - Best overall return deviation

\Delta R

and Reward attribution score for each preference is highlighted in bold.

preference	policy	$R^{1}$	$R^{2}$	$\Delta R^{1}$	$\Delta R^{2}$	$\Delta R\left(c\right)$	RAS(c)
(0.25,0.75)	expert	1007.504	2427.701	-	-	-	-
	original	912.711	2385.175	-	-	-	-
	c0	777.709	938.847	0.148	0.606	0.624	0.418
	c1	837.784	2307.194	0.082	0.033	0.088	0.004
	c2	915.246	2406.504	-0.003	-0.009	0.009	0.006
(0.5,0.5)	expert	2066.393	1940.264	-	-	-	-

Table 2: Preference Level analysis results for MO-Ant environment. Best overall return deviation

\Delta R

and Reward attribution score for each preference is highlighted in bold.

preference	policy	$R^{1}$	$R^{2}$	$\Delta R^{1}$	$\Delta R^{2}$	$\Delta R\left(c\right)$	RAS(c)
(0.25,0.75)	expert	51.324	149.375	-	-	-	-
	original	69.763	147.884	-	-	-	-
	c0	73.300	147.842	-0.051	0.000	0.051	0.013
	c1	85.556	146.778	-0.226	0.007	0.227	0.062
	c2	59.052	147.240	0.154	0.004	0.154	0.035
	c3	50.273	148.138	0.279	-0.002	0.279	0.071

Table 3: Preference Level analysis results for MO-Swimmer environment. Best overall return deviation

\Delta R

and Reward attribution score for each preference is highlighted in bold.

Panel of Figure 3 — Figure 3: MO-HalfCheetah trajectories for the clusters with (0.5, 0.5) user-preference

Panel of Figure 4 — Figure 4: MO-Ant trajectories for the clusters with (0.5, 0.5) user-preference

Figure 5: MO-Ant Cluster 2 trajectory with (0.25, 0.75) user-preference

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: MO-HalfCheetah

Panel of Figure 8 — Figure 8: MO-Swimmer

Interpretability in RL

Interpretability in reinforcement learning makes learned policies, decisions, and internal representations understandable and auditable.

See all 71 papers

BXRL: Behavior-Explainable Reinforcement Learning
Mar 24, 2026Ram Rachum, Yotam Amitai, Yonatan Nakar +2Interpretability in RL
University of California, Berkeley · Tufts University · Independent Researcher +1
Objective-Behavior Alignment: Diagnostics for MORL Policy Selection
Jun 19, 2026Antonio Mone, Zuzanna Osika, Florian Felten +4Multi-Objective Reinforcement Learning Reinforcement Learning
Delft University of Technology, the Netherlands · ETH Zurich, Switzerland & herve.review, Belgium · ETH Zurich, Switzerland
A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning
Apr 27, 2026Ying-Tu Chen, Wei Hung, Bing-Shu Wu +2Multi-Objective Reinforcement Learning Reinforcement Learning
National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Massachusetts Institute of Technology

Mar 24, 2026J/K move · Enter open · S save

BXRL: Behavior-Explainable Reinforcement Learning

Ram Rachum, Yotam Amitai, Yonatan Nakar, Reuth Mirsky+1

University of California, Berkeley · Tufts University · Independent Researcher+1

Interpretability in RL

Open paper AskPDF arXiv

A major challenge of Reinforcement Learning is that agents often learn undesired behaviors that seem to defy the reward structure they were given. Explainable Reinforcement Learning (XRL) methods can answer queries such as "explain this specific action", "explain this specific trajectory", and "explain the entire policy". However, XRL lacks a formal definition for behavior as a pattern of actions across many episodes. We provide such a definition, and use it to enable a new query: "Explain this behavior". We present Behavior-Explainable Reinforcement Learning (BXRL), a new problem formulation that treats behaviors as first-class objects. BXRL defines a behavior measure as any function

m : Π\to \mathbb{R}

, allowing users to precisely express the pattern of actions that they find interesting and measure how strongly the policy exhibits it. We define contrastive behaviors that reduce the question "why does the agent prefer

a

a'

?" to "why is

m(π)

high?" which can be explored with differentiation. We do not implement an explainability method; we instead analyze three existing methods and propose how they could be adapted to explain behavior. We present a port of the HighwayEnv driving environment to JAX, which provides an interface for defining, measuring, and differentiating behaviors with respect to the model parameters.

Abstract

Interpretability in RL

Explore similar work

BXRL: Behavior-Explainable Reinforcement Learning

Objective-Behavior Alignment: Diagnostics for MORL Policy Selection

A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning