Command-Space Counterfactual Explanations for Pareto-Conditioned Reinforcement Learning

Joanikij Chulev; Hendrik Baier

cs.LGAug 15, 2026

Command-Space Counterfactual Explanations for Pareto-Conditioned Reinforcement Learning

Authors: Joanikij Chulev, Hendrik Baier

Organizations: Centrum Wiskunde & Informatica (CWI), Amsterdam · Eindhoven University of Technology

Abstract

Pareto Conditioned Networks learn multiple multi-objective reinforcement learning behaviours by conditioning a single policy on a desired return command. However, the local mapping from command and state to action remains opaque. We propose command-space counterfactual explanations for PCNs: given a fixed state, original command, and foil action, we search, in a black-box setting, for a minimally changed desired-return command under which the same trained policy would choose the foil. Our contributions are threefold. First, we formulate PCN explanations as return-command interventions, using a return-only PCN variant that avoids the added ambiguity of horizon-conditioning. Second, we adapt adversarial machine learning methods to reinforcement-learning explanations. Third, we introduce a boundary-seeded directional search that improves over purely local optimization in the command-action landscape, resulting in our proposed approach CF-ZOO. The resulting explanations are actionable and intuitively expressed in the user's own preferences: "If your trade-off had shifted slightly towards X, the agent would have chosen Y."

Figures & tables

Figure 1: Pareto front recovered by the return-only PCN variant on Minecart. Blue points (O) show the final logged learning front; orange points (X) show rollout-achieved returns.

Figure 2: (a) Branch-Path Environment - Custom. (b) Collect-Two Environment - Custom. (c) Minecart Environment - Classic.

Env.	Method	Cases	Succ.	Rate [95% CI]	Dist.
Branch	White-CW	35	7	20.0 [10.0,35.9]	0.270
Branch	CF-ZOO	35	25	71.4 [54.9,83.7]	0.416
B-Bottles	White-CW	8	2	25.0 [7.1,59.1]	0.017
B-Bottles	CF-ZOO	8	6	75.0 [40.9,92.9]	0.030
Collect	White-CW	136	29	21.3 [15.3,28.9]	0.514
Collect	CF-ZOO	136	136	100.0 [97.3,100.0]	0.444

Table 1: Comparison of our perturbation method with a white-box C&W baseline. Success-rate intervals are Wilson score

95\%

confidence intervals. Dist. denotes the mean scaled

\ell_{2}

distance over successful counterfactuals.

Figure 3: Command-space decision landscapes for four qualitative showcase scenarios. The title explains the flip, which return was chosen, and the timestep of the rollout. We mark the original command (circle) and the C&W (cross), our CF-ZOO (star), and nearest Pareto-front (diamond) counterfactuals, with arrows for the local margin gradient. Axes show true values in rewards.

Environment	Cases	Succ.	Rate [95% CI]	Dist.
Branch-Path	44	30	68.2 [53.4,80.0]	0.466
B-Bottles	8	6	75.0 [40.9,92.9]	0.030
Collect-Two	152	152	100.0 [97.5,100.0]	0.493
DST	243	70	28.8 [23.5,34.8]	0.641
Fruit-Tree	672	672	100.0 [99.4,100.0]	0.231
Minecart	1846	1341	72.6 [70.6,74.6]	0.360

Table 2: Stress test of CF-ZOO . Each case corresponds to a state, command, and valid foil action. Success-rate intervals are Wilson score

95\%

confidence intervals. Dist. denotes the mean scaled

\ell_{2}

distance over successful counterfactuals.

Outcome	Cases	% all [95% CI]	% feasible [95% CI]
Success †	2515	78.7 [77.3,80.1]	94.4 [93.5,95.2]
Grid-infeasible foil	531	16.6 [15.4,18.0]	–
Search-limited failure	149	4.7 [4.0,5.5]	5.6 [4.8,6.5]

Table 3: CF-ZOO diagnosis. Feasibility is assessed using a grid with 200 values per command dimension, yielding

200^{d}

commands for

R\in\mathbb{R}^{d}

. Percentages in the % feasible column use only the feasible cases. † Includes 93 additional cases recovered by increasing the budget to 100k. Brackets denote 95% Wilson confidence intervals.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	White-box comparison	Stress test
Random directions	$21{,}000$	$9{,}001$
Maximum queries	$21{,}000$	$9{,}001$
Ray binary-search steps	$16$	$16$
Margin $\kappa$	$0.05$	$0.05$
Penalty coefficient $c$	$1.0$	$1.0$
ZOO-ADAM learning rate	$0.01$	$0.01$

Appendix

Table 4: CF-ZOO hyperparameters used in the reported experiments. The white-box comparison uses the same settings as the stress test except for the random-direction and query budgets.

Env.	Method	Time (s)
Branch	White-CW	9.44
Branch	CF-ZOO	53.21
B-Bottles	White-CW	8.55
B-Bottles	CF-ZOO	53.08
Collect	White-CW	7.98
Collect	CF-ZOO	45.38

Appendix

Table 5: Mean runtime per counterfactual query. The aggregate row is weighted by the number of cases in each environment. Environment abbreviations follow the notation used in the main paper.

Budget	Mean $\pm$ std (s)	Median (s)
10k	19.1 $\pm$ 0.8	19.0
20k	43.7 $\pm$ 1.8	43.5
30k	60.5 $\pm$ 2.5	60.4
50k	101.4 $\pm$ 2.6	102.1
75k	108.4 $\pm$ 3.5	107.3
100k	176.0 $\pm$ 6.4	178.0

Appendix

Table 6: Mean (

\pm

standard deviation) and median wall-clock time per counterfactual query across query budgets.

Interpretability in RL

Interpretability in reinforcement learning makes learned policies, decisions, and internal representations understandable and auditable.

See all 71 papers

Constraint-Anchored Attribution: Feasibility-Certified Counterfactuals and Bonferroni-PAC Sufficient Subsets for Neural CO Policies
May 24, 2026Sohaib LafifiFeature Attribution Counterfactual Explanations
Univ. Artois, UR 3926, Laboratoire de Génie Informatique et d’Automatique de l’Artois (LGI2A) Béthune, France
When Do Intrinsic Rewards Lead to Exploration?
Oct 1, 2026Scott W. Viteri, Laura Gomezjurado Gonzalez, Clark BarrettIntrinsic Reward Methods RL Exploration
Stanford University
TREX: Trajectory Explanations for Multi-Objective Reinforcement Learning
Mar 23, 2026Dilina Rajapakse, Juan C. Rosero, Ivana DusparicInterpretability in RL Multi-Objective Reinforcement Learning
Trinity College Dublin, Ireland

May 24, 2026J/K move · Enter open · S save

Constraint-Anchored Attribution: Feasibility-Certified Counterfactuals and Bonferroni-PAC Sufficient Subsets for Neural CO Policies

Sohaib Lafifi

Univ. Artois, UR 3926, Laboratoire de Génie Informatique et d’Automatique de l’Artois (LGI2A) Béthune, France

Feature Attribution Counterfactual Explanations Neural Combinatorial Optimization

Open paper AskPDF arXiv

We give an attribution method for neural combinatorial-optimisation (CO) policies that (i) decomposes a decision by constraint families via LP-relaxation duals, (ii) certifies counterfactuals through a combinatorial feasibility model (implemented as a CSP feasibility-decision model), and (iii) bounds the size of a PAC-sufficient explanation with a Bonferroni-corrected Hoeffding sufficient-subset test along a greedy ordering. Across three CO problems and three seeds, our LP-anchored

Λ

-attribution matches the CF-derived signal at 96.5% on CVRPTW (n_cert=344) and 77.2% on the Orienteering Problem (n_cert=281) vs 75.0% and 35.2% for proxy gradient (paired diffs +0.215 and +0.420; McNemar exact

p \le 10^{-14}

). In the rank-aligned regime of the Flexible Job-Shop Scheduling Problem, both backends agree on every CSP-certified flip (n_cert=59), confirming the no-gain prediction. Bonferroni-PAC subsets average 5.0 nodes per step (

M=70

\varepsilon=δ=0.2

k_{\max}=25

). Reference implementation: https://github.com/sohaibafifi/neuro-co-cax

Abstract

Interpretability in RL

Explore similar work

Constraint-Anchored Attribution: Feasibility-Certified Counterfactuals and Bonferroni-PAC Sufficient Subsets for Neural CO Policies

When Do Intrinsic Rewards Lead to Exploration?

TREX: Trajectory Explanations for Multi-Objective Reinforcement Learning