Offline Reinforcement Learning (RL) is an emerging field of RL in which policies are learned solely from demonstrations. Within offline RL, some environments involve balancing multiple objectives, but existing multi-objective offline RL algorithms do not provide an efficient way to find a fair compromise. FairDICE (see arXiv:2506.08062v2) seeks to fill this gap by adapting OptiDICE (an offline RL algorithm) to automatically learn weights for multiple objectives to e.g. incentivise fairness among objectives. As this would be a valuable contribution, this replication study examines the replicability of claims made regarding FairDICE. We find that many theoretical claims hold, but an error in the code reduces FairDICE to standard behaviour cloning in continuous environments, and many important hyperparameters were originally underspecified. After rectifying this, we show in experiments extending the original paper that FairDICE can scale to complex environments and high-dimensional rewards, though it can be reliant on (online) hyperparameter tuning. We conclude that FairDICE is a theoretically interesting method, but the experimental justification requires significant revision.
Figures & tables
Figure 1: Metrics on MO-FourRooms over a sweep of α and β values.
Figure 2: Random MO-MDP metrics over a sweep of α and β , replicating Fig. 2 from Kim et al. (2025a) .
Figure 3: D4MORL NSW boxplots for various losses and β , using 10 seeds with 100 evaluations per seed.
Figure 4: NSW scores and raw return trade-offs with Pareto frontiers on two multi-objective datasets. ‘Original’ graphs in second row reproduced from Fig. 4 and Fig. 5 of Kim et al. (2025a) with permission.
Figure 5: FairDICE performance with/without reward normalisation and with piecewise-log for ui , trained on Hopper and Walker2d from D4MORL. Boxplots are drawn using 10 seeds with 100 evaluations per seed.
Figure 6: Metrics on MO-FourRooms for policies trained on balanced and imbalanced datasets
α\β
0.001
0.01
0.1
1.0
10.0
100.0
0.0
3.92±0.12
3.92±0.06
3.97±0.23
4.08±0.09
3.96±0.06
3.93±0.15
1.0
4.09±0.00
4.12±0.11
4.23±0.11
4.20±0.06
3.99±0.03
4.05±0.06
Table 1: NSW on MO-Minecart-RGB across a sweep of β on utilitarian and α=1 , where the training dataset had a NSW of 1.24±3.784 . The results are averaged over 3 seeds
Figure 8
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Visualization of the MO-FourRooms environment together with a sample policy’s state visitation heatmap
Figure 10: Distribution of (individual) preference weights in all two-objective environments in D4MORL, divided over 40 bins.
Figure 11: FairDICE statistical tests, comparing the original (top) and corrected (bottom) implementations.
Hyperparameter
BC(P)
MODT(P)
MORvS(P)
Batch Size
256
256
256
Training Steps
100,000
100,000
100,000
Optimizer
Adam
Adam
Adam
Normalize Reward
True
True
True
Condition on Preference ( μ )
✓
✓
✓
Condition on Return-to-Go
×
✓(Multi-obj)
✓(Multi-obj)
Appendix
Table 2: Hyperparameters for BC(P), MODT(P), and MORvS(P) baselines.
Figure 12: Original Fig. 2 from Kim et al. (2025a) . Original caption: "Policy performance on Random MOMDP domain across different α and β values, evaluated on Nash social welfare, Utilitarian welfare, and Jain’s fairness index. Results are averaged over 1000 seeds, and reported with 95% confidence intervals". Reproduced with permission.
Figure 13: Our replication: Random MO-MDP metrics over a sweep of α and β values.
Figure 14: Nash social welfare across 30 linearly spaced preference weights on five two-objective tasks.
Figure 15: Raw return trade-offs on five two-objective tasks, with Pareto frontiers and dominated regions.
Figure 16: MO-Hopper-3obj: raw returns and Nash social welfare (NSW) evaluations with three objectives.
Figure 17: Complete version of Fig. 3 , 10 seeds with 100 evaluations per seed.
Used for Section(s)
GPU
CPU
Hours @ Wattage
g CO2eq/kWh
∼ CO2eq
4.2.2
A100 ( 21 MIG)
Xeon Platinum 8360Y
303h @ 250W
385
29.2 kg
4.2.1 , 4.3.1 , 4.3.2 , 4.3.3 ,
RTX 5090
i9-13900K
273h @ 400W
414
45.2 kg
4.1.1 , 4.1.2 , 4.3.4
A100
AMD EPYC 7543
180h @ 300W
385
20.8 kg
4.3.2
RTX 4060
Ryzen 5-7600X
3.5h @ 150W
501
0.26 kg
Appendix
Table 3: Hardware specifications for running experiments and corresponding CO2eq estimates.
Full
Traj. 0.4-0.6 filtered
Traj. 0.3-0.7 filtered
Traj. 0.2-0.8 filtered
Environment
Dataset quality
NSW
Traj. cut
NSW
Traj. cut
NSW
Traj. cut
NSW
Swimmer
Expert
11.668±0.016
24%
11.581±0.080
48.3%
11.393±0.180
72.5%
10.752±0.441
Amateur
11.820±0.004
24%
11.811±0.003
48.5%
11.811±0.001
72.7%
11.817±0.001
Walker2d
Expert
11.202±0.432
34.9%
9.845±1.227
69.6%
10.100±0.671
94.3%
10.967±0.005
Amateur
11.185±0.374
35%
11.342±0.010
69.6%
10.847±0.090
94.1%
11.114±0.012
Ant
Expert
11.529±0.028
43.1%
11.298±0.041
86.5%
11.349±0.011
100%
-
Appendix
Table 4: NSW comparison before and after filtering out balanced trajectories across all uniform environments (with two goals) and dataset qualities of FairDICE as implemented. All NSW results show the average NSW across 5 seeds. Traj. cut refers to the percentage of trajectories removed during filtering, ‘-‘ indicates NSW cannot be evaluated as no trajectories remain.
Environment
NSW (Uniform distribution)
NSW (Narrow distribution)
NSW (Wide distribution)
MO-Hopper
10.820±0.367
10.844±0.363
11.074±0.143
MO-Swimmer
11.587±0.070
11.682±0.020
11.618±0.039
Appendix
Table 5: FairDICE as described, with a β of 1 and λ of 0, trained on both the expert uniform, narrow and wide versions of Hopper-v2 and Swimmer, where the NSW is the average NSW across 5 seeds. The descriptions ‘narrow‘ and ‘wide‘ are from the D4MORL dataset from Zhu et al. (2023) .
Figure 18: Unexpected change of sign in the FairDICE derivation, page 5 of Kim et al. (2025a)
Multi-agent Reinforcement Learning (MARL) trains a team of agents that share one environment and learn their policies together. Training maximizes the team return, and a high return does not imply that the rewards are shared fairly among the agents in every episode. Testing is an established way to discover the failures of deep reinforcement learning, yet few methods address the fairness of MARL. In this work, we propose FairTest, a search-based testing approach that seeks the unfair executions of a MARL policy. The design combines search guidance with test prioritization. The guidance scores each candidate with three fitness functions. One measures the fairness of the runs already performed, another predicts the fairness from abstract states and fairness features, and the third reads the decision uncertainty from the policy. Crossover and mutation derive further candidates from the observed executions. The prioritization ranks the candidates by the predicted fairness and the decision uncertainty, so that the runs reach the candidates where failures are expected. FairTest is evaluated on three environments and two MARL algorithms, and four baselines are given the same budget. It detects the most fairness failures compared to three baselines with statistical significance and large effect sizes. The failure count exceeds that of the strongest baseline by 221% on average and coverage improves by an average of 23%.
Xiaotong Wang, Xuan Xie
School of Computer Science and Engineering, Macau University of Science and Technology, Macao SAR, China
Fairness is an important aspect of decision-making in multi-objective reinforcement learning (MORL), where policies must ensure both optimality and equity across multiple, potentially conflicting objectives. While single-policy MORL methods can learn fair policies for fixed user preferences using welfare functions such as the generalized Gini welfare function (GGF), they fail to provide the diverse set of policies necessary for dynamic or unknown user preferences. To address this limitation, we formalize the fair optimization problem in multi-policy MORL, where the goal is to learn a set of Pareto-optimal policies that ensure fairness across all possible user preferences. Our key technical contributions are threefold: (1) We show that for concave, piecewise-linear welfare functions (e.g., GGF), fair policies remain in the convex coverage set (CCS), which is an approximated Pareto front for linear scalarization. (2) We demonstrate that non-stationary policies, augmented with accrued reward histories, and stochastic policies improve fairness by dynamically adapting to historical inequities. (3) We propose three novel algorithms, which include integrating GGF with multi-policy multi-objective Q-Learning (MOQL), state-augmented multi-policy MOQL for learning non-statoinary policies, and its novel extension for learning stochastic policies. We evaluate our algorithms across various domains and compare our methods against the state-of-the-art MORL baselines. The empirical results show that our methods learn a set of fair policies that accommodate different user preferences.
Umer Siddique, Peilang Li, Yongcan Cao
Department of Electrical and Computer Engineering, University of Texas at San Antonio
Offline-to-online reinforcement learning uses offline pretraining to accelerate subsequent online learning. While many methods assume the offline dataset remains available throughout fine-tuning, there are many settings where this is not the case. Prior work attributes failures in this setting primarily to critic recalibration under distribution shift. We identify a complementary actor-side failure: once the offline data disappears, continued behavioral cloning (BC) no longer preserves the offline behavior distribution and instead refits the behavioral model to the agent's limited, and nonstationary, online experience. We demonstrate that stopping online behavioral fitting while preserving the pretrained expressive prior substantially improves no-retention fine-tuning across implicit flow Q-learning (IFQL), flow Q-learning (FQL), and action-chunked RL (QC) on seven OGBench domains. We trace this failure to a behavioral feedback loop: continuing online behavioral cloning rapidly erases the pretrained prior, inducing representational collapse and action diversity loss that starves the critic of useful exploration. Freezing the prior breaks this cycle, preserving offline coverage while allowing value learning to drive continuous improvement.
Sami Nur Islam, Johan Obando-Ceron, Doina Precup +2
Mila – Québec AI Institute · McGill University · Université de Montréal +1