Offline Reinforcement Learning (RL) is an emerging field of RL in which policies are learned solely from demonstrations. Within offline RL, some environments involve balancing multiple objectives, but existing multi-objective offline RL algorithms do not provide an efficient way to find a fair compromise. FairDICE (see arXiv:2506.08062v2) seeks to fill this gap by adapting OptiDICE (an offline RL algorithm) to automatically learn weights for multiple objectives to e.g. incentivise fairness among objectives. As this would be a valuable contribution, this replication study examines the replicability of claims made regarding FairDICE. We find that many theoretical claims hold, but an error in the code reduces FairDICE to standard behaviour cloning in continuous environments, and many important hyperparameters were originally underspecified. After rectifying this, we show in experiments extending the original paper that FairDICE can scale to complex environments and high-dimensional rewards, though it can be reliant on (online) hyperparameter tuning. We conclude that FairDICE is a theoretically interesting method, but the experimental justification requires significant revision.
Figures & tables
Figure 1: Metrics on MO-FourRooms over a sweep of α and β values.
Figure 2: Random MO-MDP metrics over a sweep of α and β , replicating Fig. 2 from Kim et al. (2025a) .
Figure 3: D4MORL NSW boxplots for various losses and β , using 10 seeds with 100 evaluations per seed.
Figure 4: NSW scores and raw return trade-offs with Pareto frontiers on two multi-objective datasets. ‘Original’ graphs in second row reproduced from Fig. 4 and Fig. 5 of Kim et al. (2025a) with permission.
Figure 5: FairDICE performance with/without reward normalisation and with piecewise-log for ui , trained on Hopper and Walker2d from D4MORL. Boxplots are drawn using 10 seeds with 100 evaluations per seed.
Figure 6: Metrics on MO-FourRooms for policies trained on balanced and imbalanced datasets
α\β
0.001
0.01
0.1
1.0
10.0
100.0
0.0
3.92±0.12
3.92±0.06
3.97±0.23
4.08±0.09
3.96±0.06
3.93±0.15
1.0
4.09±0.00
4.12±0.11
4.23±0.11
4.20±0.06
3.99±0.03
4.05±0.06
Table 1: NSW on MO-Minecart-RGB across a sweep of β on utilitarian and α=1 , where the training dataset had a NSW of 1.24±3.784 . The results are averaged over 3 seeds
Figure 8
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Visualization of the MO-FourRooms environment together with a sample policy’s state visitation heatmap
Figure 10: Distribution of (individual) preference weights in all two-objective environments in D4MORL, divided over 40 bins.
Figure 11: FairDICE statistical tests, comparing the original (top) and corrected (bottom) implementations.
Hyperparameter
BC(P)
MODT(P)
MORvS(P)
Batch Size
256
256
256
Training Steps
100,000
100,000
100,000
Optimizer
Adam
Adam
Adam
Normalize Reward
True
True
True
Condition on Preference ( μ )
✓
✓
✓
Condition on Return-to-Go
×
✓(Multi-obj)
✓(Multi-obj)
Appendix
Table 2: Hyperparameters for BC(P), MODT(P), and MORvS(P) baselines.
Figure 12: Original Fig. 2 from Kim et al. (2025a) . Original caption: "Policy performance on Random MOMDP domain across different α and β values, evaluated on Nash social welfare, Utilitarian welfare, and Jain’s fairness index. Results are averaged over 1000 seeds, and reported with 95% confidence intervals". Reproduced with permission.
Figure 13: Our replication: Random MO-MDP metrics over a sweep of α and β values.
Figure 14: Nash social welfare across 30 linearly spaced preference weights on five two-objective tasks.
Figure 15: Raw return trade-offs on five two-objective tasks, with Pareto frontiers and dominated regions.
Figure 16: MO-Hopper-3obj: raw returns and Nash social welfare (NSW) evaluations with three objectives.
Figure 17: Complete version of Fig. 3 , 10 seeds with 100 evaluations per seed.
Used for Section(s)
GPU
CPU
Hours @ Wattage
g CO2eq/kWh
∼ CO2eq
4.2.2
A100 ( 21 MIG)
Xeon Platinum 8360Y
303h @ 250W
385
29.2 kg
4.2.1 , 4.3.1 , 4.3.2 , 4.3.3 ,
RTX 5090
i9-13900K
273h @ 400W
414
45.2 kg
4.1.1 , 4.1.2 , 4.3.4
A100
AMD EPYC 7543
180h @ 300W
385
20.8 kg
4.3.2
RTX 4060
Ryzen 5-7600X
3.5h @ 150W
501
0.26 kg
Appendix
Table 3: Hardware specifications for running experiments and corresponding CO2eq estimates.
Full
Traj. 0.4-0.6 filtered
Traj. 0.3-0.7 filtered
Traj. 0.2-0.8 filtered
Environment
Dataset quality
NSW
Traj. cut
NSW
Traj. cut
NSW
Traj. cut
NSW
Swimmer
Expert
11.668±0.016
24%
11.581±0.080
48.3%
11.393±0.180
72.5%
10.752±0.441
Amateur
11.820±0.004
24%
11.811±0.003
48.5%
11.811±0.001
72.7%
11.817±0.001
Walker2d
Expert
11.202±0.432
34.9%
9.845±1.227
69.6%
10.100±0.671
94.3%
10.967±0.005
Amateur
11.185±0.374
35%
11.342±0.010
69.6%
10.847±0.090
94.1%
11.114±0.012
Ant
Expert
11.529±0.028
43.1%
11.298±0.041
86.5%
11.349±0.011
100%
-
Appendix
Table 4: NSW comparison before and after filtering out balanced trajectories across all uniform environments (with two goals) and dataset qualities of FairDICE as implemented. All NSW results show the average NSW across 5 seeds. Traj. cut refers to the percentage of trajectories removed during filtering, ‘-‘ indicates NSW cannot be evaluated as no trajectories remain.
Environment
NSW (Uniform distribution)
NSW (Narrow distribution)
NSW (Wide distribution)
MO-Hopper
10.820±0.367
10.844±0.363
11.074±0.143
MO-Swimmer
11.587±0.070
11.682±0.020
11.618±0.039
Appendix
Table 5: FairDICE as described, with a β of 1 and λ of 0, trained on both the expert uniform, narrow and wide versions of Hopper-v2 and Swimmer, where the NSW is the average NSW across 5 seeds. The descriptions ‘narrow‘ and ‘wide‘ are from the D4MORL dataset from Zhu et al. (2023) .
Figure 18: Unexpected change of sign in the FairDICE derivation, page 5 of Kim et al. (2025a)