Integrating Fairness and Explainability in a Multiple Instance Reinforcement Learning System

Bente Hinkenhuis; Seyed Sahand Mohammadi Ziabari; Ali Mohammed Mansoor Alsahag

cs.LGSep 2, 2026

Integrating Fairness and Explainability in a Multiple Instance Reinforcement Learning System

Authors: Bente Hinkenhuis, Seyed Sahand Mohammadi Ziabari, Ali Mohammed Mansoor Alsahag

Organizations: University of Amsterdam Amsterdam, The Netherlands

Abstract

Predicting student performance from educational interaction data requires models that are both accurate and sufficiently transparent to support meaningful intervention, while demographic information introduces an additional risk of unfair predictions. This study investigates a multi-objective framework that combines reinforcement learning-based multiple instance learning (RL-MIL), adversarial debiasing, and preference-conditioned hypernetworks for student-at-risk prediction. MIL represents each student as a bag of weakly labeled interactions, while an RL agent selects informative instances for downstream classification. Two hypernetwork variants are evaluated to determine whether a user-defined preference scalar can continuously control the trade-off between predictive performance and Equalized Odds. The underlying RL-MIL baseline achieves strong classification performance, but both hypernetwork extensions exhibit mode collapse: changing the preference weight produces little systematic movement along the intended fairness-performance frontier. The failure is associated with objective dominance, weak gradient propagation through the conditioning mechanism, and interactions between dynamically generated parameters. The results show that fairness objectives can be incorporated into an interpretable RL-MIL pipeline, but preference conditioning alone does not guarantee controllable multi-objective behavior. Robust fair RL-MIL therefore requires explicit mechanisms for gradient balancing, objective separation, and stability analysis.

Figures & tables

Figure 1. Overview of proposed novel architecture. Black arrows indicate data/parameter flow. Red arrows indicate backpropagation. A flowchart showing how the two reinforcement learning agents for instance selection and adversarial debiasing, the MIL classifier and a preference based parameterization hypernetwork are connected and how data flows through the architecture.

Figure 2. Simple correlation matrix calculated using the technically incorrect assumption of interval scaling for each categorical variable to simplify correlation calculation. The relationship between assessment score or click statistics and has_passed is comparatively the strongest (excluding the obviously high correlation between sum_click and click_days). Correlations involving protected features are low, but tend to be higher for scores and click statistics than for has_passed, indicating that the causal model proposed in the methodology section could have some merit. TODO

Framework	Pooling	Preference	F1 score (performance) ( $\pm$ SD)	Equalized Odds (bias) ( $\pm$ SD)
Baseline	Attention		0.9245 ( $\pm$ 0.0018)	0.1681 ( $\pm$ 0.0044)
Baseline	Max		0.9187 ( $\pm$ 0.0012)	0.0998 ( $\pm$ 0.0288)
Baseline	Mean		0.9228 ( $\pm$ 0.0017)	0.1055 ( $\pm$ 0.0160)
Baseline	Rep-The-Set		0.9280 ( $\pm$ 0.0010)	0.1426 ( $\pm$ 0.0237)
Task-agnostic Hypernetwork	Attention	0.0	0.9261 ( $\pm$ 0.0011)	0.1344 ( $\pm$ 0.0007)
Task-agnostic Hypernetwork	Attention	1.0	0.9244 ( $\pm$ 0.0020)	0.1356 ( $\pm$ 0.0176)

Table 1. Shortened list of performance metrics for the best models. Metrics for preference values between 0 and 1 are similar to those shown, which can be confirmed by looking at Table 5 in Appendix B .

Pooling	Framework	Tested Hypothesis	T value	p value
Mean	Task-agnostic Hypernetwork	Preference 0 > Preference 1	-0.1782	0.8630
Mean	Task-aware Hypernetwork	Preference 0 > Preference 1	0.1859	0.8697
Max	Task-agnostic Hypernetwork	Preference 0 > Preference 1	-0.3635	0.7257
Max	Task-aware Hypernetwork	Preference 0 > Preference 1	6.8235	0.0208 (*)
Attention	Task-agnostic Hypernetwork	Preference 0 > Preference 1	0.8337	0.4286
Attention	Task-aware Hypernetwork	Preference 0 > Preference 1	-1.2058	0.3512

Table 2. T test of classification F1-score between preference 0 and preference 1 for both hypernetwork architectures and all pooling methods.

Baseline	Task-agnostic Hypernetwork, preference 0	Task-agnostic Hypernetwork, preference 1
Age Band (98.9774 %)	VLE Clicks: homepage (97.1877 %)	VLE Clicks: homepage (97.4434 %)
Code Presentation (91.8919 %)	Assessment: TMA (86.8152 %)	Age Band (93.0241 %)
Gender (89.6640 %)	VLE Clicks: forumng (85.9021 %)	Disability (86.8517 %)
VLE Clicks: homepage (89.2988 %)	VLE Clicks: oucontent (81.3733 %)	Assessment: TMA (86.7421 %)
Region (88.8605 %)	Num Of Prev Attempts (81.3002 %)	VLE Clicks: forumng (85.9386 %)
Module Presentation Length (86.1943 %)	Imd Band (80.0950 %)	Gender (80.3871 %)

Table 3. Common instance types sorted by bag presence in percent for the two extreme preference settings 0 and 1 of the task-agnostic hypernetwork and the baseline method using mean pooling.

			Sparsity		Consistency
Model	Preference	Pooling	Gini Coeff.	HHI Score	Spearman’s $\rho$	Jaccard (K=5)	Demographics (%)
Task-agnostic	0.0	Rep-The-Set	0.4793	0.0614	0.9983	1.0000	12.0%
Task-agnostic	1.0	Rep-The-Set	0.4796	0.0613	0.9972	1.0000	12.0%
Task-aware	0.0	Rep-The-Set	0.4939	0.0559	0.9980	0.6667	15.2%
Task-aware	1.0	Rep-The-Set	0.4932	0.0558	0.9985	0.6667	15.2%
Task-agnostic	0.0	Attention	0.5275	0.0621	0.9982	1.0000	18.7%

Table 4. Sparsity, Consistency and Demographics Reliance metrics for several model configurations. Baseline values are only available for mean pooling due to unexpected maintenance on the Snellius national supercomputer.

Panel of Figure 3 — Figure 3. Schematic depiction of the expected causal patterns between observed variables in our problem setting. Arrows point from causal ancestors to their children. A schematic display of the structural causal model we will use to adjust classification reward strength.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4. Number of students for each combination of protected features. TODO

Figure 5. Summary plots of assessment scores separated by the protected features education, gender and age. The most noticeable relationship can be seen between education and assessment score, with post graduate qualification performing better and no formal qualifications performing worse than all other levels. For age and gender, no clear pattern emerges. TODO

Figure 6. Distribution of the number of click days for each resource, split by gender, age and education. Again, the most clear but very subtle pattern can be seen by comparing the peak of the no formal qualifications level against the peak of all other education levels. For the mentioned level, many resources are clicked on less often than for other levels. TODO

Figure 7. Summary plots of pass rates split by module, gender and age. Although there are stark differences between the courses, there is no clear effect for either age or gender. TODO

Framework	Pooling	Preference	F1 score (performance) ( $\pm$ SD)	Equalized Odds (bias) ( $\pm$ SD)
Baseline	Attention		0.9245 ( $\pm$ 0.0018)	0.1681 ( $\pm$ 0.0044)
Baseline	Max		0.9187 ( $\pm$ 0.0012)	0.0998 ( $\pm$ 0.0288)
Baseline	Mean		0.9228 ( $\pm$ 0.0017)	0.1055 ( $\pm$ 0.0160)
Baseline	Rep-The-Set		0.9280 ( $\pm$ 0.0010)	0.1426 ( $\pm$ 0.0237)
Task-agnostic Hypernetwork	Attention	0.0	0.9261 ( $\pm$ 0.0011)	0.1344 ( $\pm$ 0.0007)
Task-agnostic Hypernetwork	Attention	0.1	0.9266 ( $\pm$ 0.0018)	0.1408 ( $\pm$ 0.0053)

Appendix

Table 5. Full list of performance metrics for the best models. Most values are means over 5 different seeds (for mean and std). Due to maintenance on the Snellius national supercomputer, for the Task-aware Hypernetwork Mean, Max and Attention pooling had to be carried out on only one seed.

Framework	Pooling	Seed	Preference	F1 score (performance) ( $\pm$ SD)	Equalized Odds (bias) ( $\pm$ SD)
Baseline	Mean	0		0.9230 ( $\pm$ 0.0005)	0.1019 ( $\pm$ 0.0656)
Task-agnostic Hypernetwork	Mean	0	0.0	0.9323 ( $\pm$ 0.0015)	0.1690 ( $\pm$ 0.0010)
Task-agnostic Hypernetwork	Mean	0	0.1	0.9301 ( $\pm$ 0.0015)	0.1711 ( $\pm$ 0.0011)
Task-agnostic Hypernetwork	Mean	0	0.2	0.9301 ( $\pm$ 0.0011)	0.1691 ( $\pm$ 0.0003)
Task-agnostic Hypernetwork	Mean	0	0.3	0.9316 ( $\pm$ 0.0030)	0.1721 ( $\pm$ 0.0020)
Task-agnostic Hypernetwork	Mean	0	0.4	0.9319 ( $\pm$ 0.0015)	0.1707 ( $\pm$ 0.0018)

Appendix

Table 6. Performance metrics for all combinations of framework, pooling method, seed and preference. Due to maintenance on the Snellius national supercomputer, for the Task-aware Hypernetwork Mean, Max and Attention pooling had to be carried out on only one seed.

Pooling	Tested Hypothesis	T value	p value
Mean	Task-agnostic Hypernetwork > Baseline	1.6058	0.1143
Mean	Task-aware Hypernetwork > Baseline	2.5671	0.0234 (*)
Mean	Task-aware Hypernetwork > Task-agnostic Hypernetwork	1.1183	0.2681
Max	Task-agnostic Hypernetwork > Baseline	1.3277	0.1900
Max	Task-aware Hypernetwork > Baseline	2.9425	0.0114 (*)
Max	Task-aware Hypernetwork > Task-agnostic Hypernetwork	1.6701	0.1003

Appendix

Table 7. T test results of Equalized Odds Deviation for every combination of frameworks and each pooling method.

Pooling	Tested Hypothesis	T value	p value
Mean	Baseline > Task-agnostic Hypernetwork	0.2529	0.8013
Mean	Task-aware Hypernetwork > Baseline	3.3349	0.0054 (*)
Mean	Task-aware Hypernetwork > Task-agnostic Hypernetwork	1.4354	0.1566
Max	Task-agnostic Hypernetwork > Baseline	0.6903	0.4930
Max	Baseline > Task-aware Hypernetwork	8.1000	0.0000 (*)
Max	Task-agnostic Hypernetwork > Task-aware Hypernetwork	7.2982	0.0000 (*)

Appendix

Table 8. T test results of F1-score (classification performance) for every combination of frameworks and each pooling method.

Pooling	Framework	Tested Hypothesis	T value	p value
Mean	Task-agnostic Hypernetwork	Preference 0 > Preference 1	-0.9422	0.3737
Mean	Task-aware Hypernetwork	Preference 0 > Preference 1	-0.8843	0.4698
Max	Task-agnostic Hypernetwork	Preference 0 > Preference 1	0.0100	0.9922
Max	Task-aware Hypernetwork	Preference 0 > Preference 1	-0.2726	0.8107
Attention	Task-agnostic Hypernetwork	Preference 0 > Preference 1	-0.0245	0.9811
Attention	Task-aware Hypernetwork	Preference 0 > Preference 1	-0.9305	0.4503

Appendix

Table 9. T test of Equalized Odds Deviation between preference 0 and preference 1 for both hypernetwork architectures and all pooling methods.

Algorithmic Fairness

Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.

See all 420 papers

Learning Fair Pareto-Optimal Policies in Multi-Objective Reinforcement Learning
Jun 16, 2026Umer Siddique, Peilang Li, Yongcan CaoMulti-Objective Reinforcement Learning Algorithmic Fairness
Department of Electrical and Computer Engineering, University of Texas at San Antonio
FairTest: Search-Based Fairness Testing for Multi-Agent Reinforcement Learning Systems
Sep 23, 2026Xiaotong Wang, Xuan XieAutomated Software Testing Algorithmic Fairness
School of Computer Science and Engineering, Macau University of Science and Technology, Macao SAR, China
Inference-Time Policy Alignment for Fair Reinforcement Learning
Jul 31, 2026Umer Siddique, Peilang Li, Conor Wallace +1Algorithmic Fairness Inference-Time Guidance
Department of Electrical and Computer Engineering, University of Texas at San Antonio

Jun 16, 2026J/K move · Enter open · S save

Learning Fair Pareto-Optimal Policies in Multi-Objective Reinforcement Learning

Umer Siddique, Peilang Li, Yongcan Cao

Department of Electrical and Computer Engineering, University of Texas at San Antonio

Multi-Objective Reinforcement Learning Algorithmic Fairness

Open paper AskPDF arXiv

Fairness is an important aspect of decision-making in multi-objective reinforcement learning (MORL), where policies must ensure both optimality and equity across multiple, potentially conflicting objectives. While single-policy MORL methods can learn fair policies for fixed user preferences using welfare functions such as the generalized Gini welfare function (GGF), they fail to provide the diverse set of policies necessary for dynamic or unknown user preferences. To address this limitation, we formalize the fair optimization problem in multi-policy MORL, where the goal is to learn a set of Pareto-optimal policies that ensure fairness across all possible user preferences. Our key technical contributions are threefold: (1) We show that for concave, piecewise-linear welfare functions (e.g., GGF), fair policies remain in the convex coverage set (CCS), which is an approximated Pareto front for linear scalarization. (2) We demonstrate that non-stationary policies, augmented with accrued reward histories, and stochastic policies improve fairness by dynamically adapting to historical inequities. (3) We propose three novel algorithms, which include integrating GGF with multi-policy multi-objective Q-Learning (MOQL), state-augmented multi-policy MOQL for learning non-statoinary policies, and its novel extension for learning stochastic policies. We evaluate our algorithms across various domains and compare our methods against the state-of-the-art MORL baselines. The empirical results show that our methods learn a set of fair policies that accommodate different user preferences.

Abstract

Algorithmic Fairness

Explore similar work

Learning Fair Pareto-Optimal Policies in Multi-Objective Reinforcement Learning

FairTest: Search-Based Fairness Testing for Multi-Agent Reinforcement Learning Systems

Inference-Time Policy Alignment for Fair Reinforcement Learning