Hiding a Gap Does Not Close It: The Price of Removing the Mean Group Contrast from Learned Models of Learning Dynamics
Organizations: Vanguard Center, Mohammed VI Polytechnic University (UM6P), Rabat, Morocco
Abstract
Adaptive tutors use models learned from student data, and a common fairness remedy erases the group signal from these models. We turn the cost of this remedy into an audit tool. Building on mean-parity regression, we give the minimum extra error, relative to the best predictor, of removing the mean group contrast from a predictor that cannot use the group, , and the exact cost of any correction built from an estimated group model, which, for the best predictor, is never below this minimum. A three-stage audit (fit on training units, announce the cost on validation units, test on independent units) estimates this cost before the correction is used. In a controlled study, the price can be large even when the state predicts the group poorly. On three real data sets, erasing the group from a learned model of learning dynamics predicts almost no gap where one is observed. In a selected knowledge-tracing network, erasing the hidden state leaves most of the gap; a correction of the prediction removes it at an extra Brier score of 0.021 on average. On Eedi, with the network selected on separate units, the audit accepted the announced cost in all five splits, the measured cost fell inside the announced interval in all five, and the gap reduction transferred to new classes in four; on new ASSISTments schools the audit abstains. In a tutoring simulator, the objective and the help budget matter more than the erasure, and floors on each group's knowledge separate narrowing a gap from levelling down. Hiding a gap in a model does not close it.
Figures & tables
| Claim | Evidence | Main limit |
|---|---|---|
| The minimum cost of removing the mean contrast has an explicit value and a bound, and, for the best predictor, a correction built from any group model costs at least this minimum | Proofs (Propositions 1 and 3 ) | Population and ; for a learned predictor the cost of the correction can be below the minimum (Eq. ( 24 )); mean parity, not independence; fixed law of the states |
| The cost of the built correction can be announced before use, and a diagnostic screens the estimate | Controlled study with a known truth (Section 5 ); on Eedi, with the network selected on separate units: accepted in five splits, measured cost in the announced interval in five, gap reduction transfers in four (Section 7.3 ) | Conservative for the minimum when the group model misses part of the group signal; approximate intervals: nominal 95%, observed coverage 90–100% with 1,000 bootstrap draws (both learners and sample sizes), so not a 95% guarantee; on Eedi, five splits that share units, and intervals one to two times as wide as the cost; intervals conditional on the fitted predictor; thresholds are conventions |
| Removing the group direction hides observed gaps and costs accuracy | Three data sets, cluster-bootstrap intervals; score delays, reweighting, held-out presentations (Section 7.1 ) | Performance proxies inflate error ratios (1.5–1.7 on weeks with a new score); retrospective school groups |
| The loss comes from the group direction, and linear removal costs what the formula says | Rank-matched controls, LEACE, school-level shuffle; formula within 0–13% on held-out students (Section 7.2 ) | Linear removal, one feature dictionary; an iterated forecast adds a cross term |
| Erasing the hidden state of a selected knowledge-tracing network does not remove the gap from its predictions; at equal remaining gap, the built output correction has the lowest observed mean cost | Four methods, five splits, paired cluster intervals; intensity also set on validation (Section 7.3 ) | Clearly cheaper only than LEACE on the hidden state; the audit abstains on new ASSISTments schools; for the retrained network the comparison is an empirical reference |
| In control, the objective and the help budget matter more than the erasure | Reference that uses the known equations, approximately equal budgets (Section 8.2 ) | Noise-free plans by grid search; one simulator family; learned models stay below the reference at |
| Work | Predictor | Attribute | Constraint | What is given |
|---|---|---|---|---|
| Calders et al. [7] | linear | discrete | mean differences | fitted linear models |
| Wei et al. [40] | RKHS, uses the attribute | discrete | exact mean parity, and a relaxation | closed-form regressor; quadratic trade-off; upper bound on the cost |
| Chzhen et al. [9] ; Le Gouic et al. [24] | any, uses the attribute | discrete | full demographic parity | optimal regressor (Wasserstein barycentre) |
| Petrulionytė et al. [32] | neural, features only | continuous, multidimensional | penalty on the variance of the conditional mean | training algorithm (functional bilevel) |
| This paper | any function of the state | binary | exact fraction of the mean gap | minimum cost in closed form, two forms, bound; exact cost of a correction built from any group score (Prop. 3 ); three-stage audit of learned dynamic models |
| Quantity | Units |
|---|---|
| and | training (for the network of Section 7.3 : selection and early stopping on separate selection units) |
| , centre of , announced and total | validation |
| Announced interval and diagnostic D1–D3 | bootstrap of training units ( refitted; refitted when cheap) and of validation units |
| , measured cost, checks T1–T2 | test; interval: test units resampled, correction fixed |
| Measured minus announced | training, validation and test units resampled together |
| Error ratio, projected / original | ||||||
|---|---|---|---|---|---|---|
| Data set (group) | Observed | Original | Recal. | Projected | All starting points | Early starts / new-score weeks |
| ASSISTments (school performance) | (3/5) | 7.4–8.1 | 4.4 (early) | |||
| Eedi (PremiumPupil) | (5/5) | 8.4–9.0 | 4.3 (early) | |||
| OULAD (deprivation) | (5/5) | 10.6–11.0 | 1.68–1.74 (new score) | |||
| OULAD (disability) | (5/5) | 8.4–8.9 | 1.52–1.57 (new score) | |||
| OULAD (gender) | (0/5) | 7.5–9.3 | 1.46–1.58 (new score) | |||
| Raw | Group | LEACE | Shuffled | Random | Top PCA | Lowest PCA | |
|---|---|---|---|---|---|---|---|
| Rank | – | 4 | 1 | 4 | 4 | 4 | 4 |
| ASSISTments (school) | 0.0074 | 0.0575 | 0.0553 | 0.0371 | 0.0234 | 0.0543 | 0.0074 |
| Eedi (PremiumPupil) | 0.0080 | 0.0695 | 0.0677 | 0.0338 | 0.0266 | 0.0667 | 0.0081 |
| OULAD (deprivation) | 0.0030 | 0.0319 | 0.0278 | 0.0114 | 0.0109 | 0.0337 | 0.0030 |
| OULAD (disability) | 0.0030 | 0.0257 | 0.0223 | 0.0114 | 0.0109 | 0.0337 | 0.0030 |
| OULAD (gender) | 0.0030 | 0.0242 | 0.0053 | 0.0114 | 0.0109 | 0.0337 | 0.0030 |
| Setting | of group | Formula | Measured (train) | Measured (held out) | ||
|---|---|---|---|---|---|---|
| ASSISTments (school) | 0.93 | 0.030 | 0.0513 | 0.0438 | 0.0438 | 0.0437 |
| Eedi (PremiumPupil) | 0.97 | 0.017 | 0.0625 | 0.0592 | 0.0592 | 0.0593 |
| OULAD (deprivation) | 0.89 | 0.016 | 0.0310 | 0.0245 | 0.0245 | 0.0246 |
| OULAD (disability) | 0.77 | 0.004 | 0.0310 | 0.0185 | 0.0185 | 0.0187 |
| OULAD (gender) | 0.27 | 0.060 | 0.0310 | 0.0023 | 0.0023 | 0.0023 |
| ASSISTments (school) | Eedi (PremiumPupil) | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | AUC | AUC | Brier | AUC | AUC | Brier | ||
| EDMD, original | 0.659 | 2/5 | 0.687 | 5/5 | ||||
| EDMD, projected | 0.518 | 5/5 | 0.521 | 5/5 | ||||
| DKT, selected on validation | 0.676 | ref. | ref. | 1 | 0.719 | ref. | ref. | 1 |
| DKT, LEACE on the hidden state | 0.676 | 1/5 | 0.693 | 5/5 | ||||
| DKT, LEACE within each next skill | 0.670 | 4/5 | 0.685 | 5/5 | ||||
| Method | Output correction at the same gap | ||||||
|---|---|---|---|---|---|---|---|
| matched on test | set on validation | ||||||
| Method | Cost | Cross | Cost | splits | Cost | ||
| LEACE, hidden state | 0.0077 | 0.0033 | 5/5 | 0.0028 | |||
| LEACE, within next skill | 0.0098 | 0.0079 | 2/5 | 0.0076 | |||
| Mean-parity fine-tuning † | 0.0224 | 0.0200 | 2/5 | 0.0201 | |||
| Output correction (Prop. 3 ) | 0.0214 | – | – | – | – | ||
| Announced total cost | Measured | Measured | ||||
| Split | estimate | 95% interval | width | (test) | announced, 95% | Decision |
| Eedi, new classes | ||||||
| 1 | 0.0209 | [0.0134, 0.0351] | 0.0217 | 0.0198 | [-0.0038, 0.0053] | accepted; transfers, |
| 2 | 0.0228 | [0.0127, 0.0586] | 0.0460 | 0.0269 | [-0.0003, 0.0143] | accepted; does not transfer (T2), |
| 3 | 0.0176 | [0.0030, 0.0321] | 0.0291 | 0.0172 | [-0.0033, 0.0036] | accepted; transfers, |
| 4 | 0.0210 | [0.0124, 0.0396] | 0.0272 | 0.0210 | [-0.0051, 0.0052] | accepted; transfers, |
| Approximately equal budget ( of the reference) | |||||||
| Controller | |||||||
| Objective: mean knowledge, ; budget | |||||||
| Known equations, with the group | 0.667 | 0.921 | 0.414 | 0.507 | 1.00 | 0.667 | 0.300 |
| Known equations, group replaced by its mean | 0.667 | 0.921 | 0.413 | 0.508 | 1.01 | 0.667 | 0.300 |
| Learned, group-aware | 0.667 | 0.921 | 0.414 | 0.507 | 0.67 | 0.447 | 0.201 |
| Learned, original | 0.667 | 0.921 | 0.414 | 0.507 | 0.68 | 0.450 | 0.203 |
| Level and question | What must be reported | Diagnostic flag (warning sign) | Methods to compare with |
|---|---|---|---|
| 1. Structure: is the change built as specified? | Duality, idempotency and metric-duality errors; seed residual ; cross-block residuals ; leakage , ; requested, candidate, applied and numerical ranks; singular-value cutoff; the space (centered or augmented) where they are measured | an identity that should hold is broken beyond numerical error, or the removed direction comes back through the action or constant term | the original model, fitted on the same data |
| 2. Prediction: does the model still predict unseen students? | Errors at several horizons, for each variable and each group; predicted and observed gaps, with intervals at the level of the sampling unit; paired differences in error | the interval of the paired gap error excludes zero, or the model predicts worse than persistence | persistence; a linear model of the state; a nonlinear model trained on the same data; a recalibration for each group |
| 3. Policy: does a tutor using it help both groups? | Paired effects on the gap, on all students and on each group, with the worst-group change; the control budget; the uncertainty target and the calibration assumptions; the settings where the result holds | the gap, or the outcome of one group, gets worse than with the original controller | the original controller; a controller that penalizes or limits the gap; changing only the predictions afterwards |
Appendix figures & tables55 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | What it changes | What it guarantees | Dynamics |
|---|---|---|---|
| Fair representations [ 42 ] | a learned representation, trained with a parity term | no formal guarantee for later decisions | no |
| Null-space projection [ 36 ] | removes, again and again, the directions a linear classifier uses for the group | no formal guarantee; stops when a linear classifier fails | no |
| LEACE [ 5 ] | erases a linear concept in closed form, with the smallest change | no linear predictor can recover the concept | no |
| Fair regression [ 7 , 9 ] | constrains the predictions to equal group means or equal distributions | the best predictor under the constraint | no |
| Constrained MDPs [ 2 ] | adds the fairness rule to the decision problem | the constraint holds for the model used | yes (model given) |
| Fair reinforcement learning [ 18 , 41 ] | defines fairness over long-term reward or over decision processes, and learns policies | guarantees for their own definitions | yes |
| Quantity | Simulator | School-disj. | Other labels | Student split | Subsamples | Full label |
|---|---|---|---|---|---|---|
| fits | 650 | 5 | 5 | 5 | 3 | 1 |
| 0 | 0 | 0 | 0 | 0 | 0 | |
| – | – | – | – | – |
| Stage | Students | Schools | Rows |
|---|---|---|---|
| Rows with a skill and a school | 29,006 | 402 | 2,711,753 |
| At least 20 attempts | 18,366 | 243 | 2,622,847 |
| At least 26 attempts (full window ) | 16,668 | 216 | 2,584,780 |
| School has a step after (label window) | 16,661 | 211 | 2,584,598 |
| Training | Validation | Test | Group assignment | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Split | Students | Schools | Students | Schools | Students | Schools | train | test | Cutoff |
| 1 | 9,953 | 120 (60/60) | 3,340 | 41 (13/28) | 3,368 | 50 (22/28) | 0.58 | 0.81 | 0.671 |
| 2 | 9,788 | 125 (63/62) | 3,539 | 37 (17/20) | 3,334 | 49 (17/32) | 0.74 | 0.60 | 0.671 |
| 3 | 9,890 | 142 (71/71) | 3,084 | 36 (19/17) | 3,687 | 33 (17/16) | 0.52 | 0.61 | 0.689 |
| 4 | 9,853 | 113 (57/56) | 3,002 | 55 (31/24) | 3,806 | 43 (31/12) | 0.53 | 0.20 | 0.696 |
| 5 | 9,685 | 138 (69/69) | 3,637 | 31 (14/17) | 3,339 | 42 (23/19) | 0.62 | 0.51 | 0.688 |
| Split | Observed [95% CI] | Raw | Recal. | Proj. | MSE ratio | [95% CI] | TRS |
|---|---|---|---|---|---|---|---|
| 1 | 7.4 | 1.50 | |||||
| 2 | 7.7 | 1.27 | |||||
| 3 | 8.1 | 1.17 | |||||
| 4 | 7.8 | 1.31 | |||||
| 5 | 8.0 | 1.29 | |||||
| Mean [ interval] | 7.8 | 1.31 |
| Split | Observed [95% CI] | Raw | Recal. | Proj. | MSE ratio | [95% CI] | Gap error |
| 1 | 8.6 | Y/n/Y | |||||
| 2 | 8.8 | Y/n/Y | |||||
| 3 | 8.5 | Y/n/Y | |||||
| 4 | 9.0 | n/Y/Y | |||||
| 5 | 8.4 | Y/n/Y | |||||
| Mean | 8.4–9.0 | 4/1/5 of 5 |
| Attribute | Split | Observed [95% CI] | Raw | Recal. | Proj. | MSE ratio | Gap error | |
| Deprivation (IMD 0–30%) | 1 | 10.8 | Y/Y/Y | |||||
| 2 | 10.6 | Y/Y/Y | ||||||
| 3 | 10.9 | n/Y/Y | ||||||
| 4 | 11.0 | Y/Y/Y | ||||||
| 5 | 10.9 | Y/Y/Y | ||||||
| Deprivation | mean | (5/5 ) | 10.6–11.0 | 4/5/5 of 5 |
| Attribute | Delay | Observed | Original | Group-aware | Projected | Ratio, all | Ratio, new score | New-score share |
|---|---|---|---|---|---|---|---|---|
| Deprivation | 0 wk | 10.6–11.0 | 1.68–1.74 | 0.149 | ||||
| 1 wk | 10.4–10.7 | 1.71–1.79 | 0.148 | |||||
| 2 wk | 9.8–10.1 | 1.65–1.69 | 0.142 | |||||
| 4 wk | 9.3–9.8 | 1.62–1.67 | 0.128 | |||||
| Disability | 0 wk | 8.4–8.9 | 1.52–1.57 | 0.149 | ||||
| 1 wk | 8.1–8.7 | 1.57–1.62 | 0.148 |
| Predictor | Predicted gap | ||||
|---|---|---|---|---|---|
| Persistence | 0.0085 | 0.0341 | 0.0759 | 0.2356 | |
| Affine least squares | 0.0075 | 0.0249 | 0.0421 | 0.0488 | |
| Gradient boosting | 0.0070 | 0.0282 | 0.0614 | 0.1334 | |
| Raw EDMD | 0.0074 | 0.0244 | 0.0419 | 0.0485 | 0.052 |
| Recalibrated EDMD | 0.0074 | 0.0244 | 0.0418 | 0.0483 | 0.060 |
| Soft projection, | 0.0169 | 0.0480 | 0.0494 | 0.0499 | 0.029 |
| Price at | Frontier at achieved | |||||
|---|---|---|---|---|---|---|
| Setting | Linear ( ) | Nonlinear ( ) | Achieved | Measured | Nonlinear | Linear |
| ASSISTments (school) | 0.0438 (0.030) | 0.0122 (0.120) | 0.0108 | 0.0071 | 0.0203 | |
| Eedi (PremiumPupil) | 0.0592 (0.017) | 0.0412 (0.025) | 0.0391 | 0.0276 | 0.0397 | |
| OULAD (deprivation) | 0.0245 (0.016) | 0.0184 (0.021) | 0.0176 | 0.0142 | 0.0188 | |
| OULAD (disability) | 0.0185 (0.004) | 0.0090 (0.008) | 0.0081 | 0.0035 | 0.0070 | |
| OULAD (gender) | 0.0023 (0.060) | 0.0009 (0.158) | 0.0009 | 0.0011 | 0.0030 | |
| One-step error | Free-running gap | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Setting | Raw | Projected | Dyn.-only | Observed gap | Raw | Projected | Group-aware | Dyn.-only | Same-state diff. |
| ASSISTments (school) | 0.0074 | 0.0575 | 0.0074 | 0.0090 | |||||
| Eedi (PremiumPupil) | 0.0080 | 0.0695 | 0.0080 | 0.0077 | |||||
| OULAD (deprivation) | 0.0030 | 0.0319 | 0.0030 | 0.0028 | |||||
| OULAD (disability) | 0.0030 | 0.0257 | 0.0030 | 0.0025 | |||||
| OULAD (gender) | 0.0030 | 0.0242 | 0.0030 | 0.0038 | |||||
| Model | Group | Quantity | Split 1 | Split 2 | Split 3 | Split 4 | Split 5 |
|---|---|---|---|---|---|---|---|
| Raw EDMD | 0 | CITL | -4.5 [-11.8, +1.3] | -1.7 [-12.4, +2.0] | -0.7 [-10.6, +2.8] | -1.9 [-5.9, +0.5] | -1.4 [-13.2, +2.5] |
| Slope | 1.000 [0.978, 1.023] | 1.008 [0.982, 1.029] | 1.003 [0.982, 1.018] | 1.004 [0.984, 1.022] | 1.003 [0.983, 1.023] | ||
| 1 | CITL | +5.1 [+2.4, +7.9] | +1.4 [-2.0, +5.3] | +7.2 [+2.9, +10.3] | -4.0 [-7.9, +7.5] | -0.7 [-6.4, +6.9] | |
| Slope | 0.977 [0.959, 0.988] | 0.998 [0.977, 1.010] | 0.979 [0.962, 0.992] | 1.017 [0.982, 1.032] | 1.011 [0.976, 1.024] | ||
| Recalibrated EDMD | 0 | CITL | -0.5 [-7.6, +5.1] | +5.7 [-4.2, +9.3] | +2.4 [-6.9, +5.9] | +2.5 [-1.0, +4.6] | +4.3 [-7.0, +7.9] |
| Slope | 0.996 [0.974, 1.019] | 0.999 [0.973, 1.020] | 0.997 [0.976, 1.012] | 0.991 [0.971, 1.009] | 0.996 [0.976, 1.016] |
| Order | Design (re-fits) | Observed | Raw | Proj. | MSE ratio | CI | TRS | |
| Time | School-disjoint, retrospective labels (5) | 0.057 | 0.052 | 7.8 | 2/5 | 1.31 | ||
| Time | Student split, training-student labels (5, seeded) | 0.062 | 0.056 | 7.9 | 5/5 | 1.35 | ||
| Time | School-disjoint, labels from other students (5) | 0.057 | 0.052 | 7.8 | 2/5 | 1.31 | ||
| Time | Disjoint 5,000-student subsamples (3) | 0.056 | 0.050 | 7.9 | 2/3 | 1.23 | ||
| Time | Student split, full-corpus label (1) | 0.088 | 0.077 | 7.6 | 1/1 | 1.27 | ||
| Problem id | School-disjoint, retrospective labels (5) | 0.084 | 0.075 | 7.8 | 2/5 | 1.35 |
| Announced (median) | Realized (median) | Coverage of realized total | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Cell | quad. | total | quad. | total | acc. | transf. | audit | audit, acc. | cond. | cov. built, acc. | |
| Useless, | 2000 | 0.034 | 0.027 | 0.035 | 0.030 | 0.00 | 0.00 | 1.00 | – | 1.00 | – |
| Useless, | 20000 | 0.002 | 0.002 | 0.002 | 0.002 | 0.00 | 0.00 | 0.98 | – | 0.99 | – |
| Useless, | 2000 | 0.004 | 0.003 | 0.004 | 0.003 | 0.00 | 0.00 | 1.00 | – | 1.00 | – |
| Useless, | 20000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.00 | 0.00 | 1.00 | – | 1.00 | – |
| Useless, | 2000 | 0.001 | 0.000 | 0.001 | 0.000 | 0.00 | 0.00 | 1.00 | – | 0.99 | – |
| Announced (median) | Realized (median) | Coverage of realized total | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Cell | quad. | total | quad. | total | acc. | transf. | audit | audit, acc. | cond. | cov. built, acc. | |
| Useless, | 2000 | 0.043 | 0.045 | 0.043 | 0.041 | 0.00 | 0.00 | 1.00 | – | 1.00 | – |
| Useless, | 20000 | 0.002 | 0.001 | 0.002 | 0.002 | 0.00 | 0.00 | 1.00 | – | 1.00 | – |
| Useless, | 2000 | 0.002 | 0.002 | 0.002 | 0.001 | 0.00 | 0.00 | 1.00 | – | 1.00 | – |
| Useless, | 20000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.00 | 0.00 | 1.00 | – | 1.00 | – |
| Useless, | 2000 | 0.001 | 0.000 | 0.001 | 0.000 | 0.02 | 0.00 | 0.94 | 0.00 | 0.96 | 0.00 |
| Coverage of realized total | End-point change vs. | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Cell | Learner | 60 | 200 | 400 | 40 | 60 | 200 | 400 | |
| Partly aligned, | LS | 1.00 | 1.00 | 1.00 | 1.00 | 0.10 | 0.06 | 0.03 | 0.00 |
| Partly aligned, | GB | 1.00 | 1.00 | 1.00 | 1.00 | 0.10 | 0.08 | 0.03 | 0.00 |
| Aligned, | LS | 1.00 | 1.00 | 1.00 | 1.00 | 0.44 | 0.40 | 0.16 | 0.00 |
| Aligned, | GB | 1.00 | 1.00 | 1.00 | 1.00 | 0.46 | 0.29 | 0.13 | 0.00 |
| Aligned, | LS | 0.90 | 0.90 | 0.90 | 0.93 | 0.12 | 0.06 | 0.03 | 0.00 |
| Data | Split | Hidden | Learning rate | Dropout | Epochs | Validation log-loss (best; range over grid) |
|---|---|---|---|---|---|---|
| ASSISTments | 1 | 32 | 0.003 | 0.2 | 23 | 0.5558 (0.5558–0.5862) |
| 2 | 128 | 0.003 | 0.2 | 16 | 0.5998 (0.5998–0.6407) | |
| 3 | 32 | 0.003 | 0.2 | 15 | 0.5740 (0.5740–0.6095) | |
| 4 | 32 | 0.003 | 0.2 | 27 | 0.5713 (0.5713–0.6046) | |
| 5 | 32 | 0.003 | 0.2 | 21 | 0.5624 (0.5624–0.5977) | |
| Eedi | 1 | 64 | 0.003 | 0.2 | 21 | 0.5850 (0.5850–0.6012) |
| Announced (training + validation) | Measured (test) | Measured announced | |||||
|---|---|---|---|---|---|---|---|
| Split | Quadratic | Total | Total | Quadratic | Total | ||
| 1 | estimate | 0.0201 | 0.0227 | 0.0211 | 0.0232 | ||
| 95% | [0.0160, 0.0322] | [0.0181, 0.0351] | [0.0200, 0.0221] | [0.0212, 0.0252] | [-0.0006, 0.0048] | [-0.0027, 0.0065] | |
| 2 | estimate | 0.0252 | 0.0251 | 0.0267 | 0.0288 | ||
| 95% | [0.0202, 0.0427] | [0.0203, 0.0427] | [0.0254, 0.0286] | [0.0262, 0.0325] | [-0.0010, 0.0057] | [0.0001, 0.0098] | |
| 3 | estimate | 0.0179 | 0.0204 | 0.0175 | 0.0187 | ||
| Announced (training + validation) | Measured (test) | Measured announced | |||||
|---|---|---|---|---|---|---|---|
| Split | Quadratic | Total | Total | Quadratic | Total | ||
| 1 | estimate | 0.0484 | 0.0380 | 0.0535 | 0.0532 | ||
| 95% | [0.0000, 28.6582] | [-0.0002, 28.4034] | [0.0444, 0.0621] | [0.0429, 0.0624] | [-0.1969, 3.0692] | [-0.1439, 2.8946] | |
| 2 | estimate | 0.0236 | 0.0245 | 0.0244 | 0.0258 | ||
| 95% | [0.0001, 26.1249] | [-0.0001, 26.1235] | [0.0194, 0.0284] | [0.0200, 0.0314] | [-0.1266, 2.1562] | [-0.1363, 2.1172] | |
| 3 | estimate | 0.5116 | 0.5119 | 0.4850 | 0.4848 | ||
| First version (training units) | Platt recalibrated | Three-stage audit | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Data | Split | Price | slope | slope | Total announced | Accepted / transfers (failed) | |||
| ASSISTments | 1 | 0.0025 | 0.32 | 3.38 | 0.0380 | no / – (D1,D2,D3) | |||
| 2 | 0.0075 | -0.15 | -0.84 | 0.0245 | no / – (D2,D3) | ||||
| 3 | 0.0044 | -0.11 | -0.98 | 0.5119 | no / – (D2,D3) | ||||
| 4 | 0.0052 | -0.11 | -0.27 | 0.0253 | no / – (D1,D2,D3) | ||||
| 5 | 0.0072 | 0.18 | 0.76 | 0.0023 | no / – (D1,D2,D3) | ||||
| History | DKT AUC | Removal AUC | Removal Brier | Price | Measured | Interpretable |
|---|---|---|---|---|---|---|
| 25 items | 0.711 | 0.0236 | 0.0192 | 1/2 | ||
| 50 items | 0.740 | 0.0275 | 0.0245 | 1/2 | ||
| 100 items | 0.746 | 0.0259 | 0.0258 | 2/2 |
| History | Announced total | Measured total | In announced interval | Accepted | Gap reduction transfers | |
|---|---|---|---|---|---|---|
| 25 items | 0.0313 | 0.0307 | 2/2 | 2/2 | 2/2 | |
| 50 items | 0.0347 | 0.0344 | 2/2 | 2/2 | 2/2 | |
| 100 items | 0.0344 | 0.0351 | 2/2 | 2/2 | 2/2 |
| ASSISTments (school) | Eedi (PremiumPupil) | |||||
|---|---|---|---|---|---|---|
| Model | AUC | Brier | Gap | AUC | Brier | Gap |
| EDMD, raw | 0.659 (0.653) | 0.1936 | 0.687 (0.673) | 0.2145 | ||
| EDMD, projected | 0.518 (0.509) | 0.2082 | 0.521 (0.505) | 0.2377 | ||
| EDMD, group-aware | 0.660 (0.653) | 0.1935 | 0.688 (0.673) | 0.2143 | ||
| EDMD, dynamics only | 0.659 (0.653) | 0.1936 | 0.687 (0.673) | 0.2145 | ||
| DKT | 0.648 (0.640) | 0.2047 | 0.710 (0.697) | 0.2104 | ||
| Decision | Error on the gap | Increase of the MSE |
|---|---|---|
| Acceptable | the whole interval lies inside (equivalence) | the upper end of the interval is at most (non-inferiority) |
| Error that matters | the whole interval lies outside | the lower end of the interval is above |
| Inconclusive | the interval crosses or | the interval contains |
| Policy | ||||
|---|---|---|---|---|
| Zero control, raw | ||||
| Zero control, projected | ||||
| MPC, projected | ||||
| MPC, raw |
| Projected minus raw MPC | Mean effect | Sample SD | 95% interval for mean |
|---|---|---|---|
| Disparity contrast | |||
| Pooled disparity | |||
| Pooled knowledge | |||
| Knowledge, group 0 | |||
| Knowledge, group 1 |
| Arm | ratio | ratio | ||||
|---|---|---|---|---|---|---|
| Floor-constrained MPC ( ) | 1.61 | 3.09 | ||||
| Targeted top-up, equal | 1.61 | 3.08 | ||||
| Targeted top-up, equal | 1.61 | 3.09 | ||||
| Uniform increase, equal | 1.61 | 2.60 | ||||
| Floor-constrained MPC ( ) | 1.61 | 3.02 | ||||
| Targeted top-up, equal | 1.61 | 3.02 |
| Objective | Mode | Model error, with group | Model error, without group | Erasure | Random compression | Group information | |
|---|---|---|---|---|---|---|---|
| knowledge | 0.5 | equal | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] |
| [ , ] | [ , ] | [ , ] | [ , ] | [ , ] | |||
| 1.0 | equal | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] | |
| [ , ] | [ , ] | [ , ] | [ , ] | [ , ] | |||
| 2.0 | equal | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] | |
| [ , ] | [ , ] | [ , ] | [ , ] | [ , ] |
| Arm | / | |||||
| : reference , , knowledge gap | ||||||
| Disparity target, floors | 1.60 / 3.07 | |||||
| Disparity target, no floor | 1.09 / 1.91 | |||||
| Knowledge target, floors | 1.09 / 1.15 | |||||
| Knowledge target, no floor | 1.00 / 0.97 | |||||
| Top-up, same | 1.09 / 1.14 | |||||
| raw | constr. | ||||
|---|---|---|---|---|---|
| 0.5 | 1 | 0.236 | 0.233 | -0.132 [-0.134, -0.129] | -0.049 [-0.052, -0.046] |
| 0.5 | 1.5 | 0.236 | 0.217 | -0.136 [-0.138, -0.134] | -0.184 [-0.186, -0.182] |
| 0.5 | 2 | 0.236 | 0.213 | -0.135 [-0.137, -0.134] | -0.252 [-0.254, -0.250] |
| 0.5 | 3 | 0.236 | 0.208 | -0.133 [-0.134, -0.131] | -0.316 [-0.318, -0.313] |
| 0.5 | 5 | 0.236 | 0.204 | -0.130 [-0.131, -0.128] | -0.364 [-0.367, -0.361] |
| 1 | 1 | 0.287 | 0.399 | -0.228 [-0.232, -0.225] | -0.172 [-0.174, -0.169] |
| raw | arm | median | infeas. | |||||
|---|---|---|---|---|---|---|---|---|
| 0.5 | 1 | 0.236 | 0.229 | +0.071 [+0.070, +0.071] | +0.150 [+0.148, +0.153] | -0.200 [-0.203, -0.198] | 0.4 | 0.0% |
| 0.5 | 8 | 0.236 | 0.229 | +0.070 [+0.069, +0.071] | +0.148 [+0.146, +0.151] | -0.199 [-0.201, -0.196] | 2.8 | 0.0% |
| 0.5 | 64 | 0.236 | 0.229 | +0.065 [+0.064, +0.065] | +0.136 [+0.134, +0.138] | -0.185 [-0.187, -0.182] | 22.1 | 0.0% |
| 0.5 | 512 | 0.236 | 0.229 | +0.029 [+0.029, +0.030] | +0.052 [+0.050, +0.054] | -0.095 [-0.097, -0.093] | 151.3 | 0.0% |
| 0.5 | 4096 | 0.236 | 0.229 | -0.068 [-0.069, -0.067] | -0.196 [-0.200, -0.192] | +0.145 [+0.143, +0.148] | 537.3 | 0.0% |
| 0.5 | 32768 | 0.236 | 0.230 | -0.121 [-0.123, -0.119] | -0.330 [-0.336, -0.324] | +0.245 [+0.243, +0.248] | 745.8 | 0.0% |
| Scenario | Comparison | |||
|---|---|---|---|---|
| Group 1 responds half as much | Projected MPC vs. original | |||
| Gap-constrained vs. original | ||||
| Floor-constrained vs. group-aware reference | ||||
| Top-up (equal ) vs. reference | ||||
| Slower learning, twice the noise | Projected MPC vs. original | |||
| Gap-constrained vs. original |
| Arm | ratio | worst group (min) | |||||
|---|---|---|---|---|---|---|---|
| 0.5 | more control, | 1.41 | -0.029 [-0.030, -0.029] | +0.140 [+0.138, +0.143] | +0.232 [+0.229, +0.235] | +0.092 | +0.1326 |
| 0.5 | more control, | 1.78 | -0.054 [-0.055, -0.053] | +0.240 [+0.236, +0.244] | +0.274 [+0.269, +0.279] | +0.034 | +0.2293 |
| 0.5 | more control, | 2.20 | -0.081 [-0.084, -0.079] | +0.309 [+0.303, +0.315] | +0.278 [+0.272, +0.284] | -0.031 | +0.2677 |
| 0.5 | more control, | 2.58 | -0.105 [-0.109, -0.102] | +0.339 [+0.330, +0.347] | +0.279 [+0.273, +0.285] | -0.060 | +0.2678 |
| 0.5 | more control, | 2.79 | -0.120 [-0.124, -0.115] | +0.348 [+0.338, +0.358] | +0.279 [+0.273, +0.285] | -0.069 | +0.2678 |
| 0.5 | floor-constrained, | 1.00 | +0.000 [+0.000, +0.000] | +0.000 [+0.000, +0.000] | +0.000 [+0.000, +0.000] | +0.000 | +0.0000 |
| Setting | Projected: | Holm | Constrained: | Constrained: lower group | Holm |
|---|---|---|---|---|---|
| , rank 4, depth 3, , | 1.4e-02 | 6.6e-62 | |||
| , rank 4, depth 3, , | 3.9e-52 | 1.2e-33 | |||
| , rank 4, depth 3, , | 6.7e-65 | 3.5e-60 | |||
| , rank 4, depth 3, , | 5.1e-75 | 2.9e-80 | |||
| , rank 4, depth 3, , | 4.1e-66 | 8.5e-76 | |||
| , rank 4, depth 3, , | 7.0e-79 | 7.5e-03 |
| Raw MPC | Projected MPC | Gap-penalty MPC | Gap-constrained MPC | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Configuration | ||||||||||
| , | ||||||||||
| , | ||||||||||
| , (main setting) | ||||||||||
| , | ||||||||||
| , | ||||||||||
| Ridge ratio | Mean | Mean | Mean | Mean | 95% interval |
|---|---|---|---|---|---|
| 0.259805 | 0.352879 | -0.453268 | -0.136581 | ||
| 0.259749 | 0.352850 | -0.453240 | -0.136553 | ||
| 0.259734 | 0.352769 | -0.452999 | -0.136312 | ||
| 0.262496 | 0.335681 | -0.453115 | -0.136428 | ||
| 0.296710 | 0.314560 | -0.463989 | -0.147302 | ||
| 0.213994 | 0.446069 | -0.463608 | -0.146921 |
| Rule | Uses group | Low achievers | High achievers | Gap (high low) | Floor met |
|---|---|---|---|---|---|
| P0 control for all | no | yes | |||
| P1 treatment for all | no | no (low) | |||
| P3 better condition per group | yes | yes | |||
| P4 personalized | no | yes | |||
| P5 with floors | yes | yes |
| Exp. | Share | Low | High | Exp. | Share | Low | High | ||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 2909 | 0.498 | 0.88 | 0.49 | 0.51 | 16 | 901 | 0.497 | 0.89 | 0.47 | 0.56 |
| 2 | 2371 | 0.483 | 0.10 | 0.48 | 0.47 | 17 | 749 | 0.477 | 0.21 | 0.43 | 0.55 |
| 3 | 2023 | 0.499 | 0.96 | 0.49 | 0.51 | 18 | 746 | 0.483 | 0.36 | 0.47 | 0.49 |
| 4 | 1903 | 0.481 | 0.11 | 0.49 | 0.46 | 19 | 688 | 0.528 | 0.16 | 0.54 | 0.54 |
| 5 | 1881 | 0.509 | 0.46 | 0.51 | 0.52 | 20 | 467 | 0.518 | 0.46 | 0.54 | 0.51 |
| 6 | 1835 | 0.490 | 0.40 | 0.53 | 0.48 | 21 | 431 | 0.522 | 0.39 | 0.52 | 0.49 |
Group Fairness
Group fairness measures and reduces disparities in how algorithms predict, rank, allocate resources, and make decisions for different groups.
See all 156 papers