Same Model, Different People: Numerical Execution Contracts and the Reproducibility of Fairness Audits
Organizations: University of Toronto, Toronto, Canada · McMaster University, Hamilton, Canada
Abstract
Fairness audits treat a trained model as a fixed decision function, but the arithmetic that evaluates it is part of that function. This study holds the model, records, threshold and audit code fixed and varies the numerical execution contract in two tiers. Tier 1 changes only how products and sums are computed, with the serialized weights byte-identical, and is the causal test of arithmetic. Tier 2 runs reduced-precision deployment configurations (BF16 compute, FP8 and 8-bit weights), which can also round weights and change engines and kernels, so its effects belong to the configuration. It audits three tabular model families and two open-weight language models (4B and 24B parameters) on four public tasks, including hospital readmission, on CPUs and GPUs from NVIDIA, AMD and Apple. Tier 1 changed decisions near the threshold, and which ones depended on the GPU and its software stack, but no verdict under the prespecified, unweighted tests. Tier 2 configurations changed more decisions and some diagnostic verdicts, but no four-fifths verdict. Serving the 24B model with vLLM FP8 selected 54% of people rather than 45% and passed an employment-status audit by sex that its strict FP32 reference failed: its demographic-parity and equalized-odds gaps fell from 0.0567 and 0.0515 to 0.0417 and 0.0394, against a diagnostic tolerance of 0.05. At the reference's selection count FP8 failed demographic parity, and no reading gave its equalized-odds change a paired interval excluding zero. The declared audit depends on the configuration; FP8 is not shown to be fairer. At the frozen thresholds, every pass-fail verdict a configuration changed had a sampling interval containing its criterion. It proposes an audit stability protocol in which the auditor declares contracts, groups, weights and tests before scoring and records each verdict's range over them apart from its sampling uncertainty.
Figures & tables
| Contract | Tier | What changes in the arithmetic | Checked by | Hardware |
|---|---|---|---|---|
| FP64 (reference) | 1 | float64 throughout | – | CPU; H100 (LLM); MI350X (control) |
| Strict FP32 | 1 | IEEE FP32 products | read-back, canary | CPU; RTX 4090, RTX A4000, H100, MI350X |
| TF32 | 1 | operands of matrix products rounded to 10 fraction bits | read-back, canary | RTX 4090, RTX A4000, H100 |
| TF32 setting, AMD | 1 | most products as three BF16 partial products of split operands (about 17 bits per operand); an unidentified arithmetic in 1/8 of the output columns of the 4B model’s large projections | read-back, canary, probe | MI350X |
| M5 relaxed (MLX default) | 1 | both operands of matrix products truncated to 10 fraction bits | per-layer check | Apple M5 |
| M5 certified FP32 | 1 | products routed to an honest FP32 kernel | per-layer check | Apple M5 |
| Task | Train | Test | Features | Positive | AUC (LR / MLP / trees) | LLM AUC | LLM selects |
|---|---|---|---|---|---|---|---|
| ACSIncome | 156,532 | 39,133 | 667 | 0.410 | 0.896 / 0.903 / 0.905 | 0.849 | 52.5% |
| ACSEmployment | 303,054 | 75,763 | 74 | 0.454 | 0.870 / 0.902 / 0.904 | 0.821 | 0.094% |
| HMDA | 277,478 | 69,370 | 538 | 0.902 | 0.887 / 0.922 / 0.919 | 0.650 | 90.4% |
| Diabetes | 79,405 | 19,938 | 1,008 | 0.116 | 0.671 / 0.633 / 0.681 | 0.552 | 10.2% |
| LLM | MLP | ||||||
|---|---|---|---|---|---|---|---|
| Task ( ) | TF32 H100 / 4090 | BF16 | vLLM BF16 | vLLM FP8 | MI350X TF32 / BF16 / FP8 | TF32 | M5 relaxed |
| ACSIncome (39,133) | 9 / 8 | 176 | 166 | 976 | 0 / 191 / 927 | 3 | 1 |
| ACSEmployment (75,763) | 2 / 3 | 12 | 14 | 67 | 2 / 15 / 56 | 1 | 31 |
| HMDA (69,370) | 23 / 21 | 264 | 178 | 1,789 | 2 / 242 / 1,791 | 1 | 6 |
| Diabetes (19,938) | 7 / 7 | 134 | 90 | 822 | 0 / 106 / 629 | 0 | 6 |
| 24B, all tasks (89,133) | 23 / – | 580 | 479 | 3,059 | – | – | – |
| Arm | Non-control contracts | Comp. | 1 pass | 2 signif. | 3 equiv. | 4 rank | 5 4/5 | 6 moved |
| LLM, Tier 1 | TF32 (H100, 4090); TF32 setting (MI350X) | 60 | 0 / 20 | 0 / 20 | 0 / 10 | 0 / 24 | 0 / 10 | 6 / 30 |
| LLM, Tier 2 | BF16, vLLM BF16, vLLM FP8 (H100); BF16, FP8 (MI350X) | 100 | 1 / 20 | 2 (1) / 20 | 0 / 10 | 2 / 24 | 0 / 10 | 26 (1) / 30 |
| Tabular, Tier 1 | TF32 (3 NVIDIA); TF32 setting (MI350X); M5 relaxed | 200 | 0 / 40 | 0 / 40 | 0 / 20 | 0 / 24 | 0 / 20 | 6 / 60 |
| Tabular, Tier 2 | CPU BF16, CPU INT8; MLX BF16, 8-bit, MXFP8; BF16, FP8 (MI350X) | 280 | 5 / 40 | 3 / 40 | 2 / 20 | 10 (4) / 24 | 0 / 20 | 54 (2) / 60 |
| LLM 24B, Tier 1 | TF32 (H100) | 20 | 0 / 20 | 0 / 20 | 0 / 10 | 0 / 24 | 0 / 10 | 3 / 30 |
| LLM 24B, Tier 2 | BF16, vLLM BF16, vLLM FP8 (H100) | 60 | 2 / 20 | 0 / 20 | 0 / 10 | 2 / 24 | 0 / 10 | 28 / 30 |
| Raw | Matched selection | Survey-weighted | ||||||
| Contract | Selected | DP | EO | DP | EO | Selected | DP | EO |
| Strict FP32 | 0.447 | 0.0567 | 0.0515 | 0.0567 | 0.0515 | 0.429 | 0.0550 | 0.0353 |
| vLLM BF16 | 0.461 | 0.0516 | 0.0464 | 0.0581 | 0.0513 | 0.443 | 0.0515 | 0.0313 |
| vLLM FP8 | 0.544 | 0.0417 | 0.0394 | 0.0541 | 0.0481 | 0.533 | 0.0478 | 0.0271 |
| Change | ||||||||
| 95% interval | ||||||||
| Step | Report |
|---|---|
| 1. Declare | by the auditor, before scoring: the admissible contracts (deployment, strict FP32 control, and an FP64 reference or a strict FP32 one checked against FP64), each with framework, version, accelerator and precision settings † ; the comparator, tie rule, groups, any weights, the tests and the claim made |
| 2. Check | per run: the settings read back, a canary product against FP64 and the dispatched kernel names † , kept with the record |
| 3. Score | the frozen model under every contract, twice, in separate processes, or declared times if scores do not repeat |
| 4. Measure | flips against the reference per contract, their margin profile, the stability interval of every audited statistic over all runs, with its ratio to the sampling interval where one is defined, and each decision’s and verdict’s share of runs |
| 5. Classify | each verdict as numerically robust or contract-dependent and, for a gap or ratio, as settled or unsettled by its sampling interval; -values and rankings take no sampling status here |
| 6. Resolve | for a contract-dependent verdict, audit the deployment contract or tighten the contract, and say which |
| Contract | GPU | GPU-hours per | 10^{6}$ | Strict/contract | Flips vs FP64 |
|---|---|---|---|---|---|
| FP64 (reference) | H100 | 11.30 | 39.43 | 1.08 | – |
| Strict FP32 | H100 | 12.17 | 42.49 | 1.00 | 0 |
| TF32 | H100 | 2.81 | 9.81 | 4.33 | 41 |
| BF16 (transformers) | H100 | 1.61 | 5.63 | 7.55 | 586 |
| BF16 (vLLM) | H100 | 3.71 | 12.93 | 3.29 | 448 |
| FP8 (vLLM) | H100 | 3.55 | 12.40 | 3.43 | 3,654 |
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
| Strict | TF32, NVIDIA | TF32 setting, MI350X | M5 relaxed | |||||
|---|---|---|---|---|---|---|---|---|
| Task | Model | Flips | Flips | Flips | ||||
| ACSIncome | LR | 0 / 0 / 0 | 0 | 0 | ||||
| MLP | 3 / 3 / 3 | 0 | 1 | |||||
| ACSEmployment | LR | 0 / 0 / 0 | 0 | 0 | ||||
| MLP | 1 / 1 / 1 | 0 | 31 | |||||
| HMDA | LR | 0 / 0 / 0 | 0 | 0 | ||||
| Task | Model | CPU BF16 | CPU INT8 | MLX BF16 | MLX 8-bit | MLX MXFP8 | MI350X BF16 | MI350X FP8 |
|---|---|---|---|---|---|---|---|---|
| ACSIncome | LR | 39 | 98 | 91 | 41 | 190 | 39 | 230 |
| MLP | 15 | 124 | 46 | 42 | 173 | 15 | 294 | |
| ACSEmployment | LR | 185 | 266 | 49 | 300 | 957 | 185 | 1,601 |
| MLP | 96 | 480 | 150 | 59 | 152 | 96 | 860 | |
| HMDA | LR | 74 | 296 | 61 | 105 | 584 | 74 | 536 |
| MLP | 15 | 323 | 15 | 53 | 172 | 15 | 514 |
| TF32 | TF32 setting | BF16 | vLLM BF16 | vLLM FP8 | FP8 | |
| Task ( ) | H100 / 4090 | MI350X | H100 / MI350X | H100 | H100 | MI350X |
| (A) Decisions changed | ||||||
| ACSIncome (39,133) | 9 / 8 | 0 | 176 / 191 | 166 | 976 | 927 |
| ACSEmployment (75,763) | 2 / 3 | 2 | 12 / 15 | 14 | 67 | 56 |
| HMDA (69,370) | 23 / 21 | 2 | 264 / 242 | 178 | 1,789 | 1,791 |
| Diabetes (19,938) | 7 / 7 | 0 | 134 / 106 | 90 | 822 | 629 |
| Hardware | Contract | Read-back | Matrix product | One-output product | Against own operands |
| Tabular models, Tier 1 | |||||
| Apple M4 Pro CPU | FP64 | – | 0 | 0 | – |
| FP32 | – | – | |||
| RTX 4090 | CPU FP32 (Threadripper 1950X) | – | – | ||
| strict FP32 | off, highest | – | |||
| TF32 | on, high | – | |||
| Sense | Task | Model | Verdict | Contracts | Interval |
| 1 | HMDA | LLM | EO gap by sex | vLLM FP8 (H100) | contains |
| 1 | Diabetes | LR | DP gap by race | MLX 8-bit | contains |
| 1 | Diabetes | MLP | DP gap by age | CPU INT8 | contains |
| 1 | Diabetes | MLP | EO gap by race | MLX 8-bit, CPU INT8 | contains |
| 1 | HMDA | LR | DP gap by ethnicity | MLX MXFP8 | contains |
| 1 | HMDA | LR | EO gap by ethnicity | MLX 8-bit, CPU INT8, FP8 (MI350X) | contains |
| Full test set | 24B subset | |||||||
|---|---|---|---|---|---|---|---|---|
| Task | Attribute | Group | Records | Positives | Enters | Records | Positives | Enters |
| ACSIncome | race | AIAN/NHPI | 509 | 124 | S, E | 509 | 124 | S, E |
| Asian | 6,673 | 3,226 | S, E | 6,673 | 3,226 | S, E | ||
| Black | 1,716 | 566 | S, E | 1,716 | 566 | S, E | ||
| Some other race | 4,541 | 891 | S, E | 4,541 | 891 | S, E | ||
| Two or more | 1,569 | 535 | S, E | 1,569 | 535 | S, E | ||
| Task | Attribute | DP gap [95% CI] | EO gap [95% CI] | Impact ratio [95% CI] | (SR) | Verdicts |
|---|---|---|---|---|---|---|
| Logistic regression | ||||||
| ACSIncome | race | 0.3713 [0.3548, 0.3858] | 0.3575 [0.3219, 0.3933] | 0.2640 [0.2431, 0.2854] | 0 | F F S S N F |
| sex | 0.1375 [0.1282, 0.1472] | 0.0877 [0.0753, 0.1016] | 0.7102 [0.6924, 0.7272] | F F S S N F | ||
| ACSEmployment | race | 0.1668 [0.1536, 0.1865] | 0.1386 [0.1204, 0.1579] | 0.7162 [0.6856, 0.7366] | F F S S N F | |
| sex | 0.0881 [0.0812, 0.0954] | 0.1150 [0.1076, 0.1228] | 0.8458 [0.8341, 0.8572] | F F S S N P | ||
| HMDA | race | 0.0935 [0.0771, 0.1354] | 0.1430 [0.0839, 0.3446] | 0.9004 [0.8592, 0.9171] | 0.0035 | F F S S N P |
| Task | Attribute | Statistic | FP64 | Tier 1, all GPUs | All, NVIDIA | All, with MI350X |
|---|---|---|---|---|---|---|
| ACSIncome | race | DP gap | 0.4505 | [0.4504, 0.4508] | [0.4504, 0.4546] | [0.4495, 0.4546] |
| EO gap | 0.3656 | [0.3656, 0.3656] | [0.3656, 0.3856] | [0.3630, 0.3856] | ||
| Impact ratio | 0.3201 | [0.3199, 0.3204] | [0.3021, 0.3204] | [0.3021, 0.3212] | ||
| (SR) | 0 | [0, 0] | [0, 0] | [0, 0] | ||
| (TPR) | [ , ] | [ , ] | [ , ] | |||
| (TOST) | 1 | [1, 1] | [1, 1] | [1, 1] |
| Task | Attribute | Statistic | Strict FP32 | Tier 1 | All |
|---|---|---|---|---|---|
| ACSIncome | race | DP gap | 0.4456 | [0.4456, 0.4456] | [0.4456, 0.4528] |
| EO gap | 0.3845 | [0.3845, 0.3845] | [0.3824, 0.3957] | ||
| Impact ratio | 0.2896 | [0.2896, 0.2896] | [0.2826, 0.2900] | ||
| (SR) | 0 | [0, 0] | [0, 0] | ||
| (TPR) | [ , ] | [ , ] | |||
| (TOST) | 1 | [1, 1] | [1, 1] |
| Task | Model | Attribute | Statistic | FP64 [95% CI] | All Tier 2, M5 Max | BF16 only | MI350X |
|---|---|---|---|---|---|---|---|
| ACSEmployment | MLP | sex | (TOST) | 0.0012 | [ , 0.089] † | [ , 0.0012] | [0.0010, 0.0013] |
| HMDA | LR | ethnicity | DP gap | 0.0473 [0.0364, 0.0584] | [0.0446, 0.0516] † | [0.0473, 0.0479] | [0.0460, 0.0479] |
| HMDA | LR | ethnicity | EO gap | 0.0527 [0.0282, 0.1182] | [0.0415, 0.0573] † | [0.0527, 0.0547] | [0.0418, 0.0547] † |
| HMDA | MLP | race | (TPR) | 0.21 | [ , 0.21] † | [0.0034, 0.21] † | [ , 0.21] † |
| Diabetes | LR | age | (TOST) | 0.098 | [0.020, 0.13] † | [0.086, 0.11] | [0.098, 0.30] |
| Diabetes | LR | race | DP gap | 0.0463 [0.0229, 0.1012] | [0.0407, 0.0540] † | [0.0461, 0.0473] | [0.0441, 0.0473] |
| Model and contract | ACSIncome | ACSEmployment | HMDA | Diabetes |
|---|---|---|---|---|
| Tabular, Tier 1 | ||||
| MLP, strict FP32 and FP64 | 0 / 2 | 1 / 35 | 1 / 1,516 | 0 / 1 |
| MLP, TF32, RTX A4000 | 6 / 7 | 8 / 35 | 11 / 1,516 | 5 / 6 |
| MLP, TF32, RTX 4090 | 6 / 7 | 25 / 40 | 11 / 1,516 | 5 / 6 |
| MLP, TF32, H100 | 6 / 7 | 23 / 40 | 11 / 1,516 | 5 / 6 |
| MLP, TF32 setting, MI350X | 1 / 1 | 2 / 33 | 1 / 1,516 | 0 / 2 |
| Platform | Arm | Software |
|---|---|---|
| Apple M4 Pro CPU | tabular reference | macOS 27.0.1; Python 3.12.13; PyTorch 2.14.1; XGBoost 3.2.0 |
| RTX 4090, RTX A4000 | tabular Tier 1 | Linux 7.0.0; Python 3.14.4; PyTorch 2.14.1+cu130 (CUDA 13.0 build); cuDNN 9.24.0; NVIDIA driver 595.71.05; XGBoost 3.2.0 (CUDA 12.9) |
| H100 | tabular Tier 1 | Linux 5.14.0; Python 3.12.4; PyTorch 2.11.0 (CUDA 12.9 build); cuDNN 9.13.1; NVIDIA driver 580.173.02; XGBoost 3.2.0 (CUDA 12.9) |
| Apple M5 Max GPU | tabular Tiers 1 and 2 | macOS 27.0.1; Python 3.14.7; MLX 0.32.3.dev20260906, a development build: official commit ce916dbbc plus one quantized-matmul kernel fix |
| Apple M5 Max CPU | tabular Tier 2 | macOS 27.0.1; Python 3.12.13; PyTorch 2.14.1 |
| AMD MI350X | tabular and language model | Linux 7.0.0; Python 3.12.15; PyTorch 2.14.1+rocm7.2; HIP 7.2.53211; amdgpu driver 6.19.14; transformers 5.18.0 |
| Declared | Joint excluded | ||||
|---|---|---|---|---|---|
| Arm | Tier 1 | Tier 2 | Tier 1 | Tier 2 | At stake |
| Tabular LR and MLP | 0 | 3 | 0 | 1 | 36 |
| 4B, APPROVE/DENY | 0 | 1 | 0 | 0 | 24 |
| 4B, YES/NO | 0 | 3 | 0 | 2 | 24 |
| 24B | 0 | 0 | 0 | 0 | 24 |
| Contract | Flips | Observed | Null mean | Null 95% range | ||
| Language model, Tier 1 | ||||||
| TF32, H100 | 41 | 0 | 0.13 | [0, 1] | 1 | 0.8692 |
| TF32, RTX 4090 | 39 | 0 | 0.13 | [0, 1] | 1 | 0.866 |
| TF32 setting, MI350X | 4 | 0 | 0.00 | [0, 0] | 1 | 1 |
| Language model, Tier 2 | ||||||
| BF16, H100 | 586 | 1 | 0.94 | [0, 2] | 0.7224 | 0.7864 |
| Arm | Verdicts | At stake | Observed | Primary | Finer | Coarser | ||
|---|---|---|---|---|---|---|---|---|
| LLM, Tier 1 | senses 1 to 5 | 84 | 0 | 0.25 [0, 1] | 0.25 [0, 1] | 0.05 [0, 1] | 1 | 0.7542 |
| unsettled | 5 | 0 | 0.00 [0, 0] | 0.00 [0, 0] | 0.00 [0, 0] | 1 | 1 | |
| settled | 35 | 0 | 0.00 [0, 0] | 0.00 [0, 0] | 0.00 [0, 0] | 1 | 1 | |
| LLM, Tier 2 | senses 1 to 5 | 84 | 5 | 7.34 [5, 10] | 7.29 [4, 10] | 7.28 [4, 11] | 0.986 | 0.0798 |
| unsettled | 5 | 1 | 0.02 [0, 0] | 0.01 [0, 0] | 0.02 [0, 0] | 0.017 | 1 | |
| settled | 35 | 0 | 1.27 [1, 2] | 1.23 [1, 2] | 0.93 [0, 2] | 1 | 0.0002 |
| 4B LLM | Tabular | 24B LLM | ||||
| Procedure | Tier 1 | Tier 2 | Tier 1 | Tier 2 | Tier 1 | Tier 2 |
| Pooled statistics | ||||||
| Each contract’s own worst pair (prespecified) | 0 | 2 | 0 | 3 | 0 | 0 |
| Reference’s pair, unadjusted | 0 | 2 | 0 | 1 | 0 | 0 |
| Reference’s pair, Bonferroni | 1 | 4 | 0 | 1 | 0 | 2 |
| All pairs, Holm | 0 | 3 | 0 | 0 | 0 | 1 |
| Prespecified | Wald | Wald, clustered | ||||
|---|---|---|---|---|---|---|
| Test | Tier 1 | Tier 2 | Tier 1 | Tier 2 | Tier 1 | Tier 2 |
| Own worst pair | 0 | 4 | 1 | 5 | 1 | 5 |
| Reference’s pair, unadjusted | 0 | 3 | 1 | 5 | 1 | 5 |
| Reference’s pair, Bonferroni | 1 | 6 | 0 | 3 | 0 | 4 |
| All pairs, Holm | 0 | 3 | 0 | 3 | 0 | 4 |
| Equal rates (omnibus) | 0 | 2 | 0 | 3 | 0 | 4 |
| Interval contains 0.05 | Interval excludes 0.05 | |||
|---|---|---|---|---|
| Sense | Changed | Unchanged | Changed | Unchanged |
| 1, pass-fail (DP and EO gaps) | 6 | 11 | 0 | 43 |
| 3, equivalence of selection rates | 1 | 6 | 1 | 22 |
| Contract | Sel. | AUC | Acc. | Bal. acc. | Women | Men | DP | EO | DP / EO, matched |
|---|---|---|---|---|---|---|---|---|---|
| Strict FP32 | 0.447 | 0.794 | 0.707 | 0.704 | 0.419 / 0.643 / 0.253 | 0.476 / 0.694 / 0.272 | 0.0567 | 0.0515 | 0.0567 / 0.0515 |
| TF32 | 0.448 | 0.794 | 0.708 | 0.705 | 0.419 / 0.643 / 0.253 | 0.476 / 0.695 / 0.272 | 0.0568 | 0.0516 | 0.0571 / 0.0521 |
| BF16 | 0.463 | 0.796 | 0.713 | 0.711 | 0.436 / 0.666 / 0.264 | 0.491 / 0.718 / 0.279 | 0.0554 | 0.0520 | 0.0619 / 0.0590 |
| vLLM BF16 | 0.461 | 0.794 | 0.710 | 0.708 | 0.435 / 0.663 / 0.265 | 0.487 / 0.710 / 0.279 | 0.0516 | 0.0464 | 0.0581 / 0.0513 |
| vLLM FP8 | 0.544 | 0.808 | 0.735 | 0.741 | 0.523 / 0.786 / 0.328 | 0.565 / 0.825 / 0.322 | 0.0417 | 0.0394 | 0.0541 / 0.0481 |
| Contract | Flips | Verdicts changed |
|---|---|---|
| Strict FP32, H100 / RTX 4090 / MI350X | 2 / 2 / 2 | none |
| TF32, H100 / RTX 4090; TF32 setting, MI350X | 122 / 135 / 48 | none |
| BF16, H100 / MI350X | 1,791 / 1,673 | none |
| vLLM BF16, H100 | 1,243 | none |
| vLLM FP8, H100 | 10,214 | EO pass-fail; SR and TPR significance; EO ranking |
| FP8, MI350X | 7,558 | SR significance |
| Link | Selection rate | DP gap | EO gap |
|---|---|---|---|
| Strict FP32, transformers | 0.447 | 0.0567 | 0.0515 |
| BF16, transformers | 0.463 | 0.0554 | 0.0520 |
| BF16, vLLM | 0.461 | 0.0516 | 0.0464 |
| FP8, vLLM | 0.544 | 0.0417 | 0.0394 |
| Step (flips) | DP change | EO change | |
| Configuration, transformers (380) |
| Result | Status | Survives | Fails or untested |
|---|---|---|---|
| Strict FP32 changed no 4B decision at the prespecified thresholds; every contract repeated its decisions | Prespecified | three GPUs, both runs | vLLM BF16 did not repeat its scores bit for bit |
| Tier 1 flipped decisions near the threshold (TF32: 41 and 39 of the 4B model’s) | Prespecified | both runs; predicted in advance for new MLPs on the M5, an RTX 4090 and an H100 | the prediction for one MLP on an RTX A6000 |
| Tier 1 changed no verdict (84, 144 and 84 at stake) | Prespecified, unweighted tests | exact tests; tests both clustered and adjusted (Diabetes); fixed capacity | five of nine alternative tests (one sparse-cell verdict each); survey weights (one verdict at its criterion) |
| Which records flip depends on the GPU and stack (pooled Jaccard 0.38) | Prespecified measure; chance level post hoc | above-chance agreement on HMDA, under all six matchings | chance agreement on the other three tasks |
| Tier 2 changed verdicts (Table 4 ) | Prespecified senses | both runs | 7 of 9 pass-fail pairs have paired intervals including zero |
| Tier 2 changed only pass-fail verdicts whose interval contains 0.05 (6 of 17, 0 of 43) | Exploratory; 24B arm by a rule fixed in advance (2 of 5, 0 of 15) | , BCa, resampling by patient | a base-rate threshold (one change outside) |
| Flips, | Flips, | ||||||
|---|---|---|---|---|---|---|---|
| Model, task | Contract | Ties | All | Ties | All | Ties | Untied |
| 4B, ACSIncome | BF16, H100 | 147 | 176 | 41 | 241 | 106 | 135 |
| vLLM BF16, H100 | 149 | 166 | 60 | 195 | 89 | 106 | |
| vLLM FP8, H100 | 145 | 976 | 47 | 1,027 | 98 | 929 | |
| BF16, MI350X | 128 | 191 | 39 | 241 | 89 | 152 | |
| FP8, MI350X | 117 | 927 | 60 | 924 | 57 | 867 | |
| Contract | Tier | What changes in the arithmetic | Checked by | Hardware |
|---|---|---|---|---|
| FP16 (exploratory) | 2 | inputs, every layer output and the tabular weights in FP16 (largest finite 65504) | canary, range canary | Apple M4 Pro CPU; RTX 4090 (LLM) |
| Task | Model | BF16 | FP16 | E8M10 | E5M10 | Range |
|---|---|---|---|---|---|---|
| ACSIncome | LR | 39 (11) | 4 (2) | 4 (2) | 4 (2) | 0 |
| MLP | 15 (10) | 1 (3) | 1 (3) | 1 (3) | 2,240 | |
| ACSEmployment | LR | 185 (231) | 6 (20) | 6 (20) | 6 (20) | 3 |
| MLP | 96 (60) | 5 (1) | 5 (1) | 5 (1) | 11,553 | |
| HMDA | LR | 74 (19) | 18 (7) | 18 (7) | 18 (7) | 0 |
| MLP | 15 (15) | 1 (0) | 1 (0) | 1 (0) | 2,834 |
| Task | Band | BF16 | FP16 | Eager | SDPA band | E8M10 | E5M10 | Shared | Median |
|---|---|---|---|---|---|---|---|---|---|
| ACSIncome | 6,062 | 184 | 28 | 41 | 21 | 41 | 44 | 19 | 0.031 |
| ACSEmployment | 1,306 | 13 | 3 | 6 | 2 | 3 | 3 | 1 | 0.031 |
| HMDA | 4,515 | 248 | 36 | 41 | 32 | 48 | 50 | 25 | 0.063 |
| Diabetes | 4,384 | 101 | 13 | 21 | 13 | 21 | 18 | 9 | 0.031 |
| Arm | Expected | Observed |
| Tabular, relaxed default | 0.05% to 0.1% for LR and MLP | failed: LR 0; MLP 0.0026% to 0.041% |
| Tabular, TF32 | the order of the relaxed default | failed: LR 0; MLP at most 0.0077% |
| Tabular, Tier 1 audit | most checks unchanged | held: only sense 6 changed |
| Language model, TF32 | 0.1% to 1% | failed: 0.020% (H100), 0.019% (RTX 4090) |
| Language model and deployment, audit | changes, if any, most likely in senses 2, 3 and 6 | partly: the language model’s Tier 1 changed only sense 6; Tier 2 contracts changed senses 1 to 4 and 6 |
| Deployment contracts | the largest, plausibly above 1% for the language model | held: up to 4.123% (FP8 serving, Diabetes) |
| Model | Mistral-Small-3.2-24B-Instruct-2506, revision 95a6d26c , 24,011,361,280 parameters; BF16 weight fingerprint b316a3ef |
| Records | ACSEmployment test subset, 20,000 records; groups by recorded sex, each with at least 100 records and 30 positives and negatives |
| Decision | select when the margin of YES over NO at the first label token is at least , the quantile of 2,048 strict FP32 training margins that selects the training employment rate, 0.4566 |
| Tests | demographic-parity and equalized-odds gaps pass at 0.05 or less; 95% percentile intervals from 1,000 bootstrap resamples of the records |
Algorithmic Auditing
Algorithmic auditing evaluates whether automated systems are fair, reliable, robust, compliant, and accountable in their intended settings.
See all 160 papers