cs.CYNov 18, 2026

Same Model, Different People: Numerical Execution Contracts and the Reproducibility of Fairness Audits

Authors: Erzheng Zhang, Agnes Chu

Organizations: University of Toronto, Toronto, Canada · McMaster University, Hamilton, Canada

Abstract

Fairness audits treat a trained model as a fixed decision function, but the arithmetic that evaluates it is part of that function. This study holds the model, records, threshold and audit code fixed and varies the numerical execution contract in two tiers. Tier 1 changes only how products and sums are computed, with the serialized weights byte-identical, and is the causal test of arithmetic. Tier 2 runs reduced-precision deployment configurations (BF16 compute, FP8 and 8-bit weights), which can also round weights and change engines and kernels, so its effects belong to the configuration. It audits three tabular model families and two open-weight language models (4B and 24B parameters) on four public tasks, including hospital readmission, on CPUs and GPUs from NVIDIA, AMD and Apple. Tier 1 changed decisions near the threshold, and which ones depended on the GPU and its software stack, but no verdict under the prespecified, unweighted tests. Tier 2 configurations changed more decisions and some diagnostic verdicts, but no four-fifths verdict. Serving the 24B model with vLLM FP8 selected 54% of people rather than 45% and passed an employment-status audit by sex that its strict FP32 reference failed: its demographic-parity and equalized-odds gaps fell from 0.0567 and 0.0515 to 0.0417 and 0.0394, against a diagnostic tolerance of 0.05. At the reference's selection count FP8 failed demographic parity, and no reading gave its equalized-odds change a paired interval excluding zero. The declared audit depends on the configuration; FP8 is not shown to be fairer. At the frozen thresholds, every pass-fail verdict a configuration changed had a sampling interval containing its criterion. It proposes an audit stability protocol in which the auditor declares contracts, groups, weights and tests before scoring and records each verdict's range over them apart from its sampling uncertainty.

Figures & tables

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Robust Decentralized Fairness Auditing

    Oct 7, 2026Sayan Biswas, Jade Garcia Bourrée, Anne-Marie Kermarrec +2Algorithmic AuditingLLM Auditing

  2. Sequential Fairness Auditing with Limited Output Access

    Jun 29, 2026Ioannis Pitsiorlas, Martha V. Sourla, Marios KountourisAlgorithmic FairnessGroup Fairness

  3. When Post-Processing Fairness Constraints Help and When They Harm: Evidence from Eight Cross-Domain Evaluations

    Sep 22, 2026Nithin Raghava Ramachandra NarlaAlgorithmic AuditingAlgorithmic Fairness