cs.CYAug 30, 2026

Applications of Risk Science to AI Fairness Evaluation: Principles, Challenges, and Best Practices

Authors: Kyra Wilson, Sabrina Kang, Saloni Dash, Aylin Caliskan

Organizations: University of Washington Seattle, WA 98195 USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Scholarly work which aims to describe potential societal impacts (e.g., risks) of proliferating technology (especially related to artificial intelligence or other algorithmic systems) is likely to have an impact beyond the scientific communities it was written for, given that general society itself is a primary object of study. However, it is an open question whether the current practices of AI evaluation scholarship follow the principles and best practices established by risk science, which aims to systematically generate knowledge related to understanding, assessing, communicating, managing, and governing risk. In this work, we examine this in depth by conducting a literature review of scholarly works purporting to evaluate the bias or fairness of technological systems used for tasks related to hiring and employment. Through analysis of 22 common fairness evaluation metrics and studies using them, we find that most characterize the severity of bias- or fairness-related consequences but do not follow best practices to characterize the uncertainty around either the occurrence of these consequences or severity estimates. Next, we conduct a case study of fairness evaluation for an AI-mediated resume screening task and demonstrate how principles of risk science can be incorporated into such an evaluation. Finally, we propose the AI Risk Report Card, which facilitates the reporting and communication of risk assessment results to stakeholders in positions to act based on the predicted risks. The outcomes of these activities suggest that further research at the convergence of risk science and AI evaluation can lead to advancements in AI assessments of societal impact by enabling shared frameworks to evaluate and discuss AI risks both within and outside of the scientific community.

Figures & tables

Figure 1: Illustration of how both the expected severity of a consequence and associated uncertainties play a role in risk assessment. In the two leftmost panels, the expected severity μ\mu (0.85) is identical, but the probability that an event is harmful (any severity less than 0.8) is over 60% greater when the variability σ\sigma increases. If the expected severity falls within the range of harmful consequences (as in the two rightmost panels where severity is 0.75), increasing the variability has the opposite effect and decreases the probability of a severe outcome. Thus, considering both the severity and uncertainty or variability of possible consequence(s) is essential to understanding and effectively controlling or mitigating it.
MeasureBoundsValid Probability
Skew@k(- ∞\infty , ∞\infty )No (I)
Normalized Discounted Cumulative Kullback-Leibler Divergence (NDKL)[0,1]No (II)
Disparate Impact (DI)[0,1]No (II)
Demographic Disparity (DD)[-1,1]No (I)
Representation in Positive Predictive Rate (RPPR)[-1,1]No (I)
True-Positive Rate Difference (TPRD)[-1,1]No (I)
Table 1: Summary of whether the AI hiring fairness metrics listed in Fabris et al. (2025) describe severity or uncertainty of harms. If a metric is not a valid probability (all but one), the constraints it violates are given in parentheses (corresponding to criteria in Equations I and II ). The colors of rows correspond to the different kinds of fairness identified by Fabris et al. (2025) : Outcome , Accuracy , Impact , Representational , and Process .
Figure 2: Among the 28 evaluations in our analysis set, only seven provided variability estimates for their metric results. Of those, four provided variability estimates for all results: one used a confidence interval, and three used a combination of standard deviations or errors and visualizations of distributions. Two papers provided confidence intervals for some but not all results, and one provided variability estimates for some results but did not specify the type.
Figure 3: Examples of risk characterizations given in the AI resume screening evaluation conducted by Wilson and Caliskan (2024) .
Figure 4: [A] The distribution of observed Disparate Impact (DI) scores when evaluating white men’s vs. Black men’s resumes using GritLM. The red line marks the sample average; the two dashed black lines represent different possible DI thresholds at 0.666 and 0.800. [B] The number of resume screening scenarios that are classified as discriminatory vs. non-discriminatory when the DI threshold is 0.800 (based on the 80% rule). [C] The number of resume screening scenarios which are classified as discriminatory vs. not when the DI threshold is 0.666 (based on statistical significance).
Figure 5: Example of a proposed AI Risk Report Card to facilitate communication between researchers conducting AI evaluations of societal impact and stakeholders beyond the scientific community.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

UncertaintyStrength of KnowledgeCriteriaConditions
   The phenomena involved are well understood; the models used are known to give predictions with the required accuracy
   Much reliable data are available
   The assumptions made are seen as very reasonable
LowStrongAll conditions are fulfilledThere is broad agreement among experts
   Example: The phenomena involved are well understood, but the models used are considered simple
MediumModerateConditions between low and high are fulfilled to varying degreesExample: Some reliable data are available
Appendix
Table 7
RankCriteria
Very high riskPotential for extreme consequences, relatively large associated probability of such consequences and/or significant uncertainty (relatively weak background knowledge)
High riskThe potential for extreme consequences, relatively small associated probability of such consequences and moderate or weak background knowledge
Moderate riskBetween low and high risk. For example, the potential for moderate consequences, and weak background knowledge.
Low riskNo potential for serious consequences
Appendix
Table 8
MeasureEquationStudies
Skew@k\skewkg=log⁡(Ngk/kDg)\skewk_{g}=\log\left(\frac{N_{g}^{k}/k}{D_{g}}\right)( Geyik, Ambler, and Kenthapadi 2019 )
Normalized Discounted Cumulative Kullback-Leibler Divergence (NDKL)\ndkl=1Z∑k=1K1log⁡2(k+1)dKL(D,Nk)\ndkl=\frac{1}{Z}\sum_{k=1}^{K}\frac{1}{\log_{2}(k+1)}d_{\text{KL}}(D,N^{k})( Geyik, Ambler, and Kenthapadi 2019 ) ( Arafan et al. 2022 )
Disparate Impact (DI)\di=min⁡g∈SNgk/Ngmax⁡g∈SNgk/Ng\di=\frac{\min_{g\in\mathcal{S}}N_{g}^{k}/N_{g}}{\max_{g\in\mathcal{S}}N_{{g}}^{k}/N_{{g}}} =min⁡g∈SPr⁡(y^=1∣s=g)max⁡g∈SPr⁡(y^=1∣s=g)=\frac{\min_{g\in\mathcal{S}}\Pr(\hat{y}=1|s=g)}{\max_{g\in\mathcal{S}}\Pr(\hat{y}=1|s={g})}( Booth et al. 2021 ) ( Burke, Burke, and Kuljanin 2021 ) ( Köchling et al. 2021 ) ( Wilson et al. 2021 ) ( Deshpande, Pan, and Foulds 2020 )
Demographic Disparity (DD)\dd=Ngk/Ng−Ngck/Ngc\dd=N_{g}^{k}/N_{g}-N_{{g}^{\mathsf{c}}}^{k}/N_{{g}^{\mathsf{c}}} =Pr⁡(y^=1∣s=g)−Pr⁡(y^=1∣s=gc)=\Pr(\hat{y}=1|s=g)-\Pr(\hat{y}=1|s={g}^{\mathsf{c}})( Rus et al. 2022 )
Representation in Positive Predictive Rate (RPPR)\rppg=Pr⁡(s=g∣y^=1)\rpp_{g}=\Pr(s=g|\hat{y}=1) RPPD=\rppg−\rppgc=2\rppg−1\text{RPPD}=\rpp_{g}-\rpp_{{g}^{\mathsf{c}}}=2\rpp_{g}-1( Ali et al. 2019 ) ( Imana, Korolova, and Heidemann 2021 ) ( Pena et al. 2020 )
True-Positive Rate Difference (TPRD)TPRg=Pr⁡(y^=1∣y=1,s=g)\text{TPR}_{g}=\Pr(\hat{y}=1|y=1,s=g) \tprd=TPRg−TPRgc\tprd=\text{TPR}_{g}-\text{TPR}_{{g}^{\mathsf{c}}}( De-Arteaga et al. 2019 ) ( Hemamou and Coleman 2022 )
Appendix
Table 2: Summary of the AI hiring fairness metrics listed in Fabris et al. (2025) and the studies which used them. The colors of rows correspond to the different kinds of fairness identified by Fabris et al. (2025) : Outcome , Accuracy , Impact , Representational , and Process .
Figure 6: [A] The distribution of observed Disparate Impact (DI) scores when evaluating white men’s vs. Black men’s resumes using e5. The red line marks the sample average; the two dashed black lines represent different possible DI thresholds at 0.666 and 0.800. [B] The number of resume screening scenarios that are classified as discriminatory vs. non-discriminatory when the DI threshold is 0.800 (based on the 80% rule). [C] The number of resume screening scenarios which are classified as discriminatory vs. not when the DI threshold is 0.666 (based on statistical significance).
Figure 7: [A] The distribution of observed Disparate Impact (DI) scores when evaluating white men’s vs. Black men’s resumes using SFR. The red line marks the sample average; the two dashed black lines represent different possible DI thresholds at 0.666 and 0.800. [B] The number of resume screening scenarios that are classified as discriminatory vs. non-discriminatory when the DI threshold is 0.800 (based on the 80% rule). [C] The number of resume screening scenarios which are classified as discriminatory vs. not when the DI threshold is 0.666 (based on statistical significance).
More in this topic

AI Safety Evaluation

AI Safety Evaluation studies how to test whether AI systems avoid harmful behavior and remain reliable across users, tasks, and deployment settings.

See all 143 papers

Explore similar work

CardsList

  1. Overview of Risk Assessment and Management for Intelligent Systems under the AI Act and Beyond

    Jul 2, 2026Javier Irigoyen, Roberto Daza, Aythami Morales +5AI Risk ManagementAI Governance

    BiometricsAI, Universidad Aut´onoma de Madrid (UAM), Spain · GHIA, Universidad Aut´onoma de Madrid (UAM), Spain · Universidad de Las Palmas de Gran Canaria (ULPGC), Spain

  2. Measuring Biological Capabilities and Risks of AI Agents

    Jun 18, 2026Patricia Paskov, Jeffrey Lee, Kyle Brady +1AI for ScienceAI Agent Evaluation

    RAND Corporation, Santa Monica, Calif.

  3. PopResume: Causal Fairness Evaluation of LLM/VLM Resume Screeners with Population-Representative Dataset

    Mar 24, 2026Sumin Yu, Juhyeon Park, Taesup MoonAI-Assisted Decision MakingLLM Auditing

    ECE, Seoul National University · IPAI, Seoul National University · ASRI / INMC / AIIS, Seoul National University