Counterfactual Bias Testing for Application Tracking System

Sai Yashwant; Shruti Bansal; Anurag Dubey; Samaroha Chatterjee; Satyam Kumar; Shreyash Gupta; Gantala Thulsiram

cs.AIAug 27, 2026

Counterfactual Bias Testing for Application Tracking System

Authors: Sai Yashwant, Shruti Bansal, Anurag Dubey, Samaroha Chatterjee, Satyam Kumar, Shreyash Gupta, Gantala Thulsiram

Organizations: ManpowerGroup Services India Pvt. Ltd. · Indian Institute of Technology, Hyderabad

Abstract

Automated candidate-job matching systems are increasingly classified as high-risk AI under emerging regulation, yet auditing them for demographic bias is expensive: classical correspondence-audit studies require hand-crafted resumes and manual submission, which does not scale to fast pipeline retraining cycles. This paper presents a general, reusable methodology that (1) uses task-specialized LLM agents to synthesize identity-neutral base resumes and inject controlled demographic treatments across five protected-characteristic axes (sex/gender, age, residence, language, disability), producing a K x (1+N) correspondence-audit matrix; (2) qualitatively flags inferred protected characteristics per an EU AI Act-aligned prompt; (3) ranks candidates against a job description via a fine-tuned sentence-embedding model and cosine similarity; and (4) computes a nine-metric fairness suite spanning counterfactual (score delta, mean absolute rank change, flip rate), group-fairness (top-K retention, four-fifths/impact ratio), and merit-aware (Recall@K, nDCG@K, equal opportunity, equalized odds) families, each with bootstrap confidence intervals, significance tests, and Benjamini-Hochberg correction, culminating in an automated PASS/INVESTIGATE/FAIL report with a composite risk score. On an example corpus of 5 job orders, 100 base candidates, and 10 demographic treatments (90 metric x variant evaluations): score shifts, top-K retention, and merit-aware rate gaps stay within tolerance for every treatment, but a rank-stability metric (MARC) and nDCG@K each surface borderline findings - including one on the neutral baseline itself - that a score- or retention-only view would miss. The results argue for multi-metric, multi-family auditing over any single aggregate score, and for LLM-agent-generated audits as a practical, low-cost complement to human-curated audits for any candidate-job matching pipeline.

Figures & tables

Panel of Figure 1 — Figure 1: Flowchart of the end-to-end correspondence-audit generation and fairness-auditing pipeline of Algorithm 1 . Stages 2–3 (translation, optional bias flagging) repeat once per generated resume; Stage 4 (bottom five boxes) runs on demand over the full accumulated candidate pool.

Axis	Allowed values	Signal used for classification
sex_gender	Male-FR, Male-NA, Female-FR, Female-NA	Gender cue plus North-African/Arabic/immigrant-origin naming cue ( -NA ) vs. default ( -FR )
age_signal	Young, Older	Explicit age, years of experience, or “older” framing
place_of_residence	Close, Far, Very far	Commute distance / peripheral-location framing
language_signal	FR only, FR+EN, FR+AR	Mentioned language fluencies
disability_signal	Yes, No	Disability / RQTH / health-limitation mention

Table 1: Protected-characteristic axis taxonomy used for automated classification and metric stratification.

Variant	sex_gender	age_signal	place_of_residence	language_signal	disability_signal
V01	Female-FR	Older	Far	FR+EN	No
V02	Female-FR	Young	Very far	FR+AR	Yes
V03	Male-FR	Older	Close	FR+AR	Yes
V04	Female-NA	Young	Far	FR+AR	Yes
V05	Male-NA	Older	Very far	FR+EN	No
V06	Female-FR	Older	Close	FR only	Yes

Table 2: Composition of the nine bias variants

V_{01}

–

V_{09}

used in the example corpus of Section 4 , as one specific combination of levels from the protected-axis taxonomy of Table 1 per variant. BASE (Male-FR, Young, Close, FR only, No) is the neutral reference treatment against which every variant is compared.

Metric	Direction	PASS	INVESTIGATE	Statistical Basis
Score Delta ( $\|\Delta\|$ )	lower-is-better	$\leq 0.02$	$\leq 0.05$	Pre-registered tolerance on the cosine-similarity scale; Wilcoxon test + bootstrap CI (Sec. 3.4.1 )
MARC / $K$	lower-is-better	$\leq 0.50$	$\leq 0.70$	Pre-registered rank-churn tolerance, scale-invariant to $K$ ; paired bootstrap CI
Flip Rate	lower-is-better	$\leq 0.05$	$\leq 0.10$	Pre-registered boundary-crossing tolerance; Wilson CI + McNemar’s test ( Wilson, 1927 ; McNemar, 1947 )
Retention Rate	higher-is-better	$\geq 0.80$	$\geq 0.70$	Legal four-fifths rule ( Equal Employment Opportunity Commission and Civil Service Commission and Department of Labor and Department of Justice, 1978 ) applied to shortlist retention
Impact Ratio (four-fifths rule)	higher-is-better	$\geq 0.80$	$\geq 0.70$	Direct transposition of the four-fifths rule ( Equal Employment Opportunity Commission and Civil Service Commission and Department of Labor and Department of Justice, 1978 ; Feldman et al., 2015 ) ; Fisher’s exact test ( Fisher, 1922 )
Recall@ $K$	higher-is-better	$\geq 0.80$	$\geq 0.70$	Four-fifths bound extended by analogy to ranking-quality preservation

Table 3: Status-classification thresholds. For lower-is-better metrics, PASS requires value

\leq

pass and INVESTIGATE requires value

\leq

investigate (else FAIL); for higher-is-better metrics the inequalities are reversed. The Statistical Basis column states, for each metric, whether its threshold is a direct statutory transposition, an analogical extension of that statute, or a pre-registered policy choice under the dual-threshold protocol of Section 3.7 .

Metric(s)	Test	Motivation
Score Delta, MARC	Wilcoxon signed-rank ( Wilcoxon, 1945 )	Paired, non-normal continuous/ordinal statistic (scores, ranks)
Flip Rate	McNemar ( McNemar, 1947 )	Paired binary in/out-of-top- $K$ indicator, $2\times 2$ table
Impact Ratio	Fisher exact ( Fisher, 1922 )	Small-cell-count $2\times 2$ retained/not-retained comparison
Retention Rate, Flip Rate CIs	Wilson score interval ( Wilson, 1927 )	Proportion CI robust at small $n$ / extreme proportions
All effect sizes	Nonparametric bootstrap ( $N=1000$ )	Distribution-free CI without a normality assumption
All $p$ -values	Benjamini–Hochberg FDR ( Benjamini and Hochberg, 1995 )	Controls false-discovery rate across up to $90$ simultaneous tests

Table 4: Statistical test mapped to each metric family, and the property of the metric that motivates it.

Corpus	Jobs	Base identities	Variants	Description
Example corpus	5	100	BASE, V01–V09	Illustrative correspondence-audit matrix

Table 5: The example evaluation corpus analyzed in this section.

Figure 2: Score Delta by bias variant on the example corpus ( $n=500$ paired candidates per variant). Bar color encodes PASS (green); all nine variants pass the $|\Delta|\leq 0.02$ threshold.

Figure 3: Normalized Mean Absolute Rank Change (MARC $/K$ ) by bias variant; this is the statistic classified against the thresholds in Table 3 . V06 (orange) is the only variant landing in the INVESTIGATE band.

Figure 4: Flip Rate (fraction of candidates crossing the top- $10$ boundary in either direction) by bias variant. All nine variants remain under the $5\%$ PASS threshold.

Figure 5: Macro-averaged top- $10$ Retention Rate by bias variant, against a BASE value of $1.0$ by construction. All nine variants clear the $0.80$ PASS line.

Figure 6: Four-fifths Impact Ratio (top- $10$ retention ratio to BASE ) by bias variant. All nine variants clear the $0.80$ PASS line; the dashed threshold lines mark the legal four-fifths PASS boundary ( $0.80$ ) and the $0.70$ FAIL boundary.

Figure 7: Recall@ $15$ by bias variant, against a BASE value of $0.84$ . All nine variants clear the $0.80$ PASS line.

Figure 8: nDCG@ $15$ by bias variant. BASE , V04, and V07 (orange) land in the INVESTIGATE band; the BASE finding reflects baseline ranking quality rather than a bias effect.

Figure 9: Equal Opportunity gap (top- $15$ true-positive-rate gap vs. BASE ) by bias variant. All nine treatments PASS, comfortably under the $0.10$ threshold.

Figure 10: Equalized Odds gap ( $\max$ of TPR and FPR gap vs. BASE ) by bias variant, mirroring the Equal Opportunity pattern because the FPR component is negligible for every variant in this example corpus.

Algorithmic Fairness

Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.

See all 413 papers

Mar 24, 2026

PopResume: Causal Fairness Evaluation of LLM/VLM Resume Screeners with Population-Representative Dataset

We present PopResume, a population-representative resume dataset for causal fairness auditing of LLM- and VLM-based resume screening systems. Unlike existing benchmarks that rely on manually injected demographic information and outcome-level disparities, PopResume is grounded in population statistics and preserves natural attribute relationships, enabling path-specific effect (PSE)-based fairness evaluation. We decompose the effect of a protected attribute on resume scores into two paths: the business necessity path, mediated by job-relevant qualifications, and the redlining path, mediated by demographic proxies. This distinction allows auditors to separate legally permissible from impermissible sources of disparity. Evaluating four LLMs and four VLMs on PopResume's 60.8K resumes across five occupations, we identify five representative discrimination patterns that aggregate metrics fail to capture. Our results demonstrate that PSE-based evaluation reveals fairness issues masked by outcome-level measures, underscoring the need for causally-grounded auditing frameworks in AI-assisted hiring.

Sumin Yu, Juhyeon Park, Taesup Moon

ECE, Seoul National University · IPAI, Seoul National University · ASRI / INMC / AIIS, Seoul National University

AI-Assisted Decision Making LLM Auditing Algorithmic Fairness

Ask PDF

Aug 28, 2026

Fairness Beyond Anonymization? Demographic Leakage in German LLM-Generated Resumes

Large language models (LLMs) are increasingly integrated into AI-assisted hiring pipelines, including automated resume generation and screening. Under the EU AI Act, the hiring domain is classified as high-risk, making fairness and transparency critical requirements. Existing work has primarily focused on explicit hiring decisions, while less attention has been paid to whether generated resumes themselves encode recoverable demographic information. In this work, we conduct a two-stage audit of demographic leakage in German-language LLM-generated resumes. First, we use ChatGPT (GPT-4o-mini), Gemini 2.5 Flash-Lite, and multiple scales of the open-weight Qwen 3 model family (4B, 8B, and 14B) to generate resumes from real anonymized job-matching profiles, systematically varying gender- and ethnicity-associated names while holding qualifications constant. Second, we simulate a downstream resume screening scenario, where the generated resumes are first anonymized and gender-neutralized, before demographic leakage classifiers are trained on the resulting texts. We find that, despite these interventions, classifiers reliably distinguish between resumes generated with male and female names. This leakage is not driven by overtly gendered wording, but by subtle differences in the usage of semantically equivalent, formally gender-neutral terms in German. In contrast, ethnicity-related leakage remains comparatively weak across models. Our findings demonstrate that apparently neutral resume generation can still preserve highly predictive demographic signals, raising concerns about anonymization-based fairness interventions in multilingual AI hiring pipelines.

Charlotte Leininger, Helena Veit, Matthias Aßenmacher +1

Department of Statistics, LMU Munich, Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany

Algorithmic Fairness Demographic Bias in Language Models Privacy Leakage in Language Models Language Model Auditing Text Generation

Ask PDF

Sep 8, 2026

The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits

Whether a language model looks demographically biased can depend on how the audit asks its question. A charitable-aid benchmark reports that the same models favor minority applicants when rating requests one at a time and penalize some when ranking side by side. We test whether that reversal generalizes to hiring, lending, and medical triage: 40,726 requests to five models, applications differing only in the applicant's name, and a primary test fixed before collection. It does not. None of 36 planned contrasts survives correction. The rating advantage keeps its sign at roughly half the published size, and a precision extension bounds any hiring ranking penalty below the published effect, though the lending and triage ranking floors sit above that margin, so the exclusion is conclusive for hiring ranking and for rating in all three domains only. Planted disparities tracking their injected sizes and a directional replication on the original aid materials bound these nulls. The audit is livelier than the demographics: models recognize transparent audits nearly always, tie every identical-content comparison whether the varying detail is race or a hobby, and reward first-listed candidates as much as any demographic effect we measure. Audit verdicts reflect audit construction more than demographic bias.

Siddharth Vohra, Manikandan Ravikiran

Carnegie Mellon University Amazon Web Services AI Native Pittsburgh, PA, USA · Indian Institute of Technology Mandi, India

Language Model Bias Evaluation Demographic Bias in Language Models Position Bias Black-Box Model Auditing Language Model Auditing

Ask PDF

Abstract

Algorithmic Fairness

Explore similar work

PopResume: Causal Fairness Evaluation of LLM/VLM Resume Screeners with Population-Representative Dataset

Fairness Beyond Anonymization? Demographic Leakage in German LLM-Generated Resumes

The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits