Job applicants increasingly use large language models (LLMs) to write cover letters, while employers deploy LLMs to screen them. In this all-AI setting, we ask whether the choice of model affects hiring outcomes independently of candidate qualifications. We introduce a multi-model auditing framework pairing eleven LLM writer agents with nine API-hosted evaluator agents across ten job descriptions and fifty candidates per job. Four findings emerge. First, we identify a generative preference hierarchy: a consistent global ranking of writer models by evaluator acceptance, in which locally hosted models are systematically disadvantaged regardless of their license, compounded by self-preference bias where evaluators favour their own outputs. Second, evaluator models differ substantially in how much bias they introduce based on applicants' tool choice, offering actionable guidance for fairer screening design. Third, this hierarchy has measurable competitive consequences: in the most adverse pairing, approximately two (1.8) out of twenty-five qualified candidates are filtered out purely due to model mismatch. Fourth, a common mechanism explains the patterns above: writer preference and self-preference trace to a combination of how much each writer leans on specific stylistic cues and how strongly evaluators, often unanimously, reward or penalise those same cues. A further check finds little evidence that an unmeasured "writer fingerprint" remains once these same features are accounted for. We discuss implications for algorithmic fairness in employment, regulation of AI-assisted screening, and the design of auditing frameworks that can be reapplied as the model landscape evolves.
Figures & tables
Figure 1: Overview of the three-phase auditing pipeline. In the generation phase, eleven writer models (nine API-hosted, two locally hosted, dashed border) produce 5,500 cover letters across ten job descriptions and fifty candidates per job. In the evaluation phase, the nine API-hosted models also serve as evaluators, scoring candidates under three conditions (CV only, cover letter only, CV + cover letter). CV-only scoring does not depend on writer identity, so it is run once per candidate (500 candidates × 9 evaluators × 4 runs =18,000 ); the other two conditions are run once per writer instead (5,500 × 9 × 4 =198,000 each), for 18,000+198,000+198,000=414,000 evaluations in total. The analysis phase cross-references writer identity, evaluator scores, and text features to produce the four main findings previewed on the right.
Figure 2: Relative Preference Gap Δpref(e,w) : mean residual s~e,j,c,w averaged over all (job, candidate) pairs, for the cover-letter-only (left) and combined CV + Cover letter (right) conditions. Columns sorted by decreasing gap; bold values indicate p<0.05 (one-sample t -test of the 50 per-candidate gaps against 0, per cell); true for 80 – 84% of cells, so bold is mostly a significance flag: it marks gaps distinguishable from zero, regardless of magnitude. Bold black-bordered cells (a separate marker) mark the self-preference diagonal. Locally hosted models are consistently disfavoured, regardless of licensing; self-preference is strongest in the combined setting.
Figure 3: Rank gap gˉe,w as a heatmap (evaluators × writers) for the cover-letter-only (left) and combined CV + Cover letter (right) conditions; red indicates a writer is over-ranked by that evaluator, blue indicates under-ranked. Rows are ordered by increasing cumulative ∑w∣gˉe,w∣ (least to most biased), shown as the adjacent bar chart; columns follow the same order for every writer that is also an evaluator, so the bold-bordered self-preference diagonal (as in Figure 2 ) renders as an actual diagonal, with the two writer-only, locally hosted models appended at the end. Gemini 3.5 Flash is the most biased evaluator in the cover-letter-only condition but the most neutral once the CV is also available; DeepSeek V3.2 becomes the most biased evaluator in the combined condition. Locally hosted models are persistently under-ranked across all evaluators.
Figure 4: Both panels refer to the combined CV + Cover letter condition. Left: head-to-head win rates WR(w1,w2) ; triples where the evaluator matches either writer are excluded to remove self-evaluation confounds. Gemini 3.5 Flash leads (56%), closely followed by GPT-5-mini and DeepSeek V3.2 (56% and 54%); locally hosted models consistently lose. Right: net displacement rate ΔR(w2,w1) (percentage points above the same-writer baseline) for the scenario where all high-fit candidates (defined as the top 25 by each evaluator’s own CV-only ranking, recomputed per evaluator, job, and run) use writer w1 (rows), and all moderate-fit candidates (bottom 25 by the same ranking) use writer w2 (columns). Diagonal cells (grey “–”) are the control condition, w1=w2 , by definition 0 and omitted. Off-diagonal cells show the mean net rate and, beneath it, ± its standard deviation across the nine evaluators. In the most adverse pairing (Llama-3.1-8B vs. DeepSeek V3.2), approximately two (1.8) out of twenty-five high-fit candidates are displaced solely due to model mismatch.
Figure 5: Left: cross-evaluator Ridge coefficient for each feature (cover-letter-only condition; Cosine Similarity to CV excluded, see text), one dot per evaluator plus a bold bar for the cross-evaluator mean, ranked top to bottom by that mean; rows shaded when all nine evaluators agree in sign (green = unanimous reward, red = unanimous penalty). Right: each writer’s own mean use of that feature (mean ± 1 std), standardised to the same units. Row labels are coloured by feature category. As throughout, Cosine Similarity to Job Ad/CV measures the generated letter’s embedding against the job ad/CV. It is not the job ad and CV measured against each other (Figure A1 ). Cosine Similarity to Job Ad is both the most strongly and the most unanimously rewarded feature by a wide margin; Paragraph Count, Anger, and Neutral are the clearest unanimous penalties.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Cover-letter text features by writer model and candidate tier, grouped by feature class (one row per class). Emotion is reduced to Joy and Neutral, the two dominant categories; the remaining five (surprise, fear, anger, disgust, sadness) are low, largely indistinguishable across models, and, thus, omitted here. Error bars show ±1 standard deviation across candidates. The three Semantic Fit columns verify the alignment of the generated cover letter’s embedding with the ones of the job ad and CV: “Cosine Similarity to Job Ad” is letter-vs-job-ad similarity, “Cosine Similarity to CV” is letter-vs-CV similarity, and “Job − CV Similarity” is their difference, i.e. whether a given letter leans more toward echoing the job posting’s language or the candidate’s own CV.
Figure A2: Mean pairwise cosine similarity between cover-letter embeddings for the same writer, comparing pairs drawn from the same job vs. different jobs. Within-job similarity exceeds cross-job similarity for every writer, confirming that writers condition their output on the specific job description. This validates the experimental design (mentioned at the start of the Results section, Section 2 ).
Figure A3: Cohen’s d (High-Fit − Moderate-Fit) per writing-style feature and writer model, averaged across jobs. Flesch Reading Ease is the one systematically positive effect across almost every writer; other features show smaller and less consistent effects (summarised at the start of the Results section, Section 2 ).
Claude Sonnet 4.6
Claude Haiku 4.5
GPT-5
Gemini 3.5 Flash
GPT-4o mini
Gemini 3 Flash
GPT-5 mini
DeepSeek V4 Flash
DeepSeek V3.2
Mean
CV
0.04
0.13
0.21
0.23
0.29
0.23
0.36
0.37
0.40
0.25
CL
0.07
0.10
0.30
0.34
0.30
0.38
0.39
0.55
0.62
0.34
CV+CL
0.06
0.11
0.23
0.28
0.26
0.30
0.40
0.49
0.48
0.29
Avg
0.05
0.11
0.25
0.28
0.29
0.30
0.38
0.47
0.50
0.29
Appendix
Table A1: Mean intra-run σ by evaluator and condition. Evaluators are ordered left-to-right by increasing mean σ (most to least reproducible), using the same colour-coding as elsewhere in the paper. Scores on a 1–10 scale, rounded to two decimals. CV = CV only; CL = cover letter only; CV+CL = CV + cover letter.
Figure A4: (a) Mean evaluator scores ( ±1 standard deviation) for high-fit and moderate-fit candidates across the three evaluation conditions. Left (CV only): all evaluators discriminate in the expected direction (overall gap Δ=2.25 ), confirming that the stratification is universally recognized. Centre (cover letter only): scores inflate markedly and the gap narrows ( Δ=1.89 ), as AI-generated prose presents all candidates favourably regardless of merit. Right (CV + cover letter): combining documents partially restores discrimination ( Δ=2.12 ). (b) Candidate quality agreement: Spearman ρ between evaluator score vectors across all conditions. Left: CV-only ( ρˉ=0.92 ). Centre: cover-letter-only ( ρˉ=0.88 ). Right: combined CV + Cover letter ( ρˉ=0.91 ). Despite strictness differences, evaluators agree strongly on candidate quality across all conditions.
Evaluator
Overlap (of 25)
Agreement
Claude Sonnet 4.6
23.4
93.6%
Claude Haiku 4.5
23.2
92.8%
GPT-5
22.5
90.0%
Gemini 3 Flash
22.5
90.0%
GPT-4o mini
22.5
90.0%
Gemini 3.5 Flash
22.4
89.6%
Appendix
Table A2: Agreement between the fixed cosine-similarity high-fit split and the high-fit split each evaluator’s own independent CV-only score would draw instead. Overlap is the mean number (out of 25) of cosine-similarity-designated high-fit candidates that also fall in that evaluator’s own top 25 by CV-only score, averaged across the ten jobs. Evaluators are ordered by decreasing agreement.
Figure A5: Writer preference agreement: Spearman ρ on within-(job, candidate) writer ranks re,j,c,w=rankw(se,j,c,w) . By ranking within each evaluation context, this removes absolute strictness, candidate quality, and job difficulty simultaneously. Left: cover-letter-only condition (range 0.52–0.83, mean ρˉ=0.63 ). Right: combined CV + Cover letter condition (range 0.18–0.39, mean ρˉ=0.27 ).
Figure B1: Net competitive advantage (as in Figure 4 , right panel) for two individual evaluators rather than pooled across all nine: GPT-4o-mini (left), which has the single most extreme cell ( +12.8% ) of any evaluator, and DeepSeek V3.2 (right), which has the highest total displacement summed across its whole matrix. Grey “—” marks the same-writer control diagonal.
Figure C1: CV + Cover Letter condition. Left: each writer’s average letter-to-job-ad vs. letter-to-CV cosine similarity, centred on the across-writer mean; GPT-4o-mini is the clearest job-ad-leaning outlier, GPT-5 and GPT-5-mini the clearest CV-leaning ones. Right: each evaluator’s Ridge coefficient rewarding job-ad fit vs. CV fit. Points cluster tightly around the y=x diagonal (fitted slope 0.49 ), visually confirming that the two forms of semantic fit are rewarded almost identically once the CV is actually visible.
Figure C2: As writers, readability and average word length trade off sharply: no writer combines long, complex words with easy-to-read structure. As evaluators, that constraint disappears: every model rewards both higher readability and longer words simultaneously, a combination essentially no writer among the eleven actually produces.
Figure C3: As writers, dominance and arousal move together: a more assertive-sounding letter also tends to read as more energetic. As evaluators, the relationship partially reverses: every model rewards dominance, but the ones that reward it most tend to reward arousal least, favouring "confident but composed" language over "assertive and energetic."
Figure C4: Both panels show a near-perfect trade-off between Joy and Neutral tone. The evaluator panel is not mechanically constrained to show this (Ridge coefficients on the two are fit independently), yet the mirror-image relationship is, if anything, steeper there than in how writers actually write: a genuine preference structure.
Figure C5: Cover-letter-only condition; only Cosine Similarity to Job Ad shown, since Cosine Similarity to CV is excluded (Section 2.5.1 ). GPT-4o-mini’s own dot (opaque, top row) sits at the low end of its own row’s spread: every other evaluator rewards its job-ad echo more than GPT-4o-mini itself does.
Figure C6: CV + Cover Letter condition; both semantic-fit features shown. Here GPT-4o-mini’s own dot on Cosine Similarity to Job Ad sits at the high end of its row instead: the self-preference documented in Section 2.5.1 strengthens once the CV is present.
Figure C7: Cover-letter-only condition. Confirms Section 2.5.1 ’s reading at the individual-evaluator level: Claude Haiku 4.5’s long average word length and Llama-3.1-8B’s high readability are each rewarded consistently across (almost) every evaluator.
Figure C8: Self-evaluation diagonal only (cover-letter-only condition): for each feature, one triangle per dual-role model at its own contribution to its own predicted preference (own standardised style × own Ridge coefficient), ranked top to bottom by the largest ∣value∣ across models. Triangle direction shows whether that model’s own mean sits above ( ▲ ) or below ( ▼ ) the feature’s overall mean; size shows the standardised effect size of that deviation. Row labels are coloured by feature category as in Figure 5 .
Figure C9: Same analysis as Figure 5 , for the CV + Cover Letter condition. Cosine Similarity to Job Ad and Cosine Sim. to CV converge to almost identical, dominant reward levels; VAD Dominance loses unanimous-reward status and VADER Sentiment gains unanimous-penalty status relative to the cover-letter-only condition.
Figure C10: Same analysis as Figure C8 , for the CV + Cover Letter condition. GPT-4o-mini’s self-preference on Cosine Similarity to Job Ad grows further despite the feature’s shrinking overall reward; Claude Haiku 4.5’s self-preference on Average Word Length emerges as the largest style-feature self-contribution in this condition.
Figure C11: Cover-letter-only (circle) and CV + Cover Letter (square) conditions, Cosine Similarity to CV excluded from the cover-letter-only fit (see text). One point per condition per dual-role model (the nine that are both writer and evaluator; Net Self-Preference is undefined for the two locally-hosted writers, which are never evaluators and so do not appear here), connected by an arrow from cover-letter-only to CV + Cover Letter: x -axis, Net Self-Preference, the signed sum of the model’s own idiosyncratic bias across all measured features; y -axis, Total Preference Received, that writer’s own style dotted with the summed Ridge coefficients of all nine evaluators. In the cover-letter-only condition, GPT-4o-mini and GPT-5-mini sit at opposite extremes: GPT-4o-mini is the harshest critic of its own writing yet the most liked by the panel overall, while GPT-5-mini is the mirror image. Both collapse sharply toward the origin once the CV is present: GPT-4o-mini’s Net Self-Preference turns slightly positive ( +0.036 ) while its Total Preference Received falls to near zero ( +0.05 , down from +2.98 ), and DeepSeek V4 Flash becomes the writer most liked by the panel overall ( +2.44 ).
Figure C12: Nested variance decomposition, pooled across all nine evaluators, for CV Only, CV + Cover Letter, and Cover-Letter-Only. Job → Substance → Evaluator is a shared trunk; the dashed red line marks where branch A (+ Writer) and branch B (+ Text Features) split off from “+ Evaluator” as two alternative, parallel explanations of the same remaining variance. Cover-Letter-Only’s Text Features exclude letter-to-CV similarity, since the evaluator never sees the CV in that condition. CV Only has no letter, so branches A/B do not apply there (hatched rows). All panels share one R2 scale and row order.
Figure C13: Exact Shapley split of R2 on top of a job-only baseline, per evaluator, using only two blocks, Substance (that evaluator’s own CV-only score) and Writer identity (categorical), and ranked by each evaluator’s absolute Writer contribution. A large Writer share relative to Substance is a direct, regression-based measure of how little of an evaluator’s score tracks genuine candidate quality once its own substance judgment is already accounted for.
Figure C14: Exact Shapley split of R2 on top of a job-only baseline, per evaluator, ranked by each evaluator’s own Substance contribution. Substance is that evaluator’s own CV-only score for the exact candidate being judged: a self-referential quantity specific to that evaluator and that candidate; since Writer identity is not a baseline control, between-writer variation is absorbed into the Semantic/style blocks.
Figure C15: ΔR2 of individual style features on top of the same Job + Substance + Job-Ad Fit + CV Consistency baseline as Figure C14 ; category-level totals here are identical to that figure’s style-category bars, this view simply decomposes them feature by feature.
Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.
Whether a language model looks demographically biased can depend on how the audit asks its question. A charitable-aid benchmark reports that the same models favor minority applicants when rating requests one at a time and penalize some when ranking side by side. We test whether that reversal generalizes to hiring, lending, and medical triage: 40,726 requests to five models, applications differing only in the applicant's name, and a primary test fixed before collection. It does not. None of 36 planned contrasts survives correction. The rating advantage keeps its sign at roughly half the published size, and a precision extension bounds any hiring ranking penalty below the published effect, though the lending and triage ranking floors sit above that margin, so the exclusion is conclusive for hiring ranking and for rating in all three domains only. Planted disparities tracking their injected sizes and a directional replication on the original aid materials bound these nulls. The audit is livelier than the demographics: models recognize transparent audits nearly always, tie every identical-content comparison whether the varying detail is race or a hobby, and reward first-listed candidates as much as any demographic effect we measure. Audit verdicts reflect audit construction more than demographic bias.
Siddharth Vohra, Manikandan Ravikiran
Carnegie Mellon University Amazon Web Services AI Native Pittsburgh, PA, USA · Indian Institute of Technology Mandi, India
Industries are increasingly adopting AI-driven systems to automate a wide range of tasks, including high-stakes decisions in human resources, such as candidate screening and recruitment. This makes evaluating their fairness essential, particularly when decisions directly affect individuals' opportunities. Accordingly, the EU AI Act classifies recruitment AI systems as high-risk, imposing strict requirements on their deployment. In this context, compact ``System~1'' decision models offer a fast and cost-efficient alternative to general-purpose LLMs, yet their fairness remains largely unexplored. This article presents the first systematic bias and fairness audit of \jev{}, a typed-decision model targeting recruitment applications. The study evaluates how explicit demographic attributes and implicit candidate cues influence screening scores, rankings, and shortlisting outcomes. These effects are compared against a noise floor derived from name substitutions and typographical variations. The evaluation also examines output validity, latency, and cost. Counterfactual CV datasets and evaluation code are released for reproducibility.
Muhammad Uzair Ul Haq, Davide Rigoni, Alessandro Sperduti +1
Amajor SB S.p.A, Italy · Department of Mathematics “Tullio Levi-Civita”, University of Padova, Italy · Human Inspired Technology Research Centre, University of Padova, Italy +2
We audit fourteen mainstream large language models (LLMs) for hiring discrimination using the paired-resume methodology of Kline, Rose, and Walters (2022). The sole 2023-vintage model reproduces the pro-White callback gap documented in field experiments on labor market discrimination (+2.12 pp, significant at the 1% level). Every model released in 2024 or after shows either a null gap or a significant pro-Black reversal (up to −3.01 pp). The same pattern holds on the gender axis. Based on 24,024 paired postings per model across 14 models, our results document a reversal in the direction of algorithmic hiring bias across model generations.
Zhenyu Gao, Wenxi Jiang, Yutong Yan
Department of Finance, CUHK Business School, The Chinese University of Hong Kong