cs.CYOct 30, 2026
SaveAI evaluators favor job applications written by certain AI models, independent of candidate quality
Organizations: Universität Zürich, Zurich, Switzerland · ETH Zürich, Zurich, Switzerland
Abstract
Job applicants increasingly use large language models (LLMs) to write cover letters, while employers deploy LLMs to screen them. In this all-AI setting, we ask whether the choice of model affects hiring outcomes independently of candidate qualifications. We introduce a multi-model auditing framework pairing eleven LLM writer agents with nine API-hosted evaluator agents across ten job descriptions and fifty candidates per job. Four findings emerge. First, we identify a generative preference hierarchy: a consistent global ranking of writer models by evaluator acceptance, in which locally hosted models are systematically disadvantaged regardless of their license, compounded by self-preference bias where evaluators favour their own outputs. Second, evaluator models differ substantially in how much bias they introduce based on applicants' tool choice, offering actionable guidance for fairer screening design. Third, this hierarchy has measurable competitive consequences: in the most adverse pairing, approximately two (1.8) out of twenty-five qualified candidates are filtered out purely due to model mismatch. Fourth, a common mechanism explains the patterns above: writer preference and self-preference trace to a combination of how much each writer leans on specific stylistic cues and how strongly evaluators, often unanimously, reward or penalise those same cues. A further check finds little evidence that an unmeasured "writer fingerprint" remains once these same features are accounted for. We discuss implications for algorithmic fairness in employment, regulation of AI-assisted screening, and the design of auditing frameworks that can be reapplied as the model landscape evolves.
Figures & tables
Figure 1: Overview of the three-phase auditing pipeline. In the generation phase, eleven writer models (nine API-hosted, two locally hosted, dashed border) produce 5,500 cover letters across ten job descriptions and fifty candidates per job. In the evaluation phase, the nine API-hosted models also serve as evaluators, scoring candidates under three conditions (CV only, cover letter only, CV + cover letter). CV-only scoring does not depend on writer identity, so it is run once per candidate (500 candidates 9 evaluators 4 runs ); the other two conditions are run once per writer instead (5,500 9 4 each), for evaluations in total. The analysis phase cross-references writer identity, evaluator scores, and text features to produce the four main findings previewed on the right.
Figure 2: Relative Preference Gap : mean residual averaged over all (job, candidate) pairs, for the cover-letter-only (left) and combined CV + Cover letter (right) conditions. Columns sorted by decreasing gap; bold values indicate (one-sample -test of the 50 per-candidate gaps against 0, per cell); true for – of cells, so bold is mostly a significance flag: it marks gaps distinguishable from zero, regardless of magnitude. Bold black-bordered cells (a separate marker) mark the self-preference diagonal. Locally hosted models are consistently disfavoured, regardless of licensing; self-preference is strongest in the combined setting.
Figure 3: Rank gap as a heatmap (evaluators writers) for the cover-letter-only (left) and combined CV + Cover letter (right) conditions; red indicates a writer is over-ranked by that evaluator, blue indicates under-ranked. Rows are ordered by increasing cumulative (least to most biased), shown as the adjacent bar chart; columns follow the same order for every writer that is also an evaluator, so the bold-bordered self-preference diagonal (as in Figure 2 ) renders as an actual diagonal, with the two writer-only, locally hosted models appended at the end. Gemini 3.5 Flash is the most biased evaluator in the cover-letter-only condition but the most neutral once the CV is also available; DeepSeek V3.2 becomes the most biased evaluator in the combined condition. Locally hosted models are persistently under-ranked across all evaluators.
Figure 4: Both panels refer to the combined CV + Cover letter condition. Left: head-to-head win rates ; triples where the evaluator matches either writer are excluded to remove self-evaluation confounds. Gemini 3.5 Flash leads (56%), closely followed by GPT-5-mini and DeepSeek V3.2 (56% and 54%); locally hosted models consistently lose. Right: net displacement rate (percentage points above the same-writer baseline) for the scenario where all high-fit candidates (defined as the top 25 by each evaluator’s own CV-only ranking, recomputed per evaluator, job, and run) use writer (rows), and all moderate-fit candidates (bottom 25 by the same ranking) use writer (columns). Diagonal cells (grey “–”) are the control condition, , by definition 0 and omitted. Off-diagonal cells show the mean net rate and, beneath it, its standard deviation across the nine evaluators. In the most adverse pairing (Llama-3.1-8B vs. DeepSeek V3.2), approximately two (1.8) out of twenty-five high-fit candidates are displaced solely due to model mismatch.
Figure 5: Left: cross-evaluator Ridge coefficient for each feature (cover-letter-only condition; Cosine Similarity to CV excluded, see text), one dot per evaluator plus a bold bar for the cross-evaluator mean, ranked top to bottom by that mean; rows shaded when all nine evaluators agree in sign (green = unanimous reward, red = unanimous penalty). Right: each writer’s own mean use of that feature (mean 1 std), standardised to the same units. Row labels are coloured by feature category. As throughout, Cosine Similarity to Job Ad/CV measures the generated letter’s embedding against the job ad/CV. It is not the job ad and CV measured against each other (Figure A1 ). Cosine Similarity to Job Ad is both the most strongly and the most unanimously rewarded feature by a wide margin; Paragraph Count, Anger, and Neutral are the clearest unanimous penalties.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Cover-letter text features by writer model and candidate tier, grouped by feature class (one row per class). Emotion is reduced to Joy and Neutral, the two dominant categories; the remaining five (surprise, fear, anger, disgust, sadness) are low, largely indistinguishable across models, and, thus, omitted here. Error bars show standard deviation across candidates. The three Semantic Fit columns verify the alignment of the generated cover letter’s embedding with the ones of the job ad and CV: “Cosine Similarity to Job Ad” is letter-vs-job-ad similarity, “Cosine Similarity to CV” is letter-vs-CV similarity, and “Job CV Similarity” is their difference, i.e. whether a given letter leans more toward echoing the job posting’s language or the candidate’s own CV.
Figure A2: Mean pairwise cosine similarity between cover-letter embeddings for the same writer, comparing pairs drawn from the same job vs. different jobs. Within-job similarity exceeds cross-job similarity for every writer, confirming that writers condition their output on the specific job description. This validates the experimental design (mentioned at the start of the Results section, Section 2 ).
Figure A3: Cohen’s (High-Fit Moderate-Fit) per writing-style feature and writer model, averaged across jobs. Flesch Reading Ease is the one systematically positive effect across almost every writer; other features show smaller and less consistent effects (summarised at the start of the Results section, Section 2 ).
| Claude Sonnet 4.6 | Claude Haiku 4.5 | GPT-5 | Gemini 3.5 Flash | GPT-4o mini | Gemini 3 Flash | GPT-5 mini | DeepSeek V4 Flash | DeepSeek V3.2 | Mean | |
|---|---|---|---|---|---|---|---|---|---|---|
| CV | 0.04 | 0.13 | 0.21 | 0.23 | 0.29 | 0.23 | 0.36 | 0.37 | 0.40 | 0.25 |
| CL | 0.07 | 0.10 | 0.30 | 0.34 | 0.30 | 0.38 | 0.39 | 0.55 | 0.62 | 0.34 |
| CV+CL | 0.06 | 0.11 | 0.23 | 0.28 | 0.26 | 0.30 | 0.40 | 0.49 | 0.48 | 0.29 |
| Avg | 0.05 | 0.11 | 0.25 | 0.28 | 0.29 | 0.30 | 0.38 | 0.47 | 0.50 | 0.29 |
Table A1: Mean intra-run by evaluator and condition. Evaluators are ordered left-to-right by increasing mean (most to least reproducible), using the same colour-coding as elsewhere in the paper. Scores on a 1–10 scale, rounded to two decimals. CV = CV only; CL = cover letter only; CV+CL = CV + cover letter.
Figure A4: (a) Mean evaluator scores ( standard deviation) for high-fit and moderate-fit candidates across the three evaluation conditions. Left (CV only): all evaluators discriminate in the expected direction (overall gap ), confirming that the stratification is universally recognized. Centre (cover letter only): scores inflate markedly and the gap narrows ( ), as AI-generated prose presents all candidates favourably regardless of merit. Right (CV + cover letter): combining documents partially restores discrimination ( ). (b) Candidate quality agreement: Spearman between evaluator score vectors across all conditions. Left: CV-only ( ). Centre: cover-letter-only ( ). Right: combined CV + Cover letter ( ). Despite strictness differences, evaluators agree strongly on candidate quality across all conditions.
| Evaluator | Overlap (of 25) | Agreement |
|---|---|---|
| Claude Sonnet 4.6 | 23.4 | 93.6% |
| Claude Haiku 4.5 | 23.2 | 92.8% |
| GPT-5 | 22.5 | 90.0% |
| Gemini 3 Flash | 22.5 | 90.0% |
| GPT-4o mini | 22.5 | 90.0% |
| Gemini 3.5 Flash | 22.4 | 89.6% |
Table A2: Agreement between the fixed cosine-similarity high-fit split and the high-fit split each evaluator’s own independent CV-only score would draw instead. Overlap is the mean number (out of 25) of cosine-similarity-designated high-fit candidates that also fall in that evaluator’s own top 25 by CV-only score, averaged across the ten jobs. Evaluators are ordered by decreasing agreement.
Figure A5: Writer preference agreement: Spearman on within-(job, candidate) writer ranks . By ranking within each evaluation context, this removes absolute strictness, candidate quality, and job difficulty simultaneously. Left: cover-letter-only condition (range 0.52–0.83, mean ). Right: combined CV + Cover letter condition (range 0.18–0.39, mean ).
Figure B1: Net competitive advantage (as in Figure 4 , right panel) for two individual evaluators rather than pooled across all nine: GPT-4o-mini (left), which has the single most extreme cell ( ) of any evaluator, and DeepSeek V3.2 (right), which has the highest total displacement summed across its whole matrix. Grey “—” marks the same-writer control diagonal.
Figure C1: CV + Cover Letter condition. Left: each writer’s average letter-to-job-ad vs. letter-to-CV cosine similarity, centred on the across-writer mean; GPT-4o-mini is the clearest job-ad-leaning outlier, GPT-5 and GPT-5-mini the clearest CV-leaning ones. Right: each evaluator’s Ridge coefficient rewarding job-ad fit vs. CV fit. Points cluster tightly around the diagonal (fitted slope ), visually confirming that the two forms of semantic fit are rewarded almost identically once the CV is actually visible.
Figure C2: As writers, readability and average word length trade off sharply: no writer combines long, complex words with easy-to-read structure. As evaluators, that constraint disappears: every model rewards both higher readability and longer words simultaneously, a combination essentially no writer among the eleven actually produces.
Figure C3: As writers, dominance and arousal move together: a more assertive-sounding letter also tends to read as more energetic. As evaluators, the relationship partially reverses: every model rewards dominance, but the ones that reward it most tend to reward arousal least, favouring "confident but composed" language over "assertive and energetic."
Figure C4: Both panels show a near-perfect trade-off between Joy and Neutral tone. The evaluator panel is not mechanically constrained to show this (Ridge coefficients on the two are fit independently), yet the mirror-image relationship is, if anything, steeper there than in how writers actually write: a genuine preference structure.
Figure C5: Cover-letter-only condition; only Cosine Similarity to Job Ad shown, since Cosine Similarity to CV is excluded (Section 2.5.1 ). GPT-4o-mini’s own dot (opaque, top row) sits at the low end of its own row’s spread: every other evaluator rewards its job-ad echo more than GPT-4o-mini itself does.
Figure C6: CV + Cover Letter condition; both semantic-fit features shown. Here GPT-4o-mini’s own dot on Cosine Similarity to Job Ad sits at the high end of its row instead: the self-preference documented in Section 2.5.1 strengthens once the CV is present.
Figure C7: Cover-letter-only condition. Confirms Section 2.5.1 ’s reading at the individual-evaluator level: Claude Haiku 4.5’s long average word length and Llama-3.1-8B’s high readability are each rewarded consistently across (almost) every evaluator.
Figure C8: Self-evaluation diagonal only (cover-letter-only condition): for each feature, one triangle per dual-role model at its own contribution to its own predicted preference (own standardised style own Ridge coefficient), ranked top to bottom by the largest across models. Triangle direction shows whether that model’s own mean sits above ( ) or below ( ) the feature’s overall mean; size shows the standardised effect size of that deviation. Row labels are coloured by feature category as in Figure 5 .
Figure C9: Same analysis as Figure 5 , for the CV + Cover Letter condition. Cosine Similarity to Job Ad and Cosine Sim. to CV converge to almost identical, dominant reward levels; VAD Dominance loses unanimous-reward status and VADER Sentiment gains unanimous-penalty status relative to the cover-letter-only condition.
Figure C10: Same analysis as Figure C8 , for the CV + Cover Letter condition. GPT-4o-mini’s self-preference on Cosine Similarity to Job Ad grows further despite the feature’s shrinking overall reward; Claude Haiku 4.5’s self-preference on Average Word Length emerges as the largest style-feature self-contribution in this condition.
Figure C11: Cover-letter-only (circle) and CV + Cover Letter (square) conditions, Cosine Similarity to CV excluded from the cover-letter-only fit (see text). One point per condition per dual-role model (the nine that are both writer and evaluator; Net Self-Preference is undefined for the two locally-hosted writers, which are never evaluators and so do not appear here), connected by an arrow from cover-letter-only to CV + Cover Letter: -axis, Net Self-Preference, the signed sum of the model’s own idiosyncratic bias across all measured features; -axis, Total Preference Received, that writer’s own style dotted with the summed Ridge coefficients of all nine evaluators. In the cover-letter-only condition, GPT-4o-mini and GPT-5-mini sit at opposite extremes: GPT-4o-mini is the harshest critic of its own writing yet the most liked by the panel overall, while GPT-5-mini is the mirror image. Both collapse sharply toward the origin once the CV is present: GPT-4o-mini’s Net Self-Preference turns slightly positive ( ) while its Total Preference Received falls to near zero ( , down from ), and DeepSeek V4 Flash becomes the writer most liked by the panel overall ( ).
Figure C12: Nested variance decomposition, pooled across all nine evaluators, for CV Only, CV + Cover Letter, and Cover-Letter-Only. Job Substance Evaluator is a shared trunk; the dashed red line marks where branch A (+ Writer) and branch B (+ Text Features) split off from “+ Evaluator” as two alternative, parallel explanations of the same remaining variance. Cover-Letter-Only’s Text Features exclude letter-to-CV similarity, since the evaluator never sees the CV in that condition. CV Only has no letter, so branches A/B do not apply there (hatched rows). All panels share one scale and row order.
Figure C13: Exact Shapley split of on top of a job-only baseline, per evaluator, using only two blocks, Substance (that evaluator’s own CV-only score) and Writer identity (categorical), and ranked by each evaluator’s absolute Writer contribution. A large Writer share relative to Substance is a direct, regression-based measure of how little of an evaluator’s score tracks genuine candidate quality once its own substance judgment is already accounted for.
Figure C14: Exact Shapley split of on top of a job-only baseline, per evaluator, ranked by each evaluator’s own Substance contribution. Substance is that evaluator’s own CV-only score for the exact candidate being judged: a self-referential quantity specific to that evaluator and that candidate; since Writer identity is not a baseline control, between-writer variation is absorbed into the Semantic/style blocks.
Figure C15: of individual style features on top of the same Job + Substance + Job-Ad Fit + CV Consistency baseline as Figure C14 ; category-level totals here are identical to that figure’s style-category bars, this view simply decomposes them feature by feature.
More in this topic
Algorithmic Fairness
Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.
See all 440 papers