Corporate recruiters often need to screen many resumes within a limited time, which increases their burden and may cause suitable candidates to be overlooked. To address these challenges, prior work has explored LLM-based automated resume screening. However, some methods rely on commercial LLMs, which may pose data privacy risks. Moreover, since companies typically do not make resumes with evaluation results publicly available, it remains unclear which resume samples should be used during learning to improve an LLM's judgment performance. To address these problems, we propose AutoScreen-FW, an LLM-based locally and automatically resume screening framework. AutoScreen-FW uses several methods to select a small set of representative resume samples. These samples are used for in-context learning together with a persona description and evaluation criteria, enabling open-source LLMs to act as a career advisor and evaluate unseen resumes. Experiments with multiple ground truths show that the open-source LLM judges consistently outperform GPT-5-nano. Under one ground truth setting, it also surpass GPT-5-mini. Although it is slightly weaker than GPT-5-mini under other ground-truth settings, it runs substantially faster per resume than commercial GPT models. These findings indicate the potential for deploying AutoScreen-FW locally in companies to support efficient screening while reducing recruiters' burden.
Figures & tables
Fig. 1 : The comparison between (a) Human-only Screening Pipeline and (b) LLM-based Screening Pipeline.
Fig. 2 : Overview of the AutoScreen-FW.
Fig. 3 : Persona used in the prompt.
Dimension
Evaluation Perspective
Evaluation Criteria
Content
Validity of Evidence
Appropriate episodes are used to effectively convey the applicant’s strengths.
Specificity of Content
• The applicant’s motivation for the target company and their potential contributions are described concretely. • Episodes are described in sufficient detail. • Achievements, research outcomes, and related accomplishments are presented concretely using numerical values.
Structure
Explicit Conclusion
For each question, the conclusion is clearly stated at the beginning so that recruiters can understand it easily.
Focus on a Single Selling Point
For questions intended to highlight the applicant’s strengths, such as self-promotion, the response focuses on a single selling point.
Conciseness of Writing
The overall length is appropriate, and the writing is clear and concise.
Language
Grammatical Accuracy
The response contains no grammatical errors.
TABLE I: Resume evaluation items
Fig. 4 : Evaluation instruction used in the prompt.
Model
Few-shot
Time per Resume (s)
GPT-5-mini
5.09±0.22
GPT-5-nano
7.48±0.45
Qwen3-8B
3.73±0.69
Llama-3.1-8B
0.89±0.15
Qwen3-8B
✓
3.84±0.26
Llama-3.1-8B
✓
2.10±0.24
TABLE II : Per-resume judge time for each LLM model. Results are reported as mean ± standard deviation.
Model
Few-shot
Sampling Strategy
Shots
Sample Type
Attribute Type
Acc.
GPT-5-mini
N/A
0
N/A
N/A
0.6917
GPT-5-nano
N/A
0
N/A
N/A
0.6091
Qwen3-8B
N/A
0
N/A
N/A
0.6063
Llama-3.1-8B
N/A
0
N/A
N/A
0.5721
Qwen3-8B
✓
Similarity-based
3
High-quality only
Overall Judgement
0.6746
Llama-3.1-8B
✓
Diversity-based
20
High- and Low- quality
Overall Judgement and Dimension Scores
0.6101
TABLE III : Accuracy (Acc.) of each model with GPT-5.2 as the ground truth.
Model
Few-shot
Sampling Strategy
Shots
Sample Type
Attribute Type
Acc.
GPT-5-mini
N/A
0
N/A
N/A
0.8445
GPT-5-nano
N/A
0
N/A
N/A
0.8374
Qwen3-8B
N/A
0
N/A
N/A
0.8552
Llama-3.1-8B
N/A
0
N/A
N/A
0.7927
Qwen3-8B
✓
Diversity-based
5
High-quality only
Overall Judgement and Dimension Scores
0.8682
Llama-3.1-8B
✓
Clustering-based
15
High- and Low- quality
Overall Judgement and Dimension Scores
0.8485
TABLE IV : Accuracy (Acc.) of each model with GPT-5.1 as the ground truth.
Model
Few-shot
Sampling Strategy
Shots
Sample Type
Attribute Type
Acc.
GPT-5-mini
N/A
0
N/A
N/A
0.7387
GPT-5-nano
N/A
0
N/A
N/A
0.6617
Qwen3-8B
N/A
0
N/A
N/A
0.6744
Llama-3.1-8B
N/A
0
N/A
N/A
0.6280
Qwen3-8B
✓
Diversity-based
3
High- and Low- quality
Overall Judgement and Dimension Scores
0.7218
Llama-3.1-8B
✓
Clustering-based
20
High-quality only
Overall Judgement and Dimension Scores
0.6333
TABLE V : Accuracy (Acc.) of each model with GPT-o3 as the ground truth.
Fig. 5 : Score distributions and mean scores assigned by GPT-5.2, GPT-5.1, and GPT-o3 across three evaluation dimensions. Each dot denotes an individual resume, and the horizontal line indicates the mean score for each LLM model.
Fig. 6 : Score distributions and mean scores for GPT-5.2, Qwen3-8B (few-shot), and Qwen3-8B (zero-shot) across three evaluation dimensions. Notation follows Fig. 5 .
Fig. 7 : Score distributions and mean scores for GPT-o3, Qwen3-8B (few-shot), and Qwen3-8B (zero-shot) across three evaluation dimensions. Notation follows Fig. 5 .
Fig. 8 : Visualization of resume embeddings highlighting the five samples selected by each sampling strategy.
Fig. 9 : Visualization of resume embeddings highlighting the twenty samples selected by each sampling strategy.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
Resume screeners must infer job-relevant competence from resumes whose presentation can vary substantially in wording, structure, stylistic polish, and document extraction quality. Ideally, such surface variation should not change decisions when the underlying qualification evidence is unchanged. We introduce a controlled audit of this property, constructing occupation-grounded candidate profiles at controlled competence levels and rendering each profile into multiple resume presentations. A deterministic validation gate excludes variants that alter the underlying evidence before scoring. Across six open instruction-tuned LLM conditions, we find a clear disconnect between screening validity and presentation stability. Llama-3.1-8B with its native chat template achieves the strongest validity (0.781) yet reverses 29.6% of matched pairwise decisions under competence-preserving presentation changes; Mistral-7B-v0.3 reaches validity 0.644 with a 41.4% flip rate. Native chat formatting improves validity for several chat-tuned models but does not remove this instability. These results show that resume-screening evaluations should assess not only whether a system identifies stronger candidates, but also whether those decisions remain stable when the same competence evidence is presented differently.
Qiangju Chen, Yang Xiao
1Macquarie University · 2The University of Melbourne
Large language models (LLMs) are increasingly used to screen and rank job applicants, creating incentives for candidates to strategically manipulate algorithmic hiring systems. We study prompt injection in automated résumé screening, defined as subtle self-promotional text that introduces no new qualifications but is designed to influence LLM evaluations. Using controlled experiments, we show that prompt injection reliably improves applicant rankings when résumé quality is homogeneous and few candidates inject. However, its effectiveness rapidly diminishes as more candidates inject, collapsing when manipulation becomes widespread. When candidate quality is heterogeneous, prompt injection is less effective on average, but can occasionally allow lower-quality candidates to outrank higher-quality ones, raising fairness concerns. Overall, LLM-based screening is most vulnerable when manipulation is rare and candidate quality differences are small. Code and resources are publicly available at: https://github.com/preetb1199/Prompt_Injection_ACL26
Preet Baxi, Jiannan Xu, Jane Yi Jiang +1
Department of Physics, University of Michigan · Robert H. Smith School of Business, University of Maryland · Fisher College of Business, The Ohio State University +1
We present PopResume, a population-representative resume dataset for causal fairness auditing of LLM- and VLM-based resume screening systems. Unlike existing benchmarks that rely on manually injected demographic information and outcome-level disparities, PopResume is grounded in population statistics and preserves natural attribute relationships, enabling path-specific effect (PSE)-based fairness evaluation. We decompose the effect of a protected attribute on resume scores into two paths: the business necessity path, mediated by job-relevant qualifications, and the redlining path, mediated by demographic proxies. This distinction allows auditors to separate legally permissible from impermissible sources of disparity. Evaluating four LLMs and four VLMs on PopResume's 60.8K resumes across five occupations, we identify five representative discrimination patterns that aggregate metrics fail to capture. Our results demonstrate that PSE-based evaluation reveals fairness issues masked by outcome-level measures, underscoring the need for causally-grounded auditing frameworks in AI-assisted hiring.
Sumin Yu, Juhyeon Park, Taesup Moon
ECE, Seoul National University · IPAI, Seoul National University · ASRI / INMC / AIIS, Seoul National University