Corporate recruiters often need to screen many resumes within a limited time, which increases their burden and may cause suitable candidates to be overlooked. To address these challenges, prior work has explored LLM-based automated resume screening. However, some methods rely on commercial LLMs, which may pose data privacy risks. Moreover, since companies typically do not make resumes with evaluation results publicly available, it remains unclear which resume samples should be used during learning to improve an LLM's judgment performance. To address these problems, we propose AutoScreen-FW, an LLM-based locally and automatically resume screening framework. AutoScreen-FW uses several methods to select a small set of representative resume samples. These samples are used for in-context learning together with a persona description and evaluation criteria, enabling open-source LLMs to act as a career advisor and evaluate unseen resumes. Experiments with multiple ground truths show that the open-source LLM judges consistently outperform GPT-5-nano. Under one ground truth setting, it also surpass GPT-5-mini. Although it is slightly weaker than GPT-5-mini under other ground-truth settings, it runs substantially faster per resume than commercial GPT models. These findings indicate the potential for deploying AutoScreen-FW locally in companies to support efficient screening while reducing recruiters' burden.
Figures & tables
Fig. 1 : The comparison between (a) Human-only Screening Pipeline and (b) LLM-based Screening Pipeline.
Fig. 2 : Overview of the AutoScreen-FW.
Fig. 3 : Persona used in the prompt.
Dimension
Evaluation Perspective
Evaluation Criteria
Content
Validity of Evidence
Appropriate episodes are used to effectively convey the applicant’s strengths.
Specificity of Content
• The applicant’s motivation for the target company and their potential contributions are described concretely. • Episodes are described in sufficient detail. • Achievements, research outcomes, and related accomplishments are presented concretely using numerical values.
Structure
Explicit Conclusion
For each question, the conclusion is clearly stated at the beginning so that recruiters can understand it easily.
Focus on a Single Selling Point
For questions intended to highlight the applicant’s strengths, such as self-promotion, the response focuses on a single selling point.
Conciseness of Writing
The overall length is appropriate, and the writing is clear and concise.
Language
Grammatical Accuracy
The response contains no grammatical errors.
TABLE I: Resume evaluation items
Fig. 4 : Evaluation instruction used in the prompt.
Model
Few-shot
Time per Resume (s)
GPT-5-mini
5.09±0.22
GPT-5-nano
7.48±0.45
Qwen3-8B
3.73±0.69
Llama-3.1-8B
0.89±0.15
Qwen3-8B
✓
3.84±0.26
Llama-3.1-8B
✓
2.10±0.24
TABLE II : Per-resume judge time for each LLM model. Results are reported as mean ± standard deviation.
Model
Few-shot
Sampling Strategy
Shots
Sample Type
Attribute Type
Acc.
GPT-5-mini
N/A
0
N/A
N/A
0.6917
GPT-5-nano
N/A
0
N/A
N/A
0.6091
Qwen3-8B
N/A
0
N/A
N/A
0.6063
Llama-3.1-8B
N/A
0
N/A
N/A
0.5721
Qwen3-8B
✓
Similarity-based
3
High-quality only
Overall Judgement
0.6746
Llama-3.1-8B
✓
Diversity-based
20
High- and Low- quality
Overall Judgement and Dimension Scores
0.6101
TABLE III : Accuracy (Acc.) of each model with GPT-5.2 as the ground truth.
Model
Few-shot
Sampling Strategy
Shots
Sample Type
Attribute Type
Acc.
GPT-5-mini
N/A
0
N/A
N/A
0.8445
GPT-5-nano
N/A
0
N/A
N/A
0.8374
Qwen3-8B
N/A
0
N/A
N/A
0.8552
Llama-3.1-8B
N/A
0
N/A
N/A
0.7927
Qwen3-8B
✓
Diversity-based
5
High-quality only
Overall Judgement and Dimension Scores
0.8682
Llama-3.1-8B
✓
Clustering-based
15
High- and Low- quality
Overall Judgement and Dimension Scores
0.8485
TABLE IV : Accuracy (Acc.) of each model with GPT-5.1 as the ground truth.
Model
Few-shot
Sampling Strategy
Shots
Sample Type
Attribute Type
Acc.
GPT-5-mini
N/A
0
N/A
N/A
0.7387
GPT-5-nano
N/A
0
N/A
N/A
0.6617
Qwen3-8B
N/A
0
N/A
N/A
0.6744
Llama-3.1-8B
N/A
0
N/A
N/A
0.6280
Qwen3-8B
✓
Diversity-based
3
High- and Low- quality
Overall Judgement and Dimension Scores
0.7218
Llama-3.1-8B
✓
Clustering-based
20
High-quality only
Overall Judgement and Dimension Scores
0.6333
TABLE V : Accuracy (Acc.) of each model with GPT-o3 as the ground truth.
Fig. 5 : Score distributions and mean scores assigned by GPT-5.2, GPT-5.1, and GPT-o3 across three evaluation dimensions. Each dot denotes an individual resume, and the horizontal line indicates the mean score for each LLM model.
Fig. 6 : Score distributions and mean scores for GPT-5.2, Qwen3-8B (few-shot), and Qwen3-8B (zero-shot) across three evaluation dimensions. Notation follows Fig. 5 .
Fig. 7 : Score distributions and mean scores for GPT-o3, Qwen3-8B (few-shot), and Qwen3-8B (zero-shot) across three evaluation dimensions. Notation follows Fig. 5 .
Fig. 8 : Visualization of resume embeddings highlighting the five samples selected by each sampling strategy.
Fig. 9 : Visualization of resume embeddings highlighting the twenty samples selected by each sampling strategy.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
Department of Physics, University of Michigan · Robert H. Smith School of Business, University of Maryland · Fisher College of Business, The Ohio State University +1