Scalable Identification and Prioritization of Requisition-Specific Personal Competencies Using Large Language Models

Wanxin Li; Denver McNeney; Nivedita Prabhu; Charlene Zhang; Renee Barr; Matthew Kitching; Khanh Dao Duc; Anthony S. Boyce

cs.CLMar 9, 2026

Scalable Identification and Prioritization of Requisition-Specific Personal Competencies Using Large Language Models

Authors: Wanxin Li, Denver McNeney, Nivedita Prabhu, Charlene Zhang, Renee Barr, Matthew Kitching, Khanh Dao Duc, Anthony S. Boyce

Organizations: Department of Computer Science, University of British Columbia, Canada · Amazon · Department of Mathematics, University of British Columbia, Canada

Abstract

AI-powered recruitment tools are increasingly adopted in personnel selection, yet they struggle to capture the requisition (req)-specific personal competencies (PCs) that distinguish successful candidates beyond job categories. We propose a large language model (LLM)-based approach to identify and prioritize req-specific PCs from reqs. Our approach integrates dynamic few-shot prompting, reflection-based self-improvement, similarity-based filtering, and multi-stage validation. Applied to a dataset of Program Manager reqs, our approach correctly identifies the highest-priority req-specific PCs with an average accuracy of 0.76, approaching human expert inter-rater reliability, and maintains a low out-of-scope rate of 0.07.

Figures & tables

Figure 1: (A) An example output from our approach. In this example, our approach output two PCs for PMT-1: A “Domain/Team-Specific” PC “Consumer-Facing Techinical Products” with a priority rating of 9 and a “Other Functional” PC “Vendor Management” with a priority rating of 6. (B) An overview of our approach with a toy example using primary call, evaluation, regeneration, filter and validation components. PC justification is omitted from the output to save space. In this example, we want to identify req-specific PCs for PM-31. In the primary call component, we prepare the primary call prompt from the reqs and the most similar example from the example library. In the evaluation component, an LLM evaluates the outputs from the primary call and gives a suggestion to revise PC_2 definition. In the improvement component, an LLM uses the suggestion to improve PC_2 definition and corrects the priority for PC_1 using rules. In the filter component, we filter out PC_3 as it is too similar to “Ownership”, which is an explicitly defined PC to exclude (out-of-scope). In the validation component, we validate each PC label against the standardized competency library and find PC_2’label is too similar to the library PC “Program Management” but with significantly different definitions. Hence, our label refinement LLM refines PC_2 to a different label PC_2_revised to avoid confusion.

Evaluation/ Rating names	Evaluation/ Rating anchors	Used in
Out-of-scope check	Binary (0/1)	Both
Granularity appropriateness	Binary (0/1)	Both
Categorization correctness	Binary (0/1)	(b)
Justification quality	Binary (0/1)	(b)
Overlapping meanings check	Binary (0/1)	(b)
Top-1 appropriateness	1-3 scale	(a)

Table 1: Dimensions used in (a) SME-assessed ratings and in the (b) evaluation component.

	top-1	ranking	alignment	category
	precision	alignment	alignment	alignment
PM	0.88	0.68	0.88	1
PMT	0.79	0.65	0.88	1

Table 2: IRR from SME for algorithmic-determined metrics for PM and PMT reqs.

	Number of	top-1	top-2	top-3	ranking	priority	category
	reqs	precision	precision	precision	alignment	alignment	alignment
PM	17	0.77	0.86	0.91	0.60	0.82	0.85
		[0.70, 0.84]	[0.81, 0.91]	[0.86, 0.95]	[0.56, 0.65]	[0.81, 0.84]	[0.81, 0.88]
PMT	17	0.75	0.88	0.93	0.59	0.88	0.86
		[0.71, 0.79]	[0.84, 0.92]	[0.90, 0.96]	[0.56, 0.62]	[0.87, 0.89]	[0.83, 0.89]

Table 3: Averages and 95% CI for algorithmic-determined metrics on the test set. Results are averaged over 10 runs.

	Number of ratings	top-1	overall	top-1	granularity
	ratings	precision	out-of-scope rate	out-of-scope rate	appropriateness
PM	282	0.76	0.07	0.06	0.87
PMT	331	0.78	0.07	0.06	0.69

Table 4: SME-assessed ratings on the test set. Results are evaluated on 1 run.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Workflow of req-specific PC identifier. Distinct shapes and colors represent different components and flows. Actions are depicted as blue rectangles, data storage is shown as grey cylinders, model-generated label and evaluation metrics are represented by orange circles, the model itself is indicated by a red circle, and decision points are marked by green diamonds. The pipeline execution follows a sequential order: beginning with steps on the train set (purple lines), followed by steps on the dev set (green lines), and concluding with steps on the test set (blue lines).

Evaluation/Rating Name & Question	Evaluation/Rating Anchors	Level
Out-of-scope check - Is this an out-of-scope PC?	1 - No 0 - Yes
Granularity appropriateness - Does this PC have the just-right level of granularity	1 - Yes - Just right 0 - No - Too broad 0 - No - Too granular	PC
Categorization correctness - Is the PC categorized correctly as a Domain/Team-Specific or Other Functional?	1 - Yes 0 - No	PC
Justification quality - Is the PC supported by explicit evidence from the req?	1 - Yes 0 - No	PC
Overlapping meanings check - Do any PCs have overlapping or redundant meanings?	1 - No overlap 0 - Redundant/overlapping	Req
Top-1 precision - To what extend does the top-1 model-generated PC match with the top-1 SME ground-truth PC?	1 - Not appropriate; should not be top-1 2 - Acceptable but not the best 3 - Perfect as top-1	Req

Appendix

Table 5: Dimensions used in SME-assessed ratings and in the evaluation component, with anchors and level of application (PC vs. req).

	top-1	ranking	priority	category
	precision	alignment	alignment	alignment
Full approach with	0.77	0.60	0.82	0.85
dynamic few shot (baseline)	[0.70, 0.84]	[0.56, 0.65]	[0.81, 0.84]	[0.81, 0.88]
Zero shot	0.75	0.60	0.82	0.83
	[0.70, 0.81]	[0.57, 0.62]	[0.81, 0.84]	[0.79, 0.87]
Static few shot	0.57	0.55	0.83	0.80

Appendix

Table 6: Averages and 95% CI for algorithmic-determined metrics on the test set for ablation study on experimenting with different prompting techniques for PM reqs.

	top-1	ranking	priority	category
	precision	alignment	alignment	alignment
Full approach with	0.75	0.59	0.88	0.86
dynamic few shot (baseline)	[0.71, 0.79]	[0.56, 0.62]	[0.87, 0.89]	[0.83, 0.89]
Zero shot	0.71	0.57	0.87	0.82
	[0.66, 0.75]	[0.55, 0.59]	[0.86, 0.88]	[0.79, 0.86]
Static few shot	0.75	0.60	0.89	0.85

Appendix

Table 7: Averages and 95% CI for algorithmic-determined metrics on the test set for ablation study on experimenting with different prompting techniques for PMT reqs.

	top-1	ranking	priority	category
	precision	alignment	alignment	alignment
Full approach (baseline)	0.77	0.60	0.82	0.85
	[0.70, 0.84]	[0.56, 0.65]	[0.81, 0.84]	[0.81, 0.88]
Without extended reasoning	0.69	0.55	0.83	0.85
	[0.65, 0.74]	[0.53, 0.56]	[0.82, 0.84]	[0.82, 0.87]
Without evaluation and regeneration	0.74	0.55	0.84	0.87

Appendix

Table 8: Averages and 95% CI for algorithmic-determined metrics on the test set for ablation study on experimenting with removing different functionalities and components from the full approach for PM reqs.

	top-1	ranking	priority	category
	precision	alignment	alignment	alignment
Full approach (baseline)	0.75	0.59	0.88	0.86
	[0.71, 0.79]	[0.56, 0.62]	[0.87, 0.89]	[0.83, 0.89]
Without extended reasoning	0.66	0.56	0.89	0.85
	[0.61, 0.72]	[0.54, 0.58]	[0.88, 0.90]	[0.82, 0.88]
Without evaluation and regeneration	0.74	0.56	0.88	0,85

Appendix

Table 9: Averages and 95% CI for algorithmic-determined metrics on the test set for ablation study on experimenting with removing different functionalities and components from the full approach for PMT reqs.

	top-1	ranking	priority	category
	precision	alignment	alignment	alignment
Full approach with	0.77	0.60	0.82	0.85
Claude 4.0 (baseline)	[0.70, 0.84]	[0.56, 0.65]	[0.81, 0.84]	[0.81, 0.88]
Full approach with	0.71	0.53	0.84	0.85
Claude 3.7	[0.64, 0.78]	[0.51, 0.54]	[0.83, 0.85]	[0.81, 0.89]

Appendix

Table 10: Averages and 95% CI for algorithmic-determined metrics on the test set for ablation study on experimenting with Claude 3.7 from the full approach for PM reqs.

	top-1	ranking	priority	category
	precision	alignment	alignment	alignment
Full approach with	0.75	0.59	0.88	0.86
Claude 4.0 (baseline)	[0.71, 0.79]	[0.56, 0.62]	[0.87, 0.89]	[0.83, 0.89]
Full approach with	0.56	0.44	0.88	0.85
Claude 3.7	[0.51, 0.62]	[0.41, 0.47]	[0.87, 0.89]	[0.82, 0.87]

Appendix

Table 11: Averages and 95% CI for algorithmic-determined metrics on the test set for ablation study on experimenting with Claude 3.7 from the full approach for PMT reqs.

AI-Assisted Decision Making

AI-assisted decision making studies how computational systems can inform human choices while accounting for context, uncertainty, and consequences.

See all 144 papers

Agentic AI for Human Resources: LLM-Driven Candidate Assessment
Mar 17, 2026Kamer Ali Yuksel, Abdul Basit Anees, Ashraf Elneima +3Learning to Rank LLM-as-a-Judge
aiXplain, Inc., San Jose, CA, USA
Towards Position-Robust Talent Recommendation via Large Language Models
Apr 2, 2026Silin Du, Hongyan LiuRecommender Systems Large Language Model-Based Recommendation
Department of Management Science and Engineering, Tsinghua University, Beijing, China.
EHR-Bench: A Chinese Multi-Task Benchmark for Recruitment AI
Oct 12, 2026Ruijia Yang, Bing Xu, Zulong Chen +5LLM Evaluation AI and Labor
The Hong Kong University of Science and Technology (Guangzhou) · Alibaba Group · Guangzhou University +1

Mar 17, 2026J/K move · Enter open · S save

Agentic AI for Human Resources: LLM-Driven Candidate Assessment

Kamer Ali Yuksel, Abdul Basit Anees, Ashraf Elneima, Sanjika Hewavitharana+2

aiXplain, Inc., San Jose, CA, USA

Learning to Rank LLM-as-a-Judge

Open paper AskPDF arXiv

In this work, we present a modular and interpretable framework that uses Large Language Models (LLMs) to automate candidate assessment in recruitment. The system integrates diverse sources, including job descriptions, CVs, interview transcripts, and HR feedback; to generate structured evaluation reports that mirror expert judgment. Unlike traditional ATS tools that rely on keyword matching or shallow scoring, our approach employs role-specific, LLM-generated rubrics and a multi-agent architecture to perform fine-grained, criteria-driven evaluations. The framework outputs detailed assessment reports, candidate comparisons, and ranked recommendations that are transparent, auditable, and suitable for real-world hiring workflows. Beyond rubric-based analysis, we introduce an LLM-Driven Active Listwise Tournament mechanism for candidate ranking. Instead of noisy pairwise comparisons or inconsistent independent scoring, the LLM ranks small candidate subsets (mini-tournaments), and these listwise permutations are aggregated using a Plackett-Luce model. An active-learning loop selects the most informative subsets, producing globally coherent and sample-efficient rankings. This adaptation of listwise LLM preference modeling (previously explored in financial asset ranking) provides a principled and highly interpretable methodology for large-scale candidate ranking in talent acquisition.

Abstract

AI-Assisted Decision Making

Explore similar work

Agentic AI for Human Resources: LLM-Driven Candidate Assessment

Towards Position-Robust Talent Recommendation via Large Language Models

EHR-Bench: A Chinese Multi-Task Benchmark for Recruitment AI