cs.CYNov 15, 2026

Can AI Judge You? An Empirical Audit of Deployment Readiness in LLM-assisted Hiring Systems

Authors: Spandan Roy, Subhabrata Majumdar, Soudeep Deb

Organizations: Indian Institute of Management Bangalore

Abstract

Delegating resume screening to large language models (LLMs) requires organizations to establish whether they can distinguish candidates by relevant qualifications while allowing only authorized users to change screening rules or approve exceptions. To this end, we develop a system architecture that separates candidate evidence, evaluation, permission controls, and assessment record, and empirically examine two complementary dimensions of its deployment readiness. First, we assess its operational readiness on four LLMs using a controlled experiment on 210 resumes across 14 job roles and six decision interfaces. Second, we assess its compliance readiness by testing how it handles unauthorized user requests and instructions embedded in resumes. We find that operational performance depends on the choice of evaluation interface: scalar scores generally distinguish qualification changes better than simple pass/fail judgments, while comparative judgments are sensitive to presentation order. Explicit permission rules and separation of resume content from instructions reduce unauthorized compliance, to a varying degree of effectiveness across LLMs and usage contexts. Finally, we find that LLM evaluators often diverge between their decisions and offered justifications. Going beyond simply evaluating LLMs for hiring tasks, our framework supports an audit of a fully configured resume evaluation system to enable its deployment decision.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Competence-Preserving Resume Perturbations Expose Presentation Sensitivity in LLM Screening

    Sep 15, 2026Qiangju Chen, Yang XiaoLLM EvaluationLanguage Model Robustness

  2. PopResume: Causal Fairness Evaluation of LLM/VLM Resume Screeners with Population-Representative Dataset

    Mar 24, 2026Sumin Yu, Juhyeon Park, Taesup MoonAI-Assisted Decision MakingLLM Auditing

  3. Beyond the Resumé: A Rubric-Aware Automatic Interview System for Information Elicitation

    Mar 2, 2026Harry Stuart, Masahiro Kaneko, Timothy BaldwinOutput ElicitationAI and Labor