cs.CLNov 16, 2026

HireWatch: Evaluating LLM Compliance with U.S. Employment Law Under Contextual Pressure

Authors: Huy Nghiem, Maria Kostylew, Noam Kolt

Organizations: MATS Research, University of Maryland · MATS Research · Hebrew University of Jerusalem

Abstract

The deployment of large language models (LLMs) in hiring pipelines is growing, yet existing evaluations generally overlook their compliance with legal rules. To fill this gap, we introduce HireWatch, an evaluation that assesses whether LLMs comply with U.S. laws that govern the questions that job candidates may be asked. We construct a 200-item bank of lawful and unlawful interview questions, whose legal classification is based on U.S. federal statutes, regulations, and agency guidance. We instruct LLMs to select questions from this bank in preparing for job interviews, across a set of various occupations and contexts, and report the fraction of unlawful questions that LLMs select. Evaluation of 18 closed and open-weight LLMs across ~100,000 trials under different types of contextual pressure reveals that LLMs' legal compliance is brittle. While LLMs generally avoid selecting unlawful questions in low-pressure contexts, legal violations increase sharply under higher pressure. Explicit prompts to comply with law can improve compliance rates, but do not altogether eliminate legal violations. We publicly release our code and data at https://github.com/hnghiem-nlp/hirewatch_repo.

Figures & tables

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Can AI Judge You? An Empirical Audit of Deployment Readiness in LLM-assisted Hiring Systems

    Nov 15, 2026Spandan Roy, Subhabrata Majumdar, Soudeep DebAI-Assisted Decision MakingLLM Auditing

  2. AI evaluators favor job applications written by certain AI models, independent of candidate quality

    Oct 30, 2026Nicolò Pagan, Michael Andreas Ebling, Stefania Ionescu +1Algorithmic FairnessLanguage Model Bias Evaluation

  3. Can LLMs Hire Fairly? Racial Bias in Resume Screening

    Jun 27, 2026Zhenyu Gao, Wenxi Jiang, Yutong YanGender Bias in Language ModelsSocial Bias in Language Models