Paper ID: 2410.16325

This Candidate is [MASK]. Letters of Reference and Job Market Outcomes using LLMs

Fabian Slonimczyk

I implement a prompt-based learning strategy to extract measures of sentiment and other features from confidential reference letters. I show that the contents of reference letters is clearly reflected in the performance of job market candidates in the Economics academic job market. In contrast, applying traditional ``bag-of-words'' approaches produces measures of sentiment that, while positively correlated to my LLM-based measure, are not predictive of job market outcomes. Using a random forest, I show that both letter quality and length are predictive of success in the job market. Letters authored by advisers appear to be as important as those written by other referees.

Submitted: Oct 18, 2024