Prioritize, Don't Decide: What We Learned Auditing a Deployed Human-in-the-Loop CV Sorter
Organizations: AcademicTransfer, Utrecht, The Netherlands · Department of Methodology, Statistics & Data Science, Utrecht University, The Netherlands · Department of Information and Computing Sciences, Utrecht University, The Netherlands
Abstract
A single academic vacancy can draw hundreds of applications, and the first screening round consumes most of a committee's time. We describe a CV Priority Sorter deployed at a national platform for academic vacancies and integrated with its vacancy database. It proposes a reading order for a pile of CVs and leaves every decision to the committee, pairing a frozen multilingual transformer encoder with a per-session linear model that retrains after each label and is discarded when the session ends. It works: on a pool of 300 synthetic CVs, a committee reading in the tool's order reaches 95% of the relevant candidates after half the pile, where an arbitrary order needs almost all of it. This paper is about what we found when we audited that claim, rather than about the system itself. Three findings changed the product. The pool turned out to be sortable by document length alone, which makes precision-style metrics on it uninformative; regenerating it under a shared length budget removes that shortcut and leaves the reading-depth result standing. The four-level agreement score such systems report, ours included, is a binary probability cut at fixed thresholds, so it follows the classifier's confidence rather than its ranking and falls as labels accumulate. And uncertainty sampling, the obvious way to implement a "suggest the next CV" button, leaves model quality intact but raises the committee's total reading load above the no-label baseline, because the CVs it asks about are the ones that would otherwise be skipped. A fourth result shows where to look next: a hosted LLM embedding model lifts zero-label ordinal discrimination from 0.66 to 0.80 under length control, then stops mattering once the session model has two dozen labels; a local cross-encoder matches that cold-start gain without sending data out. We report the fixes each finding forced.
Figures & tables
| Ranker, no labels | nDCG | P@25 | P@75 | Read | |
|---|---|---|---|---|---|
| Document length | 0.994 | 1.00 | 1.000 | 0.640 | 0.862 |
| Line count | 0.992 | 1.00 | 1.000 | 0.697 | 0.841 |
| Venue mentions | 0.960 | 1.00 | 1.000 | 0.977 | 0.766 |
| BM25 | 0.991 | 1.00 | 0.987 | 0.723 | 0.805 |
| Zero-shot embed. | 0.918 | 0.72 | 0.787 | 0.653 | 0.634 |
| Ranker, no labels | nDCG | P@75 | Read | len. | |
|---|---|---|---|---|---|
| Document length | 0.955 | 0.827 | 0.847 | 0.395 | 1.000 |
| Venue mentions | 0.996 | 1.000 | 0.477 | 0.934 | 0.544 |
| BM25 | 0.972 | 0.933 | 0.790 | 0.670 | 0.494 |
| Zero-shot embedding | 0.881 | 0.707 | 0.717 | 0.510 | 0.000 |
| Local cross-encoder | 0.982 | 1.000 | 0.477 | 0.912 | 0.300 |
| Strategy (120 labels) | Relevant | Screened | |
|---|---|---|---|
| Random (stratified) | 0.954 | 60.0 | 0.691 |
| Random (uniform) | 0.954 | 61.0 | 0.689 |
| Uncertainty sampling | 0.910 | 22.1 | 0.817 |
| Extremes first | 0.955 | 60.0 | 0.700 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| No labels | vs len. | Read@95 | |
|---|---|---|---|
| full pool | |||
| LLM scorer | 0.955 | 0.863 | 0.580 |
| LLM vectorizer | 0.894 | 0.813 | 0.567 |
| Local cross-encoder | 0.937 | 0.869 | 0.560 |
| Deployed encoder | 0.634 | 0.596 | 0.653 |
| length-controlled band, | |||
| Perturbation | Top-75 | Read | ||
|---|---|---|---|---|
| Phone number (control) | 0.974 | 0.49 | 0 | 0.500 |
| Name placeholder | 0.970 | 0.62 | 1 | 0.503 |
| Name set A | 0.969 | 0.59 | 1 | 0.503 |
| Name set B | 0.970 | 0.57 | 1 | 0.500 |
| Name set C | 0.971 | 0.59 | 0 | 0.500 |
| Name set D | 0.967 | 0.66 | 1 | 0.500 |
| Budget | 12 | 24 | 40 | 60 | 80 | 120 |
|---|---|---|---|---|---|---|
| as reported | ||||||
| Random (strat.) | 0.270 | 0.458 | 0.857 | 0.924 | 0.906 | 0.869 |
| Uncertainty | 0.291 | 0.398 | 0.439 | 0.455 | 0.461 | 0.418 |
| , blend disabled at scoring time | ||||||
| Random (strat.) | 0.401 | 0.458 | 0.857 | 0.924 | 0.906 | 0.869 |
| Uncertainty | 0.286 | 0.473 | 0.829 | 0.878 | 0.862 | 0.805 |
Human-in-the-Loop Evaluation
Human-in-the-Loop Evaluation studies how human judgment and feedback can assess, guide, and monitor AI systems in realistic settings.
See all 174 papers