Prioritize, Don't Decide: What We Learned Auditing a Deployed Human-in-the-Loop CV Sorter
Authors: Hadi Mohammadi, Jeroen Sparla, Alexandros Paliouras, Georg Krempl, Robert A. Bagheri
Organizations: AcademicTransfer, Utrecht, The Netherlands · Department of Methodology, Statistics & Data Science, Utrecht University, The Netherlands · Department of Information and Computing Sciences, Utrecht University, The Netherlands
A single academic vacancy can draw hundreds of applications, and the first screening round consumes most of a committee's time. We describe a CV Priority Sorter deployed at a national platform for academic vacancies and integrated with its vacancy database. It proposes a reading order for a pile of CVs and leaves every decision to the committee, pairing a frozen multilingual transformer encoder with a per-session linear model that retrains after each label and is discarded when the session ends. It works: on a pool of 300 synthetic CVs, a committee reading in the tool's order reaches 95% of the relevant candidates after half the pile, where an arbitrary order needs almost all of it. This paper is about what we found when we audited that claim, rather than about the system itself. Three findings changed the product. The pool turned out to be sortable by document length alone, which makes precision-style metrics on it uninformative; regenerating it under a shared length budget removes that shortcut and leaves the reading-depth result standing. The four-level agreement score such systems report, ours included, is a binary probability cut at fixed thresholds, so it follows the classifier's confidence rather than its ranking and falls as labels accumulate. And uncertainty sampling, the obvious way to implement a "suggest the next CV" button, leaves model quality intact but raises the committee's total reading load above the no-label baseline, because the CVs it asks about are the ones that would otherwise be skipped. A fourth result shows where to look next: a hosted LLM embedding model lifts zero-label ordinal discrimination from 0.66 to 0.80 under length control, then stops mattering once the session model has two dozen labels; a local cross-encoder matches that cold-start gain without sending data out. We report the fixes each finding forced.
Figures & tables
Figure 1: The tool as the committee sees it. (a) The pile as three priority groups with a score per CV. (b) One CV: the tool explains its score and asks for a second look when it disagrees with the label. Both screens show the synthetic pool.
Ranker, no labels
nDCG
P@25
P@75
Read ↓
ρ
Document length
0.994
1.00
1.000
0.640
0.862
Line count
0.992
1.00
1.000
0.697
0.841
Venue mentions
0.960
1.00
1.000
0.977
0.766
BM25
0.991
1.00
0.987
0.723
0.805
Zero-shot embed.
0.918
0.72
0.787
0.653
0.634
Table 1: Label-free rankers over the 300-CV pool. Read is the fraction of the pile read to reach 95% of the relevant CVs; ρ is Spearman correlation with the true tier.
Ranker, no labels
nDCG
P@75
Read ↓
ρ
ρ len.
Document length
0.955
0.827
0.847
0.395
1.000
Venue mentions
0.996
1.000
0.477
0.934
0.544
BM25
0.972
0.933
0.790
0.670
0.494
Zero-shot embedding
0.881
0.707
0.717
0.510
0.000
Local cross-encoder
0.982
1.000
0.477
0.912
0.300
Table 2: Label-free rankers on pool v2, where every tier shares one length budget. The optimum Read@95 for this pool is 0.475.
Figure 2: The same runs under two statistics. Weighted κ (solid) rises and falls with the labeling budget; Spearman ρ (dashed) is flat from 24 labels on.
Strategy (120 labels)
ρ↑
Relevant
Screened ↓
Random (stratified)
0.954
60.0
0.691
Random (uniform)
0.954
61.0
0.689
Uncertainty sampling
0.910
22.1
0.817
Extremes first
0.955
60.0
0.700
Table 3: Labeling strategies at 120 labels, mean over 20 seeds. Relevant is how many of the 120 labeled CVs were relevant; Screened is the share of the pool read to reach 95% recall, counting labeled CVs as read. The no-label baseline is 0.653.
Figure 3: Share of the pool screened to reach 95% recall, by labeling budget. Dotted lines are the floors set by the labeled sets of random (stratified) and uncertainty sampling.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: The pipeline. A zero-shot score orders the pile at upload; a per-session classifier takes over as labels arrive. Nothing inside the dashed boundary outlives the session.
No labels
ρ↑
ρ vs len.
Read@95 ↓
full pool
LLM scorer
0.955
0.863
0.580
LLM vectorizer
0.894
0.813
0.567
Local cross-encoder
0.937
0.869
0.560
Deployed encoder
0.634
0.596
0.653
length-controlled band, n=149
Appendix
Table 4: Cold start, before any labels. The middle column is the correlation with document length, the confound of Section 5.1 .
Perturbation
τ
∣Δs∣
Top-75
Read
Phone number (control)
0.974
0.49
0
0.500
Name → placeholder
0.970
0.62
1
0.503
Name set A
0.969
0.59
1
0.503
Name set B
0.970
0.57
1
0.500
Name set C
0.971
0.59
0
0.500
Name set D
0.967
0.66
1
0.500
Appendix
Table 5: Ranking movement under identity perturbations. τ is Kendall correlation with the unperturbed ranking; ∣Δs∣ is the mean absolute score change on the 0 to 100 scale; Top-75 counts CVs entering or leaving the top group; Read is reading depth for 95% recall.
Budget
12
24
40
60
80
120
κw as reported
Random (strat.)
0.270
0.458
0.857
0.924
0.906
0.869
Uncertainty
0.291
0.398
0.439
0.455
0.461
0.418
κw , blend disabled at scoring time
Random (strat.)
0.401
0.458
0.857
0.924
0.906
0.869
Uncertainty
0.286
0.473
0.829
0.878
0.862
0.805
Appendix
Table 6: Robustness of the comparison in Section 5.4 . Random labeling is blended only at a budget of 12, so its row changes in one column; uncertainty sampling is blended at every budget.