Skill extraction benchmarks score models against a few human-annotated positives and treat every other skill in the vocabulary as equally wrong. At the scale of ESCO (13,939 skills) this binary view can reward models that rank the annotated positive high for the wrong reason, and hides semantically nonsensical predictions. The WorkRB Challenge at RecSys-HR 2026 replaces it with graded relevance: every candidate skill in a model's ranking window carries a 0 to 4 grade, and the leaderboard scores nDCG@100 against these grades. Since ternary judgements proved too coarse for objective annotation, relevance is decomposed into a cascade of four binary questions (same domain, activity mentioned, right granularity, replacement of a gold positive) that a smaller LLM judge can answer reliably; the labels are validated with blind human and frontier-model re-annotation. This paper documents the annotation methodology, the construction of five benchmark datasets (four with vacancy sentences as queries, one with lexically distant skill surface forms), the validation of the LLM cascade, and insights from the submissions. The field converged on multi-stage retrieve-and-rerank pipelines, whose gains come from clearing nonsense out of the ranking window rather than finding more positives: at the top of the leaderboard, the graded ordering is uncorrelated with what binary evaluation would have said. Graded labels expose much bigger gaps between top submissions and baselines than binary evaluation does, and show that semantic-understanding skill recommendation is still wide open. The clear next direction is bringing efficiency back: every submitted system we could size uses at least 42x, typically several hundred times, more parameters than the baselines it beats.
Figures & tables
Figure 1: The annotation cascade, with a worked example on every exit. A pair descends the gates in order and the first “no” fixes its grade, while human gold positives bypass the cascade entirely. The six red skills are real ESCO entries judged against one real House query, and each is placed by its ESCO description rather than its label. design hair style (“design the styles of hair based on the preference of the customer”) shares only the word “design”, the kind of surface overlap a retriever scores highly and the domain gate exists to reject. test materials is engineering work, so it clears the domain gate, but its description covers testing the composition and characteristics of materials, never designing anything or using software. use ICT systems (“select and use ICT systems for a variety of complex tasks”) is true of the job and of thousands of others, an umbrella too broad to be core. design engineering components (“design engineering parts, assemblies, products, or systems”) is correct and genuinely additional, describing the design act itself rather than the drafting tool the gold names. The two grade-4 exits differ only in provenance: use CADD software is the human annotation, and use CAD software is the jury judging a near-paraphrase of it, the two differing only by the drafting suffix. Section 7 turns that last distinction into the submission screen.
Figure 2: Blind validation of the cascade labels, test manifest. Left : row-normalised confusion matrix (%) between the released cascade grade and the blind Opus reviewer. Middle : the same, against the blind human expert. Both show mass on the diagonal or one cell to its left with a nearly empty upper-right triangle: the disagreement is one-directional for both reviewers, human included. Right : mean blind grade per cascade grade, Opus (both splits) and the human expert (test only). All three curves sit below the identity line, and the human curve is markedly less extreme at grade 1 but has the same overall shape. Grades 3 and 4 are reported together throughout, since grade 4 is defined against the gold-positive list the blind reviewer cannot see.
Judge
QWK
ρ
Exact
MAE
Gen./Str.
Cascade (released)
0.686
0.686
0.586
0.529
29/12
One call, rationale
0.618
0.621
0.525
0.654
25/22
One call, grade only
0.631
0.627
0.461
0.675
22/32
Table 1: Cascade versus one-shot direct grading by the same judge, against the blind human expert on the 280-pair test manifest (grades 0 to 3). Gen./Str. is the share of pairs the judge grades above or below the human.
Queries
Judged
Test, per query
Dataset
Val
Test
pairs
gr. ≥2
gr. 4
House
61
261
0.64M
22.7
2.7
Tech
75
338
0.83M
15.8
1.9
TechWolf
n/a
324
0.65M
17.5
2.3
SkillSkape
100
500
1.20M
27.6
3.2
SkillNorm
50
450
1.00M
9.3
1.1
Table 2: The five released benchmarks. Queries are ranked against the same corpus of 13,891 ESCO v1.1.0 skills, so the released qrels are the full query × corpus grid, with every pair the prefilter did not surface carried at grade 0. Judged pairs counts the pairs the cascade actually graded (queries × the top-2000 window, 4.3 M in total). The last two columns are test-split means per query: targets graded ≥2 , and targets graded 4. TechWolf ships no validation split.
Preferred label
Alternative label
Div.
Cos.
Retained
provide periodontal treatment
manage gum conditions
1.00
0.64
install crane equipment
build lifting machinery
1.00
0.66
pest management techniques
extermination methods
1.00
0.56
monitor crops
checking crop growth
0.75
0.74
welcome restaurant guests
greet and seat guests
0.83
0.68
Table 3: ESCO preferred/alternative pairs at each filtering stage. Divergence ≥0.7 removes morphological variants (middle block), and cosine ≥0.5 removes pairs sharing no semantic content despite different wording (bottom block). The top block satisfies both.
#
Team
Macro
House
Tech
TechWolf
SkillSkape
SkillNorm
1
thylinao s
0.8458
0.8387
0.8390
0.8347
0.8683
0.8480
2
qihang-wang
0.8323
0.8405
0.8337
0.8197
0.8585
0.8092
3
yeyang s
0.8224
0.8226
0.8261
0.8030
0.8586
0.8018
4
aiden-lee
0.8191
0.8114
0.7903
0.8109
0.8316
0.8514
5
elara96
0.8182
0.8015
0.8157
0.8022
0.8443
0.8274
6
carolzz
0.8132
0.7982
0.8106
0.8026
0.8442
0.8101
Table 4: Filtered leaderboard: graded nDCG@100 per task and macro-averaged, best valid submission per account, plus the organiser baselines. Teams are named by their CodaBench handle throughout. The winning entry is the work of Maksim Silchenko (National University of Singapore), competing solo as team thylinao , who documents the system in [ 20 ] . s self-declared student team. The ConTeXT-match row is the pre-released checkpoint. Retrained with a tuned schedule the same method reaches 0.620 (Section 6 ).
Figure 3: Graded nDCG@100 against MAP on the public binary positives, both macro-averaged over the four extraction tasks (SkillNorm has no binary counterpart and is excluded from both axes). One point per valid submission. The two metrics agree on the broad strokes and disagree exactly where a leaderboard matters: the graded winner is 12th by binary MAP, and the binary-MAP leader (leonshangguan) is 16th on the graded board.
Figure 4: Grade composition of the 100 slots that nDCG@100 reads (mean per query, macro over the four extraction tasks) for the ideal ranking, the best and median valid submissions, and the organiser baselines. The ideal window contains essentially no grade-0 items, and the ranking systems differ mostly in how many grade-0 items they let in (9 for the winner, 17 at the median, 34–37 for the trained baselines, 79 for BM25) while everyone finds nearly all ≈ 2.5 grade-4 slots.
System
nDCG@100
Backbones
Params (B)
× baseline
thylinao ∗
0.8458
6
13.9
127
qihang-wang ∗
0.8323
6
35.8
327
yeyang
0.8224
6
45.6
417
aiden-lee
0.8191
9
243.7
2226
elara96 ∗
0.8182
7
53.7
490
carolzz
0.8132
7
52.2
476
Table 5: Inference-time size of every submitted system whose code could be read. Params is the base-checkpoint footprint (distinct pretrained backbones, billions), and × baseline is the ratio to CurriculumMatch. † a lower bound: the pipeline also calls a hosted frontier model with no published weight count. ∗ includes at least one checkpoint sized from file bytes rather than exact metadata.
Figure 5: Size against score, log scale. Hollow dotted markers are estimates or lower bounds. BM25 ( 0.2250 , no parameters) cannot be placed on a log axis and is omitted.
Information Retrieval (IR) Evaluation measures how well search systems find, rank, and cover useful evidence across queries, collections, and settings.