Skill extraction benchmarks score models against a few human-annotated positives and treat every other skill in the vocabulary as equally wrong. At the scale of ESCO (13,939 skills) this binary view can reward models that rank the annotated positive high for the wrong reason, and hides semantically nonsensical predictions. The WorkRB Challenge at RecSys-HR 2026 replaces it with graded relevance: every candidate skill in a model's ranking window carries a 0 to 4 grade, and the leaderboard scores nDCG@100 against these grades. Since ternary judgements proved too coarse for objective annotation, relevance is decomposed into a cascade of four binary questions (same domain, activity mentioned, right granularity, replacement of a gold positive) that a smaller LLM judge can answer reliably; the labels are validated with blind human and frontier-model re-annotation. This paper documents the annotation methodology, the construction of five benchmark datasets (four with vacancy sentences as queries, one with lexically distant skill surface forms), the validation of the LLM cascade, and insights from the submissions. The field converged on multi-stage retrieve-and-rerank pipelines, whose gains come from clearing nonsense out of the ranking window rather than finding more positives: at the top of the leaderboard, the graded ordering is uncorrelated with what binary evaluation would have said. Graded labels expose much bigger gaps between top submissions and baselines than binary evaluation does, and show that semantic-understanding skill recommendation is still wide open. The clear next direction is bringing efficiency back: every submitted system we could size uses at least 42x, typically several hundred times, more parameters than the baselines it beats.
Figures & tables
Figure 1: The annotation cascade, with a worked example on every exit. A pair descends the gates in order and the first “no” fixes its grade, while human gold positives bypass the cascade entirely. The six red skills are real ESCO entries judged against one real House query, and each is placed by its ESCO description rather than its label. design hair style (“design the styles of hair based on the preference of the customer”) shares only the word “design”, the kind of surface overlap a retriever scores highly and the domain gate exists to reject. test materials is engineering work, so it clears the domain gate, but its description covers testing the composition and characteristics of materials, never designing anything or using software. use ICT systems (“select and use ICT systems for a variety of complex tasks”) is true of the job and of thousands of others, an umbrella too broad to be core. design engineering components (“design engineering parts, assemblies, products, or systems”) is correct and genuinely additional, describing the design act itself rather than the drafting tool the gold names. The two grade-4 exits differ only in provenance: use CADD software is the human annotation, and use CAD software is the jury judging a near-paraphrase of it, the two differing only by the drafting suffix. Section 7 turns that last distinction into the submission screen.
Figure 2: Blind validation of the cascade labels, test manifest. Left : row-normalised confusion matrix (%) between the released cascade grade and the blind Opus reviewer. Middle : the same, against the blind human expert. Both show mass on the diagonal or one cell to its left with a nearly empty upper-right triangle: the disagreement is one-directional for both reviewers, human included. Right : mean blind grade per cascade grade, Opus (both splits) and the human expert (test only). All three curves sit below the identity line, and the human curve is markedly less extreme at grade 1 but has the same overall shape. Grades 3 and 4 are reported together throughout, since grade 4 is defined against the gold-positive list the blind reviewer cannot see.
Judge
QWK
ρ
Exact
MAE
Gen./Str.
Cascade (released)
0.686
0.686
0.586
0.529
29/12
One call, rationale
0.618
0.621
0.525
0.654
25/22
One call, grade only
0.631
0.627
0.461
0.675
22/32
Table 1: Cascade versus one-shot direct grading by the same judge, against the blind human expert on the 280-pair test manifest (grades 0 to 3). Gen./Str. is the share of pairs the judge grades above or below the human.
Queries
Judged
Test, per query
Dataset
Val
Test
pairs
gr. ≥2
gr. 4
House
61
261
0.64M
22.7
2.7
Tech
75
338
0.83M
15.8
1.9
TechWolf
n/a
324
0.65M
17.5
2.3
SkillSkape
100
500
1.20M
27.6
3.2
SkillNorm
50
450
1.00M
9.3
1.1
Table 2: The five released benchmarks. Queries are ranked against the same corpus of 13,891 ESCO v1.1.0 skills, so the released qrels are the full query × corpus grid, with every pair the prefilter did not surface carried at grade 0. Judged pairs counts the pairs the cascade actually graded (queries × the top-2000 window, 4.3 M in total). The last two columns are test-split means per query: targets graded ≥2 , and targets graded 4. TechWolf ships no validation split.
Preferred label
Alternative label
Div.
Cos.
Retained
provide periodontal treatment
manage gum conditions
1.00
0.64
install crane equipment
build lifting machinery
1.00
0.66
pest management techniques
extermination methods
1.00
0.56
monitor crops
checking crop growth
0.75
0.74
welcome restaurant guests
greet and seat guests
0.83
0.68
Table 3: ESCO preferred/alternative pairs at each filtering stage. Divergence ≥0.7 removes morphological variants (middle block), and cosine ≥0.5 removes pairs sharing no semantic content despite different wording (bottom block). The top block satisfies both.
#
Team
Macro
House
Tech
TechWolf
SkillSkape
SkillNorm
1
thylinao s
0.8458
0.8387
0.8390
0.8347
0.8683
0.8480
2
qihang-wang
0.8323
0.8405
0.8337
0.8197
0.8585
0.8092
3
yeyang s
0.8224
0.8226
0.8261
0.8030
0.8586
0.8018
4
aiden-lee
0.8191
0.8114
0.7903
0.8109
0.8316
0.8514
5
elara96
0.8182
0.8015
0.8157
0.8022
0.8443
0.8274
6
carolzz
0.8132
0.7982
0.8106
0.8026
0.8442
0.8101
Table 4: Filtered leaderboard: graded nDCG@100 per task and macro-averaged, best valid submission per account, plus the organiser baselines. Teams are named by their CodaBench handle throughout. The winning entry is the work of Maksim Silchenko (National University of Singapore), competing solo as team thylinao , who documents the system in [ 20 ] . s self-declared student team. The ConTeXT-match row is the pre-released checkpoint. Retrained with a tuned schedule the same method reaches 0.620 (Section 6 ).
Figure 3: Graded nDCG@100 against MAP on the public binary positives, both macro-averaged over the four extraction tasks (SkillNorm has no binary counterpart and is excluded from both axes). One point per valid submission. The two metrics agree on the broad strokes and disagree exactly where a leaderboard matters: the graded winner is 12th by binary MAP, and the binary-MAP leader (leonshangguan) is 16th on the graded board.
Figure 4: Grade composition of the 100 slots that nDCG@100 reads (mean per query, macro over the four extraction tasks) for the ideal ranking, the best and median valid submissions, and the organiser baselines. The ideal window contains essentially no grade-0 items, and the ranking systems differ mostly in how many grade-0 items they let in (9 for the winner, 17 at the median, 34–37 for the trained baselines, 79 for BM25) while everyone finds nearly all ≈ 2.5 grade-4 slots.
System
nDCG@100
Backbones
Params (B)
× baseline
thylinao ∗
0.8458
6
13.9
127
qihang-wang ∗
0.8323
6
35.8
327
yeyang
0.8224
6
45.6
417
aiden-lee
0.8191
9
243.7
2226
elara96 ∗
0.8182
7
53.7
490
carolzz
0.8132
7
52.2
476
Table 5: Inference-time size of every submitted system whose code could be read. Params is the base-checkpoint footprint (distinct pretrained backbones, billions), and × baseline is the ratio to CurriculumMatch. † a lower bound: the pipeline also calls a hosted frontier model with no published weight count. ∗ includes at least one checkpoint sized from file bytes rather than exact metadata.
Figure 5: Size against score, log scale. Hollow dotted markers are estimates or lower bounds. BM25 ( 0.2250 , no parameters) cannot be placed on a log axis and is omitted.
Information Retrieval (IR) Evaluation measures how well search systems find, rank, and cover useful evidence across queries, collections, and settings.
As large language model agents gain access to increasingly large skill libraries, retrieving the right skill becomes critical to reliable capability selection and execution. Existing retrievers often treat skill contents as ordinary documents, overlooking their highly regular structure: shared descriptive patterns recur across many skills while providing little evidence for distinguishing the required capability. We show that this shared descriptive background is reflected in dense relevance scores, induces a pronounced energy gap between queries and skill documents, and obscures discriminative signals, especially for structurally similar hard negatives. Based on this observation, we propose SkillSight, a training-free retrieval framework that calibrates shared background in both semantic and lexical spaces. Semantic Background Calibration estimates a background subspace from generic tokens identified by IDF, reducing similarity induced by shared descriptive patterns, while Lexical Evidence Calibration downweights shared background tokens to recover discriminative token-level evidence. Experiments on SRA-Bench and SkillBench-Supp demonstrate consistent improvements across retrieval metrics, with SkillSight improving Recall@10 by up to 20.21 percentage points over the original dense retriever. It is up to 1,248 times faster than the Dense + Reranker baseline. In end-to-end evaluation, SkillSight achieves the best overall performance across three agent models and outperforms LLM Selection by up to 4.97 percentage points. These results identify shared descriptive background as a source of ranking interference in skill retrieval and demonstrate that calibrating it enables accurate and efficient skill selection without additional training. Our code can be found at https://github.com/xiaojinying/SkillSight
Jinying Xiao, Bin Li, Xiaopeng Li +7
1National University of Defense Technology · 2Qinghai Normal University · 3Xizang University
Skills provide LLM agents with reusable procedural knowledge, making appropriate skill selection essential for reliable agent behavior. As skill libraries expand, distinguishing semantically similar candidates becomes increasingly difficult. Existing methods encode or rewrite skill descriptions for relevance-based ranking, yet heterogeneous descriptions can obscure capability differences. To address this challenge, we propose SKALOS, a graph-native reranker that compares skills via explicit capability evidence. Its multi-hierarchy graph aligns equivalent expressions across skills while preserving skill-specific capability evidence. Since individually relevant capability elements may not jointly support a query, semiring-based capability witness learning scores each branch by its weakest evidence, checks joint support for the extracted requirements within one implementation, and selects the strongest alternative. To reduce the influence of extraction errors, we develop ranking-feedback graph calibration that adjusts unreliable evidence for candidate comparison. Experiments on SkillRet show that SKALOS improves NDCG@10 by 2.68 percentage points over a matched text-only reranker. Witness-guided skill selection also increases ALFWorld task success rate from 22% to 29%.
Wenhan Xiao, Ziwei Zhang, Xingcheng Fu +5
Beihang University, Beijing, China · Guangxi Normal University, Guilin, China · Nankai University, Tianjin, China +1
Open-ended reward modeling requires judges that can follow subtle, domain-specific preferences when verifiable answers are unavailable. Existing rubric-based methods often address this by generating criteria online for each query, but the extra generation step can add inference overhead and produce rigid or misaligned guidance. We introduce Eval-Skill, an exploration-guided method that synthesizes reusable evaluation skills for reward modeling and reframes reward guidance as context evolution rather than parameter training or per-query rubric generation. Using only 100 cases per domain for skill evolution, Eval-Skill synthesizes reusable domain-level evaluation skills through two progressive stages, workflow generation followed by principle generation, with exploration and selection interleaved across both stages. Once generated, a skill is directly injected into the judge context. Across multiple RM benchmarks, Eval-Skill consistently improves diverse judge backbones; on RewardBench 2, it yields significant gains over vanilla judging for each main backbone (+13.44% for Qwen3-8B, and 18.51% for DeepSeek-V4-Flash). Further analyses of evolution-time scaling, generalizability, and transferability show that compact evaluation skills offer an efficient new paradigm for LLM-based evaluation. Code is available at https://github.com/xing-stellus-yue/Eval-Skill.