cs.AINov 2, 2026

Grading the Unannotated: Insights from the WorkRB Skill Extraction Challenge at RecSys-HR 2026

Authors: Warre Veys, Luis Gasco, Jens-Joris Decorte

Organizations: TechWolf, 9000 Gent, Belgium · Ghent University – imec, 9052 Gent, Belgium · Avature, Barcelona, Spain

Abstract

Skill extraction benchmarks score models against a few human-annotated positives and treat every other skill in the vocabulary as equally wrong. At the scale of ESCO (13,939 skills) this binary view can reward models that rank the annotated positive high for the wrong reason, and hides semantically nonsensical predictions. The WorkRB Challenge at RecSys-HR 2026 replaces it with graded relevance: every candidate skill in a model's ranking window carries a 0 to 4 grade, and the leaderboard scores nDCG@100 against these grades. Since ternary judgements proved too coarse for objective annotation, relevance is decomposed into a cascade of four binary questions (same domain, activity mentioned, right granularity, replacement of a gold positive) that a smaller LLM judge can answer reliably; the labels are validated with blind human and frontier-model re-annotation. This paper documents the annotation methodology, the construction of five benchmark datasets (four with vacancy sentences as queries, one with lexically distant skill surface forms), the validation of the LLM cascade, and insights from the submissions. The field converged on multi-stage retrieve-and-rerank pipelines, whose gains come from clearing nonsense out of the ranking window rather than finding more positives: at the top of the leaderboard, the graded ordering is uncorrelated with what binary evaluation would have said. Graded labels expose much bigger gaps between top submissions and baselines than binary evaluation does, and show that semantic-understanding skill recommendation is still wide open. The clear next direction is bringing efficiency back: every submitted system we could size uses at least 42x, typically several hundred times, more parameters than the baselines it beats.

Figures & tables

Explore similar work

CardsList
  1. SkillSight: Calibrating Generic Content Bias for Skill Retrieval

    Jul 21, 2026Jinying Xiao, Bin Li, Xiaopeng Li +7Lexical RetrievalLLM Agent Skill Retrieval

  2. SKALOS: Learning to Distinguish Similar Skills with Capability Graphs

    Oct 26, 2026Wenhan Xiao, Ziwei Zhang, Xingcheng Fu +5LLM Agent Skill RetrievalTool Selection for Language Model Agents

  3. Beyond Rubrics: Exploration-Guided Evaluation Skills for Reward Modeling

    Jun 5, 2026Xing Yue, Linjuan Wu, Daoxin Zhang +2Reward ModelingLLM Evaluation