From Lexical Baselines to Agentic Retrieval-Augmented Generation: Structured Skill and Responsibility-Level Extraction with the SFIA Framework
Authors: Ranuga Disansa, U. S. Samarasinghe, Lasith Gunawardena
Organizations: Informatics Institute of Technology, Colombo, Sri Lanka · Department of Information Technology, University of Sri Jayewardenepura, Colombo, Sri Lanka
Automated skill extraction underpins workforce planning, yet most systems represent skills as flat labels with no notion of the responsibility level at which a skill is practiced. The Skills Framework for the Information Age (SFIA) captures exactly this dimension, defining 147 professional skills across seven responsibility levels, but no automated LLM-based extraction targeting SFIA has been reported. We formalize the task as structured prediction of (skill, level) pairs from free text and ask three questions: how accurately can text be mapped onto SFIA's closed vocabulary, which strategies reliably predict the level alongside the skill, and do agentic designs improve on simpler retrieval and prompting? We evaluate five strategies (a lexical baseline, dense retrieval with LLM reranking, a zero-shot schema-constrained LLM, single-agent agentic RAG, and a three-agent retriever--matcher--verifier crew) against expert-mapped European ICT role profiles, all drawing on an SFIA~9 corpus built by a fully automated agentic pipeline that we release. Retrieval-based matching identifies the most skills while generative strategies are markedly more precise; only strategies assigning the level as an explicit decision predict it reliably, with similarity-based selection more than twice as inaccurate; and the crew doubles latency without improving accuracy, so added agent roles do not automatically benefit closed-taxonomy matching. These results provide the first reproducible baseline for structured, level-aware skill extraction against SFIA.
Figures & tables
Fig. 1: Fully automated corpus-construction pipeline: a serverless crawler stores each documentation page as markdown and HTML; a context-scouting agent reads the non-skill pages and a synthesizer agent condenses its notes into a standing briefing; an extractor agent emits (code, level, text) records per skill page; a deterministic guard checks every pair against the SFIA 9 skill–level chart; a verifier agent enforces verbatim fidelity, with rejections dropped and counted.
Figure 2
Strategy
P
R
F1
MAE
Exact
W1
s/rec
BM25 lexical
0.141
0.133
0.128
0.94
0.260
0.875
0.001
Dense + rerank
0.266
0.243
0.232
1.60
0.181
0.436
6.17
Zero-shot LLM
0.436
0.117
0.176
0.74
0.368
0.895
0.23 a
Agentic RAG
0.361
0.172
0.213
0.68
0.526
0.816
167.1
Multi-agent crew
0.398
0.218
0.212
1.16
0.319
0.717
327.1
Macro-averages over the 30 roles; F1 is the mean of per-role F1, not the harmonic mean of the P and R columns. MAE/Exact/W1 = level MAE, exact and within-one level accuracy on correctly named skills; s/rec = mean wall-clock seconds per record. a From the sweep run; the final run repeated identical temperature-0 prompts and was largely served from the gateway response cache (0.05 s/record).
TABLE I: Best configuration per strategy on the 30-role harness. Each row is a single reporting run of the model its sweep selected; the sweeps (Tables II – IV ) are separate selection runs.
Model
F1
Lvl exact
Lvl MAE
failed
Gemma-4-26B-A4B
0.176
0.368
0.74
0
Nemotron-3-120B-A12B
0.174
0.438
0.88
0
Llama-3.1-8B-fast
0.166
0.194
1.64
0
Llama-4-Scout-17B
0.153
0.694
0.36
0
Qwen3-30B-A3B
0.152
0.400
0.93
0
Mistral-Small-3.1-24B
0.139
0.393
0.75
0
TABLE II: Zero-shot LLM model sweep (30 roles).
Model
F1
Lvl exact
Lvl MAE
s/rec
failed
GLM-4.7-Flash
0.236
0.37
0.95
227
1
Mistral-Small-3.1-24B
0.224
0.25
0.95
14
1
Kimi-K2.6
0.216
0.52
0.67
151
0
Nemotron-3-120B-A12B
0.188
0.53
0.65
155
0
Gemma-4-26B-A4B
0.176
0.38
0.84
244
4
GLM-5.2
0.086
0.50
0.50
20
22
TABLE III: Single-agent agentic RAG model sweep (30 roles).
Model
F1
Lvl exact
Lvl MAE
s/rec
failed
Kimi-K2.6
0.189
0.29
1.20
342
0
Nemotron-3-120B-A12B
0.172
0.32
0.87
401
3
GLM-4.7-Flash
0.160
0.34
0.92
315
1
GLM-5.2
0.073
0.50
0.50
44
24
Kimi-K2.7-Code
0.059
0.40
0.60
78
21
Failed roles score as empty predictions (see Table II note).
TABLE IV: Multi-agent crew model sweep (30 roles, one shared model).