SNO-PI: Label-Scarce SNOMED CT Entity Linking with Pre-trained Encoders and Curated Terminology Knowledge
Organizations: The University of Queensland, Greater Brisbane Clinical School, Brisbane, Australia
Abstract
We present SNO-PI, the top-performing submission on SNOMED International's SNOMED CT Entity Linking Benchmark as of 18 October 2026. SNO-PI achieves a macro character IoU of 0.5387, outperforming the organisers' reference system (0.4582; based on the 2024 challenge winner's approach) by 18% relative. Linking clinical mentions to SNOMED CT is a zero- and few-shot problem over a very large label space: the 272 annotated MIMIC-IV discharge summaries cover only 6,595 of the roughly 219,000 eligible concepts, and a third of those occur only once. This motivates a design built on transfer from pre-trained models and human-curated knowledge. SNO-PI combines high-recall span proposal, SapBERT candidate retrieval and listwise ensemble reranking with three BERT-family encoders, each further pre-trained with masked language modelling on unlabelled MIMIC-IV notes and then on SNOMED CT synonyms and hierarchy before task fine-tuning. We find that curated knowledge helps most on concepts never seen in training, yet these concepts remain the main source of error.
Figures & tables
| Knowledge source | Where it enters | Measured effect |
|---|---|---|
| SNOMED terms and synonyms | span proposal, retrieval index, option text | only source of candidates for unseen concepts (R@10 .77) |
| UMLS synonym sets (via SapBERT) | retrieval | R@10 .969 overall |
| Pre-trained encoders (general and biomedical) | reranker initialisation, ensemble | prior that SNOMED pre-training builds on (SNOMED pre-training row); BioLinkBERT best on unseen in a controlled base-size comparison (.784) |
| 50k unlabelled notes (MLM) | tagger and encoders | tagger F1 +.009; small but consistent reranker gains |
| SNOMED pre-training | reranker initialisation | concept acc. .26 .85 / .71 (seen / unseen) with no labels (Table 2 ); +.015 macro after fine-tuning (ablation) |
| SNOMED hierarchy | scope filter, parent in option text, pre-training hard negatives | scope via is-a keeps the 22% of gold annotations a semantic-tag filter would drop; part of SNOMED pre-training gain (+.015, above) |
| Concept acc. | Linked | ||
|---|---|---|---|
| Stage | seen | unseen | (unseen) |
| SapBERT retriever (top-1) | .285 | .550 | .216 |
| C-DAPT, raw encoder | .262 | .266 | .000 |
| + SNOMED stage | .849 | .705 | .106 |
| + note fine-tuning | .948 | .829 | .725 |
| System | Macro | Weighted |
|---|---|---|
| SNO-PI (rank 1) | 0.5387 | 0.6629 |
| Rank 2 entry | 0.4985 | 0.6423 |
| Rank 3 entry | 0.4695 | 0.6273 |
| Rank 4 entry | 0.4657 | 0.6165 |
| Organisers’ reference (Super Dictionary) | 0.4582 | 0.6234 |
| Rank 6 entry | 0.4214 | 0.5831 |
| Local | Hidden test | |
|---|---|---|
| Development stage | dev / test | macro / wtd |
| Three-encoder ensemble | .504 / .511 | .500 / .628 |
| + six members, wider context | .532 / .530 | – |
| + retrained on all 272 notes | – | .535 / .658 |
| + Qwen3-8B member (final) | – | .539 / .663 |
| Dev | Test | |
|---|---|---|
| Gold spans in span set (exact) | 95.2% | 94.2% |
| Retrieval recall@10 (seen / unseen) | .974 / .768 | .969 / .766 |
| Linked to gold, seen concepts | .888 | .858 |
| Linked to gold, unseen concepts | .678 | .685 |
| Non-gold spans correctly rejected | 91.0% | 91.6% |
| Macro IoU, as is | .532 | .530 |
| Idea | Outcome |
|---|---|
| sap4 : our own SapBERT retrained on UMLS 2025AB with harder negatives, then mention-tuned on gold spans | UMLS held-out acc@1 69.2% 80.8%, but almost all of the gain is outside SNOMED, and harder negatives lowered unseen-concept recall. Mention tuning gave the best retriever measured (R@10 .955 .962; unseen .738 .785), yet only +.001 / +.002 in the ensemble. |
| SNOMED is-a hierarchy encoder (HiT [ 21 ] ) as retriever | Worse than SapBERT: R@10 .950 vs .955, unseen .716 vs .738. |
| Hierarchy-aware SNOMED pre-training loss (soft targets, relation margins); pre-training on every in-scope concept | Within seed noise; unseen accuracy unchanged. |
| KRISSBERT [ 22 ] : contextual retrieval as a second retriever, with source-agreement features | Recovers 366 of 1,058 SapBERT misses (only 85 unseen); +.005 per three-model ensemble, 0 in the six-model ensemble. |
| Hierarchy near-miss resolution (parent/child pooling, specificity preference) | Oracle +.085; every rule +.0005, as annotators split a phrase between neighbouring concepts. |