UMCR: Universal Medical Concept Recognition through Guideline Integration and Agentic Linking

Fabrice Lamarche; Amal Zouaq

cs.CLOct 12, 2026

UMCR: Universal Medical Concept Recognition through Guideline Integration and Agentic Linking

Authors: Fabrice Lamarche, Amal Zouaq

Organizations: LAMA-WeST Lab, Polytechnique Montréal, Mila – Quebec AI Institute

Abstract

Medical Concept Recognition is the task of extracting and grounding spans in medical text to entries in a standard ontology. Current methods are generally developed towards a single ontology and source text type (clinical notes, drug labels, etc.) The central challenge is therefore to build methods that adapt to any target ontology without requiring ontology-specific retraining. More recent approaches like REAL and AutoPCR make direct use of Large Language Models and can theoretically adapt to any ontology choice but suffer from a lack of generalization. We propose UMCR, a pipeline that allows extraction and linking to any target ontology using any source text variety, utilizing direct guideline integration and an agentic linking module. We evaluate our method on the recent MedPath benchmark, covering a wide variety of medical text types and vocabularies, and show that our method achieves strong performance across these tasks. This ability unlocks interoperability in the healthcare domain, allowing for extraction and alignment between different medical standards. Our method also reduces dependence on parametric LLM memory, thanks to improved interaction with the target ontology compared to other proposed approaches.

Figures & tables

Panel of Figure 1 — Figure 1: Concept Recognition Example.

Panel of Figure 2 — Figure 2: UMCR pipeline.

Name (abbrev.)	Sents	Domain	Target ontology
ShAReCLEF (SC) Suominen et al. (2013)	7,957	Hospital discharge notes	SNOMED CT
Mantra-GSC (MAN) Kors et al. (2015)	599	Patents, drug labels	UMLS
MedMentions (MedM) Mohan and Li (2019)	8,863	PubMed abstracts	UMLS
COMETA (COM) Basaldella et al. (2020)	4,226	Reddit, Twitter posts	SNOMED CT
MIMIC-IV-EL (MIM) Davidson et al. (2025)	20,404	ICU discharge notes	SNOMED CT
TAC-ADR (ADR) Roberts et al. (2017)	8,157	FDA drug labels	MedDRA, RxNorm

Table 1: Evaluation datasets.

	MedPath									Phenotype		Overall
Model	SC	MAN	MedM	COM	ADR	CAD	MIM	NCBI	CDR	BioC	GSC	Avg	Rank
LLM-based
UMCR	0.56	0.61	0.44	0.34	0.66	0.27	0.35	0.50	0.58	0.70	0.60	0.51	1.18
AutoPCR	0.40	0.46	0.35	0.15	0.38	0.23	0.19	0.40	0.43	0.69	0.69	0.40	2.55
REAL	0.26	0.41	0.11	0.30	0.10	0.26	0.19	0.27	0.26	0.64	0.36	0.29	4.00
Rule-based

Table 2: Overlap Linking F1 Score. Bold marks the best measured score per dataset, underline the second best. Grey cells mark in-distribution results. Avg represents the average score across the datasets (

\uparrow

), and Rank is the average ranking per dataset for each model (

\downarrow

). Qwen3.8-27B is the backbone model for all LLM-based methods.

Dataset	bare	compiled	full
ShAReCLEF	0.48	0.54	0.50
NCBI Disease	0.35	0.39	0.38
BioC-GS	0.62	0.64	0.62
GSC+	0.54	0.59	0.53

Table 3: Guideline Compilation Ablation: overlap linking F1 Score after Stage 3. Bold marks the best configuration.

Dataset	Stage 2	Stage 3	Stage 4
Overlap Mention Detection F1 Score
ShAReCLEF	0.70	0.70	0.70
NCBI Disease	0.63	0.67	0.79
BioC-GS	0.81	0.82	0.85
GSC+	0.63	0.63	0.65
Overlap Linking F1 Score

Table 4: Stage Ablation Results. Stage 2: Mention Detection, Stage 3: Agentic Linking, Stage 4: Filtering.

Dataset	Agentic Linking	Fixed
ShAReCLEF	0.54	0.49
NCBI Disease	0.39	0.31
BioC-GS	0.64	0.52
GSC+	0.59	0.55

Table 5: Linking Type Ablation: Overlap linking F1 Score after Stage 3.

Figure 3: Percentage of graph exploration tool calls per mention.

Dataset	Optional	Encouraged	Mandatory
ShAReCLEF	0.50	0.51	0.54
NCBI Disease	0.38	0.39	0.39
BioC-GS	0.59	0.60	0.64
GSC+	0.56	0.58	0.59

Table 6: Agentic Linking Ablation (Prompt Choice): Overlap linking F1 Score after Stage 3

Figure 4: Backbone LLM Ablation: Overlap linking F1 Score.

\Delta

measures difference between both models.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Calls/sent.	$\approx$ s/sent
ShAReCLEF	0.4	$\sim$ 2s
NCBI Disease	3.1	$\sim$ 16s
BioC-GS	3.6	$\sim$ 18s
GSC+	2.6	$\sim$ 13s

Appendix

Table 7: Calls and estimated processing time per sentence.

Parameter	Value
Precision	BF16
Thinking mode	off
temperature	0.7
top_p	0.80
top_k	20
min_p	0.0

Appendix

Table 8: Qwen3.8-27B generation configuration.

Tool	Arguments	Returns
search	query , k (=10)	Top-k candidate concepts (exact matching + vector search + BM25)
get_concept	id	Name, definition, synonyms, and semantic types of id concept
get_parents	id	Parent concepts of a candidate
get_children	id	Children concepts of a candidate
get_siblings	id	Siblings concepts of a candidate
adjust_span	text	Refines span boundaries of current mention

Appendix

Table 9: Tools available to the Stage 3 Agentic Linking Module.

Dataset	Agent %	Calls	search	get_concept	Graph	commit/unlink
ShAReCLEF	20.2	5.26	27.6	37.9	18.9	15.6
NCBI Disease	13.0	3.63	34.7	30.3	7.5	27.6
BioC-GS	35.9	4.81	34.6	26.3	21.5	17.6
GSC+	24.7	6.01	47.3	20.4	21.6	10.8

Appendix

Table 10: Stage 3 Tool Use per dataset. Agent % = percentage of Stage 2 mentions that enter Stage 3; Calls = Average # of calls per mention; Last four columns are tool call proportion over Stage 3.

	MedPath									Phenotype		Overall
Model	SC	MAN	MedM	COM	ADR	CAD	MIM	NCBI	CDR	BioC	GSC	Avg	Rank
LLM-based
UMCR	0.73	0.73	0.66	0.49	0.74	0.48	0.61	0.78	0.82	0.85	0.67	0.69	1.18
AutoPCR	0.52	0.64	0.64	0.22	0.48	0.37	0.38	0.70	0.65	0.78	0.74	0.56	3.09
REAL	0.39	0.60	0.20	0.46	0.17	0.61	0.34	0.42	0.40	0.75	0.44	0.43	4.45
Rule-based

Appendix

Table 11: Overlap Mention-Detection F1 Score. Bold marks the best measured score per dataset, underline the second best. Grey cells mark in-distribution results. Avg represents the average score across the datasets, and Rank is the average ranking per dataset for each model.

	MedPath									Phenotype		Overall
Model	SC	MAN	MedM	COM	ADR	CAD	MIM	NCBI	CDR	BioC	GSC	Avg	Rank
LLM-based
UMCR	0.61	0.66	0.51	0.42	0.67	0.28	0.42	0.58	0.75	0.62	0.62	0.56	1.45
AutoPCR	0.45	0.54	0.53	0.18	0.41	0.24	0.31	0.51	0.59	0.62	0.70	0.46	2.55
REAL	0.32	0.48	0.15	0.40	0.15	0.47	0.25	0.31	0.32	0.47	0.38	0.34	4.36
Rule-based

Appendix

Table 12: Exact-match Mention-Detection F1 Score. Bold marks the best measured score per dataset, underline the second best. Grey cells mark in-distribution results. Avg represents the average score across the datasets, and Rank is the average ranking per dataset for each model.

	MedPath									Phenotype		Overall
Model	SC	MAN	MedM	COM	ADR	CAD	MIM	NCBI	CDR	BioC	GSC	Avg	Rank
LLM-based
UMCR	0.54	0.60	0.40	0.31	0.62	0.22	0.30	0.40	0.56	0.56	0.58	0.46	1.36
AutoPCR	0.38	0.44	0.32	0.14	0.36	0.20	0.19	0.34	0.41	0.60	0.67	0.37	2.55
REAL	0.24	0.39	0.09	0.28	0.09	0.24	0.17	0.21	0.23	0.43	0.32	0.24	4.09
Rule-based

Appendix

Table 13: Exact-match Linking F1 Score. Bold marks the best measured score per dataset, underline the second best. Grey cells mark in-distribution results. Avg represents the average score across the datasets, and Rank is the average ranking per dataset for each model.

Panel of Figure 5 — Figure 5: Stage 2 mention-extraction prompt.

Panel of Figure 6 — Figure 6: Stage 3 agentic-linking prompt. Lower panel: graph-exploration variants from the agentic ablation (Optional, Encouraged, Mandatory=default).

Panel of Figure 7 — Figure 7: Stage 4 guideline-filter prompt (brief arm).

Panel of Figure 8 — Figure 8: Guideline sources: paper sentence (MedMentions), a paper section (Mantra GSC), and a full PDF manual (ShAReCLEF).

Panel of Figure 9 — Figure 9: Compiled Guideline JSON format (placeholders; no dataset content).

Panel of Figure 10 — Figure 10: Guideline compilation: source guideline to compiled JSON (Stage 0).

Panel of Figure 11 — Figure 11: Bare guideline: stage goals only, no compiled brief.

Biomedical Entity Linking

See all 11 papers

MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction
Apr 5, 2026Zhichao Yang, Gregory D. Lyng, Sanjit Singh Batra +1Clinical Information Extraction Clinical Language Model Evaluation
Optum AI
MedUAG: Unified Understanding and Generation for Medical Multimodal Models
Aug 19, 2026Zijie Meng, Yuncheng Zhang, Hualiang Wang +8Unified Multimodal Models Multimodal Generation
Zhejiang University · Hong Kong University of Science and Technology · Tsinghua University +1
CUICurate: A GraphRAG-based Framework for Automated Clinical Concept Curation for NLP applications
Feb 20, 2026Victoria Blake, Jamie Novak, Mathew Miller +2Clinical NLP Graph Retrieval-Augmented Generation
Centre for Big Data Research in Health, University of New South Wales, Sydney, New South Wales, Australia · Eastern Heart Clinic, Prince of Wales Hospital, Randwick, NSW, Australia · Department of Cardiology, Prince of Wales Hospital, South Eastern Sydney Local Health District, Randwick, NSW, Australia +3

Apr 5, 2026J/K move · Enter open · S save

MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction

Zhichao Yang, Gregory D. Lyng, Sanjit Singh Batra, Robert E. Tillman

Optum AI

Clinical Information Extraction Clinical Language Model Evaluation LLM Grounding Implicit Reasoning in Language Models

Open paper AskPDF arXiv

Medical concept extraction from electronic health records underpins many downstream applications, yet remains challenging because medically meaningful concepts are frequently implied rather than explicitly stated in medical narratives. Existing benchmarks with human-annotated evidence spans underscore the importance of grounding extracted concepts in medical text. However, they predominantly focus on explicitly stated concepts instead of implicit concepts. We present MedicalBench, a benchmark for medical concept extraction with evidence grounding that evaluates implicit medical reasoning. MedicalBench formulates medical concept extraction as a verification task over medical note-concept pairs, coupled with sentence-level evidence identification. Built from MIMIC-IV discharge summaries and human-verified ICD-10 codes, the dataset is curated through a multi-stage large language model (LLM) triage pipeline followed by medical annotation and expert review. It deliberately includes implicit positives, semantically confusable negatives, and cases where LLM judgments disagree with medical expert assessments. We define two complementary evaluation tasks: (1) medical concept extraction and (2) sentence-level evidence retrieval, enabling assessment of both correctness and interpretability. Benchmarking state-of-the-art LLMs reveals that performance remains modest, highlighting the difficulty of extracting implicitly expressed concepts. We further show that performance is largely invariant to note length, indicating that MedicalBench isolates reasoning difficulty rather than superficial confounders. MedicalBench provides the first systematic benchmark for implicit, evidence-grounded medical concept extraction, offering a foundation for developing medical language models that can both identify medically relevant concepts and justify their predictions in a transparent and medically faithful manner.

Abstract

Biomedical Entity Linking

Explore similar work

MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction

MedUAG: Unified Understanding and Generation for Medical Multimodal Models

CUICurate: A GraphRAG-based Framework for Automated Clinical Concept Curation for NLP applications