Are We Done with UMLS Injection into Language Models for Medical Question Answering?

Yiran Zeng; Giacomo Frisoni; Lorenzo Molfetta; Gianluca Moro

cs.AIOct 24, 2026

Are We Done with UMLS Injection into Language Models for Medical Question Answering?

Authors: Yiran Zeng, Giacomo Frisoni, Lorenzo Molfetta, Gianluca Moro

Organizations: Department of Computer Science and Engineering, University of Bologna, Italy

Abstract

Maybe not. The Unified Medical Language System (UMLS) has become an oracle for millions of clinicians and health system developers worldwide. Researchers have increasingly used knowledge graphs derived from UMLS to enhance language models' factual accuracy and reasoning traceability in high-stakes tasks. Medical Question Answering (MQA) is a prime testbed for integration, with benchmarks usually involving professional exercises such as disease diagnosis and drug prescription from patient history. Despite the assumed benefits, empirical evidence shows that UMLS-based graph injection typically yields modest performance gains. This paper reviews existing MQA work, critically analyzing the selected UMLS sources and their usage with language models, and probes each of these choices in isolation. We uncover methodological issues never discussed in prior work, from low UMLS coverage to weak subgraph retrieval pipelines. Based on our findings, we provide actionable recommendations to unlock the untapped potential of UMLS as an external knowledge source.

Figures & tables

Figure 1: Review pipeline and venue profile of the surveyed literature. NLP-oriented conferences account for the majority of the corpus.

Panel of Figure 2 — Figure 2: UMLS data inconsistency across publications and widespread under-reporting. (a) Reporting of UMLS release and filtering strategy. (b) Difference in CUI and vocabulary count over seven consecutive releases.

Panel of Figure 3 — Figure 3: The gap between UMLS expressiveness and its use in language model injection. Top: the 6 CUI-centric tables of the UMLS Metathesaurus (2025AB release) illustrated on the concept Esophageal Achalasia; the remaining 17 tables are omitted as they serve infrastructure, versioning, or cross-vocabulary alignment purposes rather than encoding concept-level knowledge. Bottom: usage frequency of each table across surveyed publications.

Figure 4: Construction pattern for UMLS use in MQA. Surveyed systems ground input text to UMLS concepts, construct a task-specific graph or evidence object, and expose the resulting representation through textual, structural, or training-time injection.

Panel of Figure 5 — Figure 5: Linked UMLS 2020AA concepts per instance on the test sets of three multiple-choice MQA benchmarks under two entity linkers. Dots denote mean; bars span the minimum and maximum. $\mathcal{Q}$ and $\mathcal{A}$ denote question stems and answer options, respectively.

Linker	Links	C	G	W	C/Links	Grounded
MetaMap	180	64	45	71	35.6%	40/40
scispaCy	122	57	23	42	46.7%	36/40
Union	219	76	50	93	34.7%	40/40

Table 1: Manual linking precision on MedQA dev answer options (UMLS 2023AA). 83 of the 219 links are returned by both linkers. Grounded : options with at least one link labeled C.

	MedQA (dev, $n{=}1{,}272$ )				PubMedQA (all)
	Question		Options		Question
Configuration	$\mu$	distinct	$\mu$	distinct	$\mu$	distinct
UMLS release (all sources)
2020AA †	57.67	9,338	9.96	6,913	68.46	13,536
2021AA	71.76	11,760	12.46	8,794	78.77	16,453
2022AB	67.18	11,232	11.98	8,487	75.54	15,851

Table 2: Linked UMLS concepts per instance across four releases and two source subsets. scispaCy with the mention model held fixed and only the concept knowledge base swapped

\dagger

: its shipped default).

\mu

: mean per instance, with answer-side counts taken as the union across the four options. PubMedQA contributes question-side coverage only.

Panel of Figure 6 — Figure 6: Growth of UMLS neighborhoods under $K$ -hop expansion. Mean node and edge counts over PubMedQA test instances, seeded by scispaCy, for full UMLS 2025AB and the DRAGON-filtered subset. Error bars denote mean $\pm$ one standard deviation. Log scale.

KG	Connectivity rule	Nodes ( $\mu\pm\sigma$ )	Edges ( $\mu\pm\sigma$ )
UMLS 2025AB	All linked concepts	$46{,}871\pm 24{,}465$	$385{,}342\pm 268{,}713$
	Question $\to$ answer only	$537\pm 1{,}007$	$4{,}981\pm 16{,}715$
	Ratio	87 $\times$	77 $\times$
DRAGON	All linked concepts	$5{,}750\pm 3{,}197$	$16{,}823\pm 9{,}444$
	Question $\to$ answer only	$96\pm 58$	$94\pm 143$
	Ratio	60 $\times$	179 $\times$

Table 3: MedQA subgraph scale under two connectivity rules. Node and edge counts at

K{=}2

under pooled all-linked-concept versus question-to-answer retrieval.

		Release				Subset		No Graph
		20AA	21AA	22AB	23AA	Lv-0	DrugBank +DDB	Empty
QA-GNN	Dev	38.8 $\pm$ .3	38.2 $\pm$ .8	38.4 $\pm$ 1.0	38.0 $\pm$ .7	37.8 $\pm$ .6	38.3 $\pm$ .3	38.9 $\pm$ .6
QA-GNN	Test	38.1 $\pm$ .4	37.6 $\pm$ .6	38.7 $\pm$ .5	38.0 $\pm$ .9	37.5 $\pm$ 1.0	38.5 $\pm$ .6	37.4 $\pm$ .7
GREASELM	Test	39.3 $\pm$ 2.2	38.4 $\pm$ .6	37.7 $\pm$ 1.4	38.5 $\pm$ .8	38.1 $\pm$ 1.0	38.8 $\pm$ .5	36.7 $\pm$ .2

Table 4: End-to-end accuracy of two systems across UMLS constructions. MedQA accuracy (%), mean

\pm

standard deviation over three seeds. Releases abbreviated (e.g., 20AA for 2020AA). Empty keeps only the placeholder concepts used on linking failure.

Panel of Figure 7 — Figure 7: Record-level UMLS evidence contribution across readers and benchmarks. Accuracy gain (+%) and 95% CI over zero-shot on the baseline-wrong subset, for 10 readers from 5 families under 3 KI .

Model	Evidence view (+%)
Model	Rel.	$\mathcal{QA}$	+Meta
Qwen3.5-4B	25.7	22.9	22.9
Qwen3.5-9B	18.5	33.3	37.0
Llama-3.1-8B	21.2	18.2	15.2
MedGemma-4B	11.6	25.6	27.9
MedGemma-27B	15.4	23.1	19.2

Table 5: Evidence view under chain-of-thought prompting. Recovery rate (%) on a chain-of-thought pilot over a subsample of the MedQA test split, under scispaCy linking, best view per row in bold . Rel. : MRCONSO+MRREL neighborhoods;

\mathcal{QA}

: the same restricted to question-to-answer paths; +Meta : paths with MRDEF definitions and MRSTY semantic types.

Panel of Figure 8 — Figure 8: Question-to-answer path length effect on the Qwen3.5 ladder. Accuracy gain (+%) and 95% CI on MedQA over zero-shot across KSR configurations. Each $k$ setting uses only pairs where every answer option is reachable from a question node by an exact $k$ -hop path.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Structured search query. Used to retrieve literature on knowledge graphs and language models for question answering and reasoning from Scopus.

Dataset	Example
BioASQ	Is the monoclonal antibody Trastuzumab (Herceptin) of potential use in the treatment of prostate cancer? (A) Yes (B) No
PubMedQA	Are group 2 innate lymphoid cells ( ILC2s ) increased in chronic rhinosinusitis with nasal polyps or eosinophilia? (A) Yes (B) No
MedQA	The only immunoglobulin found as a dimer has what primary function? (A) Protect against invasive helminth infection? (B) Protect against viral infections (C) Inhibiting bacterial adherance and colonization of mucous membranes (D) Provides the most specific recognition to circulating antigens in the bloodstream
MedMCQA	Chronic urethral obstruction due to benign prismatic hyperplasia can lead to the following change in kidney parenchyma? (A) Hyperplasia (B) Hyperophy (C) Atrophy (D) Dyplasia
MedicationQA	what meloxicam look like? Product Characteristics Color YELLOW (light yellow) Score no score Shape OVAL Size 3mm Imprint Code S160
CovidQA	What plays the crucial role in the Mother to Child Transmission of HIV-1 and what increases the risk? DC-SIGNR plays a crucial role in MTCT of HIV-1 and that impaired placental DC-SIGNR expression increases risk of transmission.

Appendix

Table 6: Verbatim representative items from each QA benchmark used in the reviewed studies. Correct or reference answers are highlighted in green .

Stage	Report	See
UMLS	$\square$ Release version	§ 4
UMLS	$\square$ Subset: source vocabularies and filters on the concepts and relations	§ 4 , 5.1
	$\square$ Metadata: fields attached to each concept, e.g. definitions, semantic types	§ 3 , 6
NEL	$\square$ Linker: tool, non-default settings	§ 5.1
NEL	$\square$ Linking result: entity coverage, unlinked items	App. D , E
KSR	$\square$ Connectivity rule	§ 5.2

Appendix

Table 8: Minimum reporting checklist for UMLS-augmented MQA. Each item changes the UMLS evidence reaching the model and should be reported for a gain to be attributable to UMLS.

Span	System	Mean	SD	Min	Max	Var
Question stems	MetaMap	113.57	42.68	3	334	1821.34
	scispaCy	57.67	22.67	1	172	514.00
	Diff (M-S)	55.90	25.02	1	164	626.06
Answer options	MetaMap	13.82	11.46	0	109	131.29
	scispaCy	9.96	6.33	0	45	40.02
	Diff (M-S)	3.86	8.61	$-$ 20	64	74.12

Appendix

Table 9: Per-question entity counts on MedQA dev (

n{=}1{,}272

) using UMLS 2020AA. Answer-option counts are the union of the concepts linked across the four options, not their sum. Diff(M-S) gives the per-question difference between MetaMap and scispaCy.

System	Split	Answer-side Links		Question-side Links		Both Empty
System	Split	Present	Empty	Present	Empty	Both Empty
QA-GNN	Dev	2,056	3,032	4,872	216	148
	Test	2,096	2,996	4,924	168	119
	Train	16,115	24,597	38,836	1,876	1,320
GREASELM	Dev	2,055	3,033	4,872	216	148
	Test	2,090	3,002	4,924	168	119

Appendix

Table 10: Linking coverage on MedQA preprocessing artifacts from QA-GNN, GREASELM, and DRAGON, and our MetaMap processing. Present : at least one UMLS concept linked. Empty : no concept linked. Both Empty : neither question stem nor answer-option set produced any linked concept.

System	Scoring	Pre-cap	Final cap
QA-GNN	RoBERTa-large MLM	none	200
GREASELM	RoBERTa-large MLM	none	200
DRAGON	GloVe-50 dot prod.	300	200

Appendix

Table 11: Subgraph pruning across QA-GNN, GREASELM, and DRAGON.

Class	Paradigm	How UMLS reaches the model	$n$
I	Textual linearization	Serialized factual context	5
II	Structural injection	Explicit graph at inference	9
III	Parameter encoding	Training-time supervision	7
	Total		21

Appendix

Table 12: Injection mechanism choices. Counts are restricted to systems with explicit UMLS injection mechanisms.

Panel of Figure 10 — Figure 10: Zero-shot prompt with UMLS evidence for medical multiple-choice question answering.

Setting	Value
Serving	vLLM endpoint with prefix caching and chunked prefill
Context length	max_model_len=65536
Batching	max_num_seqs=64 , max_num_batched_tokens=65536
Cache	FP8 key-value cache
Decoding	temperature=1.0 , top_p=0.95 , top_k=20 , min_p=0.0 , presence_penalty=1.5 , repetition_penalty=1.0

Appendix

Table 13: Inference configuration. Serving and decoding settings used for all reader evaluations.

Knowledge Augmentation for Language Models

Knowledge augmentation for language models adds external or newly learned information to improve factuality, reasoning, and domain performance.

See all 178 papers

Injecting Structured Biomedical Knowledge into Language Models: Continual Pretraining vs. GraphRAG
Apr 3, 2026Jaafer Klila, Sondes Bannour Souihi, Rahma Boujelben +2Knowledge Augmentation for Language Models Continued Pretraining
Université Paris-Saclay, CEA, List Palaiseau, France · University of Sfax Sfax, Tunisia
Multilingual Medical Reasoning for Question Answering with Large Language Models
Dec 5, 2025Pietro Ferrazzi, Aitor Soroa, Rodrigo AgerriKnowledge Augmentation for Language Models Healthcare
Fondazione Bruno Kessler, Trento, Italy · HiTZ Center - Ixa, University of the Basque Country EHU, Spain
A Systematic Study of Retrieval Pipeline Design for Retrieval-Augmented Medical Question Answering
Apr 8, 2026Nusrat Sultana, Abdullah Muhammad Moosa, Kazi Afzalur Rahman +1RAG Evaluation Medical QA
Department of Mechatronics & Industrial Engineering, Chittagong University of Engineering & Technology, Chittagong, Bangladesh. · Department of Mechanical Engineering, Chittagong University of Engineering & Technology, Chittagong, Bangladesh.

Apr 3, 2026J/K move · Enter open · S save

Injecting Structured Biomedical Knowledge into Language Models: Continual Pretraining vs. GraphRAG

Jaafer Klila, Sondes Bannour Souihi, Rahma Boujelben, Nasredine Semmar+1

Université Paris-Saclay, CEA, List Palaiseau, France · University of Sfax Sfax, Tunisia

Knowledge Augmentation for Language Models Continued Pretraining Graph Retrieval-Augmented Generation Biomedical KGs Biomedical QA

Open paper AskPDF arXiv

The injection of domain-specific knowledge is crucial for adapting language models (LMs) to specialized fields such as biomedicine. While most current approaches rely on unstructured text corpora, this study explores two complementary strategies for leveraging structured knowledge from the UMLS Metathesaurus: (i) Continual pretraining that embeds knowledge into model parameters, and (ii) Graph Retrieval-Augmented Generation (GraphRAG) that consults a knowledge graph at inference time. We first construct a large-scale biomedical knowledge graph from UMLS (3.4 million concepts and 34.2 million relations), stored in Neo4j for efficient querying. We then derive a ~100-million-token textual corpus from this graph to continually pretrain two models: BERTUMLS (from BERT) and BioBERTUMLS (from BioBERT). We evaluate these models on six BLURB (Biomedical Language Understanding and Reasoning Benchmark) datasets spanning five task types and evaluate GraphRAG on the two QA (Question Answering) datasets (PubMedQA, BioASQ). On BLURB tasks, BERTUMLS improves over BERT, with the largest gains on knowledge-intensive QA. Effects on BioBERT are more nuanced, suggesting diminishing returns when the base model already encodes substantial biomedical text knowledge. Finally, augmenting LLaMA 3-8B with our GraphRAG pipeline yields over than 3 points accuracy on PubMedQA and 5 points on BioASQ without any retraining, delivering transparent, multi-hop, and easily updated knowledge access. We release the processed UMLS Neo4j graph to support reproducibility.

Abstract

Knowledge Augmentation for Language Models

Explore similar work

Injecting Structured Biomedical Knowledge into Language Models: Continual Pretraining vs. GraphRAG

Multilingual Medical Reasoning for Question Answering with Large Language Models

A Systematic Study of Retrieval Pipeline Design for Retrieval-Augmented Medical Question Answering