Injecting Structured Biomedical Knowledge into Language Models: Continual Pretraining vs. GraphRAG

Jaafer Klila; Sondes Bannour Souihi; Rahma Boujelben; Nasredine Semmar; Lamia Hadrich Belguith

cs.CLApr 3, 2026

Injecting Structured Biomedical Knowledge into Language Models: Continual Pretraining vs. GraphRAG

Authors: Jaafer Klila, Sondes Bannour Souihi, Rahma Boujelben, Nasredine Semmar, Lamia Hadrich Belguith

Organizations: Université Paris-Saclay, CEA, List Palaiseau, France · University of Sfax Sfax, Tunisia

Abstract

The injection of domain-specific knowledge is crucial for adapting language models (LMs) to specialized fields such as biomedicine. While most current approaches rely on unstructured text corpora, this study explores two complementary strategies for leveraging structured knowledge from the UMLS Metathesaurus: (i) Continual pretraining that embeds knowledge into model parameters, and (ii) Graph Retrieval-Augmented Generation (GraphRAG) that consults a knowledge graph at inference time. We first construct a large-scale biomedical knowledge graph from UMLS (3.4 million concepts and 34.2 million relations), stored in Neo4j for efficient querying. We then derive a ~100-million-token textual corpus from this graph to continually pretrain two models: BERTUMLS (from BERT) and BioBERTUMLS (from BioBERT). We evaluate these models on six BLURB (Biomedical Language Understanding and Reasoning Benchmark) datasets spanning five task types and evaluate GraphRAG on the two QA (Question Answering) datasets (PubMedQA, BioASQ). On BLURB tasks, BERTUMLS improves over BERT, with the largest gains on knowledge-intensive QA. Effects on BioBERT are more nuanced, suggesting diminishing returns when the base model already encodes substantial biomedical text knowledge. Finally, augmenting LLaMA 3-8B with our GraphRAG pipeline yields over than 3 points accuracy on PubMedQA and 5 points on BioASQ without any retraining, delivering transparent, multi-hop, and easily updated knowledge access. We release the processed UMLS Neo4j graph to support reproducibility.

Figures & tables

Figure 1: Overview of the UMLS-Based Knowledge Injection Pipeline: Continual Pretraining on BERT/BioBERT and GraphRAG.

Figure 2: Pipeline for constructing a biomedical knowledge graph from UMLS Metathesaurus.

Figure 3: Example of generating a textualized triple from the KG.

Dataset	Task	Train	Dev	Test	Evaluation Metrics	Number of Classes
EBM PICO	PICO	339,167	85,321	16,364	Macro F1 word-level	3
BIOSSES	Sentence Similarity	64	16	20	Pearson	N/A (labels continus)
HoC	Document Classification	1,295	186	371	Average Micro F1	10
DDI	Relation Extraction	25,296	2,496	5,716	Micro F1	18
PubMedQA	Question Answering	450	50	500	Accuracy	3
BioASQ	Question Answering	670	75	140	Accuracy	2

Table 1: Summary of the biomedical datasets used with the number of classes and evaluation metrics.

Dataset	BERT	BERTUMLS
PubMedQA	$50.72\pm 0.17$	$\textbf{53.00}\pm 0.0$
BioASQ	$66.57\pm 0.93$	$\textbf{71.57}\pm 1.17$
EBM PICO	$71.95\pm 0.06$	$\textbf{72.18}\pm 0.05$
DDI	$77.99\pm 0.19$	$\textbf{78.67}\pm 0.27$
BIOSSES	$82.31\pm 4.74$	$\textbf{82.54}\pm 1.14$
HoC	$\textbf{80.60}\pm 0.35$	$80.57\pm 0.88$

Table 2: BERT vs BERTUMLS (ours) test results (%) across biomedical NLP tasks. Evaluation metrics are listed in Table 1 .

Dataset	BioBERT	BioBERTUMLS
PubMedQA	$52.80\pm 0.0$	$\textbf{54.20}\pm 0.0$
BioASQ	$\textbf{79.71}\pm 0.38$	$75.57\pm 1.25$
EBM PICO	$73.56\pm 0.01$	$\textbf{73.66}\pm 0.04$
DDI	$\textbf{81.64}\pm 0.19$	$80.97\pm 0.33$
BIOSSES	$88.37\pm 2.02$	$\textbf{90.36}\pm 0.45$
HoC	$82.82\pm 0.22$	$\textbf{83.03}\pm 0.25$

Table 3: BioBERT vs BioBERTUMLS (ours) test results (%) across biomedical NLP tasks. Evaluation metrics are listed in Table 1 .

Model	Number of words	Domain
BERT	3.4B	General
BioBERT	4.5B	Biomedical
BERTUMLS	100M	Biomedical
BioBERTUMLS	100M	Biomedical

Table 4: Pretraining corpora of biomedical and general language models.

Dataset	LLaMA 3-8B	LLaMA 3-8B + GraphRAG
PubMedQA	51.2	54.4
BioASQ	75.0	80.0

Table 5: Comparison of accuracy (%) on PubMedQA and BioASQ between LLaMA 3-8B and our GraphRAG-enhanced model.

Figure 5: A real-world biomedical question from BioASQ and the corresponding subgraph extracted from our Neo4j knowledge graph. Using a LLama model alone often yields an incorrect answer (‘No’). By injecting structured biomedical knowledge from the UMLS graph as context, the model correctly answers ‘Yes’.

Knowledge Augmentation for Language Models

Knowledge augmentation for language models adds external or newly learned information to improve factuality, reasoning, and domain performance.

See all 178 papers

Knowledge-Graph Grounding Helps LLMs Only for Out-of-Training Knowledge: A Controlled Study on Clinical Question Answering
Jun 21, 2026Madhulatha Mandarapu, Sandeep KunkunuruFactual Knowledge in Language Models LLM Reasoning with Graphs
samyama
Are We Done with UMLS Injection into Language Models for Medical Question Answering?
Oct 24, 2026Yiran Zeng, Giacomo Frisoni, Lorenzo Molfetta +1Knowledge Augmentation for Language Models Medical QA
Department of Computer Science and Engineering, University of Bologna, Italy
RAGU: A Multi-Step GraphRAG Engine with a Compact Domain-Adapted LLM
Jul 13, 2026Mikhail Komarov, Ivan Bondarenko, Stanislav Shtuka +5RAG Evaluation KG Construction
1ITMO University · 2Novosibirsk State University · 3Far Eastern Federal University +1

Jun 21, 2026J/K move · Enter open · S save

Knowledge-Graph Grounding Helps LLMs Only for Out-of-Training Knowledge: A Controlled Study on Clinical Question Answering

Madhulatha Mandarapu, Sandeep Kunkunuru

samyama

Factual Knowledge in Language Models LLM Reasoning with Graphs Clinical QA LLM Grounding Clinical Language Model Evaluation Biomedical KGs+1

Open paper AskPDF arXiv

A recent Nature Medicine study reports that general-purpose frontier LLMs outperform specialized retrieval-augmented clinical tools on medical benchmarks, and that retrieval can hurt strong models. We ask the natural follow-up: does structured knowledge-graph (KG) grounding change this, and when does grounding help at all? We contribute two results. First, a reproduction: the study's headline HealthBench score (~88) is the Consensus variant, not full HealthBench, where frontier models and ideal completions both score ~46-47 under a physician-calibrated grader (agreement 82.5%); we reproduce GPT-5.2 Consensus =90.9 and flag a score-deflating grader bug. Second, a knowledge-boundary result. Using a graph+vector engine (samyama-graph) over the public biomedical KG PrimeKG, neither naive triple retrieval nor an agentic natural-language-to-Cypher loop (82% successful queries) improves MedQA across a weak-to-strong model ladder (all |Delta| <= 3.4). On a synthetic counterfactual KG, and on a hybrid benchmark mixing known and novel facts, the identical pipeline lifts out-of-training accuracy from chance to ~100% (+68 to +79) while adding nothing on known facts (a no-LLM arm answers both). Across three regimes (no-knowledge, graph-aided, hybrid), grounding helps only insofar as the decisive fact lies outside the model's training -- public-KG facts are redundant, private and novel data are where it pays -- matching the study's institutional-data caveat.

Abstract

Knowledge Augmentation for Language Models

Explore similar work

Knowledge-Graph Grounding Helps LLMs Only for Out-of-Training Knowledge: A Controlled Study on Clinical Question Answering

Are We Done with UMLS Injection into Language Models for Medical Question Answering?

RAGU: A Multi-Step GraphRAG Engine with a Compact Domain-Adapted LLM