Injecting Structured Biomedical Knowledge into Language Models: Continual Pretraining vs. GraphRAG

Jaafer Klila; Sondes Bannour Souihi; Rahma Boujelben; Nasredine Semmar; Lamia Hadrich Belguith

cs.CLApr 3, 2026

Injecting Structured Biomedical Knowledge into Language Models: Continual Pretraining vs. GraphRAG

Authors: Jaafer Klila, Sondes Bannour Souihi, Rahma Boujelben, Nasredine Semmar, Lamia Hadrich Belguith

Organizations: Université Paris-Saclay, CEA, List Palaiseau, France · University of Sfax Sfax, Tunisia

Abstract

The injection of domain-specific knowledge is crucial for adapting language models (LMs) to specialized fields such as biomedicine. While most current approaches rely on unstructured text corpora, this study explores two complementary strategies for leveraging structured knowledge from the UMLS Metathesaurus: (i) Continual pretraining that embeds knowledge into model parameters, and (ii) Graph Retrieval-Augmented Generation (GraphRAG) that consults a knowledge graph at inference time. We first construct a large-scale biomedical knowledge graph from UMLS (3.4 million concepts and 34.2 million relations), stored in Neo4j for efficient querying. We then derive a ~100-million-token textual corpus from this graph to continually pretrain two models: BERTUMLS (from BERT) and BioBERTUMLS (from BioBERT). We evaluate these models on six BLURB (Biomedical Language Understanding and Reasoning Benchmark) datasets spanning five task types and evaluate GraphRAG on the two QA (Question Answering) datasets (PubMedQA, BioASQ). On BLURB tasks, BERTUMLS improves over BERT, with the largest gains on knowledge-intensive QA. Effects on BioBERT are more nuanced, suggesting diminishing returns when the base model already encodes substantial biomedical text knowledge. Finally, augmenting LLaMA 3-8B with our GraphRAG pipeline yields over than 3 points accuracy on PubMedQA and 5 points on BioASQ without any retraining, delivering transparent, multi-hop, and easily updated knowledge access. We release the processed UMLS Neo4j graph to support reproducibility.

Figures & tables

Figure 1: Overview of the UMLS-Based Knowledge Injection Pipeline: Continual Pretraining on BERT/BioBERT and GraphRAG.

Figure 2: Pipeline for constructing a biomedical knowledge graph from UMLS Metathesaurus.

Figure 3: Example of generating a textualized triple from the KG.

Dataset	Task	Train	Dev	Test	Evaluation Metrics	Number of Classes
EBM PICO	PICO	339,167	85,321	16,364	Macro F1 word-level	3
BIOSSES	Sentence Similarity	64	16	20	Pearson	N/A (labels continus)
HoC	Document Classification	1,295	186	371	Average Micro F1	10
DDI	Relation Extraction	25,296	2,496	5,716	Micro F1	18
PubMedQA	Question Answering	450	50	500	Accuracy	3
BioASQ	Question Answering	670	75	140	Accuracy	2

Table 1: Summary of the biomedical datasets used with the number of classes and evaluation metrics.

Dataset	BERT	BERTUMLS
PubMedQA	$50.72\pm 0.17$	$\textbf{53.00}\pm 0.0$
BioASQ	$66.57\pm 0.93$	$\textbf{71.57}\pm 1.17$
EBM PICO	$71.95\pm 0.06$	$\textbf{72.18}\pm 0.05$
DDI	$77.99\pm 0.19$	$\textbf{78.67}\pm 0.27$
BIOSSES	$82.31\pm 4.74$	$\textbf{82.54}\pm 1.14$
HoC	$\textbf{80.60}\pm 0.35$	$80.57\pm 0.88$

Table 2: BERT vs BERTUMLS (ours) test results (%) across biomedical NLP tasks. Evaluation metrics are listed in Table 1 .

Dataset	BioBERT	BioBERTUMLS
PubMedQA	$52.80\pm 0.0$	$\textbf{54.20}\pm 0.0$
BioASQ	$\textbf{79.71}\pm 0.38$	$75.57\pm 1.25$
EBM PICO	$73.56\pm 0.01$	$\textbf{73.66}\pm 0.04$
DDI	$\textbf{81.64}\pm 0.19$	$80.97\pm 0.33$
BIOSSES	$88.37\pm 2.02$	$\textbf{90.36}\pm 0.45$
HoC	$82.82\pm 0.22$	$\textbf{83.03}\pm 0.25$

Table 3: BioBERT vs BioBERTUMLS (ours) test results (%) across biomedical NLP tasks. Evaluation metrics are listed in Table 1 .

Model	Number of words	Domain
BERT	3.4B	General
BioBERT	4.5B	Biomedical
BERTUMLS	100M	Biomedical
BioBERTUMLS	100M	Biomedical

Table 4: Pretraining corpora of biomedical and general language models.

Dataset	LLaMA 3-8B	LLaMA 3-8B + GraphRAG
PubMedQA	51.2	54.4
BioASQ	75.0	80.0

Table 5: Comparison of accuracy (%) on PubMedQA and BioASQ between LLaMA 3-8B and our GraphRAG-enhanced model.

Figure 5: A real-world biomedical question from BioASQ and the corresponding subgraph extracted from our Neo4j knowledge graph. Using a LLama model alone often yields an incorrect answer (‘No’). By injecting structured biomedical knowledge from the UMLS graph as context, the model correctly answers ‘Yes’.

Knowledge Augmentation for Language Models

Knowledge augmentation for language models adds external or newly learned information to improve factuality, reasoning, and domain performance.

See all 178 papers

Jun 21, 2026

Knowledge-Graph Grounding Helps LLMs Only for Out-of-Training Knowledge: A Controlled Study on Clinical Question Answering

A recent Nature Medicine study reports that general-purpose frontier LLMs outperform specialized retrieval-augmented clinical tools on medical benchmarks, and that retrieval can hurt strong models. We ask the natural follow-up: does structured knowledge-graph (KG) grounding change this, and when does grounding help at all? We contribute two results. First, a reproduction: the study's headline HealthBench score (~88) is the Consensus variant, not full HealthBench, where frontier models and ideal completions both score ~46-47 under a physician-calibrated grader (agreement 82.5%); we reproduce GPT-5.2 Consensus =90.9 and flag a score-deflating grader bug. Second, a knowledge-boundary result. Using a graph+vector engine (samyama-graph) over the public biomedical KG PrimeKG, neither naive triple retrieval nor an agentic natural-language-to-Cypher loop (82% successful queries) improves MedQA across a weak-to-strong model ladder (all |Delta| <= 3.4). On a synthetic counterfactual KG, and on a hybrid benchmark mixing known and novel facts, the identical pipeline lifts out-of-training accuracy from chance to ~100% (+68 to +79) while adding nothing on known facts (a no-LLM arm answers both). Across three regimes (no-knowledge, graph-aided, hybrid), grounding helps only insofar as the decisive fact lies outside the model's training -- public-KG facts are redundant, private and novel data are where it pays -- matching the study's institutional-data caveat.

Madhulatha Mandarapu, Sandeep Kunkunuru

samyama

Factual Knowledge in Language Models LLM Reasoning with Graphs Clinical QA LLM Grounding Clinical Language Model Evaluation Biomedical KGs+1

Ask PDF

Oct 24, 2026

Are We Done with UMLS Injection into Language Models for Medical Question Answering?

Maybe not. The Unified Medical Language System (UMLS) has become an oracle for millions of clinicians and health system developers worldwide. Researchers have increasingly used knowledge graphs derived from UMLS to enhance language models' factual accuracy and reasoning traceability in high-stakes tasks. Medical Question Answering (MQA) is a prime testbed for integration, with benchmarks usually involving professional exercises such as disease diagnosis and drug prescription from patient history. Despite the assumed benefits, empirical evidence shows that UMLS-based graph injection typically yields modest performance gains. This paper reviews existing MQA work, critically analyzing the selected UMLS sources and their usage with language models, and probes each of these choices in isolation. We uncover methodological issues never discussed in prior work, from low UMLS coverage to weak subgraph retrieval pipelines. Based on our findings, we provide actionable recommendations to unlock the untapped potential of UMLS as an external knowledge source.

Yiran Zeng, Giacomo Frisoni, Lorenzo Molfetta +1

Department of Computer Science and Engineering, University of Bologna, Italy

Knowledge Augmentation for Language Models Medical QA Biomedical KGs

Ask PDF

Jul 13, 2026

RAGU: A Multi-Step GraphRAG Engine with a Compact Domain-Adapted LLM

Graph retrieval-augmented generation (GraphRAG) enhances large language models with structured knowledge, yet existing systems construct knowledge graphs in a single extraction pass, producing noisy entities and brittle retrieval. RAGU, an open-source modular GraphRAG engine, addresses this by separating extraction from consolidation: entities and relations pass through two-stage typed extraction, DBSCAN-backed deduplication, LLM summarization, and Leiden community detection. A key insight motivates a compact extractor: the skills an in-pipeline LLM needs - comprehension, extraction, reasoning over context - are language skills that grow only weakly with model size, unlike factual world knowledge. Accordingly, we train Meno-Lite-0.1, a 7B model optimized for language skills, which outperforms Qwen2.5-32B on knowledge-graph construction (+12.5% relative harmonic mean) and matches it on English GraphRAG tasks. On GraphRAG-Bench (Medical), RAGU retrieves the most complete context at every factoid level (evidence recall up to 0.84 vs.

\leq

0.76) and overtakes HippoRAG2 on synthesis tasks; on multi-hop factoid QA, the apparent HippoRAG2 advantage is shown to be largely an answer-format artifact. RAGU is installable via

\texttt{pip install graph_ragu}

, runs on a single GPU, and is released under MIT. The source code is publicly available at https://github.com/RaguTeam/RAGU, and the Meno-Lite-0.1 model can be obtained from https://huggingface.co/bond005/meno-lite-0.1.

Mikhail Komarov, Ivan Bondarenko, Stanislav Shtuka +5

1ITMO University · 2Novosibirsk State University · 3Far Eastern Federal University +1

RAG Evaluation KG Construction LLM Reasoning with Graphs Graph Retrieval-Augmented Generation Small Language Models Knowledge Representation+2

Ask PDF

Abstract

Knowledge Augmentation for Language Models

Explore similar work

Knowledge-Graph Grounding Helps LLMs Only for Out-of-Training Knowledge: A Controlled Study on Clinical Question Answering

Are We Done with UMLS Injection into Language Models for Medical Question Answering?

RAGU: A Multi-Step GraphRAG Engine with a Compact Domain-Adapted LLM