A Hybrid Knowledge-Grounded Framework for Safety and Traceability in Prescription Verification

Yichi Zhu; Kan Ling; Xu Liu; Hengrun Zhang; Huiqun Yu; Guisheng Fan

cs.AIMar 11, 2026

A Hybrid Knowledge-Grounded Framework for Safety and Traceability in Prescription Verification

Authors: Yichi Zhu, Kan Ling, Xu Liu, Hengrun Zhang, Huiqun Yu, Guisheng Fan

Organizations: School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China

Abstract

Medication errors pose a significant threat to patient safety, making pharmacist verification (PV) a critical, yet heavily burdened, final safeguard. The direct application of Large Language Models (LLMs) to this zero-tolerance domain is untenable due to their inherent factual unreliability, lack of traceability, and weakness in complex reasoning. To address these challenges, we introduce PharmGraph-Auditor, a novel system designed for safe and evidence-grounded prescription auditing. The core of our system is a trustworthy Hybrid Pharmaceutical Knowledge Base (HPKB), implemented under the Virtual Knowledge Graph (VKG) paradigm. This architecture strategically unifies a relational component for set constraint satisfaction and a graph component for topological reasoning via a rigorous mapping layer. To construct this HPKB, we propose the Iterative Schema Refinement (ISR) algorithm, a framework that enables the co-evolution of both graph and relational schemas from medical texts. For auditing, we introduce the KB-grounded Chain of Verification (CoV), a new reasoning paradigm that transforms the LLM from an unreliable generator into a transparent reasoning engine. CoV decomposes the audit task into a sequence of verifiable queries against the HPKB, generating hybrid query plans to retrieve evidence from the most appropriate data store. Experimental results demonstrate robust knowledge extraction capabilities and show promises of using PharmGraph-Auditor to enable pharmacists to achieve safer and faster prescription verification.

Figures & tables

Figure 1. The workflow and the role of PharmGraph-Auditor.

† $O(1)$ denotes constant time per relationship traversal, irrespective of $\|V\|$ .
Dimension	Relational ( $\mathcal{R}$ )	Graph ( $\mathcal{G}$ )
Data Nature	Atomic, Numerical, Conditional	Associative, Hierarchical, Transitive
Logic Type	Set Constraint Satisfaction	Topological Traversal
Access Cost	Index Scan: $O(\log N)$	Index-free Adjacency: $O(1)^{\dagger}$
Complexity	Dependent on Dataset Size ( $N$ )	Independent of Dataset Size ( $N$ )
Typical Audit	Dosage Checks, Contraindications	Interactions, Allergies, Duplicate Therapy

Table 1. The Knowledge Stratification Framework

Figure 2. System architecture of PharmGraph-Auditor.

Figure 3. The KB-grounded Chain of Verification (CoV) framework.

Figure 4. The Patient Profile-driven Evidence Selection Tree (P-EST) logic for pruning conflicting dosage rules.

Metric	Relational Data	Graph Relations	Total
# Documents	100	100	100
# Extracted Records	2,951	923	3,874
Avg. # Records / Doc	29.51	9.23	38.74

Table 2. Statistics of the Golden Standard HPKB

Method	Component	Precision	Recall	F1-score
PharmGraph-Auditor (Ours, GPT-4o)	Relational	0.7973	0.8243	0.8106
	Graph	0.9565	0.9565	0.9565
	Overall (Micro)	0.8260	0.8491	0.8374
PharmGraph-Auditor (Ours, Deepseek-V3)	Relational	0.7948	0.8378	0.8157
	Graph	0.9782	0.9782	0.9782
	Overall (Micro)	0.8235	0.8603	0.8415

Table 3. Performance on Knowledge Population Task

Figure 5. Types of prescription errors.

Method	Precision	Recall	F1-Score
Experience Review (Human)	100.0%	45.9%	62.9%
Rule Review (CDSS)	52.1%	67.6%	58.8%
Proposed Method (Ours)	74.3%	70.3%	72.2%

Table 4. Performance Comparison

Figure 6. Fine-grained performance analysis by risk category. The top chart illustrates Precision, and the bottom chart illustrates Recall.

Setting	Precision	Recall	F1	Cost
Proposed Method	0.7924	0.9504	0.8642	$0.0225
w/o CoV	0.5757	0.7645	0.6561	$0.0250
w/o CoV & Knowledge	0.3927	0.5233	0.4487	$0.0055

Table 5. Ablation Study Results

Knowledge Graphs

Knowledge graphs represent entities and relationships as structured, queryable networks for integrating data and supporting reasoning.

See all 232 papers

Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops
Jun 12, 2026Muhammad Osama, Maheera Amjad, Zartasha Mustansar +2Multi-Agent LLM Systems LLM Hallucination Mitigation
Data Science and Machine Learning Lab, SINES, NUST · CEME, NUST
SafeRx-Agent: A Knowledge-Grounded Multi-Agent Framework for Safe and Explainable Medication Recommendation
May 27, 2026Xinyu Wang, Hanwei Wu, Zhenghan Tai +7Medication Recommendation Multi-Agent LLM Systems
McGill University · McMaster University · University of Toronto +1
Reason and Verify: A Framework for Faithful Retrieval-Augmented Generation
Mar 10, 2026Eeham Khan, Luis Rodriguez, Marc QueudotRAG Citation Faithfulness Retrieval-Augmented Generation
Concordia University, Montréal, QC, Canada · Centre de recherche informatique de Montréal (CRIM), Montréal, QC, Canada

Jun 12, 2026J/K move · Enter open · S save

Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops

Muhammad Osama, Maheera Amjad, Zartasha Mustansar, Arslan Shaukat+1

Data Science and Machine Learning Lab, SINES, NUST · CEME, NUST

Multi-Agent LLM Systems LLM Hallucination Mitigation LLM Auditing Clinical QA Medical LLMs

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly deployed in healthcare settings, yet their tendency to hallucinate poses risks when clinical decisions are involved. This study examine whether LLMs recommend recently banned or withdrawn pharmaceuticals when answering clinical questions and tests an agent-based method for reducing such errors. We developed a five-agent "Trust but Verify" system using a single LLM backbone. To measure regulatory knowledge obsolescence, we created an adversarial dataset of 103 clinical MCQs where historically correct answers now refer to banned substances. This scale ensures statistical significance across various therapeutic classes. We evaluated three open-access model families (GPT-OSS, Llama-3, Falcon-3) under vanilla and agentic conditions. Performance was measured via pointwise score, label accuracy, Hallucination Error Rate (HER), and Component Fidelity (CF) score. We also observed clinical safety regression in proprietary models. In default configurations, all models showed high hallucination rates, consistently selecting banned drugs that matched training data patterns. Our proposed agentic architecture reduced HER by approximately 53% across models. Pointwise scores shifted from -0.25 (unsafe recommendation) toward 0.0 (appropriate refusal). The safety audit intercepted dangerous outputs even when models' parametric knowledge favored the banned substance. The proposed multi-agent framework offers a model-agnostic method for enforcing regulatory compliance that prioritizes patient safety over fluent text generation. Our work demonstrates a practical approach for deploying autonomous AI systems in safety-critical healthcare settings. It shows how real-time regulatory data can be integrated into LLM pipelines to support clinical decision-making.

Abstract

Knowledge Graphs

Explore similar work

Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops

SafeRx-Agent: A Knowledge-Grounded Multi-Agent Framework for Safe and Explainable Medication Recommendation

Reason and Verify: A Framework for Faithful Retrieval-Augmented Generation