Trust the Edge, Doubt the Arrow: Auditing Validation Signals for Text-Mined Clinical Causal Knowledge Graphs

Su Lan; Shicheng Zhou; Yiqi Wang; Yanming Zhu; Xuefei Yin; Alan Wee-Chung Liew

cs.AIOct 17, 2026

Trust the Edge, Doubt the Arrow: Auditing Validation Signals for Text-Mined Clinical Causal Knowledge Graphs

Authors: Su Lan, Shicheng Zhou, Yiqi Wang, Yanming Zhu, Xuefei Yin, Alan Wee-Chung Liew

Organizations: School of Information and Communication Technology, Griffith University, Gold Coast, QLD, Australia

Abstract

Clinical causal retrieval-augmented generation systems increasingly use knowledge graphs (KGs) mined from notes, yet extracted assertions remain candidates that require audit for patient-level EHR support and direction. We construct a directed candidate graph from 328,839 MIMIC-IV discharge summaries using LLM extraction and UMLS normalization, and evaluate three signals against SemMedDB: within-admission co-occurrence lift, cross-admission diagnosis-coding temporal precedence, and literature corroboration. Lift filters unsupported co-mentions and provides useful EHR support, but not causal proof. Coding order is a weak direction signal: among the 686 candidates where it commits, it reaches 54.2% agreement with SemMedDB, below the extractor's 69.4% on the same subset. Literature corroboration supplies independent direction evidence but covers only 30.4% of candidate relations. A MIMIC-III replication yields the same pattern: extractor direction remains stable (70.5%), whereas coding-order direction is near chance (50.9%). Thus, text-mined clinical causal candidate KGs are best treated as EHR-supported but direction-uncertain. In a leakage-controlled causal QA benchmark, lift-only retrieval achieves 35.7% accuracy; conservative causal demotion achieves 37.2% while reducing evidence-unsupported answers from 17.3% to 6.1% at fixed retrieval recall. We release code, aggregate summaries, and scripts to regenerate restricted outputs.

Figures & tables

Figure 1: Overview of the proposed audit-and-retrieval framework. Build: causal candidates are mined from clinical notes and normalized to UMLS concepts. Audit: co-occurrence lift provides EHR support, coding-order time is unreliable for direction, and SemMedDB provides partial literature corroboration. Apply: support-aware causal RAG retrieves from the EHR-supported candidate graph and demotes candidates that fail the causal screen, reducing evidence-unsupported answers at fixed recall.

	Directed	Patient	EHR	Direction	QA
	Candidates	Text	Support	Audited	Benchmark
GraphCare [ 7 ] , Med. GraphRAG [ 31 ]	$\times$	$\sim$	$\times$	$\times$	$\times$
CausalRAG [ 30 ]	$\checkmark$	$\times$	$\times$	$\times$	$\times$
Lit. causal KGs [ 11 , 22 , 18 , 29 ]	$\checkmark$	$\times$	$\sim$	$\times$	$\times$
Ours	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 1: Positioning against representative prior work. ✓ indicates yes,

\sim

partial, and

\times

no. Ours is the only setting that combines directed causal candidate mining from patient narratives, structured-EHR support filtering, direction-signal auditing, and a patient-grounded causal QA benchmark.

Metric	Rule-based	LLM(ours)
Raw surface edges	142,890	1,173,644
UMLS link rate	21.9%	49.3%
CUI nodes	8,572	4,318
CUI edges	19,171	19,035
Giant component	92.3%	97.1%
Lift-supported candidates	736	2,047

Table 2: Candidate-KG construction on the full pre-split corpus: rule-based vs. LLM extraction.

Direction signal	Acc. vs. SemMedDB
Raw LLM extraction direction	73.6%
Temporal precedence (committed, $n{=}686$ )	54.2%
LLM direction on those 686	69.4%

Table 3: Direction-signal accuracy vs. SemMedDB (

n=4{,}943

candidate relations). Temporal precedence abstains on 86% of candidates; on the 686 where it commits it is below the extractor’s own direction and near chance.

Figure 2: Direction-signal audit. Left: on the 686 candidate relations where cross-admission temporal precedence commits to a direction, it agrees with SemMedDB less often (54.2%) than the extractor’s own direction (69.4%) and barely above chance. Right: of temporal’s 304 “reversed” verdicts, 204 are false reversals (coding artifacts) and only 100 are true corrections.

Figure 3: The support-filtering recipe as a funnel. Recurrent LLM candidates are normalized to UMLS concepts, filtered to the EHR lift-supported subgraph, and partitioned by literature corroboration into a higher-confidence core; direction is taken from the extractor rather than coding-order time.

Figure 4: Cross-release replication on MIMIC-III. Compared with the main MIMIC-IV audit, MIMIC-III has lower lift support and lower SemMedDB coverage, but the direction pattern is stable: the extractor’s direction remains substantially above chance, whereas coding-order direction remains near chance.

Screen status	Slots	%	Unique pairs	Qs $\geq$ 1	Mean rank before $\to$ after
Pass ( $z\geq 1.64$ )	1,718	18.6	127	630 (67.6)	4.63 $\to$ 3.78
Fail ( $z<1.64$ )	4,510	48.7	494	807 (86.6)	7.09 $\to$ 8.39
Untested	3,025	32.7	377	780 (83.7)	5.54 $\to$ 4.08
Total	9,253	100	–	932	–

Table 4: Causal-screen coverage and ranking effect in the lift-only top-12 retrieval lists. The screen is applied after lift-only retrieval and only reorders the selected candidates. Lower mean rank indicates earlier placement in the evidence list.

System	WHY	WHATCAUSES	Overall	Faith.	Unsup.	Recall
1 Closed-book	22.5	9.0	15.8	N/A	N/A	N/A
2 Naive TF-IDF text-RAG	18.2	7.5	12.9	N/A	N/A	N/A
3 SemMedDB-only	24.9	13.5	19.2	83.8	16.2	23.9
4 Assoc-graph	35.4	16.3	25.9	73.9	26.1	72.7
5 Raw-causal	35.8	18.7	27.3	83.2	16.8	72.3
6 Lift + temporal	47.2	24.0	35.6	86.3	13.7	77.8

Table 5: Causal QA on held-out patients (%).

n=932

paired. Faith. denotes answer faithfulness to retrieved evidence; Unsup. denotes evidence-unsupported answers.

Figure 5: Causal-QA performance on held-out patients. Left: answer correctness. Right: evidence-unsupported answers for systems where evidence support can be measured. The final system uses lift-only retrieval and demotes candidates that fail the causal screen. The main downstream effect is fewer evidence-unsupported answers at fixed recall; the exact-accuracy change over lift-only is a small non-significant trend.

Biomedical KGs

Biomedical knowledge graphs organize evidence about genes, diseases, drugs, pathways, and patients for computational discovery and clinical reasoning.

See all 71 papers

ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV
May 11, 2026Alex StinardRAG Evaluation Clinical QA
Department of Clinical Sciences, College of Medicine · University of Central Florida, Orlando, FL 32816
PubMedCausal: A Span-Level Annotated Corpus for Causal Relation Extraction in Biomedical Text
May 27, 2026Ifeoluwa Kunle-John, Josiah Paul, Oluwatosin Agbaakin +3Relation Extraction Clinical NLP
Edyah Limited · University of Ibadan, NG · Indiana University, Indianapolis, IN +1
EviDAG: Auditable Causal DAG Authoring with Biomedical Literature
Jul 23, 2026Yi-han Sheu, Michael R. Steigman, Yu Zhou +3Data Provenance Evidence-Grounded Reasoning
Massachusetts General Hospital, USA · Harvard University, USA · Northeastern University, USA

May 11, 2026J/K move · Enter open · S save

ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV

Alex Stinard

Department of Clinical Sciences, College of Medicine · University of Central Florida, Orlando, FL 32816

RAG Evaluation Clinical QA Biomedical IR Healthcare Knowledge Graph Question Answering Biomedical KGs+1

Open paper AskPDF arXiv

Reasoning benchmarks measure clinical performance on clean inputs. We evaluate the step before reasoning: retrieval over real EHR notes, where negation, temporality, and family-versus-patient attribution can flip a correct answer to a wrong one. EpiKG carries an assertion label and a temporality tag with every fact in a patient knowledge graph, then routes retrieval by question intent. ClinicalBench is a 400-question test over 43 MIMIC-IV patients across 9 assertion-sensitive categories. A 7-condition ablation tests each piece of EpiKG across six LLMs (Claude Opus 4.6, GPT-OSS 20B, MedGemma 27B, Gemma 4 31B, MedGemma 1.5 4B, Qwen 3.5 35B). Three physicians blindly adjudicated 100 paired items. The author-blind primary endpoint, leave-author-out paired exact McNemar on 50 unanimous-strict items rated by two external physicians, yields +22.0 percentage points (95 percent Newcombe CI [+5.1, +31.5], p=0.0192). The architectural novelty, intent-aware KG-RAG over a Contriever dense-RAG baseline (C2b to C4g_kw on the change-excluded n=362 endpoint), is +8.84 percentage points (paired McNemar p=1.79e-3); +12.43 percentage points under oracle intent. Sensitivities agree directionally: three-rater physician majority +24.0 percentage points (subject to single-author circularity); deterministic keyword reproducibility proxy +39.5 percentage points. Across the six models, the gain shrinks as the LLM-alone baseline rises (beta=-1.123, r=-0.921, p=0.009). With n=6 this looks more like regression to the mean than encoding substituting for model size. Physician adjudication identified 56 percent of auto-generated reference answers as defective, a methodological finding indicating that NLP-pipeline clinical-QA benchmarks require physician adjudication to be usable. ClinicalBench, the frozen evaluator, three-rater adjudication data, and the EpiKG output stack are publicly released.

Abstract

Biomedical KGs

Explore similar work

ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV

PubMedCausal: A Span-Level Annotated Corpus for Causal Relation Extraction in Biomedical Text

EviDAG: Auditable Causal DAG Authoring with Biomedical Literature