cs.CLApr 7, 2026

A Multi-Stage Validation Framework for Trustworthy Large-scale Clinical Information Extraction using Large Language Models

Authors: Maria Mahbub, Gregory M. Dams, Josh Arnold, Caitlin Rizy, Sudarshan Srinivasan, Elliot M. Fielstein, Minu A. Aghevli, Kamonica L. Craig, 

Organizations: Oak Ridge National Laboratory, Oak Ridge, TN, USA · Program Evaluation and Resource Center, Office of Mental Health and Office of Suicide Prevention, Department of Veterans Affairs, Menlo Park, CA, USA · Vanderbilt University Medical Center, Nashville, TN, USA · VA Maryland Health Care System, Baltimore, MD, USA · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) show promise for extracting clinically meaningful information from unstructured health records, yet their translation into real-world settings is constrained by the lack of scalable and trustworthy validation approaches. Conventional evaluation methods rely heavily on annotation-intensive reference standards or incomplete structured data, limiting feasibility at population scale. We propose a multi-stage validation framework for LLM-based clinical information extraction that enables rigorous assessment under weak supervision. The framework integrates prompt calibration, rule-based plausibility filtering, semantic grounding assessment, targeted confirmatory evaluation using an independent higher-capacity judge LLM, selective expert review, and external predictive validity analysis to quantify uncertainty and characterize error modes without exhaustive manual annotation. We applied this framework to extraction of substance use disorder (SUD) diagnoses across 11 substance categories from 919,783 clinical notes. Rule-based filtering and semantic grounding removed 14.59% of LLM-positive extractions that were unsupported, irrelevant, or structurally implausible. For high-uncertainty cases, the judge LLM's assessments showed substantial agreement with subject matter expert review (Gwet's AC1=0.80). Using judge-evaluated outputs as references, the primary LLM achieved an F1 score of 0.80 under relaxed matching criteria. LLM-extracted SUD diagnoses also predicted subsequent engagement in SUD specialty care more accurately than structured-data baselines (AUC=0.80). These findings demonstrate that scalable, trustworthy deployment of LLM-based clinical information extraction is feasible without annotation-intensive evaluation.

Figures & tables

Figure 1: A multi-stage framework for trustworthy large-scale LLM extraction of clinical information
Figure 2: Direct vs chain-of-thought prompting to extract SUD diagnoses information from clinical notes.
Figure 3: Prompt for judge LLM
Reasoning typeNumber of shotsPrecision (relaxed)Recall (relaxed)F1 (relaxed)F1 (strict)
Direct00.87820.86910.86470.8203
10.87720.85640.85840.8113
20.86480.85170.85140.8031
Chain-of-thought00.91890.89730.90040.8582
Table 2: Prompt calibration results on the annotated dataset comparing zero-, one-, and two-shot prompting under direct and chain-of-thought strategies.
Substance categoryPre-filter LLM+Count of flagged LLM+ extractionsPost-filter LLM+
Rule-basedSemantic grounding
0.500.550.600.650.70
Alcohol261037296929721064126017082270229637
Opioid63123141238248559471057132647943
Cannabis7177759171065109311501315157964545
Cocaine6579640668738889201025131760705
Table 3: LLM+ extraction counts and flagged extraction counts from rule-based filtering and semantic grounding assessment by substance category.
Substance categoryPrecision (relaxed)Recall (relaxed)F1 (relaxed)F1 (strict)
Alcohol0.86780.87590.86890.8446
Opioid0.84100.84480.83940.8106
Cannabis0.80260.80760.80270.7805
Cocaine0.76900.77350.76820.7401
Amphetamine0.61550.61550.61390.5934
Caffeine0.67450.67390.67300.6542
Table 4: Information extraction performance for SUD diagnoses as assessed by the judge LLM, considering only instances where the primary LLM produced an extraction.
Substance categoryOverall Agreement (%)Gwet’s AC1
Alcohol900.87
Opioid920.9
Cannabis920.9
Cocaine910.88
Amphetamine830.79
Caffeine950.95
Table 5: Inter-annotator agreement (Gwet’s AC1) between the judge LLM and SME for a random sample of 100 clinical notes across substance categories.
Figure 4: Predictive validity of LLM-extracted SUD diagnoses shown by ROC curves comparing outcome prediction using LLM-extracted, ICD-10–based, and combined features across cohorts (a) full study population, (b) concordant cohort, and (c) narrative-only cohort.
More in this topic

Clinical Information Extraction

Clinical Information Extraction turns clinical notes, reports, and patient narratives into structured, searchable facts and relations.

See all 106 papers

Explore similar work

CardsList
Jun 30, 2026

CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning

Large Language Models (LLMs) achieve strong results on many medical benchmarks, but their clinical reasoning remains difficult to evaluate reliably. A central risk is an evaluation illusion: fluent and well-structured explanations can appear clinically convincing even when the final diagnosis is incorrect. We introduce CLExEval, a human-in-the-loop framework for evaluating LLM clinical reasoning under progressive information masking. CLExEval combines 5,600 expert-physician annotations with 200 clinical reasoning traces derived from 40 rare diagnostic cases. Our analysis identifies three recurring failure patterns: (i) verbosity bias, where GPT-4o-mini's diagnostic accuracy drops from 95.0% to 32.5% under information scarcity; (ii) a hidden knowledge paradox, where a specialist model reaches 92.5% maximum diagnostic potential but fails to retrieve that knowledge reliably in verbose contexts; and (iii) a 68.6% reasoning-to-output mismatch, where correct diagnoses appear in reasoning traces but are not reflected in final answers. We further evaluate the LLM-as-a-Judge paradigm on a human-verified failure set (n = 142). GPT-4o-mini approved 47.9% of clinically incorrect outputs, while HuatuoGPT-o1 approved all validly scored failures and showed a positive self-preference bias. These results suggest that standalone automated clinical evaluations can substantially overestimate clinical reliability without expert-grounded validation.

Ajmal M., Abin Roy, Afthab Salam Kanniyan +4

1MBZUAI · 2IIT Madras · 3Calicut Medical College

LLM-as-a-JudgeHuman-in-the-Loop EvaluationLLM ReliabilityClinical ReasoningClinical Language Model Evaluation
AskPDF
Apr 3, 2026

A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework

As large language models (LLMs) increasingly generate and process clinical text, scalable evaluation has become critical. LLM-as-a-Judge (LaaJ), which uses LLMs to evaluate model outputs, offers a scalable alternative to costly expert review, but its healthcare adoption raises safety and bias concerns. We conducted a PRISMA-ScR scoping review of six databases (January 2020-January 2026), screening 11,727 studies and including 49. The landscape was dominated by evaluation and benchmarking applications (n=37, 75.5%), pointwise scoring (n=42, 85.7%), and GPT-family judges (n=36, 73.5%). Despite growing adoption, validation rigor was limited: among 36 studies with human involvement, the median number of expert validators was 3, while 13 (26.5%) used none. Risk of bias testing was absent in 36 studies (73.5%), only 1 (2.0%) examined demographic fairness, and none assessed temporal stability or patient context. Deployment remained limited, with 1 study (2.0%) reaching production and four (8.2%) prototype stage. Importantly, these gaps may interact: when judges and evaluated systems share training data or architectures, they may inherit similar blind spots, and agreement metrics may fail to distinguish true validity from shared errors. Minimal human oversight, limited bias assessment, and model monoculture together represent a governance gap where current validation may miss clinically significant errors. To address this, we propose MedJUDGE (Medical Judge Utility, De-biasing, Governance and Evaluation), a risk-stratified three-pillar framework organized around validity, safety, and accountability across clinical risk tiers, providing deployment-oriented evaluation guidance for healthcare LaaJ systems.

Chenyu Li, Zohaib Akhtar, Mingu Kwak +13

Department of Biomedical Informatics, School of Medicine, University of Pittsburgh, Pittsburgh, PA, USA · Department of Health Information Management, School of Health and Rehabilitation Sciences, University of Pittsburgh, Pittsburgh, PA, USA · OpenCura, Health Innovation Consortium +6

LLM-as-a-JudgeClinical Language Model EvaluationLanguage Model Safety EvaluationBias in LLM-as-a-Judge
AskPDF
Mar 1, 2026

Conformal Prediction for Risk-Controlled Medical Entity Extraction Across Clinical Domains

Large Language Models (LLMs) are increasingly used for medical entity extraction, yet their confidence scores are often miscalibrated, limiting safe deployment in clinical settings. We present a conformal prediction framework that provides finite-sample coverage guarantees for LLM-based extraction across two clinical domains. First, we extract structured entities from 1,000 FDA drug labels across eight sections using GPT-4.1, verified via FactScore-based atomic statement evaluation (97.7% accuracy over 128,906 entities). Second, we extract radiological entities from MIMIC-CXR reports using the RadGraph schema with GPT-4.1 and Llama-4-Maverick, evaluated against physician annotations (entity F1: 0.81 to 0.84). Our central finding is that miscalibration direction reverses across domains: on well-structured FDA labels, models are underconfident, requiring modest conformal thresholds (τ≈0.06τ\approx 0.06), while on free-text radiology reports, models are overconfident, demanding strict thresholds (ττ up to 0.99). Despite this heterogeneity, conformal prediction achieves target coverage (≥90%\geq 90\%) in both settings with manageable rejection rates (9--13%). These results demonstrate that calibration is not a global model property but depends on document structure, extraction category, and model architecture, motivating domain-specific conformal calibration for safe clinical deployment.

Manil Shrestha, Edward Kim

Department of Computer Science, Drexel University, Philadelphia, USA

Named Entity RecognitionLanguage Model CalibrationConformal PredictionMedical LLMsClinical Information Extraction
AskPDF