Institution-Specific LLM Prompting Recovers PHI That De-identification Systems and Their Gold Standards Both Miss
Authors: Daniel Palacios, Matthew Brady Neeley, Angel Adetomike Otto, Shalini Dhamodharan, John P. Woodhouse, Chi-fan Lin, Mark Zobeck, Zhandong Liu, Hyun-Hwan Jeong
Organizations: Quantitative and Computational Biosciences, Baylor College of Medicine, Houston, Texas, USA · Department of Pediatrics, Baylor College of Medicine, Houston, Texas, USA · Jan and Dan Duncan Neurological Research Institute, Texas Children’s Hospital, Houston, Texas 77030, USA · Data Science Center, Texas Children’s Hospital, Houston, Texas 77030, USA · Section of Hematology-Oncology, Department of Pediatrics, Baylor College of Medicine, Houston, Texas, USA
Secondary use of electronic health records requires de-identification, yet existing systems miss institutionally situated protected health information (PHI) such as hospital abbreviations, building names, and internal codes whose status is locally determined. We ask whether large language models (LLMs) with in-context learning (ICL) can close this gap and control the precision--recall trade-off. On 100 annotated pediatric oncology notes (5,322 PHI spans) from Texas Children's Hospital, we benchmarked eight LLMs against two purpose-built systems (Stanford TiDE, OpenMed PII) and two pattern-based baselines. Each LLM ran under three prompts of increasing specificity: (1) a HIPAA-aligned baseline, (2) baseline plus the institutional PHI categories it missed, and (3) prompt 2 plus instructions against over-redacting clinical content. We then compared 14multi-agent and ensemble configurations against the best single prompt, with recall the primary safety metric. LLMs outperformed the purpose-built systems (best F1=0.918±0.001 vs.\ TiDE 0.779), with advantages concentrated in contextual categories. Naming the missed categories recovered 79% (48/61) of them, and discouraging over-redaction restored precision. No agentic architecture beat calibrated single-pass prompting (F1 0.906--0.907), but LLM outputs surfaced 414candidate annotation gaps; re-annotation confirmed 227~PHI spans, against which the final prompt reached recall=0.981 (F1=0.907±0.002). Well-calibrated ICL resolves both the institutional PHI gap and the precision--recall trade-off in one LLM call per note. LLMs cost more to run than traditional methods, but that cost buys a way to audit the reference standard. LLMs are a legitimate, adaptable alternative to purpose-built de-identification systems; institution-specific prompt development should be the primary adaptation strategy.
Figures & tables
System
Type
Family
Parameters
Claude Opus 4.8
LLM
Anthropic
Undisclosed
Claude Sonnet 4.6
LLM
Anthropic
Undisclosed
GPT-oss-120B
LLM
OpenAI
120B
GPT-oss-20B
LLM
OpenAI
20B
GLM-5
LLM
Zhipu AI
754B
Kimi K2.5
LLM
Moonshot
1.1T
Table 1: LLMs and purpose-built systems evaluated. The two pattern-based baselines (regex-only; spaCy NER + regex) are specified in the text and are included in all reported comparisons.
Figure 1: Study design and corpus characteristics. (A) Study design: 8 LLMs (top 4 shown in main figures) + traditional baselines evaluated on 100 pediatric oncology notes (5,322 spans) under 3 prompt conditions. (B) Synthetic clinical note illustrating canonical HIPAA PHI (names, dates; blue) vs. institutionally situated PHI (facility abbreviations, building names; orange). (C) Gold standard PHI distribution over the six categories with N≥20 (four further categories hold ≤5 spans each; all ten, with their denominators, are in Supplementary Table S4): Date (72.1%) and Name (22.3%) dominate, while institutionally enriched categories (Other Unique ID, Geographic Data) comprise 3.7% of spans. This distribution is that of the original gold standard; because that annotation under-counted institutional identifiers (see Results), 3.7% is a lower bound on their true prevalence.
Figure 2: LLMs outperform traditional de-identification approaches. (A) Recall, precision, and F1 for the top 4 LLMs and 4 traditional baselines ( Baseline prompt, 100 notes, 5,322 spans; all 8 LLMs in Table S3 and Supplementary Figure 1). (B) Per-category recall heatmap for the same systems, with categories ordered by gold-standard frequency and systems by overall F1, reveals shared weakness on the institutionally enriched categories (Other Unique ID [OUI], Geographic Data); the four categories with ≤ 5 gold spans are omitted here and reported in Table S4. (C) Sonnet 4.6 false negative decomposition on the primary Baseline run (Table S3; 211 FN): 29% are institutionally situated PHI (61 of 211; institution abbreviations 38, building/facility 15, department/clinic codes 8; see taxonomy in Table S7); the remaining 150 are canonical, of which the internal staff codes (pager and Voalte extensions, 4) are shown separately because Table S7 lists them as borderline.
Figure 3: In-context learning enables control over the precision–recall trade-off. (A) Precision, recall, and F1 for the top 4 LLMs across the three prompt versions ( Baseline ∙ , Targeted ■ , Precision ▲ ; single run per model, all 8 in Table S5). Each horizontal line spans the minimum-to-maximum of a model’s three stage values, so a non-monotonic Targeted stage visibly stretches the line rather than being hidden; Baseline and Precision values are labeled. Effect magnitude and direction are model-dependent. (B) Problem-specific deltas (single run), same model order and colors as (A), isolating what each prompt revision buys: Δ Recall( Targeted − Baseline ) is the recall gain from institutional targeting, and Δ Precision( Precision − Targeted ) is the precision recovery from the anti-over-redaction instructions. The Precision prompt lifts precision for all models (cross-model mean over the 7 models with valid output: +0.054) and acts as a corrective: gains are largest for models that over-redact under the baseline prompt (GPT-oss-120B 0.198→0.924 , Kimi 0.613→0.816 , DeepSeek 0.698→0.806 across the full Baseline → Precision progression; full values in Tables S5 and S6) and near-zero for models whose native precision is already high. Sonnet 4.6 reproducibility (5 trials per stage on the full 100-note corpus; SD ≤ 0.007 at every stage) confirms these are systematic prompt effects, not run-to-run noise (Table S9).
Figure 4: Enhanced gold standard validation and agentic architecture comparison. (A) Configuration landscape: F1 for the 16 configurations that the panel displays, colour-coded by class (single-pass, agentic, ensemble, traditional) and ranked on the enhanced gold standard; 10 further configurations are omitted for legibility and appear in Table S12, which lists all 26. The omitted set is the weakest-performing tail, except that TiDE is retained as the traditional-system reference. Single-pass Sonnet ( Precision ) (dashed line) matches or exceeds every multi-agent variant, and self-refinement and ensemble voting fail to beat it. (B) Reproducibility: 5-trial strip plots for the top 3 methods. All three achieve closely comparable mean F1 (0.906–0.908) with overlapping 95% trial-resampled CIs, but the Scrubber–Auditor ( Precision ) pipeline on Opus shows high variance (SD F1 =0.018) vs. near-deterministic single-pass Sonnet (SD F1 =0.002). (C) Sonnet 4.6 prompt progression on the enhanced gold standard (10-note subset, 1,758 spans; 5 independent trials per version). Lines show mean; shading shows ± SD; points show individual trials. Baseline recall is 0.847 here versus 0.958 on the original 100-note gold standard (a different note set and a different gold standard) because it misses newly-annotated institutional terms; Targeted recovers recall to 0.980; Precision maintains recall while improving precision (F1=0.907 ± 0.002). Terminology: single-pass = one LLM call per note; dual-pass = two sequential LLM calls; structured pass 2 = a second pass that ingests the first pass’s output in a structured format; Scrubber–Auditor = an agentic pipeline in which a generator LLM (scrubber) is checked by a second auditor LLM. Baseline , Targeted , and Precision denote the three prompt versions.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Name in text
Bedrock model identifier
Provider
Claude Opus 4.8
us.anthropic.claude-opus-4-8 †
Anthropic
Claude Sonnet 4.6
us.anthropic.claude-sonnet-4-6 †
Anthropic
GPT-oss-120B
openai.gpt-oss-120b-1:0
OpenAI
GPT-oss-20B
openai.gpt-oss-20b-1:0
OpenAI
GLM-5
zai.glm-5
Zhipu AI
Kimi K2.5
moonshotai.kimi-k2.5
Moonshot
Appendix
Table E1: Exact AWS Bedrock model identifiers for the eight benchmarked LLMs. All models were invoked through the Bedrock Converse API in region us-east-1 at temperature=0.0 (except Opus 4.8, which rejects the parameter), max_tokens=65,000 , with up to 3 retries under exponential backoff.
System
Latency
Cost (49 notes)
Cost/note
Platform
Claude Opus 4.8
6.8 min
$2.142
$0.0437
Bedrock API
Claude Sonnet 4.6
4.6 min
$0.428
$0.0087
Bedrock API
GPT-oss-120B
5.6 min
$0.400
$0.0082
Bedrock API
DeepSeek V3.2
6.4 min
$0.166
$0.0034
Bedrock API
Kimi K2.5
3.3 min
$0.166
$0.0034
Bedrock API
GLM-5
11.5 min
$0.166
$0.0034
Bedrock API
Appendix
Table E2: Cost and latency comparison on the USDHUB validation dataset (49 notes, sequential processing). LLM costs reflect AWS Bedrock on-demand pricing; local systems have zero marginal API cost (compute is pre-provisioned on the evaluation instance).
List of suppfigures E1 Per-note recall distribution (Baseline). Each point represents one clinical note. Most top models cluster near perfect recall (1.0), but all exhibit a tail of low-recall notes; these correspond to notes rich in institutionally situated PHI.
List of suppfigures E2 Perfect de-identification rate (Baseline). Fraction of notes with zero leaked PHI spans. DeepSeek and GLM-5 achieve perfect de-identification on ∼ 60% of notes; no model exceeds 65%, reflecting the long tail of institutionally situated identifiers.
List of suppfigures E3 Per-category recall change under targeted prompting (Targeted − Baseline). The targeted prompt yields concentrated gains in Other Unique ID and Name (institutionally situated categories) for responsive models, while MiniMax M2.5 shows broad regressions. Geographic Data shows mixed effects.
List of suppfigures E4 Overall performance under precision-focused prompt (Precision). The "do-not-over-redact" instructions improve precision across most models relative to Baseline/Targeted, with GPT-oss-120B showing the most dramatic improvement. GPT-oss-20B suffers output failures (reasoning exhausts token budget).
List of suppfigures E5 Per-category recall change (Precision − Baseline). Unlike Targeted (Supplementary Figure E3 ), the Precision prompt shows smaller recall deltas for strong models, confirming that precision can be improved without substantial recall degradation. GPT-oss-20B and MiniMax M2.5 are outliers with large regressions.
List of suppfigures E6 Perfect de-identification rate (Precision). Under the precision-focused prompt, DeepSeek V3.2 and Kimi K2.5 maintain the highest perfect de-id rates ( ∼ 60%), while GPT-oss-20B and MiniMax M2.5 show substantial degradation. Compare with Supplementary Figure E2 for baseline.
List of suppfigures E7 Per-note recall distribution (Precision). Compared to Baseline (Supplementary Figure E1 ), the distribution remains concentrated near 1.0 for strong models but shows increased variance for GPT-oss-20B and MiniMax M2.5 under the precision-focused prompt.
List of suppfigures E8 Cross-dataset recall correlation. Each point is one system; x-axis is recall on the primary benchmark, y-axis is recall on USDHUB. Points near the diagonal indicate consistent performance across corpora. TiDE shows the largest positive deviation (higher recall on USDHUB than primary). On USDHUB, TiDE was additionally run with known-PHI matching enabled (a supplied per-note dictionary of identifiers; see Supplement 1, Supplementary Note 4.1), a near-oracle configuration not available to the LLMs, which accounts for much of this deviation; USDHUB also contains standard demographic PHI without the institutionally situated identifiers that challenge pattern-based systems.
List of suppfigures E9 Cost–performance tradeoff (USDHUB, Baseline). F1 score vs. total API cost for the 49 notes (log scale). Traditional baselines (orange squares) incur zero marginal API cost. Among LLMs (blue circles), Opus 4.8 achieves the highest F1 at the highest cost (2.14forthecorpus,0.044/note), while DeepSeek, Kimi, GLM-5 and MiniMax all cost 0.17forthecorpus(0.003/note) at F1 0.67–0.84. The cost–F1 Pareto frontier is GPT-oss-20B (0.04),GLM−5(0.17), GPT-oss-120B (0.40)andOpus4.8(2.14); DeepSeek, Kimi and MiniMax are dominated. GPT-oss-20B is cheapest only because reasoning-token exhaustion truncated some outputs, so it is not a usable operating point despite lying on the frontier.
List of suppfigures E10 Latency–performance tradeoff (USDHUB, 49 notes, sequential). End-to-end processing time vs. F1. Local systems (TiDE, spaCy, OpenMed, regex) complete in under 1 minute with no API dependency. LLMs range from 3–62 minutes depending on model and API stability. GPT-oss-20B’s 62-minute runtime reflects retry overhead from persistent API errors, not inference speed.
List of suppfigures E11 Overall performance on USDHUB validation (49 notes, 409 gold spans, Baseline). LLMs are evaluated under identical conditions as the primary benchmark. TiDE achieves the highest recall (0.976 fully unprovisioned; 0.971 maximally provisioned with a known-PHI dictionary and per-note header, a difference of < 0.6 recall points on this canonical-PHI corpus; Supplement 1, Supplementary Note 4.1) while Opus 4.8 leads on F1 (0.894). The LLM–traditional gap is clearly visible, with traditional methods clustered below F1=0.70 except TiDE.
List of suppfigures E12 Per-category recall heatmap, USDHUB validation. PHI categories in USDHUB (Name, Date, ID, Phone, Address, Other) show similar patterns to the primary benchmark: LLMs achieve near-perfect recall on most categories while traditional baselines show systematic gaps, particularly on Names.
List of suppfigures E13 Cross-dataset performance comparison. Side-by-side recall (A) and F1 (B) for all systems on the primary benchmark (100 notes, blue) vs. USDHUB validation (49 notes, orange). Both datasets use the Baseline prompt. Performance rankings are largely preserved across datasets, supporting generalizability of the findings.
De-identification of clinical text is a prerequisite for the secondary use of electronic health records. Existing public benchmarks such as the i2b2 2006 and 2014 corpora are over a decade old and lack the semantic and demographic diversity of modern clinical narratives. Large Language Models (LLMs) reach state-of-the-art zero-shot extraction, but their use at enterprise scale is limited by computational cost and by hospital data governance that restricts sending Protected Health Information (PHI) to cloud APIs. We introduce SHIELD (Synthetic Human-annotated Identifier-replaced Entries for Learning and De-identification), a diverse clinical note dataset of 1,381 notes with 10,229 gold-standard PHI spans across 9 categories, built with set-cover diversity sampling across demographic and document-type strata and human-in-the-loop adjudication. We evaluate four LLMs (two proprietary, two open-weight) to establish a performance ceiling on SHIELD, then show that a teacher-student distillation framework transfers these capabilities into locally deployable Small Language Models. Our best distilled model reaches micro-averaged span-level precision of 0.89 and recall of 0.88 while running on standard workstation hardware. It trails its cloud teacher on per-category recall (0.90 vs. 0.81 macro-averaged) but remains competitive given its lower cost and on-premise deployability. Cross-dataset evaluation shows that diversity-trained models generalize well on universal structured PHI categories, while institution-specific entities remain hard to transfer in both directions, which suggests pairing broad-coverage models with specialized models for high-volume, semi-structured note types. We publicly release the SHIELD dataset and the distilled DeBERTa v3 model to provide an accurate, cost-effective de-identification pipeline deployable entirely behind institutional firewalls.
Jose D. Posada, David Love, Somalee Datta +1
Technology & Digital Solutions Stanford Medicine · Stanford Medicine
De-identifying clinical notes with a hosted language model discloses the information the step is meant to protect. We present EdgeDeID, a 0.6B-parameter decoder-only de-identifier that runs locally and is trained without real patient text: three LLM teachers write synthetic notes, a single labeler re-annotates them into one convention, and the model learns to emit a typed XML schema of the identifiers it finds. On a 100-note synthetic in-domain benchmark with manually verified annotations it reaches 0.994 micro-F1 and leaves no missed identifier in 99% of notes, against 0.857 and 12% for the strongest of eight off-the-shelf systems. Quantized, it runs on CPU in 615 MB. Of three additions to supervised training, only a verifier adapter helps: it separates complete from incomplete outputs of other systems at 0.94 AUROC, whereas reinforcement learning with a recall-weighted reward and a rule-based post-processing layer leave in-domain results unchanged. On MIMIC ICU nursing notes F1 falls to 0.48. The errors follow two properties our synthetic notes lack, telegraphic style and low identifier density, and three hand-written in-context examples recover 0.08 F1 without annotated target data.
Sina Daneshgar, Raya Namazi, Amirali Khosravi +2
Sharif University of Technology · Department of Radiology, Mayo Clinic · Department of Radiology and Biomedical Imaging, Yale University
Clinical notes contain personally identifiable information (PII), restricting reuse for research and medical AI, especially when data cannot leave an institution. We developed MedDeID, an on-premises framework combining in-house annotation and synthetic-note generation with model training, inference, pseudonymisation and evaluation. On an independently annotated, adjudicated 300-note Dutch hospital benchmark, a hospital-trained compact transformer detected 98.9% of identifying text while redacting 0.24% of text outside annotated identifiers; a synthetic-only counterpart detected 96.1%. On 100 primary-care notes, the synthetic-trained model achieved higher recall than the hospital-trained model (90.3% versus 87.0%) and greater robustness to identifier-format perturbations. An English instantiation trained without real text detected 99.7% and 98.9% of annotated identifier characters on two external synthetic benchmarks. These results demonstrate transfer of the workflow to another language, but not clinical English performance. MedDeID provides a route to locally governed de-identification using real or synthetic training data.
Stig Hellemans, Tom Stroobants, Elyne Scheurwegs +3
Adrem Data Lab, Department of Computer Science, University of Antwerp, Antwerp, Belgium · Antwerp University Hospital (UZA), Edegem, Belgium · Laboratory of Experimental Medicine and Pediatrics (LEMP), University of Antwerp, Antwerp, Belgium