cs.CLAug 17, 2026
SaveInstitution-Specific LLM Prompting Recovers PHI That De-identification Systems and Their Gold Standards Both Miss
Organizations: Quantitative and Computational Biosciences, Baylor College of Medicine, Houston, Texas, USA · Department of Pediatrics, Baylor College of Medicine, Houston, Texas, USA · Jan and Dan Duncan Neurological Research Institute, Texas Children’s Hospital, Houston, Texas 77030, USA · Data Science Center, Texas Children’s Hospital, Houston, Texas 77030, USA ·
Abstract
Secondary use of electronic health records requires de-identification, yet existing systems miss institutionally situated protected health information (PHI) such as hospital abbreviations, building names, and internal codes whose status is locally determined. We ask whether large language models (LLMs) with in-context learning (ICL) can close this gap and control the precision--recall trade-off. On 100 annotated pediatric oncology notes (5,322 PHI spans) from Texas Children's Hospital, we benchmarked eight LLMs against two purpose-built systems (Stanford TiDE, OpenMed PII) and two pattern-based baselines. Each LLM ran under three prompts of increasing specificity: (1) a HIPAA-aligned baseline, (2) baseline plus the institutional PHI categories it missed, and (3) prompt 2 plus instructions against over-redacting clinical content. We then compared 14multi-agent and ensemble configurations against the best single prompt, with recall the primary safety metric. LLMs outperformed the purpose-built systems (best F1=0.9180.001 vs.\ TiDE 0.779), with advantages concentrated in contextual categories. Naming the missed categories recovered 79% (48/61) of them, and discouraging over-redaction restored precision. No agentic architecture beat calibrated single-pass prompting (F1 0.906--0.907), but LLM outputs surfaced 414candidate annotation gaps; re-annotation confirmed 227~PHI spans, against which the final prompt reached recall=0.981 (F1=0.9070.002). Well-calibrated ICL resolves both the institutional PHI gap and the precision--recall trade-off in one LLM call per note. LLMs cost more to run than traditional methods, but that cost buys a way to audit the reference standard. LLMs are a legitimate, adaptable alternative to purpose-built de-identification systems; institution-specific prompt development should be the primary adaptation strategy.
Figures & tables
| System | Type | Family | Parameters |
| Claude Opus 4.8 | LLM | Anthropic | Undisclosed |
| Claude Sonnet 4.6 | LLM | Anthropic | Undisclosed |
| GPT-oss-120B | LLM | OpenAI | 120B |
| GPT-oss-20B | LLM | OpenAI | 20B |
| GLM-5 | LLM | Zhipu AI | 754B |
| Kimi K2.5 | LLM | Moonshot | 1.1T |
Table 1: LLMs and purpose-built systems evaluated. The two pattern-based baselines (regex-only; spaCy NER + regex) are specified in the text and are included in all reported comparisons.
Figure 1: Study design and corpus characteristics. (A) Study design: 8 LLMs (top 4 shown in main figures) + traditional baselines evaluated on 100 pediatric oncology notes (5,322 spans) under 3 prompt conditions. (B) Synthetic clinical note illustrating canonical HIPAA PHI (names, dates; blue) vs. institutionally situated PHI (facility abbreviations, building names; orange). (C) Gold standard PHI distribution over the six categories with (four further categories hold spans each; all ten, with their denominators, are in Supplementary Table S4): Date (72.1%) and Name (22.3%) dominate, while institutionally enriched categories (Other Unique ID, Geographic Data) comprise 3.7% of spans. This distribution is that of the original gold standard; because that annotation under-counted institutional identifiers (see Results), 3.7% is a lower bound on their true prevalence.
Figure 2: LLMs outperform traditional de-identification approaches. (A) Recall, precision, and F1 for the top 4 LLMs and 4 traditional baselines ( Baseline prompt, 100 notes, 5,322 spans; all 8 LLMs in Table S3 and Supplementary Figure 1). (B) Per-category recall heatmap for the same systems, with categories ordered by gold-standard frequency and systems by overall F1, reveals shared weakness on the institutionally enriched categories (Other Unique ID [OUI], Geographic Data); the four categories with 5 gold spans are omitted here and reported in Table S4. (C) Sonnet 4.6 false negative decomposition on the primary Baseline run (Table S3; 211 FN): 29% are institutionally situated PHI (61 of 211; institution abbreviations 38, building/facility 15, department/clinic codes 8; see taxonomy in Table S7); the remaining 150 are canonical, of which the internal staff codes (pager and Voalte extensions, 4) are shown separately because Table S7 lists them as borderline.
Figure 3: In-context learning enables control over the precision–recall trade-off. (A) Precision, recall, and F1 for the top 4 LLMs across the three prompt versions ( Baseline , Targeted , Precision ; single run per model, all 8 in Table S5). Each horizontal line spans the minimum-to-maximum of a model’s three stage values, so a non-monotonic Targeted stage visibly stretches the line rather than being hidden; Baseline and Precision values are labeled. Effect magnitude and direction are model-dependent. (B) Problem-specific deltas (single run), same model order and colors as (A), isolating what each prompt revision buys: Recall( Targeted Baseline ) is the recall gain from institutional targeting, and Precision( Precision Targeted ) is the precision recovery from the anti-over-redaction instructions. The Precision prompt lifts precision for all models (cross-model mean over the 7 models with valid output: +0.054) and acts as a corrective: gains are largest for models that over-redact under the baseline prompt (GPT-oss-120B , Kimi , DeepSeek across the full Baseline Precision progression; full values in Tables S5 and S6) and near-zero for models whose native precision is already high. Sonnet 4.6 reproducibility (5 trials per stage on the full 100-note corpus; SD 0.007 at every stage) confirms these are systematic prompt effects, not run-to-run noise (Table S9).
Figure 4: Enhanced gold standard validation and agentic architecture comparison. (A) Configuration landscape: F1 for the 16 configurations that the panel displays, colour-coded by class (single-pass, agentic, ensemble, traditional) and ranked on the enhanced gold standard; 10 further configurations are omitted for legibility and appear in Table S12, which lists all 26. The omitted set is the weakest-performing tail, except that TiDE is retained as the traditional-system reference. Single-pass Sonnet ( Precision ) (dashed line) matches or exceeds every multi-agent variant, and self-refinement and ensemble voting fail to beat it. (B) Reproducibility: 5-trial strip plots for the top 3 methods. All three achieve closely comparable mean F1 (0.906–0.908) with overlapping 95% trial-resampled CIs, but the Scrubber–Auditor ( Precision ) pipeline on Opus shows high variance (SD =0.018) vs. near-deterministic single-pass Sonnet (SD =0.002). (C) Sonnet 4.6 prompt progression on the enhanced gold standard (10-note subset, 1,758 spans; 5 independent trials per version). Lines show mean; shading shows SD; points show individual trials. Baseline recall is 0.847 here versus 0.958 on the original 100-note gold standard (a different note set and a different gold standard) because it misses newly-annotated institutional terms; Targeted recovers recall to 0.980; Precision maintains recall while improving precision (F1=0.907 0.002). Terminology: single-pass = one LLM call per note; dual-pass = two sequential LLM calls; structured pass 2 = a second pass that ingests the first pass’s output in a structured format; Scrubber–Auditor = an agentic pipeline in which a generator LLM (scrubber) is checked by a second auditor LLM. Baseline , Targeted , and Precision denote the three prompt versions.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Name in text | Bedrock model identifier | Provider |
| Claude Opus 4.8 | us.anthropic.claude-opus-4-8 † | Anthropic |
| Claude Sonnet 4.6 | us.anthropic.claude-sonnet-4-6 † | Anthropic |
| GPT-oss-120B | openai.gpt-oss-120b-1:0 | OpenAI |
| GPT-oss-20B | openai.gpt-oss-20b-1:0 | OpenAI |
| GLM-5 | zai.glm-5 | Zhipu AI |
| Kimi K2.5 | moonshotai.kimi-k2.5 | Moonshot |
Table E1: Exact AWS Bedrock model identifiers for the eight benchmarked LLMs. All models were invoked through the Bedrock Converse API in region us-east-1 at temperature=0.0 (except Opus 4.8, which rejects the parameter), max_tokens=65,000 , with up to 3 retries under exponential backoff.
| System | Latency | Cost (49 notes) | Cost/note | Platform |
|---|---|---|---|---|
| Claude Opus 4.8 | 6.8 min | $2.142 | $0.0437 | Bedrock API |
| Claude Sonnet 4.6 | 4.6 min | $0.428 | $0.0087 | Bedrock API |
| GPT-oss-120B | 5.6 min | $0.400 | $0.0082 | Bedrock API |
| DeepSeek V3.2 | 6.4 min | $0.166 | $0.0034 | Bedrock API |
| Kimi K2.5 | 3.3 min | $0.166 | $0.0034 | Bedrock API |
| GLM-5 | 11.5 min | $0.166 | $0.0034 | Bedrock API |
Table E2: Cost and latency comparison on the USDHUB validation dataset (49 notes, sequential processing). LLM costs reflect AWS Bedrock on-demand pricing; local systems have zero marginal API cost (compute is pre-provisioned on the evaluation instance).
List of suppfigures E1 Per-note recall distribution (Baseline). Each point represents one clinical note. Most top models cluster near perfect recall (1.0), but all exhibit a tail of low-recall notes; these correspond to notes rich in institutionally situated PHI.
List of suppfigures E2 Perfect de-identification rate (Baseline). Fraction of notes with zero leaked PHI spans. DeepSeek and GLM-5 achieve perfect de-identification on 60% of notes; no model exceeds 65%, reflecting the long tail of institutionally situated identifiers.
List of suppfigures E3 Per-category recall change under targeted prompting (Targeted Baseline). The targeted prompt yields concentrated gains in Other Unique ID and Name (institutionally situated categories) for responsive models, while MiniMax M2.5 shows broad regressions. Geographic Data shows mixed effects.
List of suppfigures E4 Overall performance under precision-focused prompt (Precision). The "do-not-over-redact" instructions improve precision across most models relative to Baseline/Targeted, with GPT-oss-120B showing the most dramatic improvement. GPT-oss-20B suffers output failures (reasoning exhausts token budget).
List of suppfigures E5 Per-category recall change (Precision Baseline). Unlike Targeted (Supplementary Figure E3 ), the Precision prompt shows smaller recall deltas for strong models, confirming that precision can be improved without substantial recall degradation. GPT-oss-20B and MiniMax M2.5 are outliers with large regressions.
List of suppfigures E6 Perfect de-identification rate (Precision). Under the precision-focused prompt, DeepSeek V3.2 and Kimi K2.5 maintain the highest perfect de-id rates ( 60%), while GPT-oss-20B and MiniMax M2.5 show substantial degradation. Compare with Supplementary Figure E2 for baseline.
List of suppfigures E7 Per-note recall distribution (Precision). Compared to Baseline (Supplementary Figure E1 ), the distribution remains concentrated near 1.0 for strong models but shows increased variance for GPT-oss-20B and MiniMax M2.5 under the precision-focused prompt.
List of suppfigures E8 Cross-dataset recall correlation. Each point is one system; x-axis is recall on the primary benchmark, y-axis is recall on USDHUB. Points near the diagonal indicate consistent performance across corpora. TiDE shows the largest positive deviation (higher recall on USDHUB than primary). On USDHUB, TiDE was additionally run with known-PHI matching enabled (a supplied per-note dictionary of identifiers; see Supplement 1, Supplementary Note 4.1), a near-oracle configuration not available to the LLMs, which accounts for much of this deviation; USDHUB also contains standard demographic PHI without the institutionally situated identifiers that challenge pattern-based systems.
List of suppfigures E9 Cost–performance tradeoff (USDHUB, Baseline). F1 score vs. total API cost for the 49 notes (log scale). Traditional baselines (orange squares) incur zero marginal API cost. Among LLMs (blue circles), Opus 4.8 achieves the highest F1 at the highest cost (0.044/note), while DeepSeek, Kimi, GLM-5 and MiniMax all cost 0.003/note) at F1 0.67–0.84. The cost–F1 Pareto frontier is GPT-oss-20B (0.17), GPT-oss-120B (2.14); DeepSeek, Kimi and MiniMax are dominated. GPT-oss-20B is cheapest only because reasoning-token exhaustion truncated some outputs, so it is not a usable operating point despite lying on the frontier.
List of suppfigures E10 Latency–performance tradeoff (USDHUB, 49 notes, sequential). End-to-end processing time vs. F1. Local systems (TiDE, spaCy, OpenMed, regex) complete in under 1 minute with no API dependency. LLMs range from 3–62 minutes depending on model and API stability. GPT-oss-20B’s 62-minute runtime reflects retry overhead from persistent API errors, not inference speed.
List of suppfigures E11 Overall performance on USDHUB validation (49 notes, 409 gold spans, Baseline). LLMs are evaluated under identical conditions as the primary benchmark. TiDE achieves the highest recall (0.976 fully unprovisioned; 0.971 maximally provisioned with a known-PHI dictionary and per-note header, a difference of 0.6 recall points on this canonical-PHI corpus; Supplement 1, Supplementary Note 4.1) while Opus 4.8 leads on F1 (0.894). The LLM–traditional gap is clearly visible, with traditional methods clustered below F1=0.70 except TiDE.
List of suppfigures E12 Per-category recall heatmap, USDHUB validation. PHI categories in USDHUB (Name, Date, ID, Phone, Address, Other) show similar patterns to the primary benchmark: LLMs achieve near-perfect recall on most categories while traditional baselines show systematic gaps, particularly on Names.
List of suppfigures E13 Cross-dataset performance comparison. Side-by-side recall (A) and F1 (B) for all systems on the primary benchmark (100 notes, blue) vs. USDHUB validation (49 notes, orange). Both datasets use the Baseline prompt. Performance rankings are largely preserved across datasets, supporting generalizability of the findings.
More in this topic
PII Detection
Personally identifiable information (PII) detection finds sensitive personal details in text, conversations, images, and other data.
See all 39 papers