Accessed, never disclosed: Silent access to protected health information by medical artificial intelligence agents

Jae Wang Lee; Abhirup Banerjee; Yeong Chan Lee

cs.AINov 2, 2026

Accessed, never disclosed: Silent access to protected health information by medical artificial intelligence agents

Authors: Jae Wang Lee, Abhirup Banerjee, Yeong Chan Lee

Organizations: Department of Software, College of Computing and Informatics, Ajou University, Suwon, Republic of Korea · Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, OX3 7DQ, United Kingdom · Department of Physiology, Ajou University School of Medicine, Suwon, Republic of Korea

Abstract

Medical artificial intelligence agents should retrieve patient identifiers only when a task requires them, the data-minimization principle behind the HIPAA minimum necessary standard for protected health information (PHI). An agent can retrieve identifiers without revealing them. Such silent access remains invisible to evaluations that inspect only the output. We embedded canaries in synthetic patient records and linked each tool invocation to canary exposure in the final answers across five language models, four boundary levels, and seven attack frames for identifier-free clinical tasks. Three models invoked the PHI tools unnecessarily, whereas Claude Opus 4.8 and GPT-5.5 did not. Stronger boundaries reduced this access, and the residual access under the strictest policy was confined to frames in which the user overrode the policy or asserted approval. A pre-execution gate given the task-necessity criterion nearly eliminated this access. Output-only evaluation cannot verify minimum-necessary access; boundaries, gates, and access-log audits are required together.

Medical AI Agents

See all 49 papers

BodhiPromptShield: Pre-Inference Prompt Mediation for Surface-Form Privacy Propagation in LLM Agent Pipelines
Apr 7, 2026Bo Ma, Jinsong Wu, Weiqi YanData Leakage Privacy-Preserving Language Models
Auckland University of Technology Auckland, New Zealand · Guilin University of Electronic Technology Guilin, China
Clinically Grounded Privacy Evaluation of Medical LMs
Jun 8, 2026Sasha Ronaghi, Sana Tonekaboni, Lena Stempfle +6Data Leakage Memorization in Language Models
1Stanford University · 2Massachusetts Institute of Technology · 3American Board of Family Medicine
Institution-Specific LLM Prompting Recovers PHI That De-identification Systems and Their Gold Standards Both Miss
Aug 17, 2026Daniel Palacios, Matthew Brady Neeley, Angel Adetomike Otto +6PII Detection Electronic Health Records
Quantitative and Computational Biosciences, Baylor College of Medicine, Houston, Texas, USA · Department of Pediatrics, Baylor College of Medicine, Houston, Texas, USA · Jan and Dan Duncan Neurological Research Institute, Texas Children’s Hospital, Houston, Texas 77030, USA +2

Apr 7, 2026J/K move · Enter open · S save

BodhiPromptShield: Pre-Inference Prompt Mediation for Surface-Form Privacy Propagation in LLM Agent Pipelines

Bo Ma, Jinsong Wu, Weiqi Yan

Auckland University of Technology Auckland, New Zealand · Guilin University of Electronic Technology Guilin, China

Data Leakage Privacy-Preserving Language Models LLM Agent Security Privacy Leakage in Language Models PII Detection Privacy Protection for LLM Agents

Open paper AskPDF arXiv

In LLM agent pipelines, prompt privacy risk propagates beyond a single model call: raw user content enters retrieval queries, memory writes, tool arguments, OCR-derived text, and logs, and every downstream copy inherits what the first write contained. Existing de-identification pipelines protect document boundaries but not this cross-stage surface. We present BodhiPromptShield, a policy-aware mediation layer that detects sensitive spans before they propagate, replaces each with a typed placeholder, a semantic abstraction, or a secure symbolic token under a configured policy, and defers restoration to authorized execution boundaries. We evaluate it under one protocol against Presidio, Casper-style sanitization, an LLM sanitizer, and transformer and learned detectors, on 300 AI4Privacy documents, 493 PrivacyLens trajectories, 200 PrivacyLens tasks scored by that benchmark's own judge, and AgentDojo tasks under injection. Three findings result. Identifier propagation is controllable: residual exposure falls to 7.4% on AI4Privacy and 1.8% on PrivacyLens, and exact identifiers in an agent's final action fall from 13.7% to 2.1-3.1%. Restoration timing governs what every stage upstream of the authorized boundary sees: deferring it leaves 1.6% of protected values readable in the released context against 51.0%, and 2.7% against 4.8% in what the agent emits, for 0.11 helpfulness points. Measuring factual disclosure is harder: a word-overlap metric and an LLM judge both report that mediation leaves facts intact, and both disagree with blind human annotation (kappa = 0.25 and 0.09). The human labels reverse that: inferability falls from 100% to 24-53% under mediation, so semantic-leakage measures need human validation before they are trusted. These are systems results on English text with open-weight models, not formal guarantees.

Abstract

Medical AI Agents

Explore similar work

BodhiPromptShield: Pre-Inference Prompt Mediation for Surface-Form Privacy Propagation in LLM Agent Pipelines

Clinically Grounded Privacy Evaluation of Medical LMs

Institution-Specific LLM Prompting Recovers PHI That De-identification Systems and Their Gold Standards Both Miss