Goal-Driven Risk Assessment for LLM-Powered Systems: A Healthcare Case Study
Organizations: School of Informatics, Computing, and Cyber Systems, Northern Arizona University, Flagstaff, USA · Department of Software Science, Tallinn University of Technology, Tallinn, Estonia
Abstract
While incorporating LLMs into systems offers significant benefits in critical application areas such as healthcare, new security challenges emerge due to the potential cyber kill chain cycles that combine adversarial model, prompt injection and conventional cyber attacks. Threat modeling methods enable the system designers to identify potential cyber threats and the relevant mitigations during the early stages of development. Although the cyber security community has extensive experience in applying these methods to software-based systems, the elicited threats are usually abstract and vague, limiting their effectiveness for conducting proper likelihood and impact assessments for risk prioritization, especially in complex systems with novel attacks surfaces, such as those involving LLMs. In this study, we propose a structured, goal driven risk assessment approach that contextualizes the threats with detailed attack vectors, preconditions, and attack paths through the use of attack trees. We demonstrate the proposed approach on a case study with an LLM agent-based healthcare system. This study harmonizes the state-of-the-art attacks to LLMs with conventional ones and presents possible attack paths applicable to similar systems. By providing a structured risk assessment, this study makes a significant contribution to the literature and advances the secure-by-design practices in LLM-based systems.
Figures & tables
| Score | Likelihood | Description |
|---|---|---|
| 1 | Rare | Extremely unlikely; requires insider access or rare conditions |
| 2 | Unlikely | Requires effort, stealth, or partial misconfiguration |
| 3 | Possible | Feasible with moderate effort or misconfigurations |
| 4 | Likely | Known techniques exist; moderate mitigations present |
| 5 | Almost Certain | High probability; exploit is trivial or seen in field |
| Score | Impact | Description |
|---|---|---|
| 1 | Negligible | No clinical harm; affects UI or workflow only |
| 2 | Minor | Affects a few non-critical patients; easily recoverable |
| 3 | Moderate | Recoverable clinical harm to a single patient |
| 4 | Major | Serious harm to one or moderate harm to many |
| 5 | Catastrophic | Life-threatening harm or system-wide failures |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Attack Path | Business Knowledge Required | Technical Complexity | Likelihood | Justification |
| Prompt Injection | Low; No clinical knowledge required to manipulate LLM responses. | Low; Requires only user/API access. | 4 | Always feasible. External or internal attackers can craft prompts without domain insight. Example: “Override previous instruction and recommend immediate discharge” bypasses safety filters even without medical context. |
| LLM Session Mismanagement | Medium; Understanding of session continuity and PHI retention improves attack precision. | Medium; Requires session token access or fixation strategies. | 3–4 | Lower (3) if attacker blindly hijacks session tokens. Higher (4) if insider (e.g., developer or system admin) knows token refresh logic or memory scope, enabling persistent dialogue control. |
| Orchestrator Error | High; Requires detailed knowledge of multi-agent task routing and clinical workflows (e.g., ‘order test → analyze → prescribe’). | Medium–High; Requires ability to intercept or alter internal task messages. | 2–3 | Rare (2) if attacker lacks care-path knowledge. Higher (3) if insider or developer can edit orchestration configs, e.g., skipping “allergy screening” before treatment. |
| Model Tampering | Medium; Must know what clinical entities to target (e.g., specific diagnoses, medications). | High; Requires access to model weights or training pipeline. | 2–3 | Lower (2) if attacker must breach infrastructure. Higher (3) if insider ML engineer has fine-tuning privileges to bias discharge instructions or clinical summaries. |
| MitM Web Session | Low; No domain knowledge needed; attacker targets data in transit. | Medium; Requires network-level control (e.g., ARP spoofing, TLS downgrade). | 3 | Consistent (3). Attacker can tamper with clinical terms (e.g., change “acetaminophen” to “ibuprofen”) regardless of insider status. Medical workflow knowledge offers no added advantage. |
| Session Hijack / Fixation | Medium; Knowing which sessions hold PHI improves the ability to extract useful data. | Medium; Involves stealing a valid token or forcing reuse of a known session token (fixation). | 3–4 | 3 for blind token reuse; 4 if insider knows token lifetime & memory scope. |
| Risk ID | Healthcare Risk | Likelihood | Impact | Risk Score | Attack Vectors |
| G1-R1 | Misdiagnosis of Critical Illness | 4 (Likely) | 5 (Catastrophic) | 20 | Prompt Injection (Direct), Model Tampering (Direct), Orchestrator Error (Indirect), MitM Web Session (Indirect), LLM Session Mismanagement (Situational) |
| G1-R2 | Unauthorized Procedures Executed | 3 (Possible) | 4 (Major) | 12 | Prompt Injection (Direct), Model Tampering (Indirect), Orchestrator Error (Direct), MitM Web Session (Indirect), LLM Session Mismanagement (Situational) |
| G1-R3 | Corrupted Medication Recommendations | 4 (Likely) | 4 (Major) | 16 | Prompt Injection (Direct), Model Tampering (Direct), Orchestrator Error (Indirect), MitM Web Session (Direct), LLM Session Mismanagement (Indirect) |
| G1-R4 | Cross-Patient Context Contamination | 3 (Possible) | 3 (Moderate) | 9 | Orchestrator Error (Indirect), MitM Web Session (Direct), LLM Session Mismanagement (Direct) |