cs.AINov 14, 2026

LABEL: Literature Asymmetry between Efficacy and Limits in Drug Knowledge and Language Models

Authors: S M Rayeed, Charles V. Stewart, Yao Ma

Organizations: Rensselaer Polytechnic Institute, Troy, NY

Abstract

Every medication comes with two types of medical facts: indications, which describe what conditions the drug treats (benefits), and contraindications, which describe who must avoid it to prevent serious harm (safety limits). Although contraindications are critical for patient safety and FDA regulations require listing them as rigorously as indications, we find that contraindications are far less documented than indications in general prose. In this paper, we measure how often these texts contain both facts across 574 drugs from PrimeKG by comparing a drug's indications against its own contraindications. Across Wikipedia articles and PubMed abstracts, we find that reference prose names a drug's benefits about twice as often as its safety limits (within-drug odds ratio 2.19). We call this documentation gap LABEL (Literature Asymmetry between Efficacy and Limits). Language models mirror this gap. As we scale Qwen2.5 from 0.5B to 14B, accuracy on indications rises by 18.3 points, whereas accuracy on contraindications remains flat near random guessing. However, when we provide models with a retrieved passage that states the answer, they use safety facts just as effectively as benefits. Furthermore, routing safety questions to official FDA labels boosts accuracy by 15.8 points whenever the label states the fact. Overall, our findings show that models fail on medication safety not because they cannot reason about risk, but because free-text corpora lack safety facts. Clinical AI systems should therefore route medication-safety questions to official regulatory labels and structured databases rather than relying on general public records. Code and data are available at https://github.com/SMRayeed/LABEL.

Figures & tables

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Aug 4, 2026

Evaluating Counterfactual Sensitivity to Patient Information in Medication-Safety Reasoning

Applying a valid medication-safety rule when its patient-specific conditions are not met can produce an incorrect decision. Existing medical evaluations largely use isolated and fixed scenarios. A model may therefore answer correctly by recalling a drug-risk association without showing that it used patient information to decide whether the rule applies. To address this gap, we introduce MedPIC-Bench, a benchmark of source-verifiable recommendations and expert-validated questions for patient-specific medication-safety reasoning. It combines guideline-following questions with paired counterfactual questions in which a controlled change in patient information changes whether a rule applies. The benchmark contains 467 questions annotated along six clinical and reasoning dimensions. Across 28 medical-specific, general, and proprietary LLMs, every model performs worse on counterfactual questions, with mean accuracy falling from 63.6% to 45.1%. Models perform well when an explicit patient attribute directly signals a familiar contraindication, but struggle when patient information must narrow or withdraw a safety warning. Model rationales often acknowledge the changed patient information, yet the final answers retain the previous safety judgment. This vulnerability persists among medical-specific LLMs, whose average CF performance trails that of general LLMs. MedPIC-Bench therefore makes conditional rule application measurable and highlights the limitations of static medication-safety accuracy for assessing patient-specific reliability.
Jun 10, 2026

DrugBench: Evaluating AI Control Protocols for Medication Harm Mitigation

Large Language Models have the potential to expand and improve the access to clinical information by enabling new ways of interacting with medical knowledge in natural language. However, their deployment in medical question-answering settings is safety-critical, since misaligned outputs can lead to severe patient harm. AI control is an emerging approach that introduces external safeguards to mitigate unsafe behaviours in misaligned systems and has been shown to be effective in domains such as code generation. However, its applicability and effectiveness in medical settings have not been systematically studied. In this work, we present a pipeline for evaluating AI control protocols to mitigate medication-related harm. To this end, we introduce DrugBench, an AI control evaluation benchmark which combines 3,671 multi-turn medical conversations from HealthBench with drug information from official FDA labels, covering four categories of medication-related harm: drug interactions, contraindications, dosing constraints, and patient action restrictions. Furthermore, inspired by the medical domain, we argue that safety should account for the severity of unsafe outputs, not just their probability. Under this revised definition, we show that existing control protocols can be subverted and propose severity-based monitoring to address this limitation.
Nov 17, 2026

MedGuardBench: A Patient-Oriented Benchmark for Personalized Guardrail Development

Safety is personal: the same action can be safe for one person but unsafe for another. Yet current guardrails judge content without considering who is affected. Using medication safety, we introduce patient-conditioned medication guarding, which predicts a safety verdict and 17 risk categories from a patient profile and prescription. We release MedGuardBench, a drug-disjoint benchmark of 5{,}001 scenarios across 87 medications. We find that standard guardrails rely on surface cues, such as wording and profile presence, rather than patient facts. To address this, we develop personalized reasoning guardrails with three components. Personalized Reasoning Distillation links risks to relevant patient fields. Context Unrolling (CU) makes models check each field themselves, reducing false alarms on most backbones; controls show that providing the check does not yield the same benefit. A rule-based reward (guided GRPO) improves unsafe recall and complex-risk detection, including allergies and adverse reactions, on some backbones, but increases false alarms on others. Our models outperform base models and matched no-profile controls in accuracy, while reducing false alarms relative to no-profile controls. Profile removal and swapping confirm that gains come from patient information, not fine-tuning alone.