cs.AINov 14, 2026

From Mutation to Medication: Characterizing Failures of Clinical LLMs to Respect Genomic Resistance

Authors: Negin Ashrafi, Roxana Daneshjou

Organizations: Stanford University, Stanford, CA, USA

Abstract

Language models are increasingly proposed to support molecular tumour boards, but may recommend cancer therapies despite genomic evidence of resistance. We evaluated ten models on 1,464 resistance cases from five curated oncology knowledge bases, collecting 58,910 judgments. Models endorsed resisted therapies in 21.8-81.4% of cases. However, raw rates can reflect a tendency to agree rather than an ability to distinguish resistance from drug sensitivity: one open-weight model endorsed 81.4% of resistance cases and 82.3% of sensitivity controls. We therefore compared endorsement within matched gene and evidence-level groups. This analysis is exploratory because it was specified after the pre-registered analysis, which yielded null-to-weak results. Measured failures depended strongly on the reference knowledge base and evidence strength. When a second source independently confirmed resistance, four models across three vendors endorsed at most 2.4% of cases; at weaker evidence tiers, discrimination was not measurable. Models sometimes acknowledged resistance yet recommended the therapy. Repeating a model's own resistance conclusion reduced endorsement to 0.0-3.5% on cases where it had stated resistance. Falsely attributing the opposite conclusion increased endorsement above baseline for all seven hosted models, reaching 95.2% for GPT-4o; an instruction without an attributed conclusion changed endorsement by only 0.9 percentage points. Providing the correct curated evidence reduced unqualified strong recommendations to 0.3-3.5%, testing evidence use under an oracle condition rather than retrieval.

Explore similar work

CardsList
  1. OpenMTB-Audit: Exposing Over-Refusal and Clinical Expert Perspectives in LLM-Based Molecular Tumor Board Safety Evaluation

    Oct 1, 2026Negin Ashrafi, Jia Luo, Stacey M. Frumm +1LLM Safety BenchmarksLLM Auditing

  2. Agentic systems for breast cancer treatment recommendations

    Jul 13, 2026Vinicius Anjos de Almeida, Nícolas Henrique Borges, Leonardo Vicenzi +5Multi-Agent LLM SystemsClinical Decision Support

  3. Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts

    Sep 29, 2026Abinitha Gourabathina, Haoran Zhang, Yuexing Hao +2Clinical TriageClinical Decision-Making