cs.LGNov 18, 2026

Auditing black-box clinical prediction models with a large language model through queries alone: a blinded test with planted flaws

Authors: Zhiyu Liu, Xinyu Yang

Organizations: Tianjin Medical University, Tianjin, China · Tianjin Medical University General Hospital, Tianjin, China

Abstract

Large language models (LLMs) are used to explain clinical prediction models, on the promise that they can say what a model computes, find its faults and even reveal new knowledge in it. We tested the three promises where the answers could be checked. From the International Stroke Trial we trained four kinds of model (logistic regression, explainable boosting machine, gradient-boosted trees, neural network) for two outcomes on patients randomised earlier, planted one flaw in each of 24 copies (an under-predicted subgroup, a reversed risk factor or an implausible step) and mixed them with the 8 clean models. GPT-6 Astra audited each model three times, blind, through a tool that only returned the model's predictions; the judging rules were committed before any run and applied by program. Its surrogate models, with a median of 128 terms, reproduced the audited models' log-odds with a mean R2R^2 of 0.976 (0.959 for an additive surrogate of 42 terms fitted by an analyst). It found the planted flaw in 69 of 72 runs (96%; 95% CI 80% to 99%); a script written for the purpose found all 24. It also reported many more problems: 60% of 1,089 checkable reports failed the pre-specified check against the model's average behaviour; post hoc, most of these were real but small steps, and 15% of the checkable reports were not found in the model. Of 601 unexpected relationships it read out of the clean models, 11% held in patients randomised later (random pairs 6%; a regression scan's shorter list of its strongest candidates, 28%); after a check on the training data, 33% against 30%. With query access alone, an LLM can describe a black-box model faithfully and find nearly every planted flaw, as a purpose-written script also did; its reports of problems must be checked on the model and its discoveries on data.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Computation, not weight access: large language models as auditors of clinical prediction networks

    Nov 5, 2026Zhiyu Liu, Xinyu YangLLM AuditingLLM Evaluation

  2. Measurement Under Selection: Decoy-Calibrated Failure Audits for Language Models

    Jun 8, 2026Vyzantinos Repantis, Ameya Gawde, Harshvardhan SinghLLM EvaluationLanguage Model Error Detection

  3. Stress-testing medical large language models reveals latent safety pathology beyond benchmark accuracy

    Jun 6, 2026Yuan Shen, Xiaojun Wu, Linghua YuLLM EvaluationLanguage Model Safety Evaluation