Auditing black-box clinical prediction models with a large language model through queries alone: a blinded test with planted flaws
Organizations: Tianjin Medical University, Tianjin, China · Tianjin Medical University General Hospital, Tianjin, China
Abstract
Large language models (LLMs) are used to explain clinical prediction models, on the promise that they can say what a model computes, find its faults and even reveal new knowledge in it. We tested the three promises where the answers could be checked. From the International Stroke Trial we trained four kinds of model (logistic regression, explainable boosting machine, gradient-boosted trees, neural network) for two outcomes on patients randomised earlier, planted one flaw in each of 24 copies (an under-predicted subgroup, a reversed risk factor or an implausible step) and mixed them with the 8 clean models. GPT-6 Astra audited each model three times, blind, through a tool that only returned the model's predictions; the judging rules were committed before any run and applied by program. Its surrogate models, with a median of 128 terms, reproduced the audited models' log-odds with a mean of 0.976 (0.959 for an additive surrogate of 42 terms fitted by an analyst). It found the planted flaw in 69 of 72 runs (96%; 95% CI 80% to 99%); a script written for the purpose found all 24. It also reported many more problems: 60% of 1,089 checkable reports failed the pre-specified check against the model's average behaviour; post hoc, most of these were real but small steps, and 15% of the checkable reports were not found in the model. Of 601 unexpected relationships it read out of the clean models, 11% held in patients randomised later (random pairs 6%; a regression scan's shorter list of its strongest candidates, 28%); after a check on the training data, 33% against 30%. With query access alone, an LLM can describe a black-box model faithfully and find nearly every planted flaw, as a purpose-written script also did; its reports of problems must be checked on the model and its discoveries on data.
Figures & tables
| C statistic, test set | : mean (range over models) | Mean absolute difference | ||||
|---|---|---|---|---|---|---|
| Audited model | Death or dependency | Death | LLM | Analyst | LLM | Analyst |
| Logistic regression | 0.794 | 0.794 | 0.995 (0.962–1.000) | 0.988 (0.968–1.000) | 0.010 | 0.077 |
| Explainable boosting machine | 0.794 | 0.794 | 0.990 (0.962–0.998) | 0.973 (0.950–0.985) | 0.085 | 0.155 |
| Gradient-boosted trees | 0.792 | 0.792 | 0.977 (0.965–0.985) | 0.948 (0.934–0.962) | 0.141 | 0.209 |
| Neural network | 0.791 | 0.789 | 0.941 (0.923–0.958) | 0.925 (0.904–0.945) | 0.269 | 0.302 |
| All 32 models | 0.976 (0.923–1.000) | 0.959 (0.904–1.000) | 0.126 | 0.186 | ||
| LLM | Audit script | |||
|---|---|---|---|---|
| Planted flaw | Models | Runs that found it | Models found in all 3 runs | Models found |
| Subgroup under-prediction | 8 | 24 of 24 | 8 of 8 | 8 of 8 |
| Reversed direction | 8 | 21 of 24 | 7 of 8 | 8 of 8 |
| Implausible step | 8 | 24 of 24 | 8 of 8 | 8 of 8 |
| All | 24 | 69 of 72 | 23 of 24 | 24 of 24 |
| Failed the check | |||||||
|---|---|---|---|---|---|---|---|
| Kind of problem | Reported | Not checkable | Passed the check | All | Smaller | Some patients | Not found |
| Step | 921 | 0 | 378 | 543 | 409 | 50 | 84 |
| Direction | 156 | 43 | 22 | 91 | 0 | 29 | 62 |
| Subgroup | 55 | 0 | 34 | 21 | 3 | 0 | 18 |
| All | 1,132 | 43 | 434 | 655 | 412 | 79 | 164 |
| As proposed | After the check on the training set | |||
|---|---|---|---|---|
| Source | Proposed | Hold in later patients | Passed | Hold in later patients |
| LLM, not on the known list | 601 | 68 (11%; 9–14) | 103 | 34 (33%; 25–43) |
| Regression scan, not on the known list | 101 | 28 (28%; 20–37) | 82 | 25 (30%; 22–41) |
| Strongest interactions in the model (H statistic) | 348 | 35 (10%; 7–14) | 79 | 18 (23%; 15–33) |
| Random pairs (negative control) | 348 | 20 (6%; 4–9) | 17 | 2 (12%; 3–34) |
| Regression scan, on the known list (positive control) | 17 | 17 (100%; 82–100) | 17 | 17 (100%; 82–100) |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Outcome | Flaw | What was planted | Share changed | Mean change | Runs found | Script |
|---|---|---|---|---|---|---|---|
| EBM | Death | Reversed | effect of age reversed | 96% | 1.15 | 3 of 3 | yes |
| EBM | Death or dependency | Reversed | effect of leg_foot_deficit reversed | 75% | 1.53 | 3 of 3 | yes |
| GBT | Death | Reversed | effect of drowsy reversed | 22% | 1.54 | 3 of 3 | yes |
| GBT | Death or dependency | Reversed | effect of unconscious reversed | 1% | 3.85 | 3 of 3 | yes |
| LR | Death | Reversed | effect of visuospatial_disorder reversed | 18% | 0.58 | 0 of 3 | yes |
| LR | Death or dependency | Reversed | effect of drowsy reversed | 22% | 2.18 | 3 of 3 | yes |
| Step | Commit | Time (UTC+8) |
|---|---|---|
| Plan, programs and judging rules, before any model was trained | 09a9166 | 09 Nov 2026 10:43:27 |
| Audited models, sealed flaw key and prompts | 5901dc2 | 09 Nov 2026 13:06:59 |
| Amendments, before any run | 173bd59 | 16 Nov 2026 04:36:19 |
| First run started; last run started | 16 Nov 2026 04:36:24; 16 Nov 2026 09:18:02 | |
| Replies frozen, before any parsing or judging | 00a49a6 | 17 Nov 2026 00:23:38 |
| Parsed replies and the reading of the tool calls | 514cd51 | 17 Nov 2026 00:24:44 |
Black-Box Model Auditing
Black-box model auditing evaluates AI systems through their inputs and outputs to assess privacy, fairness, safety, reliability, and provenance.
See all 103 papers