Does Your Trial Simulator Know What the Drug Does? An Intervention-Blind Test of Interventional Validity on 7,207 Registered Trials
Organizations: Westcliff University
Abstract
Patient world models are increasingly proposed as simulators of clinical trials, and recent surveys rank their capabilities on a ladder from temporal prediction to counterfactual rollouts. Position on that ladder is currently assigned by reading papers rather than by measurement. We turn the counterfactual rung into a measured quantity. From 30,119 completed phase 2/3 trials with results posted to ClinicalTrials.gov we construct 9,996 audited within-trial contrasts from 7,207 trials, and we score a simulator on how much of its accuracy depends on knowing which intervention was tested. The instrument is an intervention-blind reference: the same model, trained on the same trials, with every intervention-identifying field withheld. Interventional skill is the error the full model removes relative to that reference. On a temporal split (878 test contrasts from 655 trials) a registry-trained gradient-boosting simulator removes a fifth of the error of a trivial baseline, but its interventional skill is -0.12 (95% CI -0.21 to -0.05): withholding the intervention makes it better, and no alternative featurisation, stratum, temporal cut or same-molecule retrieval yields an interval above zero. A frontier language model reading only the registered protocol has interventional skill 0.20 [0.08, 0.31]; smaller and open-weight models sit at the limit of detection. Perturbation tests show the large model uses the intervention as such: swapping the arms reverses the sign of its prediction in 86% of contrasts. The blind reference is itself not drug-blind (a language model recovers the withheld drug class 45% of the time from the blind prompt), so interventional skill is a conservative lower bound. A reliability layer shows that ranking by predicted error lowers it but that calibrated error control does not transfer under temporal shift. Benchmark, code, prompts and per-call model outputs are released.
Figures & tables
| Task | System | skill( blind ) | skill( full ) | interventional skill |
|---|---|---|---|---|
| Risk difference | global mean | – | 0 | – |
| gbt (5 seeds) | 0.297 [ , ] | 0.211 [ , ] | [ , ] | |
| lm large, protocol-only | 0.462 [ , ] | 0.570 [ , ] | 0.201 [ , ] | |
| lm large, posted-record | 0.494 [ , ] | 0.629 [ , ] | 0.266 [ , ] | |
| lm small, protocol-only | 0.293 [ , ] | 0.351 [ , ] | 0.081 [ , ] | |
| lm small, posted-record | 0.313 [ , ] | 0.389 [ , ] | 0.112 [ , ] |
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| Step | count |
|---|---|
| Completed phase 2/3 interventional studies with posted results | 30,119 |
| Studies with a single-class numeric primary outcome and denominators | 23,755 |
| Arm records with and finite outcome (binary / continuous) | 40,450 (18,548 / 21,902) |
| trials contributing arm records | 18,077 |
| v2 contrasts (text-based pairing; after duplicate and removal) | 11,274 |
| v2 trials (binary / continuous) | 8,011 (3,225 / 4,786) |
| Group | Fields |
|---|---|
| Endpoint text | outcome title, outcome description, outcome timeframe, condition list |
| Intervention text | treatment arm: intervention names, intervention descriptions, protocol arm description, results-group title and description; control arm: intervention names, arm description, group title and description |
| Population/design (categorical) | phase, sponsor class, allocation, masking, primary purpose, therapeutic area, sex eligibility |
| Population (numeric) | , , mean age, fraction female, results-posting year |
| Comparator type | inert control vs active comparator |
| Intervention type | drug / biological / device / procedure / behavioural / …(treatment arm) |
| run | family | test set | trials | skill(A) | skill(B) | coverage A / B | ||
| Sonnet 5, protocol | binary | v2.1 | 878 | 655 | 0.570 | 0.462 | [ , ] | 0.70 / 0.69 |
| Sonnet 5, posted | binary | v2.1 | 878 | 655 | 0.629 | 0.494 | [ , ] | 0.74 / 0.70 |
| Haiku 4.5, protocol | binary | v2.1 | 878 | 655 | 0.351 | 0.293 | [ , ] | 0.58 / 0.59 |
| Haiku 4.5, posted | binary | v2.1 | 878 | 655 | 0.389 | 0.313 | [ , ] | 0.57 / 0.61 |
| Llama 3.1 70B, protocol | binary | v2.1 | 878 | 655 | 0.404 | 0.356 | [ , ] | 0.59 / 0.63 |
| Llama 3.1 70B, posted | binary | v2.1 | 878 | 655 | 0.417 | 0.360 | [ , ] | 0.66 / 0.69 |
| model | boundary | stratum | contrasts | trials | protocol-only | posted-record | gbt |
|---|---|---|---|---|---|---|---|
| Llama 3.1 70B | training cutoff (2024-01-01) | uncontaminated | 217 | 152 | [ , ] | [ , ] | [ , ] |
| complement | 661 | 503 | [ , ] | [ , ] | [ , ] | ||
| Haiku 4.5 | training cutoff (2025-08-01) | uncontaminated | 5 | 3 | |||
| complement | 873 | 652 | [ , ] | [ , ] | [ , ] | ||
| Haiku 4.5 | reliable-knowledge cutoff (2025-03-01) | uncontaminated | 52 | 33 | [ , ] | [ , ] | [ , ] |
| complement | 826 | 622 | [ , ] | [ , ] | [ , ] |
| set | stratum / split | prompt | (trials) | |
|---|---|---|---|---|
| v2.1 | all | protocol-only | [ , ] | 878 (655) |
| v2.1 | all | posted-record | [ , ] | 878 (655) |
| v2.1 | uncontaminated | protocol-only | [ , ] | 217 (152) |
| v2.1 | uncontaminated | posted-record | [ , ] | 217 (152) |
| v2.1 | complement | protocol-only | [ , ] | 661 (503) |
| v2.1 | complement | posted-record | [ , ] | 661 (503) |
| seed | blind - | blind | full | intervention only | ( full vs blind ) |
|---|---|---|---|---|---|
| 0 | 0.261 | 0.295 | 0.216 | 0.030 | [ , ] |
| 1 | 0.250 | 0.301 | 0.210 | 0.069 | [ , ] |
| 2 | 0.261 | 0.311 | 0.209 | 0.050 | [ , ] |
| 3 | 0.249 | 0.283 | 0.208 | 0.043 | [ , ] |
| 4 | 0.262 | 0.295 | 0.212 | 0.078 | [ , ] |
| mean | 0.257 | 0.297 | 0.211 | 0.054 | [ , ] |
| setting | mode | contrasts | trials | skill( blind ) | skill( full ) | |
|---|---|---|---|---|---|---|
| headline (seed 0) | headline | 878 | 655 | 0.295 | 0.216 | [ , ] |
| cut=2021 | retrained | 1406 | 1047 | 0.273 | 0.245 | [ , ] |
| cut=2022 | retrained | 1102 | 821 | 0.263 | 0.219 | [ , ] |
| cut=2024 | retrained | 645 | 474 | 0.315 | 0.240 | [ , ] |
| ctrl_kind=inert_control | evaluated | 644 | 470 | 0.300 | 0.215 | [ , ] |
| ctrl_kind=inert_control | retrained | 644 | 470 | 0.268 | 0.244 | [ , ] |
| architecture | seeds | skill( blind ) | skill( full ) | |
|---|---|---|---|---|
| Headline GBT, shared SVD (120) | 5 | 0.30 | 0.21 | -0.12 [-0.20, -0.05] |
| Headline GBT, intervention text only | 5 | – | 0.05 | – |
| GBT, separate SVD blocks (120+120) | 5 | 0.31 | 0.28 | -0.04 [-0.08, -0.01] |
| Ridge on sparse TF-IDF, no SVD ( 1/1) | 1 | 0.27 | 0.24 | -0.04 [-0.10, ] |
| Ridge, intervention text only ( 1) | 1 | – | 0.05 | – |
| Two-stage: ridge blind + ridge(intervention) on OOF residuals ( 100) | 1 | 0.27 | 0.27 | [ , ] |
| model | reference | skill |
|---|---|---|
| design-only | train_mean | 0.069 [ , ] |
| design endpoint text | design-only | 0.134 [ , ] |
| design endpoint text | train_mean | 0.195 [ , ] |
| design condition text | design-only | 0.063 [ , ] |
| design condition text | train_mean | 0.128 [ , ] |
| design population covariates | design-only | 0.028 [ , ] |
| matching | stratum | contrasts | trials | skill( blind ) | skill( full ) | |
|---|---|---|---|---|---|---|
| exact normalized name | seen | 286 | 230 | 0.425 | 0.296 | [ , ] |
| exact normalized name | unseen | 592 | 435 | 0.251 | 0.180 | [ , ] |
| token-Jaccard 0.5 | seen | 387 | 308 | 0.375 | 0.288 | [ , ] |
| token-Jaccard 0.5 | unseen | 491 | 356 | 0.252 | 0.166 | [ , ] |
| model | reference | skill |
|---|---|---|
| blind | train_mean | 0.245 [ , ] |
| blind_norole | train_mean | 0.225 [ , ] |
| full | train_mean | 0.212 [ , ] |
| full | blind | [ , ] |
| full | blind_norole | [ , ] |
| variant | contrasts | trials | retrieval | gbt , same subset |
|---|---|---|---|---|
| name_sameCtrl_k3_residual | 246 | 201 | [ , ] | [ , ] |
| name_sameCtrl_k3_level | 246 | 201 | [ , ] | [ , ] |
| name_anyCtrl_k3_residual | 286 | 230 | [ , ] | [ , ] |
| name_anyCtrl_k3_level | 286 | 230 | [ , ] | [ , ] |
| variant | contrasts | trials | retrieval | gbt , same subset |
|---|---|---|---|---|
| primary_union_sameCtrl_k3_residual | 297 | 240 | [ , ] | [ , ] |
| primary_union_sameCtrl_k3_level | 297 | 240 | [ , ] | [ , ] |
| union_anyCtrl_k3_residual | 330 | 265 | [ , ] | [ , ] |
| union_anyCtrl_k3_level | 330 | 265 | [ , ] | [ , ] |
| mesh_sameCtrl_k3_residual | 193 | 165 | [ , ] | [ , ] |
| mesh_sameCtrl_k3_level | 193 | 165 | [ , ] | [ , ] |
| system | seed SD | |
|---|---|---|
| all test : 878 contrasts, 655 trials | ||
| gbt | [ , ] | 0.017 |
| Sonnet 5, protocol-only | [ , ] | – |
| Sonnet 5, posted-record | [ , ] | – |
| Haiku 4.5, protocol-only | [ , ] | – |
| Llama 3.1 70B, protocol-only | [ , ] | – |
| system | seed SD | |
|---|---|---|
| all test : 1,329 contrasts, 978 trials | ||
| gbt | [ , ] | 0.019 |
| Sonnet 5, posted-record | [ , ] | – |
| randomized parallel-group : 1,305 contrasts, 961 trials | ||
| gbt | [ , ] | 0.019 |
| Sonnet 5, posted-record | [ , ] | – |
| statistic | value | pre-registered criterion / outcome | |
| (a) prompt condition, v2.1 | |||
| Claude Sonnet 5: IS, protocol-only | [ , ] | 878 | |
| Claude Sonnet 5: IS, posted-record | [ , ] | 878 | |
| Claude Sonnet 5: IS difference, posted-record minus protocol-only | [ , ] | 878 | |
| Claude Haiku 4.5: IS, protocol-only | [ , ] | 878 | |
| Claude Haiku 4.5: IS, posted-record | [ , ] | 878 | |
| statistic | observed | chance | ratio | (trials) |
| (a) language model: treatment class guessed from the blind prompt (Claude Haiku 4.5, v2.1 binary) | ||||
| drug class | 0.449 [ , ] | 0.023 [ , ] | 19.7 | 878 (655) |
| mechanism | 0.454 [ , ] | 0.019 [ , ] | 23.4 | 878 (655) |
| exact agent | 0.208 [ , ] | 0.007 [ , ] | 30.4 | 878 (655) |
| (b) registry blind text: same-molecule training contrast retrieved (v2.1) | ||||
| binary, tfidf, top-1 | 0.232 [ , ] | 0.003 | 82.2 | 228 (188) |
| regime | recipe | interval | coverage | width | MAE | coverage, one contrast per calibration trial | |
|---|---|---|---|---|---|---|---|
| in_distribution | split | constant | 640 | 0.889 [ , ] | 0.626 | 0.138 | 0.880 [ , ] |
| in_distribution | split | normalized | 640 | 0.845 [ , ] | 0.528 | 0.138 | 0.858 [ , ] |
| in_distribution | crossfit | constant | 640 | 0.906 [ , ] | 0.598 | 0.124 | 0.893 [ , ] |
| in_distribution | crossfit | normalized | 640 | 0.867 [ , ] | 0.514 | 0.124 | 0.868 [ , ] |
| temporal_2023plus | split | constant | 878 | 0.888 [ , ] | 0.602 | 0.136 | 0.875 [ , ] |
| temporal_2023plus | split | normalized | 878 | 0.867 [ , ] | 0.551 | 0.136 | 0.854 [ , ] |
| regime | target MAE | threshold chosen on | answered (calibration) | answered (contrasts) | answered (trials) | realised MAE, answered | MAE if all answered |
|---|---|---|---|---|---|---|---|
| in_distribution | 0.10 | calibration | 0.678 | 0.756 | 0.813 | 0.106 | 0.124 |
| in_distribution | 0.10 | evaluated (oracle) | – | 0.658 | 0.708 | 0.100 | 0.124 |
| in_distribution | 0.08 | calibration | 0.397 | 0.483 | 0.538 | 0.095 | 0.124 |
| in_distribution | 0.08 | evaluated (oracle) | – | 0.289 | 0.338 | 0.080 | 0.124 |
| temporal_2023plus | 0.10 | calibration | 0.759 | 0.837 | 0.850 | 0.113 | 0.130 |
| temporal_2023plus | 0.10 | evaluated (oracle) | – | 0.574 | 0.606 | 0.100 | 0.130 |
| regime | target MAE | threshold chosen on | answered (calibration) | answered (contrasts) | answered (trials) | realised MAE, answered | MAE if all answered |
|---|---|---|---|---|---|---|---|
| in_distribution | 0.10 | calibration | 0.000 | 0.000 | 0.000 | – | 0.138 |
| in_distribution | 0.10 | evaluated (oracle) | – | 0.009 | 0.011 | 0.047 | 0.138 |
| in_distribution | 0.08 | calibration | 0.000 | 0.000 | 0.000 | – | 0.138 |
| in_distribution | 0.08 | evaluated (oracle) | – | 0.009 | 0.011 | 0.047 | 0.138 |
| temporal_2023plus | 0.10 | calibration | 0.336 | 0.468 | 0.493 | 0.111 | 0.136 |
| temporal_2023plus | 0.10 | evaluated (oracle) | – | 0.158 | 0.171 | 0.098 | 0.136 |
| area | contrasts | trials | skill( full ) | skill( blind ) | |
| protocol-only prompts | |||||
| cardiometabolic | 45 | 34 | 0.332 | 0.289 | 0.060 [ , ] |
| immune/inflam | 138 | 88 | 0.401 | 0.077 | 0.352 [ , ] |
| infectious | 205 | 148 | 0.674 | 0.558 | 0.262 [ , ] |
| neuro/psych | 48 | 37 | 0.382 | 0.201 | 0.226 [ , ] |
| oncology | 89 | 81 | 0.563 | 0.470 | 0.174 [ , ] |
| prompt | 0.10 | 0.25 | 0.50 | 0.75 | 1.00 |
|---|---|---|---|---|---|
| protocol-only | 0.018 | 0.039 | 0.066 | 0.080 | 0.098 |
| posted-record | 0.013 | 0.038 | 0.056 | 0.077 | 0.091 |