When Copying Is Hard to Beat: A Negative Result in Clinical Assessment Forecasting
Organizations: Department of Computational Linguistics Heidelberg University, Germany · HIDSS4Health — Helmholtz Information & Data Science School for Health, Karlsruhe/Heidelberg, Germany. · Department of Computational Linguistics & Interdisciplinary Center for Scientific Computing (IWR) Heidelberg University, Germany
Abstract
We report a negative result in next-day clinical assessment forecasting: the evaluated models show no consistent advantage over strong persistence-based baselines. Clinicians can view the previous day's questionnaire answers, a plausible source of documentation persistence. Predicting 45 items on 29,715 ICU day-pairs with patient-disjoint splits, a physiology-informed gated model improves item-macro-F1 over a one-line clinical rule by +0.0019 [+0.0006,+0.0031], while losing accuracy. No macro-F1 advantage is established when the physiology window ends at the previous form's reference time (+0.0001 [-0.0006,+0.0008]), or when scoring excludes the 68.5% of cells derived from parent answers; combining both changes yields a small disadvantage. Four checkpoints across three families underperform persistence, and a fine-tuned one differs from it on only two cells. A categorical read-out raises changed-cell accuracy eightfold, but a state-only read-out scores higher overall than forecast-only decoding and matches a transition matrix. This negative result highlights how the documentation process, input cutoff, observation mask and decoding rule shape conclusions about forecasting recorded assessments.
Figures & tables
| Predictor | Accuracy | Macro-F1 | ||
|---|---|---|---|---|
| Questionnaire input only | ||||
| Persistence (copy) | 0.9619 | 0.8705 | 0.0000 | 0.0000 |
| Persistence rule | 0.9632 | 0.8711 | 0.0349 | 0.0000 |
| Transition matrix | 0.9634 | 0.8681 | 0.0723 | 0.0013 |
| Questionnaire physiology, late cutoff | ||||
| GRU without baseline gate | 0.9339 | 0.8033 | 0.2074 | 0.0373 |
| Cutoff / mask | Labels | Macro-F1 | Accuracy |
|---|---|---|---|
| late / filled | corrected | ||
| late / filled | raw | ||
| matched / filled | corrected | ||
| late / explicit | corrected | ||
| matched / explicit | corrected |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Quantity | Count |
|---|---|
| Source item entries / encounters / questionnaire days | 585,791 / 3,736 / 39,148 |
| Questionnaire versions (v1 / v2 / missing) | 213,455 / 372,334 / 2 |
| Items completed per form, median (IQR) | 13 (9–18) |
| Retained consecutive day-pairs / encounters / patients | 29,715 / 2,789 / 2,694 |
| Training day-pairs / encounters / patients | 18,688 / 1,775 / 1,724 |
| Validation day-pairs / encounters / patients | 4,708 / 453 / 431 |
| Row | baseline path | width | layers | ff | lr | batch | drop |
|---|---|---|---|---|---|---|---|
| GRU without baseline gate | none | 128 | 2 | 512 | 64 | 0.05 | |
| Gate over persistence | copy, , | 128 | 2 | 512 | 64 | 0.05 | |
| Gate over rule (a) | table, , | 64 | 3 | 512 | 32 | 0.3 | |
| Gate over rule (b) | table, , | 64 | 3 | 512 | 32 | 0.3 | |
| Gate over rule (c) | table, , | 128 | 2 | 512 | 64 | 0.05 | |
| Change-weighted objective | table, , | 128 | 2 | 512 | 64 | 0.05 |
| Cutoff / mask | Labels | difference |
|---|---|---|
| late / filled | corrected | |
| late / filled | raw | |
| matched / filled | corrected | |
| late / explicit | corrected | |
| matched / explicit | corrected |
| Predictor | Seed mean sd | Ensemble | ||
|---|---|---|---|---|
| Accuracy | Macro-F1 | Accuracy | Macro-F1 | |
| Gate over persistence | 0.9590 | 0.8721 | ||
| Gate over rule, macro-F1 selected | 0.9624 | 0.8749 | ||
| GRU without baseline gate | 0.9399 | 0.8154 | ||
| Change-weighted objective | 0.9389 | 0.8540 | ||
| Predictor | Corrected labels | Raw labels | ||
|---|---|---|---|---|
| Accuracy | Macro-F1 | Accuracy | Macro-F1 | |
| Persistence | 0.9619 | 0.8705 | 0.9619 | 0.8719 |
| Persistence + rule | 0.9632 | 0.8711 | 0.9632 | 0.8711 |
| Transition matrix | 0.9634 | 0.8681 | 0.9633 | 0.8680 |
| Gate over rule, macro-F1 selected | 0.9609 | 0.8730 | 0.9599 | 0.8729 |
| Predictor | Late cutoff | Matched cutoff | ||
|---|---|---|---|---|
| Accuracy | Macro-F1 | Accuracy | Macro-F1 | |
| Gate over persistence | 0.9557 | 0.8687 | 0.9557 | 0.8688 |
| Gate over rule, macro-F1 selected | 0.9609 | 0.8730 | 0.9618 | 0.8712 |
| GRU without baseline gate | 0.9339 | 0.8033 | 0.9330 | 0.7998 |
| Change-weighted objective | 0.9293 | 0.8425 | 0.9336 | 0.8460 |
| Item | chg% | Persist. | Rule | Gate | |||
|---|---|---|---|---|---|---|---|
| Daily rating of illness severity | 6,319 | 21.59 | 0.7234 | 0.7234 | 0.7245 | +0.0011 | 0.012 |
| Patient’s development in last 24 h | 6,317 | 30.22 | 0.5819 | 0.5819 | 0.5846 | +0.0026 | 0.016 |
| Working diagnosis | 6,319 | 11.81 | 0.8489 | 0.8489 | 0.8479 | -0.0010 | 0.001 |
| Suspicion of infection | 6,169 | 6.84 | 0.8697 | 0.8697 | 0.8675 | -0.0022 | 0.016 |
| Focus of infection: abdominal | 6,270 | 3.08 | 0.9665 | 0.9665 | 0.9660 | -0.0005 | 0.006 |
| Focus of infection: endocarditis | 6,270 | 0.11 | 0.8701 | 0.8701 | 0.8701 | 0.0000 | 0.000 |
| Row | Repository | Revision |
|---|---|---|
| Chronos-Bolt small | amazon/chronos-bolt-small | 772f3d25 |
| Chronos-Bolt base | amazon/chronos-bolt-base | 5d9f166d |
| Chronos-2 | amazon/chronos-2 | 29ec3766 |
| MIRA | MIRA-Mode/MIRA | 932caadc |
| Model | Accuracy | Macro-F1 | F1 Δ | Agreement | |
| Persistence (harness check) | 0.9619 | 0.8705 | 0.0000 | 0.0000 | 100.0% |
| Chronos-Bolt small, zero-shot | 0.9599 | 0.8542 | 0.0070 | 0.0013 | 99.7% |
| Chronos-Bolt base, zero-shot | 0.9601 | 0.8532 | 0.0118 | 0.0041 | 99.7% |
| Chronos-2, zero-shot | 0.9570 | 0.8495 | 0.0193 | 0.0110 | 99.3% |
| MIRA checkpoint, zero-shot | 0.9500 | 0.8231 | 0.0433 | 0.0220 | 98.4% |
| Chronos-Bolt small, fine-tuned | 0.9619 | 0.8705 | 0.0000 | 0.0000 | 100.0% |
| Quantity | Random split | Chronological split |
|---|---|---|
| Scored cells | 281,952 | 250,682 |
| Change prevalence | 3.81% | 3.89% |
| Pooled AUROC (item + state) | 0.8310 | 0.8149 |
| Pooled average precision | 0.2749 | 0.2789 |
| Item identity only, AUROC | 0.7858 | — |
| Within-item (state only), AUROC | 0.6497 | — |
| Predictor | Accuracy | Macro-F1 | ||
|---|---|---|---|---|
| Persistence | 0.9611 | 0.8641 | 0.0000 | 0.0000 |
| Persistence + rule | 0.9624 | 0.8645 | 0.0324 | 0.0000 |
| Transition matrix | 0.9629 | 0.8617 | 0.0753 | 0.0012 |
| GRU without baseline gate | 0.9222 | 0.7815 | 0.1999 | 0.0486 |
| Gate over persistence | 0.9551 | 0.8627 | 0.0200 | 0.0070 |
| Gate over rule, macro-F1 selected | 0.9594 | 0.8666 | 0.0485 | 0.0037 |