Structured Clinical Memory as Context for Probabilistic Decision Models: A Synthetic Evaluation with Jev
Abstract
Longitudinal clinical context is often distributed across noisy, repetitive forms and narrative records. We study whether a compact structured memory can improve a probabilistic decision model's answers when the underlying patient vignette is already available. We created 30 fully synthetic cases spanning twelve clinical domains, paired each with synthetically generated structured memory, and evaluated 120 typed questions with Jev. The raw-vignette arm achieved 90.0% accuracy (95% normal-approximation confidence interval: 84.6%-95.4%); adding memory achieved 93.3% (88.9%-97.8%). Five questions changed from incorrect to correct and one changed in the opposite direction (exact McNemar ). Choice log loss decreased from 0.258 to 0.175 and binary Brier score from 0.054 to 0.022. Memory added 310 input tokens per call on average, while observed mean latency was unchanged at approximately 0.5 seconds. These results are directionally positive rather than clinically validating: the cases are synthetic, the labels are single-author judgments, and the memory is an oracle upper bound rather than extracted production output. The pilot supports evaluating structured clinical memory as a reusable state interface for decision models.
Figures & tables
| Output | Accuracy rule | Probability metric |
|---|---|---|
| Choice | Exact option | Log loss |
| Binary | P(yes) 0.5 | Brier score |
| Score | Absolute error 1 | Mean absolute error |
| Condition | Correct | Accuracy | 95% CI |
|---|---|---|---|
| Arm A: vignette | 108 / 120 | 90.0% | 84.6%–95.4% |
| Arm B: + memory | 112 / 120 | 93.3% | 88.9%–97.8% |
| Difference | +4 | +3.3 pp | Not estimated |
| Question type | n | Arm A | Arm B |
|---|---|---|---|
| Choice accuracy | 56 | 89.3% | 91.1% |
| Binary accuracy | 36 | 94.4% | 97.2% |
| Score within 1 | 28 | 85.7% | 92.9% |
| Metric (lower is better) | Arm A | Arm B | Change |
|---|---|---|---|
| Choice log loss | 0.258 | 0.175 | 0.083 |
| Binary Brier score | 0.054 | 0.022 | 0.032 |
| Score MAE | 0.521 | 0.469 | 0.052 |
| Confidence bin | Arm A | Arm B | ||
|---|---|---|---|---|
| n | acc. | n | acc. | |
| 0.5–0.6 | 9 | 88.9% | 8 | 87.5% |
| 0.6–0.7 | 12 | 91.7% | 8 | 100% |
| 0.7–0.8 | 23 | 78.3% | 18 | 88.9% |
| 0.8–0.9 | 16 | 81.2% | 14 | 100% |
| 0.9–1.0 | 55 | 100% | 65 | 98.5% |
| Efficiency outcome | Arm A | Arm B |
|---|---|---|
| Mean input tokens | 553.9 | 863.9 |
| Mean latency | 0.51 s | 0.50 s |
| Calls | 30 | 30 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Domain label | Cases | Questions |
|---|---|---|
| Cardiology | 3 | 12 |
| Diabetes | 3 | 12 |
| Respiratory | 3 | 12 |
| Infectious disease | 3 | 12 |
| Neurology | 3 | 12 |
| Mental health | 3 | 12 |
| Field | Meaning in this pilot |
|---|---|
| feature_type | Condition or medication |
| condition | Normalized feature label |
| status | Active, history_of, or resolved |
| onset | Synthetic onset date or period |
| certainty | Author-assigned value from 0 to 1 |
| source_form | Originating synthetic form type |
Clinical Decision Support
Clinical Decision Support uses data, medical knowledge, and AI to help clinicians assess patients and choose tests, diagnoses, and treatments.
See all 256 papers