Medication recommendation is a multi-label prediction task. However, existing prediction-set evaluation and calibration methods rely mainly on exact-overlap metrics and therefore treat all missed medications as equally severe, overlooking how closely the prediction set matches them within the medication hierarchy. To address this, we introduce Conformal Severity Control (CSC), a post-hoc framework for calibrating medication prediction sets under hierarchy-aware mismatch severity. CSC uses the Anatomical Therapeutic Chemical (ATC) hierarchy to quantify mismatch severity and controls patient-level severity without retraining the predictor, supporting both average severity and upper-quantile severity for the highest-severity mismatches. Experiments on MIMIC-IV show that CSC tracks user-specified severity tolerances and shifts prediction sets away from distant mismatches toward exact or close matches.
Figures & tables
Figure 1: Motivation for Conformal Severity Control (CSC) . Prediction sets can achieve the same recall while differing substantially in ATC mismatch severity. CSC selects a score threshold on calibration data so that the expected severity stays below a user-specified tolerance.
Figure 2: Closest ATC match for disease-associated medications. For four disease cohorts, each selected ground-truth medication is assigned its closest predicted medication and categorized as exact, close, moderate, or distant. LLM sets threshold the predicted probabilities at 0.25; CSC sets use CSC-Avg with α=0.125 , whose cutoff is lower. The shift toward exact and close matches therefore reflects set expansion. Details are in Appendix D.1 .
Figure 3: Overview of CSC. Given patient input, a predictive model assigns scores to candidate medications, inducing a nested family of prediction sets as the threshold λ varies. CSC evaluates these candidate sets using an ATC-based hierarchy-aware mismatch severity and calibrates λ to control the expected upper-tail patient-level severity.
Figure 4: Empirical ATC taxonomy-based mismatch severity across target tolerances. For each α , CSC selects λ^ on the full calibration split, and the resulting prediction sets are evaluated once on the original test split. The left panel shows average severity (CSC-Avg); the right panel shows within-patient empirical CVaR at β=0.75 (CSC-Upper). The dashed line marks y=α . Each point is a single-split empirical estimate.
Figure 5: CSC produces patient-specific prediction sets under a single calibrated threshold λ^ . (a) CSC matches fixed Top- k baseline in average set size but produces a broad distribution of ki across patients. (b) Larger ground-truth medication sets correspond to larger calibrated sets, while fixed Top- k returns the same set size for all patients without considering patient-specific mismatch severity.
Method
α
λ^
Set Size
Controlled Loss
Precision
Recall
F1
CSC-Avg
0.125
0.9325
28.53
0.1218
0.315
0.813
0.438
0.25
0.8225
16.38
0.2439
0.443
0.661
0.510
0.35
0.7175
11.66
0.3465
0.522
0.551
0.512
CSC-Upper (β=0.75)
0.125
0.9875
62.87
0.1105
0.166
0.945
0.274
0.25
0.9675
40.83
0.2341
0.240
0.888
0.366
0.35
0.9425
30.94
0.3449
0.298
0.833
0.423
Table 1: Calibration results across target tolerances. For each α , λ^ is selected on the full calibration split, and all other entries are point estimates on the original test split. The controlled loss is average severity for CSC-Avg and within-patient empirical CVaR at β=0.75 for CSC-Upper. Precision, recall, and F1 are averaged across patients.
CSC
Plug-in
n
Loss [quantiles]
Size
Exceed
Loss [quantiles]
Size
Exceed
25
0.214 [0.154, 0.287]
18.67
13%
0.245 [0.171, 0.319]
16.60
40%
50
0.230 [0.183, 0.284]
17.44
19%
0.245 [0.198, 0.303]
16.45
41%
100
0.238 [0.203, 0.275]
16.80
28%
0.246 [0.210, 0.287]
16.35
38%
3,826
0.244
16.38
–
0.246
16.23
–
Table 2: Effect of the finite-sample correction at α=0.25 (CSC-Avg). Plug-in omits the +1 correction in Eq. ( 7 ). For n≤100 , entries are means over 100 calibration subsamples evaluated on the same fixed test split; brackets give empirical 2.5–97.5% quantiles of test loss across subsamples, and Exceed is the fraction of subsamples with test loss above α . The full-calibration row is a single point estimate.
Method
Set size
Savg
S0.75
Precision
Recall
F1
CSC-Avg (= matched CRC-FNR)
16.38
0.2439
0.6247
0.443
0.661
0.510
Score-top- k ( k=16 )
16.00
0.2206
0.6032
0.414
0.677
0.462
CSC-Upper (= matched CRC-FNR)
40.83
0.0680
0.2341
0.240
0.888
0.366
Score-top- k ( k=40 )
40.00
0.0649
0.2361
0.235
0.889
0.346
Table 3: Matched baseline diagnostics at α=0.25 . All rules use the same predictor scores. Matching parameters are selected on the calibration split, and all entries are point estimates on the original test split. CRC-FNR matched to CSC’s calibration recall or mean set size selects the same cutoff as CSC and returns identical sets. Score-top- k is matched to CSC’s mean calibration set size. Bold marks the loss that each CSC variant controls.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Description
Spaces and label sets
X
Input space.
K
Medication label space.
Y
Output space of label sets, e.g., subsets of K .
k∈K
Candidate or predicted medication label.
Y⊆K
Ground-truth medication label set for a patient.
Appendix
Table 4: Summary of notation used throughout the paper.
Level
Meaning
# Concepts
Example Code
Example Code Description
1
Anatomical main group
14
A
Alimentary tract and metabolism
2
Therapeutic main group
90
A10
Drugs used in diabetes
3
Pharmacological subgroup
248
A10B
Blood glucose lowering drugs, excluding insulins
4
Chemical subgroup
840
A10BA
Biguanides
5
Chemical substance
5,495
A10BA02
Metformin
Appendix
Table 5: Structure of the ATC medication hierarchy used in this study. # Concepts denotes the number of unique medication concepts at each level.
Dataset
#patients
#visits
#visits/patient
#conditions/patient
#procedures/patient
#drugs/patient
MIMIC-IV
19,135
35,974
1.88
21.74
4.70
43.89
Appendix
Table 6: Basic statistics of the pre-processed MIMIC-IV EHR dataset. “#” denotes counts and “/patient” denotes the average number of records per-patient.
Fixed threshold
Separate calibration
β
Test loss
λ^
Test loss
Set size
0.50
0.1335
0.9275
0.2463
27.48
0.75
0.2341
0.9675
0.2341
40.83
0.90
0.3521
0.9825
0.2364
54.08
Appendix
Table 7: Effect of β at α=0.25 : fixed threshold versus separate calibration. The fixed-threshold column keeps the CSC-Upper threshold calibrated at β=0.75 ( λ^=0.9675 , mean set size 40.83) and changes only β in the loss. The separate-calibration columns select a new threshold for each β . Losses are patient-averaged within-patient empirical CVaR; all entries are point estimates on the original test split.
CSC
Plug-in
n
Loss [quantiles]
Size
Exceed
Loss [quantiles]
Size
Exceed
25
0.214 [0.154, 0.287]
18.67
13%
0.245 [0.171, 0.319]
16.60
40%
50
0.230 [0.183, 0.284]
17.44
19%
0.245 [0.198, 0.303]
16.45
41%
100
0.238 [0.203, 0.275]
16.80
28%
0.246 [0.210, 0.287]
16.35
38%
250
0.242 [0.220, 0.262]
16.53
23%
0.245 [0.224, 0.266]
16.36
30%
500
0.244 [0.229, 0.259]
16.36
20%
0.246 [0.229, 0.261]
16.28
26%
Appendix
Table 8: Finite-sample correction across calibration sizes at α=0.25 (CSC-Avg). For n≤2,000 , loss and size are means over 100 calibration subsamples on a fixed test split; brackets give empirical 2.5–97.5% quantiles of test loss, and Exceed is the fraction of subsamples with test loss above α . Exceed is a fixed-test empirical diagnostic, not a failure probability of Theorem 3.3 . The row n=3,826 uses the full calibration split once.
Figure 6: ATC level-2 branch composition for average-severity CSC at α=0.25 . The left panel shows true positives and false positives among the most frequently selected ATC branches, while the right panel reports branch-level recall and precision.
Figure 7: ATC level-2 branch composition for upper-quantile severity CSC at α=0.25 and β=0.75 . Compared with average-severity calibration, upper-tail calibration selects larger sets and increases branch-level recall in many medication families, at the cost of lower branch-level purity.
Rule
Mean size
Median [IQR]
Savg
S0.75
Precision
Recall
F1
CSC-Avg, α=0.125 ( λ^=0.9325 )
CSC-Avg (= matched CRC-FNR)
28.53
27 [17, 37]
0.1218
0.3833
0.315
0.813
0.438
Score-top- k (size, k=28 )
28.00
–
0.1108
0.3754
0.303
0.821
0.405
Score-top- k (recall, k=27 )
27.00
–
0.1171
0.3917
0.310
0.813
0.410
Prevalence-top- k (size, k=28 )
28.00
–
0.2146
0.6128
0.246
0.655
0.327
Prevalence-top- k (recall, k=50 )
50.00
–
0.1043
0.3529
0.175
0.817
0.270
Appendix
Table 9: Matched baseline diagnostics at all tolerances. Matching parameters are selected on the calibration split, and all entries are point estimates on the original test split. CRC-FNR matched to CSC’s calibration recall or mean set size returns the same sets as CSC in every block. “Size” and “recall” mark the matching criterion. Median [IQR] summarizes set sizes across test patients; fixed- k rules give every patient the same size. Bold marks the loss that CSC controls in each block.
Method
α
λ^
Savg
S0.75
Set size
Precision
Recall
F1
CSC-Avg
0.125
0.9375
0.1197
0.3708
43.86
0.204
0.816
0.316
CSC-Avg
0.25
0.8575
0.2464
0.6091
24.73
0.309
0.659
0.399
CSC-Avg
0.35
0.7700
0.3462
0.7316
16.73
0.385
0.541
0.419
CSC-Upper
0.125
0.9825
0.0312
0.1103
93.81
0.105
0.940
0.185
CSC-Upper
0.25
0.9600
0.0740
0.2436
58.86
0.159
0.876
0.262
CSC-Upper
0.35
0.9450
0.1054
0.3337
47.63
0.191
0.834
0.301
Appendix
Table 10: CSC with DrugDoctor-based scores. Each row calibrates on all 3,826 calibration patients and reports point estimates on the 3,827 test patients. Bold marks the loss that each CSC variant controls.
Cost
λ^
Test loss
Set size
Precision
Recall
F1
d
0.8550
0.2464
18.64
0.412
0.700
0.499
d (equal-edge)
0.8225
0.2439
16.38
0.443
0.661
0.510
d2
0.7700
0.2440
13.68
0.485
0.603
0.515
Branch-dependent IC
0.6950
0.2477
10.94
0.537
0.529
0.508
Appendix
Table 11: Taxonomy-cost sensitivity for CSC-Avg at α=0.25 . Each cost is calibrated separately on the full calibration split, and entries are point estimates on the original test split. Each test loss is on its own scale and is compared only with the 0.25 target.