Quantifying uncertainty in clinical predictions is critical for high-stakes diagnosis tasks. Conformal prediction offers a principled approach by providing prediction sets with theoretical coverage guarantees. However, in practice, patient distribution shifts violate the i.i.d. assumptions underlying standard conformal methods, leading to poor coverage in healthcare settings. In this work, we evaluate several conformal prediction approaches on EEG seizure classification, a task with known distribution shift challenges and label uncertainty. We demonstrate that personalized calibration strategies can improve coverage by over 20 percentage points while maintaining comparable prediction set sizes. Our implementation is available via PyHealth, an open-source healthcare AI framework: https://github.com/sunlabuiuc/PyHealth.
Figures & tables
Figure 1: EEG Classification Challenges. EEG tasks often violate key aspects of a typical machine learning pipeline. (a) Their annotation process consistently leaves room for uncertainty, which is then passed onto the models. (b) Training, validation, and test distributions are not i.i.d. due to patient distribution shift [ 13 ] . Both issues make EEG classification a very challenging machine learning problem.
Figure 2: Empirical coverage under random (top) and patient (bottom) splits. The dotted line is target coverage 1−α . Under the random split, NCP substantially outperforms non-personalized baselines on TUEV. Under the patient split, all methods fall short of target coverage, reflecting the difficulty of cross-patient distribution shift.
Figure 3: Average prediction set sizes under random (top) and patient (bottom) splits. NCP consistently maintains smaller prediction sets than non-personalized methods despite achieving competitive or higher coverage.
Figure 4: Empirical coverage vs. α for NCP using varying calibration set sizes k . Shaded regions denote ±1 std. The dashed line is target coverage 1−α .
Department of Data Sciences and Operations University of Southern California Los Angeles, CA 90089 · AI Foundations, Capital One New York, NY 10017 · Department of Data Sciences and Operations Thomas Lord Department of Computer Science University of Southern California Los Angeles, CA 90089