Evaluating utility of clinical prediction models: A comparison of decision curve analysis and cost curves

Classification models typically predict a binary outcome and use one or more decision thresholds to produce a classification or inform a clinical decision. Decision curves and cost curves are model evaluation approaches that assess net benefit and loss across decision thresholds. In this paper we compare decision curves and cost curves, demonstrating that decision curves are closely related to a specific type of cost curve called a Brier curve. Users of both approaches set the classification threshold assuming calibrated model scores, using the relative clinical importance of correct positive and negative classifications. A given decision curve has a corresponding Brier curve such that reference lines common in each space can be included in either (e.g. the 'treat none' decision curve can be translated into an equivalent 'treat none' Brier curve). We suggest the upper envelope decision curve as a useful comparison to a model decision curve showing the maximum achievable net benefit through recalibration. We show that net benefit (used for decision curves) and Brier loss (used for Brier curves) will always choose the same model as optimal at any given threshold, but cannot be compared or aggregated across thresholds except under specific assumptions; doing so otherwise requires custom weightings.