PRISM: A Post-Prediction Reliability Framework for Evidence-Guided Correction and Selective Abstention
Authors: Omm Swarup, Shrishail M. Hiremath
Organizations: Indian Institute of Technology Madras, Chennai, Tamil Nadu, India · Department of Electronics and Communication Engineering, National Institute of Technology Rourkela, Rourkela, Odisha, India
A model may produce a prediction with high confidence and still get the outcome wrong. This is a challenge in sequential applications, where the system evolves over time and deployment data may differ from the data used during training. To address this issue, we propose PRISM (Prediction Reliability and Intelligent Self-Correction Mechanism), a framework that evaluates prediction reliability after the initial output is generated. It estimates the risk of error and examines predictions identified as potentially unreliable. For these cases, PRISM analyses available signals, retrieves similar situations from historical data, and verifies whether an alternative prediction is supported by the evidence. It can then retain the original prediction, apply a verified correction, or abstain when the evidence is insufficient. We evaluate PRISM on second-innings match prediction in the Indian Premier League (IPL). Ball-by-ball records are converted into over-level observations to capture changes in match conditions throughout an innings. To preserve temporal order, earlier IPL seasons are used for training, IPL 2024 for validation, and IPL 2025-2026 for out-of-time testing. The dataset contains 8,665 over-level observations from 475 matches, with the test partition comprising 2,568 observations from 142 matches. In one experimental record, Logistic Regression, Random Forest, and HistGradientBoosting achieve accuracies of 65.2%, 68.4%, and 70.2%, respectively. The corresponding PRISM evaluation reports a selective accuracy of 82.4% at 88.5% coverage, with a Brier score of 0.126 and an expected calibration error (ECE) of 0.019.
Figures & tables
Fig. 1: PRISM architecture. The upper row generates and diagnoses the original prediction. The lower row performs historical retrieval and independent verification before the controller selects acceptance, correction, or abstention.
Fig. 2: Documented chronological evaluation protocol for the IPL case study.
Model
Accuracy (%)
Brier
ECE
Logistic Regression
65.2
0.218
0.042
Random Forest
68.4
0.198
0.058
HistGradientBoosting
70.2
0.186
0.049
TABLE I: Baseline results
Fig. 3: Baseline accuracy values from the PRISM
Metric
Reported value
Selective accuracy
82.4%
Coverage
88.5%
Brier score
0.126
Calibration error
0.019
Harm prevention
92.4%
Historical states indexed
8,665
TABLE II: PRISM metrics
Metric
Reported value
Baseline accuracy
80.76%
PRISM selective accuracy
86.21%
TABLE III: Separate 2025 results II .
Fig. 4: Accuracy of PRISM ablation variants on the 2025 and 2026 test splits.
Fig. 5: Coverage and selective risk of the baseline and full PRISM on the 2025 and 2026 test splits.
Selective prediction lets models answer only when sufficiently reliable, trading coverage for lower error through abstention, referral, or extra evidence gathering.
Rare-event prediction is critical in domains such as healthcare, finance, reliability engineering, customer support, aviation safety, where positive outcomes are infrequent yet potentially catastrophic. Extreme class imbalance biases conventional models toward majority-class predictions, limiting recall, calibration, and operational usefulness. We propose LPCORP (Low-Prevalence CORrector for Prediction)*, a two-stage framework that combines reasoning-enhanced prediction with confidence-based outcome correction. A reasoning model first produces enriched predictions from narrative inputs, after which a lightweight classifier evaluates and selectively corrects these outputs to mitigate prevalence-driven bias. In this study we used Logistic-Regression (LR) and a simple Multilayer Perceptron (MLP) classifiers for this purpose. We evaluate LPCORP on real-world datasets from medical and consumer service domains. The results show that this method transforms the original rare-event prediction problem into a more balanced supervised correction task without discarding or resampling observations. Test-set evaluation demonstrates substantially improved performance, particularly in precision, which is a known weakness in low-prevalence data. We further provide a cost-reduction analysis comparing the expenses associated with rare-event damage control without preventive measures to those incurred when low-cost, prediction-based preventive interventions are applied that showed up to 40+% reduction in some cases.
Vitaly Bulgakov, Alexander Turchin
Mass General Brigham, Boston, MA, USA · Profiteya LLC, Boston, MA, USA
Many high-stakes regression decisions ask three things of a model at once: a point estimate to act on, an interval that bounds the risk of acting, and the option to abstain when neither can be trusted. Deep learning provides these in isolation: prediction-interval methods return an interval, and at best a value estimate inside it, for every input, while selective-prediction methods learn when to abstain, but only for point predictions and with no statement of how far the truth may lie from the estimate. We introduce SelPI, which trains interval, value and abstention heads jointly under a single loss with an explicit acceptance-rate budget. A natural objection is that interval width already tells us when to abstain. We show that it does not: across 18 tabular benchmarks, width predicts large point errors well (AUROC 0.72) but is anti-correlated with interval misses (AUROC 0.38), so abstaining on wide intervals discards precisely the inputs the interval got right. SelPI therefore selects with two gates, one trained on the interval loss and one on the value loss, coupled by a Fr'echet penalty that ties their separate budgets to the joint acceptance rate they deliver. With the base predictor held fixed, the learned gate beats width thresholding on a proper interval score on all 18 tabular datasets, closing a median 87% of the gap to an oracle selector, and it also beats selection by deep-ensemble disagreement on every dataset. The advantage grows with the heteroscedasticity of the task, and we report the settings in which it disappears.
Lidor Rubi, Gilad Katz
Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Beer-Sheva, Israel
Many high-stakes decisions depend on forecasts made before outcomes are known. In this future prediction setting, the central challenge is that public evidence evolves over time, while the main supervision signal arrives only after resolution: the realized outcome mainly assesses final correctness, offering only coarse guidance on what to track, what to verify, and which judgments to leave uncertain along the way. Our key observation is that revisiting the same unresolved question over time creates informative temporal contrasts across evolving evidence and repeated forecasts, exposing what earlier attempts missed before resolution and yielding a diagnostic signal we call the pre-resolution signal. We instantiate this idea in Milkyway, a future prediction agent with a persistent future prediction harness, an editable external state that stores reusable procedural guidance across revisits to the same unresolved question. As the same unresolved question is revisited, Milkyway extracts pre-resolution signals from evolving evidence and repeated forecasts, uses them to update the harness, and improves later forecasts on that question before resolution. After resolution, the realized outcome serves as a post-resolution check of provisional updates. On the FutureX and FutureWorld benchmarks, Milkyway achieves strong performance against competitive baselines, and a mechanism study suggests that the gains stem from harness evolution driven by pre-resolution signals rather than repeated prediction alone.
Chuyang Wei, Maohang Gao, Zhixin Han +12
University of Science and Technology of China · Zhongguancun Academy, Beijing, China · IIIS, Tsinghua University