FLARE: A Systematic, Uncertainty-Aware Framework for Evidence-Based Adoption of Artificial Intelligence in Healthcare

Jacob Idoko; Siddhartha Paudel; Mariana Bento; Roberto Souza; Gouri Ginde

cs.AIAug 24, 2026

FLARE: A Systematic, Uncertainty-Aware Framework for Evidence-Based Adoption of Artificial Intelligence in Healthcare

Authors: Jacob Idoko, Siddhartha Paudel, Mariana Bento, Roberto Souza, Gouri Ginde

Organizations: University of Calgary, Department of Electrical and Software Engineering, Calgary, Alberta, Canada · University of Calgary, Department of Biomedical Engineering, Calgary, Alberta, Canada

Abstract

Artificial intelligence is increasingly being introduced into healthcare workflows, yet most evaluations emphasize model accuracy rather than whether adoption is economically worthwhile in real clinical settings. This study proposes FLARE, a systematic and uncertainty-aware framework for evaluating the financial and operational implications of adopting AI in healthcare. FLARE combines fuzzy logic, time-driven activity-based costing, and return on investment analysis to estimate the cost of clinical service delivery, the cost of AI development and operation, and the economic consequences of workflow integration under uncertainty. The framework was demonstrated through an early health technology assessment case study of AI-assisted large vessel occlusion detection in the CT stroke pathway for acute ischemic stroke. The case study shows how FLARE can quantify conventional pathway cost, AI-related development and recurring costs, and AI-enabled service savings within a unified activity-based model. Under expected assumptions, the analysis identified a break-even threshold of approximately 3,992 patients per year, with positive first-year return on investment at typical annual stroke volumes of about 5,000 patients. The results further show that economic benefit depends not only on algorithmic performance, but also on patient volume, verification time, infrastructure choices, and workflow design. FLARE provides a transparent and practical decision-support framework for early-stage evaluation of AI adoption in healthcare. By making uncertainty, resource use, and implementation trade-offs explicit, it helps clinicians, administrators, and policymakers determine when AI deployment is economically viable and where operational changes may improve value.

Figures & tables

Figure 1. Evolution of costing methodologies from traditional systems to fuzzy logic–enhanced approaches. Adapted from Ostadi et al. ( 25 ) Diagram tracing the progression of costing methods in four stages: traditional volume- and department-based costing, activity-based costing (ABC), time-driven activity-based costing (TDABC), and fuzzy logic-enhanced FL-TDABC. Each stage is annotated with the limitation of the preceding method that motivated it.

Figure 2. Seven-step FL-TDABC framework integrating fuzzy time estimates within the TDABC cost model. Obtained from Koster et al. ( 18 ) Flow diagram of the seven sequential FL-TDABC steps: select the medical condition or service, define the care delivery value chain, develop process maps, obtain time estimates as triangular fuzzy numbers, estimate resource costs, compute the capacity cost rate, and calculate the total fuzzy cost of care.

Figure 3. CT pathway for acute ischemic stroke patients with suspected LVO, obtained from Sangha et al. Flowchart of the conventional CT stroke pathway across three stages. In ED triage the patient is placed on a stretcher, a code stroke is called, and the stroke team performs an NIHSS assessment. In the CT suite an NCCT and then a CTA are acquired and interpreted, and IV-tPA eligibility and the presence of a large vessel occlusion are confirmed. Eligible patients are transferred to the neuro-interventional radiology suite for groin puncture and endovascular thrombectomy.

Cohort	Description	Sample size ( $N$ )
Heidelberg training cohort	Internal data used for model training	835 (71%)
Heidelberg internal test cohort	Internal test data for validation	344 (29%)
External FAST cohort	Pseudo-prospective validation	Not reported
External UKB cohort	External pseudo-prospective validation	Not reported

Table 1. Cohorts used in Brugnara et al.

Monthly costs
Component	Estimated Cost (CAD)
AWS SageMaker infrastructure ( $C_{\text{AWS}}$ )	8,050.86
ML Engineer (40 hr/week @ 44.10 CAD/hr)	7,644.00
Total monthly recurring cost	15,694.86
Annualized cost (12 months)
AWS SageMaker (12 months)	96,610.32

Table 2. Monthly and annual recurring cost components for the LVO AI solution (production phase).

Figure 4. Integration of AI-assisted CTA interpretation within the CT suite. The AI replaces the manual interpretation step, while expert verification remains unchanged. Block diagram of the CT stroke pathway showing where the AI-assisted CTA interpretation module is inserted between image acquisition and expert verification, while all other workflow steps remain unchanged.

Figure 5. Cumulative ROI over five years under pessimistic, expected, and optimistic scenarios as one-time development costs are amortized. The expected case demonstrates steady positive returns, reaching approximately 44% by Year 5. The pessimistic scenario remains negative throughout, while the optimistic scenario shows substantial gains exceeding 150% by Year 5. Line chart with years one through five on the horizontal axis and cumulative ROI as a percentage on the vertical axis. Three lines are plotted. The optimistic line rises steeply to above 150 percent by Year 5, the expected line rises from about 25 percent in Year 1 to roughly 44 percent by Year 5, and the pessimistic line stays below zero across all five years.

Figure 6. ROI vs. annual patient volume across pessimistic, expected, and optimistic scenarios. Line chart with annual patient volume from 1,000 to 10,000 cases on the horizontal axis and Year 1 ROI as a percentage on the vertical axis. Three lines are plotted for the pessimistic, expected, and optimistic scenarios; all rise as volume increases. The expected line crosses zero ROI at approximately 3,992 patients per year, the optimistic line crosses zero near 3,000 patients, and the pessimistic line crosses at a higher volume.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Resource (Personnel)	Hourly Rate (CAD)	CCR (CAD/min)
ED clerk	24.50	0.41
ED technician	44.08	0.73
ED physician	130.24	2.17
ED nurse	47.38	0.79
NIR nurse	58.51	0.98
NIR technologist	44.61	0.74

Appendix

Table 3. Personnel hourly rates and capacity cost rates (CCR) used for FL–TDABC modeling. All rates are estimates for a Canadian academic hospital setting..

Material	Unit Cost (CAD)	Source
Lab tests	0	–
Contrast (CTA)	125	Mayfair Diagnostics
IV–tPA (Alteplase 100 mg)	2,746	Federal Patented Medicine Prices Review Board (PMPRB)
Stroke pack	5,500	–

Appendix

Table 4. Material cost estimates used in the CT pathway cost model (in CAD).


Activity	Sub-tasks	Personnel	Equipment	Material	a	b	c	Expected Time (min)	Expected Cost (CAD)	Cost Low	Cost High
ED Triage
ED assessment with positive stroke symptoms	Placed on stretcher	ED clerk; ED nurse	Stretcher; Monitor	–	7	12	60	26.33	31.59	8.4	72
	Code stroke called by triage	ED nurse; ED physician	–	–	5	10	60	25	74	14.8	177.6
Patient monitoring	Placed on stretcher	ED nurse; ED technician	Monitor; Stretcher	–	5	10	60	25	38	7.6	91.2
	Hooked up to monitor

Appendix

Table 5. Process map of the conventional CT pathway. The activity structure and workflow steps were obtained from Sangha et al. ( 31 ) , while all time estimates and cost parameters were obtained through a time–motion study conducted with a practicing radiologist in a tertiary academic stroke centre.

Personnel role	Hourly rate (CAD)	CCR (CAD/min)
PACS / IT Technician	40.12	0.67
ML Engineer	44.10	0.74
Data Scientist	45.67	0.76
Radiologist	44.61	0.74

Appendix

Table 6. Personnel hourly rates and capacity cost rates (CCR) used for AI development costing.

Activity	Personnel (CCR, CAD/min)	$a$ (min)	$b$ (min)	$c$ (min)	$x^{*}=\frac{a+b+c}{3}$	Expected Cost (CAD)	Low / High (CAD)
Data gathering	PACS/IT (0.67)	3.00	4.00	5.00	4.00	2.68	2.01 / 3.35
Data Preprocessing	ML Engineer (0.735)	1.12	1.38	1.88	1.46	1.07	0.82 / 1.38
Data Labeling	Radiologist (0.74)	17.00	22.00	27.00	22.00	16.28	12.58 / 19.98
Model training $\dagger$	ML Engineer (0.735)	6.00	7.00	8.00	7	5.18	4.41 / 5.88
Total per sample						25.22	19.86 / 30.64

Appendix

Table 7. Process map for the Heidelberg training cohort (

N{=}835

Activity	Personnel (CCR, CAD/min)	$a$ (min)	$b$ (min)	$c$ (min)	$x^{*}=\frac{a+b+c}{3}$	Expected Cost (CAD)	Low / High (CAD)
Data gathering	PACS/IT (0.67)	3.00	4.00	5.00	4.00	2.68	2.01 / 3.35
Data Preprocessing	ML Engineer (0.735)	1.12	1.38	1.88	1.46	1.08	0.83 / 1.39
Data Labeling	Radiologist (0.74)	17.00	22.00	27.00	22.00	16.28	12.58 / 19.98
Inference	ML Engineer (0.735)	0.27	0.33	0.47	0.36	0.27	0.20 / 0.35
Evaluation / review	Radiologist (0.74)	5.00	7.00	9.00	7.00	5.18	3.70 / 6.66
Total per sample (priced)						25.49	19.32 / 31.73

Appendix

Table 8. Process map for the Heidelberg internal test cohort (

N{=}344

Section	Parameter	Recommended / Selected Value
Real-time inference instances	Number of models deployed	1
	Number of models per endpoint	1
	Number of instances per endpoint	1
	Endpoint hours per day	24
	Endpoint days per month	30
	Selected instance type	ml.g5.12xlarge (GPU-enabled)

Appendix

Table 9. AWS SageMaker deployment configuration and estimated monthly cost (Canada–Central region).


Activity	Sub-tasks	Personnel	Equipment	Material	a	b	c	Expected Time (min)	Expected Cost (CAD)	Cost Low	Cost High
CT Suite
Patient brought to CT	Patient transported by ED nurse	ED nurse; CT nurse	Stretcher; Monitor	–	10	15	30	18.33	32.44	17.7	53.1
	Monitored by: a) ED nurse, b) stroke team, c) CT tech
	d) ED physician e) ED tech
NCCT scan	NCCT done by CT tech	CT technologist; CT nurse	CTH (CT suite only); Monitor; Stretcher	–	4	6	8	6	10.32	6.88	13.76

Appendix

Table 10. CT suite process map for the cost of delivering care along the conventional CT pathway estimated using FL-TDABC

Feb 28, 2026

Real-World AI Evaluation: How FRAME Generates Systematic Evidence to Resolve the Decision-Maker's Dilemma

Organizational leaders are being asked to make high-stakes decisions about AI deployment without dependable evidence of what these systems actually do in the environments they oversee. The predominant AI evaluation ecosystem yields scalable but abstract metrics that reflect the priorities of model development. By smoothing over the heterogeneity of real-world use, these model-centric approaches obscure how behavior varies across users, workflows, and settings, and rarely show where risk and value accumulate in practice. More user-centric studies reveal rich contextual detail, yet are fragmented, small-scale and loosely coupled to the mechanisms that shape model behavior. The Forum for Real-World AI Measurement and Evaluation (FRAME) aims to address this gap by combining large-scale trials of AI systems with structured observation of how they are used in context, the outcomes they generate, and how those outcomes arise. By tracing the path from an AI system's output through its practical use and downstream effects, FRAME turns the heterogeneity of AI-in-use into a measurable signal rather than a trade-off for achieving scale. The Forum establishes two core assets to achieve this: a Testing Sandbox that captures AI-in-use under real workflows at scale and a Metrics Hub that translates those traces into actionable indicators.

Reva Schwartz, Gabriella Waters

Civitaas Insights · Center for Responsible AI, Virginia State University, and Civitaas Insights

Human-Centered AI AI Governance

Ask PDF

Aug 31, 2026

VFR-Audit: Verdict-Level Reliability for Fairness Audits in Hospital Length-of-Stay Prediction

Fairness audits in clinical Artificial Intelligence convert continuous fairness metrics into binary pass-or-fail verdicts against operational thresholds, where hospital governance boards, payers, and regulators act on the resulting verdicts. Such audits are repeated over time and across hospital sites, thus the same verdict can flip between pass and fail across audits. Existing uncertainty methods such as Bayesian posteriors, bootstrap confidence intervals, and permutation tests address verdict instability only at the continuous-metric level. Converting metric-level uncertainty into a verdict-stability claim remains a manual step that scales poorly across the (model, metric, attribute) cells an audit covers. Existing uncertainty methods also leave open whether bias-mitigation steps, such as reweighing or per-group threshold shifts, yield a stable passing verdict at the cost of model discrimination measured as AUROC or AUPRC.To address this verdict-stability gap, we propose VFR-Audit, a framework built around the Verdict Flip Rate (VFR), a scalar bounded between 0 and 0.5 that measures the probability of verdict reversal under stratified bootstrap resampling. VFR-Audit reports VFR alongside three reliability axes, namely within-cohort resampling stability, audit-size sensitivity, and cross-hospital verdict agreement via Fleiss' kappa.

Md Jannatul Rakib Joy, Viet Vo, Caslon Chua

School of Science, Computing and Emerging Technologies

Algorithmic Auditing

Ask PDF

Apr 15, 2026

Deployment of AI-Assisted Interventions: Capacity Constraints and Noisy Compliance

AI tools increasingly drive targeted interventions in various service settings, including healthcare, education, and public services. Algorithms score individuals, trigger outreach to those above a threshold (e.g., high-risk or high-value), and encourage them to request service; then providers deliver service to those who request. Much of the work in this area has focused on improving predictive accuracy, implicitly assuming that better predictions lead to better outcomes. We show that predictions are only one component of a larger service system: when service capacity is limited and behavioral responses to outreach are probabilistic, system efficacy depends on operational forces that predictive accuracy does not capture. In such settings, the optimal score threshold must balance two effects: ensuring all capacity is filled (utilization) and, when capacity is constrained, preventing low-value requests from crowding out high-value ones (cannibalization). We characterize the optimal threshold and prove that thresholds based solely on predictive accuracy are generally suboptimal. Further, algorithm selection metrics such as AUC can be misaligned with operational performance: they weight all thresholds equally, while optimal deployment uses a subset of thresholds that depends on both capacity and compliance behavior. We introduce a new metric, Operational AUC (OpAUC), and show that it identifies the efficacy-optimal algorithm. Finally, we conduct a case study on sepsis early warning data that illustrates the magnitude of the improvement available from better threshold selection and shows that a predictor with lower AUC can achieve higher system efficacy under optimal deployment.

Carri W. Chan, Yi Han, Hannah Li +1

Decision, Risk, and Operations, Columbia Business School · Department of Statistics, Columbia University · Division of Pulmonary, Allergy, and Critical Care Medicine, Department of Medicine, Vagelos College of Physicians and Surgeons, Columbia University

AI-Assisted Decision Making Constrained Optimization

Ask PDF

Abstract

Explore similar work

Real-World AI Evaluation: How FRAME Generates Systematic Evidence to Resolve the Decision-Maker's Dilemma

VFR-Audit: Verdict-Level Reliability for Fairness Audits in Hospital Length-of-Stay Prediction

Deployment of AI-Assisted Interventions: Capacity Constraints and Noisy Compliance