Which AI Use Cases Deserve a Pilot? A Risk and Value Framework for Funding, Scoping, and Scaling AI
Organizations: Whiting School of Engineering, Johns Hopkins University, Baltimore, MD 21218
Abstract
A composite score can rank AI use cases while leaving the permitted operating scope and the evidence required for expansion unspecified. We treat the unit of decision as a use case at a proposed scope (authority, human mediation, limits, volume, environment, population) and define three decisions: screen the scope, fund the experiment, and authorize expansion only when residual risk is acceptable and evidence of incremental net value applies at that scope. A risk track (R1-R4) is implemented as an explicit decision table with vetoes and documented precedence, and a value track follows four evidence stages with each transition needing its own evidence. Funding uses expected value of perfect information as an upper bound and the value of sample information for the specific experiment. A quantified expected-loss check, required where policy says so, computes the containment an expansion needs and credits a control only on evidence measured at the proposed scope; in a synthetic scenario the highest-value control design fails that test while a smaller scope passes. A reproducible Python package with 279 passing tests demonstrates the framework on synthetic refund, reconciliation and knowledge-assistant use cases, including a synthetic matched-scope fixture and a transferability stress test (value evidence is simulated, not realized), and compares it with a public AWS sample. Thresholds and appetite are prototype policy requiring field validation.
Figures & tables
| R2 control level | R3 = 1 | R3 = 2 | R3 = 3 |
|---|---|---|---|
| 1 | LOW | LOW | MEDIUM |
| 2 | LOW | MEDIUM | HIGH |
| 3 | MEDIUM | HIGH | CRITICAL |
| Use case | V1/E1 | V2/E2 | V3/E3 | V4/E4 | Level | Durable |
|---|---|---|---|---|---|---|
| RA | measured | measured | measured | measured | E4 | yes |
| RB | measured | measured | measured | measured | E4 | yes |
| K1 | measured | measured | measured | measured | E4 | yes |
| P1 | measured | measured | weak | none | E2 | no |
| RC | measured | none | none | none | E1 | no |
| Use case | Mechanism | Decision influence | Observable outcome |
|---|---|---|---|
| RA | decision_change | applicable | Cost per eligible case: handling minutes, leakage from wrong refunds, rework. |
| RB | speed | not_applicable | Handling minutes per case after the approval decision, net of approver minutes. |
| RC | capacity | not_applicable | Share of refunds completed without staff time; errors and recovery cost. |
| P1 | speed | not_applicable | Minutes to complete an unchanged reconciliation, before versus after. |
| K1 | decision_change | applicable | Cost per contact: handling minutes, errors, rework; plus CSAT as guardrail. |
| D1 | decision_change | unknown | Cost per dispute. |
| Cases/day (V) | Actions/yr (N = 365 V) | Expected loss, no control | Required containment | Achieved, sampled review | Expected loss, sampled review |
|---|---|---|---|---|---|
| 300 | 109,500 | $219,000 | 0.0% | 85.0% | $32,850 |
| 1,644 | 600,060 | $1,200,120 | 75.0% | 31.0% | $827,820 |
| Case (RA at 1,644 cases/day; quantified check required) | Quantified check | Expected loss, $/yr (credited) | Decision |
|---|---|---|---|
| no quantified inputs | insufficient (no inputs) | – | Extend pilot |
| evidence at pilot scope, no transport | insufficient (evidence not applicable) | – | Extend pilot |
| pilot-scope evidence, value-only transport record, 1,451 reviews/day | insufficient (no risk-side rule) | – | Extend pilot |
| pilot-scope evidence, risk-side recall ratio 0.85, 1,451 reviews/day | insufficient (evidence too weak) | $434,826 | Extend pilot |
| sampled review, 600 reviews/day | fail | $827,820 | Redesign |
| sampled review, 1,450 reviews/day | fail | $300,395 | Redesign |
| Design | Scope | d | Residual expected loss | Control cost | False blocks/yr | Net value | Planning test | Matched evidence |
|---|---|---|---|---|---|---|---|---|
| D0 | proposed scope | 0.000 | $1,200,120 | $0 | 0 | $1,200,120 | fail | fail |
| D1a | proposed scope | 0.310 | $827,820 | $394,200 | 0 | $740,220 | fail | fail |
| D1b | scope change: served volume restricted to B | 0.850 | $65,700 | $394,200 | 0 | -$21,900 | queue | pass |
| D2 | proposed scope | 0.723 | $333,033 | $42,568 | 594 | $2,009,787 | fail | fail |
| D3 | proposed scope | 0.000 | $1,013,970 | $394,200 | 0 | $992,070 | fail | fail |
| D4 | scope change: limit $100 | 0.723 | $166,517 | $42,568 | 594 | $1,576,244 | pass | pass |
| Design | Scope | E0: pilot-scope evidence, value-side transfer record | E1: matched scope, 59/59 | E2: matched scope, 0/59 | Zero-miss attempts needed |
|---|---|---|---|---|---|
| D0 | proposed scope | fail / Redesign | fail / Redesign | fail / Redesign | – |
| D1a | proposed scope | insufficient (no risk-side rule) / Extend pilot | fail / Redesign | fail / Redesign | – |
| D1b | proposed scope | insufficient (no risk-side rule) / Extend pilot | fail (queue) / Redesign | fail (queue) / Redesign | – |
| D1b | scope change: served volume 600/day | insufficient (no risk-side rule) / Extend pilot | pass / Scale | fail / Redesign | – |
| D1c | proposed scope | insufficient (no risk-side rule) / Extend pilot | pass / Scale | fail / Redesign | 19 |
| D2 | proposed scope | insufficient (no risk-side rule) / Extend pilot | fail / Redesign | fail / Redesign | – |
| Containment to show | Attempts, 0 missed | 1 missed | 2 missed | 3 missed | Expected traffic at q = 1% | Traffic for a 95% chance of enough attempts |
|---|---|---|---|---|---|---|
| 90% | 29 | 46 | 61 | 76 | 2,900 | 3,834 |
| 95% | 59 | 93 | 124 | 153 | 5,900 | 7,211 |
| Use case | Pilot rung | Residual risk, pilot scope | Residual risk, target scope |
|---|---|---|---|
| RA | HUMAN_APPROVED_LIVE | LOW [R1 2 | R2 1 | R3 2] | HIGH [R1 2 | R2 2 | R3 3] |
| RB | HUMAN_APPROVED_LIVE | HIGH [R1 2 | R2 2 | R3 3] | CRITICAL [R1 2 | R2 3 | R3 3] |
| RC | SHADOW | HIGH [R1 2 | R2 2 | R3 3] | CRITICAL [R1 2 | R2 3 | R3 3] |
| Use case | Cost | EVPI bound | EVSI | EVSI belongs to | Information case | Recommendation |
|---|---|---|---|---|---|---|
| RB | $140,000 | $341,766 | $335,104 | live pilot; the modelled scale decision cannot reach SCALE at the proposed scope | supported | redesign the scope, then reassess pilot funding |
| K1 | $70,000 | $227,354 | $226,455 | live pilot, permitted | supported | fund live pilot |
| RA | $80,000 | $157,211 | $151,051 | live pilot, permitted | supported | fund live pilot |
| RC | $200,000 | $88,629 | $86,105 | hypothetical live design; the screen does not permit it | EVPI below cost | shadow run only |
| P1 | $60,000 | $111,678 | $110,154 | live pilot, permitted | supported | fund live pilot |
| D1 | $90,000 | n/a | n/a | none | not eligible |
| Use case | Experiment | Decision that could change | Independent rationale |
|---|---|---|---|
| RA | live pilot at the pilot scope (3000 cases per arm, 8 weeks; a concurrent control is needed | scale versus extend, redesign or stop at the proposed scope (the value gate) | direct operating benefit during the pilot (assumption) |
| RB | live pilot at the pilot scope (2000 cases per arm, 8 weeks; a concurrent control is needed | a redesigned, lower-scope expansion decision. The pilot cannot earn approval at the proposed scope: best-case risk tier CRITICAL exceeds the HIGH appetite even with every required control credited | direct operating benefit during the pilot (assumption) |
| RC | shadow run on live traffic: the AI decides, nothing is executed | the pilot screen for the next rung (whether a live approval-gated pilot may be authorized), not the scale decision | none stated |
| P1 | live pilot at the pilot scope (600 cases per arm, 6 weeks; a concurrent control is needed | scale versus extend, redesign or stop at the proposed scope (the value gate) | direct operating benefit during the pilot (assumption); Regulatory exam finding on reconci |
| K1 | live pilot at the pilot scope (6000 cases per arm, 8 weeks; a concurrent control is needed | scale versus extend, redesign or stop at the proposed scope (the value gate) | direct operating benefit during the pilot (assumption) |
| Use case | Design | Evid. | Req. | Value | Risk | Decision | Annual value (low / base / high) |
|---|---|---|---|---|---|---|---|
| RA | randomized | E4 | E3 | PASS | PASS | SCALE | 465,234 / $710,174 |
| RB | randomized | E4 | E3 | INSUFFICIENT | FAIL | REDESIGN | 255,999 / $393,111 |
| K1 | randomized | E4 | E3 | FAIL | INSUFFICIENT | REDESIGN | -161,789 / $-99,561 |
| P1 | pre_post | E2 | E3 | INSUFFICIENT | INSUFFICIENT | EXTEND_PILOT | -107,026 / $21,740 |
| RC | shadow | E1 | E4 | INSUFFICIENT | FAIL | REDESIGN | n/a / n/a / n/a |
| Use case | Status | Executive action | Lower-scope experiment? |
|---|---|---|---|
| RA | SCALE | Deploy/Scale | - |
| RB | REDESIGN | Redesign | a redesigned (lower) scope is a new candidate and is screened afresh |
| K1 | REDESIGN | Redesign | a redesigned (lower) scope is a new candidate and is screened afresh |
| P1 | EXTEND_PILOT | Extend pilot with constraints | the extended pilot is the lower-scope experiment |
| RC | REDESIGN | Redesign | a redesigned (lower) scope is a new candidate and is screened afresh |
| Use case | 153-test revision | This revision | Changed |
|---|---|---|---|
| RA | SCALE (value PASS, risk PASS) | SCALE (value PASS, risk PASS) | no |
| RB | REDESIGN (value INSUFFICIENT, risk FAIL) | REDESIGN (value INSUFFICIENT, risk FAIL) | no |
| K1 | REDESIGN (value FAIL, risk INSUFFICIENT) | REDESIGN (value FAIL, risk INSUFFICIENT) | no |
| P1 | EXTEND_PILOT (value INSUFFICIENT, risk INSUFFICIENT) | EXTEND_PILOT (value INSUFFICIENT, risk INSUFFICIENT) | no |
| RC | REDESIGN (value INSUFFICIENT, risk FAIL) | REDESIGN (value INSUFFICIENT, risk FAIL) | no |
| Mutation of the passing case | Before | After | Executive action |
|---|---|---|---|
| none (control) | SCALE (value PASS, risk PASS) | SCALE (value PASS, risk PASS) | Deploy/Scale |
| 1. add a hard blocker | SCALE (value PASS, risk PASS) | BLOCKED (value PASS, risk PASS) | Stop |
| 2. operating scope unknown | SCALE (value PASS, risk PASS) | DEFER (value PASS, risk PASS) | none (deferred) |
| 3. remove all guardrail results | SCALE (value PASS, risk PASS) | EXTEND_PILOT (value INSUFFICIENT, risk PASS) | Extend pilot with constraints |
| 4. weaken mediation per-item to sampled, reuse evidence | SCALE (value PASS, risk PASS) | REDESIGN (value INSUFFICIENT, risk FAIL) | Redesign |
| Proposed scope change to RA | Decision | Value | Risk |
|---|---|---|---|
| as evaluated (limit $500, 1,644/day, per-item) | SCALE | PASS | PASS |
| volume x10 (16,440/day) | EXTEND_PILOT | INSUFFICIENT | INSUFFICIENT |
| limit $1,000 | EXTEND_PILOT | INSUFFICIENT | INSUFFICIENT |
| uncapped | EXTEND_PILOT | INSUFFICIENT | INSUFFICIENT |
| sampled review | REDESIGN | INSUFFICIENT | FAIL |
| execute with approval | REDESIGN | INSUFFICIENT | FAIL |
| Case | Decision | Relies on transfer | Evidence scope |
|---|---|---|---|
| matched scope (evidence at 1,644/day) | SCALE | no | 1,644/day |
| transferred (evidence at 300/day, proposal 1,644/day) | SCALE | yes | 300/day |
| Variant | Decision | Value gate | Low end ($/yr) |
|---|---|---|---|
| as evaluated (volume transport, haircut 0.85, hurdle $100,000) | SCALE | PASS | 164750.0 |
| transfer assumption removed | EXTEND_PILOT | INSUFFICIENT | |
| assumption stated but not evaluated | EXTEND_PILOT | INSUFFICIENT | |
| effect haircut 1.00 | SCALE | PASS | 220294.0 |
| effect haircut 0.85 | SCALE | PASS | 164750.0 |
| effect haircut 0.70 | SCALE | PASS | 109206.0 |
| Variant | Composite | Risk | Readiness | Verdict |
|---|---|---|---|---|
| AWS as implemented (NO GO needs all three) | 4.4 | 25 | 5.0 | CONDITIONAL GO |
| AWS with documented OR rule (any one) | 4.4 | 25 | 5.0 | NO GO |
| Use case | State | Composite | Risk | AWS as implemented | AWS OR rule | This framework |
|---|---|---|---|---|---|---|
| RA | intake | 3.38 | 9 | CONDITIONAL GO | CONDITIONAL GO | HUMAN_APPROVED_LIVE |
| RA | post_pilot | 3.56 | 6 | GO | GO | SCALE |
| RB | intake | 2.9825 | 12 | CONDITIONAL GO | CONDITIONAL GO | HUMAN_APPROVED_LIVE |
| RB | post_pilot | 3.0575 | 12 | CONDITIONAL GO | CONDITIONAL GO | REDESIGN |
| RC | intake | 3.045 | 25 | CONDITIONAL GO | NO GO | SHADOW |
| RC | post_pilot | 3.075 | 25 | CONDITIONAL GO | NO GO | REDESIGN |
| Inventory key | Use case | Benefit claim | Pilot rung | Median rank |
|---|---|---|---|---|
| VA|VA-25-5578 | AICES | qualitative | SIMULATION | 11.0 |
| VA|VA-25-3732 | Modeling to Learn (MTL) | qualitative | SIMULATION | 11.0 |
| VA|VA-25-3480 | Smart Pension Automation | quantified | SIMULATION | 8.0 |
| SBA|SBA-30 | SBA.gov External Customer Chatbot | quantified | HOLD | 3.0 |
| SBA|SBA-35 | AI Incident Summarization | quantified | HOLD | 3.0 |
| SBA|SBA-13 | Generative AI for SBA.gov product training | qualitative | HOLD | 5.0 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Case (RA at 1,644 cases/day) | Before the evidence standard | After | |
|---|---|---|---|
| 1 | pipeline d1 = 1, evidence = None | pass / Scale | insufficient (evidence not applicable) / Extend pilot |
| 2 | 1,451 reviews/day, observed 0/59 caught | pass / Scale | fail / Redesign |
| 3 | 1,451 reviews/day, pilot-scope 59/59, transport record effect factor 0.85 | pass / Scale | insufficient (no risk-side rule) / Extend pilot |
AI Risk Management
AI risk management studies how to identify, assess, reduce, and monitor harms from AI across development, deployment, and wider social systems.
See all 112 papers