When accuracy hides risk: risk-controlled confirmation of the critical view of safety under expert disagreement
Organizations: Department of Biomedical Engineering, Vanderbilt University, 2301 Vanderbilt Place, Nashville, TN, 37235, USA · Xiangshu MedTech, Hangzhou, China
Abstract
Automated assessment of the critical view of safety (CVS) in laparoscopic cholecystectomy has advanced rapidly, but a deployed check must decide, before the first clip, whether to confirm CVS. The standard rule calibrates a threshold so that false confirmations are rare on average. We show that this rule concentrates its errors on disputed videos, which one of three expert raters judged sufficient, and that the failure is structural. When reference labels are votes of a panel sampled from a larger expert pool, the rule that maximizes sensitivity at a fixed average false-confirmation rate also maximizes false confirmation of disputed videos, and for recognizers that recover expert judgement with less noise this rate rises with accuracy. We propose certified confirmation, a risk-controlled monitoring rule whose threshold is selected by multiple testing so that, with probability at least 1 - δ, false confirmation is at most α overall, among disputed videos, within vendor and country groups, and before CVS is visible, using disputed labels only to calibrate. In a pre-specified, hash-locked evaluation on the 300 test videos (18 countries) of the SAGES CVS Challenge, standard calibration falsely confirmed 8 of 39 disputed videos (0.21; one-sided p = 0.037 against α = 0.10), whereas certified confirmation confirmed 2 (one-sided 95% upper bound 0.15). On development data, protection held for vendor and country groups left out of calibration. The cost was confirming 10% rather than 35% of sufficient videos on time. Average accuracy hides this risk; disputed cases must be protected explicitly.
Figures & tables
| Dataset | Videos | Sufficient | Disputed | Clear | Use |
|---|---|---|---|---|---|
| SAGES CVS Challenge 2024, training | 700 | 205 | 135 | 360 | development (5-fold cross-fitting) |
| SAGES CVS Challenge 2024, test | 300 | 62 | 39 | 199 | single pre-specified evaluation |
| Endoscapes2023, development | 155 | 46 | 20 | 89 | replication of the paradox (hypothesis-generating) |
| Cholec80-CVS, development | 56 | 14 | 19 | 23 | replication of the paradox (hypothesis-generating) |
| Status | Analyses | Where |
|---|---|---|
| Pre-specified, hash-locked, single test run | Scorers, thresholds and endpoints E1–E6; development results of the standard and certified rules for the frame and temporal scorers; method variants that failed | Tables 3 , 4 ; Figs. 1 c (top), d, 2 a,b, 4 ; Appendix C |
| Development, after the lock | Endoscapes2023 and Cholec80-CVS replication; frozen-scorer results; further comparators (fixed probability, Youden, PAC marginal, group-wise conformal, weighted risk control); the eleven-scorer association; left-out vendors and countries; resampling illustration; ablation; other levels ; positive predictive value and false-discovery-rate certification; model-based safety quantities and identification bounds | Tables 3 , 5 , 1 – 3 ; Figs. 1 c (bottom), 2 d, 3 a,c, 5 b |
| Post hoc on test labels (logged) | Group-wise conformal rule; model curve refitted to test votes; vote-count reference; premature confirmation among videos with an observed onset; positive predictive value; country-clustered intervals; split calibration; missing criteria; comparison at matched disputed rate; label-shift tests | Sections 7.4 – 7.7 ; Table 6 ; Fig. 5 a |
| False confirmation | Worst | Timely | |||||
| Scorer | Rule | all | disputed | clear | Premature | group | sensitivity |
| Frozen | standard (conformal) | 0.101 | 0.170 | 0.075 | 0.122 | 0.143 | 0.229 |
| certified | 0.028 | 0.059 | 0.017 | 0.024 | 0.048 | 0.112 | |
| Frame | probability | 0.057 | 0.148 | 0.022 | 0.093 | 0.059 | 0.254 |
| Youden | 0.319 | 0.563 | 0.228 | 0.429 | 0.419 | 0.351 | |
| standard (conformal) | 0.099 | 0.207 | 0.058 | 0.166 | 0.114 | 0.332 | |
| Endpoint | Result | Outcome |
|---|---|---|
| E1a: standard disputed false confirmation (frame) | 8/39 = 0.205 (95 % CI 0.093–0.365), | met |
| E1b: certified overall, disputed and premature risk | Holm ; disputed 2/39 (UB 0.153), overall 7/238 (UB 0.055), premature 2/62 (UB 0.098) | no violation found |
| E2: standard premature confirmation (frame) | 7/62 = 0.113, | inconclusive |
| E3: vendor and country groups | certified, frame: vendor 0 = 5/36 (UB 0.270); other groups at most 3 | reported |
| E4: temporal vs frame, timely (certified) | 13/62 vs 6/62, McNemar 8:1, | met; exploratory (see text) |
| E5: guidance false alerts | frame 0.06–0.09; temporal 0.06–0.12 | partly met |
| Left out | Resampling | |||
|---|---|---|---|---|
| Scorer | Rule | disputed FC | any | Timely |
| Frame | standard | 49/238 | 1.000 | 0.337 |
| PAC marginal | – | 1.000 | 0.325 | |
| group-wise | 23/238 | 0.318 | 0.178 | |
| certified | 12/238 | 0.016 | 0.127 | |
| Temporal | standard | 51/238 | 1.000 | 0.329 |
| Data | Scorer | Rule | Conf. | False | Disp. | PPV |
|---|---|---|---|---|---|---|
| Test | frame | standard | 18.0 | 8.3 | 2.7 | 0.54 (29/54) |
| certified | 5.0 | 2.3 | 0.7 | 0.53 (8/15) | ||
| temporal | standard | 18.3 | 8.7 | 4.0 | 0.53 (29/55) | |
| certified | 6.3 | 2.0 | 1.0 | 0.68 (13/19) | ||
| Dev. | frame | standard | 21.6 | 7.0 | 4.0 | 0.68 (102/151) |
| certified | 5.4 | 1.1 | 1.0 | 0.79 (30/38) |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Frame | Temporal | |||
|---|---|---|---|---|
| Certified risks | Timely | FR | Timely | FR |
| overall only | 0.332 | 0.178 | 0.312 | 0.170 |
| overall + disputed | 0.107 | 0.052 | 0.200 | 0.059 |
| overall + premature | 0.205 | 0.111 | 0.195 | 0.059 |
| overall + groups | 0.176 | 0.111 | 0.176 | 0.044 |
| all (certified rule) | 0.107 | 0.052 | 0.176 | 0.044 |
| Frame | Temporal | ||||
|---|---|---|---|---|---|
| Rule | FR | Timely | FR | Timely | |
| 0.05 | standard | 0.126 | 0.234 | 0.089 | 0.254 |
| group-wise | 0.037 | 0.112 | 0.044 | 0.137 | |
| certified | 0.022 | 0.088 | 0.015 | 0.073 | |
| 0.20 | standard | 0.393 | 0.351 | 0.393 | 0.341 |
| group-wise | 0.200 | 0.341 | 0.185 | 0.317 | |
| FC | AUROC vs | |||||
|---|---|---|---|---|---|---|
| Recognizer | mAP | disp. | clear | all | disp. | clear |
| ResNet-18, frozen | 0.284 | 3 | 8 | 0.54 | 0.49 | 0.55 |
| DINOv3-B, frozen | 0.384 | 3 | 7 | 0.68 | 0.59 | 0.70 |
| DINOv3-B, SA + DA | 0.392 | 4 | 7 | 0.71 | 0.61 | 0.73 |
| DINOv2-B, frozen | 0.401 | 6 | 6 | 0.66 | 0.52 | 0.69 |
| DINOv3-B, SA | 0.420 | 4 | 5 | 0.71 | 0.59 | 0.73 |
Surgical Video Understanding
Surgical video understanding uses computer vision and multimodal AI to interpret procedures, instruments, actions, and events over time.
See all 134 papers