From Failure Prediction to Component Selection in Semantic Segmentation
Organizations: School of Power Engineering, Jiangxi Electric Vocational and Technical College, Nanchang, Jiangxi, China · College of Information Science and Engineering, Shandong Agricultural University, Tai’an, Shandong, China · College of Aviation Manufacturing Industry, Nanchang Hangkong University, Jingdezhen, Jiangxi, China
Abstract
Reliable scene understanding requires deciding which predictions to trust, retain, or inspect. In semantic segmentation, component failure predictors support these decisions, but their predictive performance does not fully describe the value of the resulting selection. We develop a decision-centered evaluation framework that connects three questions: what an available score predicts, what can be learned from its existing fields, and what its ranking delivers. Across three segmentation systems on Cityscapes and ACDC at 90% retention, adapted MetaSeg reduces retained failure relative to forecast entropy in all twelve architecture-population comparisons, but does not establish a failure advantage over direct probability ranking. Image-nested source experiments reveal a different opportunity: reusing the full field set improves predictive and retained-loss performance under both learners in all three systems, while the raw OCRNet probe improves prediction with unresolved selective gains. Mapping a fixed prediction pair to conditional log-loss risk produces mixed selection effects, which we examine through the components exchanged by the two rankings. These findings distinguish the performance of a fitted score from the value obtainable from its inputs and the consequences of its use. The framework provides a practical basis for deciding whether to adopt an additional score, relearn existing fields, or revise a selection rule, and for explaining the gains and failures of component-level reliability methods.
Figures & tables
| Population | Images | AUROC | AUROC | CE | CE | |
|---|---|---|---|---|---|---|
| SegFormer | ||||||
| P0 | 300 | 18,728 | 0.8105 | 0.7157 | 0.4993 | 0.6076 |
| P1 | 200 | 12,209 | 0.8115 | 0.7361 | 0.5121 | 0.5927 |
| P2 | 406 | 32,429 | 0.7562 | 0.7090 | 0.5834 | 0.6825 |
| P3 | 400 | 29,614 | 0.7403 | 0.6973 | 0.5953 | 0.6880 |
| DeepLab | ||||||
| LL [nats] | Failures/ | ||||||
|---|---|---|---|---|---|---|---|
| Scene | Native | Mapped | Native | Mapped | |||
| Source 1 | 118 | 107 | 10 | 0.381016 | 0.397064 | 19/107 | 23/107 |
| Source 2 | 91 | 82 | 9 | 0.345995 | 0.325607 | 12/82 | 16/82 |
| Fog | 116 | 105 | 9 | 0.543883 | 0.575054 | 63/105 | 61/105 |
| Night | 241 | 217 | 22 | 0.541156 | 0.533147 | 164/217 | 169/217 |
| Rain | 178 | 161 | 17 | 0.535679 | 0.501192 | 83/161 | 89/161 |
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
| 1 | Input the original five image folds and fixed upstream/downstream learner definitions. |
|---|---|
| 2 | For each outer validation fold , set to the other four folds. |
| 3 | For each image fold , fit upstream F2/F3 on , including preprocessing, and predict . |
| 4 | Join the inner-held-out rows to form downstream training features on . All fits exclude labels from . |
| 5 | Fit upstream F2/F3 on all and generate outer-validation features on . |
| 6 | Fit each diagnostic’s transformations and learner on the cross-fitted rows in , using its specified image weights. |
| 7 | Apply the frozen transformations and predictors to . |
| Index | Field | Formula | Unit |
|---|---|---|---|
| 1 | area | size(K) | pixels |
| 2 | boundary_size | size(B) | pixels |
| 3 | interior_size | size(I) | pixels |
| 4 | mean_max_probability | mean_K u | probability |
| 5 | mean_entropy | mean_K h | nats |
| 6 | mean_probability_margin | mean_K g | probability difference |
| Object | Features | Target | Learner |
|---|---|---|---|
| Base forecast (F2) | 18 numeric + 19 indicators | Y | fixed L1 liblinear |
| Frozen score F3 | 11 descriptors | global-minus-image affine opportunity | robust-scaled ridge |
| Descriptors (G3) | p + raw 11 descriptors | Y | D1-R / D2-R |
| All fields (G4) | p + raw 37 fields | Y | D1-R / D2-R |
| MetaSeg | 15 E/D/S + 19 class means | Y | nested-C L1 saga |
| Fixed-pair mapping | available frozen (p,m) | no fit | deterministic mapping |
| Fields | Learner | CE vs G0 | CE vs PC | LL vs G0 | LL vs Entropy |
|---|---|---|---|---|---|
| SegFormer (A1) | |||||
| G1 | Spline | +0.004 [-0.386, +0.371] | +0.126 [-0.246, +0.487] | +0.086 [-0.087, +0.272] | -0.028 [-0.260, +0.202] |
| G1 | Boosting | -1.454 [-2.627, -0.163] | -1.454 [-2.627, -0.163] | -0.888 [-1.718, -0.119] | -1.274 [-2.191, -0.411] |
| G2 | Spline | -0.010 [-0.222, +0.203] | +0.112 [-0.150, +0.381] | +0.060 [-0.099, +0.217] | -0.055 [-0.267, +0.164] |
| G2 | Boosting | -0.708 [-1.942, +0.558] | -0.708 [-1.942, +0.558] | -0.493 [-1.240, +0.251] | -0.878 [-1.578, -0.168] |
| G3 | Spline | -0.043 [-0.893, +0.838] | +0.174 [-0.671, +1.105] | +0.013 [-0.218, +0.262] | -0.101 [-0.363, +0.171] |
| Learner / fields | PC columns | control minus Gz LL [95% CI] |
|---|---|---|
| SegFormer (A1) | ||
| Spline / G1 | 1 | +0.095 [-0.087, +0.289] |
| Spline / G2 | 1 | +0.068 [-0.100, +0.232] |
| Spline / G3 | 11 | +0.028 [-0.221, +0.278] |
| Spline / G4 | 37 | +2.238 [+0.418, +4.260] |
| Boosting / G1 | 1 | -0.888 [-1.718, -0.119] |
| Population / eligibility | N components | MetaSeg AUROC | AUROC | MetaSeg CE | CE |
|---|---|---|---|---|---|
| SegFormer (A1) | |||||
| Source / All | 18728 | 0.715702 | 0.810513 | 0.607565 | 0.499287 |
| Source / Interior | 15869 | 0.798600 | 0.803125 | 0.485110 | 0.483277 |
| Src. dev. / All | 12209 | 0.736090 | 0.811533 | 0.592712 | 0.512086 |
| Src. dev. / Interior | 10277 | 0.805113 | 0.803678 | 0.503272 | 0.501892 |
| Adv. dev. / All | 32429 | 0.708956 | 0.756237 | 0.682546 | 0.583443 |
| Population / eligibility | MetaSeg minus : AUROC [95% CI] |
|---|---|
| SegFormer (A1) | |
| Source / All | -94.811 [-103.044, -86.755] |
| Source / Interior | -4.525 [-7.242, -1.720] |
| Src. dev. / All | -75.443 [-85.645, -65.133] |
| Src. dev. / Interior | +1.435 [-2.501, +5.529] |
| Adv. dev. / All | -47.281 [-54.037, -40.452] |
| Population | LL vs entropy | LL vs | Failure vs entropy | Failure vs |
|---|---|---|---|---|
| SegFormer (A1) | ||||
| Source | -8.404 [-10.368, -6.455] | +10.100 [+7.773, +12.596] | +1.885 [+1.511, +2.252] | -1.147 [-1.414, -0.866] |
| Src. dev. | -8.010 [-10.215, -5.774] | +11.683 [+8.312, +15.035] | +1.896 [+1.426, +2.340] | -1.018 [-1.353, -0.679] |
| Adv. dev. | -18.003 [-19.594, -16.344] | +6.758 [+4.350, +9.058] | +1.771 [+1.501, +2.032] | -0.254 [-0.478, -0.031] |
| Final | -16.282 [-17.984, -14.515] | +4.607 [+1.878, +7.553] | +1.617 [+1.360, +1.888] | -0.106 [-0.345, +0.137] |
| DeepLab (A2) | ||||
| Population | Reference | Failure gain [95% CI] | LL gain [95% CI] |
|---|---|---|---|
| SegFormer (A1) | |||
| Source | Entropy | 1.885 [1.530, 2.251] | -8.404 [-10.356, -6.360] |
| Probability | -1.147 [-1.428, -0.873] | 10.100 [7.635, 12.470] | |
| Src. dev. | Entropy | 1.896 [1.436, 2.331] | -8.010 [-10.287, -5.819] |
| Probability | -1.018 [-1.342, -0.657] | 11.683 [8.309, 14.798] | |
| Adv. dev. | Entropy | 1.771 [1.504, 2.044] | -18.003 [-19.515, -16.347] |
| Population | Reference | Failure gain [95% CI] | LL gain [95% CI] |
|---|---|---|---|
| SegFormer (A1) | |||
| Source | Entropy | 2.075 [1.675, 2.473] | -9.217 [-11.313, -7.129] |
| Probability | -0.289 [-0.506, -0.081] | 1.776 [0.667, 2.988] | |
| Src. dev. | Entropy | 1.721 [1.256, 2.162] | -8.177 [-10.676, -5.758] |
| Probability | -0.141 [-0.400, 0.122] | 1.336 [-0.306, 2.989] | |
| Adv. dev. | Entropy | 1.850 [1.557, 2.141] | -19.219 [-20.954, -17.398] |
| Population | All: native-map LL | All: Entropy-map LL | Interior: native-map LL | All: native-map fail |
|---|---|---|---|---|
| SegFormer (A1) | ||||
| Source | -5.547 [-9.010, -2.389] | -13.952 [-16.651, -11.387] | +6.420 [+4.218, +8.601] | +0.643 [+0.277, +0.997] |
| Src. dev. | -8.863 [-12.422, -5.104] | -16.873 [-20.011, -13.581] | +8.757 [+6.083, +11.598] | +0.712 [+0.326, +1.094] |
| Adv. dev. | -0.813 [-3.323, +1.855] | -18.816 [-21.148, -16.560] | +24.582 [+22.424, +26.871] | -0.197 [-0.453, +0.055] |
| Final | -0.926 [-3.876, +2.083] | -17.209 [-20.057, -14.374] | +23.236 [+20.954, +25.544] | -0.126 [-0.386, +0.125] |
| DeepLab (A2) | ||||
| Population | All-valid: minus mapped | Interior native minus map |
|---|---|---|
| SegFormer (A1) | ||
| Source | -0.504 [-0.716, -0.301] | -1.596 [-2.006, -1.178] |
| Src. dev. | -0.306 [-0.498, -0.107] | -1.409 [-1.856, -0.973] |
| Adv. dev. | -0.450 [-0.598, -0.297] | -1.566 [-1.849, -1.278] |
| Final | -0.232 [-0.382, -0.089] | -1.377 [-1.671, -1.082] |
| DeepLab (A2) | ||
| Population / eligibility | 50% | 60% | 70% | 80% | 90% | 95% | 100% |
|---|---|---|---|---|---|---|---|
| SegFormer (A1) | |||||||
| Source / All | -5.962 | -8.611 | -9.861 | -7.419 | -5.547 | -2.931 | +0.000 |
| Source / Interior | +6.837 | +9.285 | +10.218 | +9.392 | +6.420 | +4.299 | +0.000 |
| Src. dev. / All | -9.458 | -11.981 | -10.937 | -9.857 | -8.863 | -3.773 | +0.000 |
| Src. dev. / Interior | +9.006 | +12.366 | +14.174 | +12.687 | +8.757 | +5.282 | +0.000 |
| Adv. dev. / All | +6.653 | +6.239 | +4.849 | +0.521 | -0.813 | -0.801 | +0.000 |
| Population / eligibility | native-map LL | Entropy-map LL | native-map failure |
|---|---|---|---|
| SegFormer (A1) | |||
| Adv. dev. / All | -0.813 [-3.759, +1.597] | -18.816 [-21.481, -16.916] | -0.197 [-0.421, +0.076] |
| Adv. dev. / Interior | +24.582 [+21.474, +27.545] | +5.363 [+3.087, +7.252] | -1.566 [-1.749, -1.314] |
| Final / All | -0.926 [-3.916, +2.537] | -17.209 [-19.616, -14.402] | -0.126 [-0.412, +0.135] |
| Final / Interior | +23.236 [+21.019, +25.543] | +5.288 [+3.925, +6.676] | -1.377 [-1.682, -1.087] |
| DeepLab (A2) | |||
| Primary all-valid quantity | Minimum | Maximum |
|---|---|---|
| Changed-image fraction (%) | +98.500 | +100.000 |
| Mean exchanged retained fraction (%) | +9.356 | +10.348 |
| Component ID | [nats] | |
| Native-only | ||
| 38 | 0 | 0.414215 |
| 43 | 0 | 0.282678 |
| 44 | 0 | 0.435519 |
| 45 | 0 | 0.628635 |
| 79 | 0 | 0.399822 |
| Population | F1 | F2=p | F3 |
|---|---|---|---|
| SegFormer (A1) | |||
| Src. dev. | -10.929 [-13.887, -7.947] | -19.693 [-23.135, -16.199] | -0.167 [-0.912, +0.568] |
| Adv. dev. | -19.496 [-21.347, -17.682] | -24.761 [-27.233, -22.264] | -0.168 [-0.648, +0.315] |
| Final | -18.026 [-20.176, -15.839] | -20.890 [-23.930, -17.662] | -0.911 [-1.369, -0.416] |
| DeepLab (A2) | |||
| Src. dev. | -6.826 [-8.958, -4.561] | -7.703 [-9.963, -5.242] | +0.194 [-0.341, +0.756] |
| Cell | Gain x1000 nats [90% CI] | Positive single-fold operators |
|---|---|---|
| Seg_dev | -1.340 [-2.706, +0.061] | 0/5 |
| Deep_dev | +1.467 [+0.313, +2.692] | 5/5 |
| Seg_pros | -0.213 [-0.597, +0.196] | 1/5 |
| Deep_pros | -0.245 [-0.715, +0.234] | 1/5 |
| Contrast | Point [interval] |
|---|---|
| Image-weighted / T_Seg | 1.126 [-0.446, 2.898] |
| Image-weighted / T_Deep | -1.712 [-3.248, -0.202] |
| Image-weighted / Cdev | 2.807 [0.616, 5.123] |
| Image-weighted / Clater | -0.031 [-0.771, 0.738] |
| Image-weighted / D | -2.838 [-5.269, -0.500] |
| Equal-city / Seg_dev | -1.340 [-2.996, 0.137] |
| Research line and unit | Established evaluation target | Connection to this study |
|---|---|---|
| Predictive uncertainty; samples or pixels [ 1 , 2 , 3 , 4 , 5 ] | Model and data uncertainty; calibration, error detection and distribution shift. | Frozen segmentation outputs; component losses and per-image retention budgets. |
| Failure prediction; pixels, components or images [ 6 , 7 , 8 , 9 , 10 ] | Correctness or quality forecasts; confidence aggregation to the decision unit. | Component MetaSeg; entropy and probability references; prediction and selection assessed separately. |
| Selective prediction; samples with rejection [ 11 , 12 , 13 , 14 , 15 , 16 , 17 ] | Risk–coverage, conditional-risk selection, joint predictor/rejector learning and shift. | A fixed predictor and matched retained counts within each image. |
| Probability calibration; class forecasts [ 18 , 19 , 20 , 21 , 22 ] | Class-probability calibration, spatial scaling and decision-specific calibration. | Component-failure forecasts; diagnostic CE; retained LL of unchanged base . |
| Evaluation; accepted sets and thresholds [ 23 , 24 , 25 , 26 , 27 ] | Calibration, discrimination and selection measures; rejection; multi-threshold alignment. | Absolute retained levels, paired gains, coverage and eligibility sensitivity. |
| Loss alignment; forecasts and image-level segmentation [ 28 , 15 , 29 ] | Proper probability scores, conditional-risk ordering and Dice-aligned confidence. | Component failure versus forecast loss; native versus LL-risk ranking on fixed . |
Selective Prediction
Selective prediction lets models answer only when sufficiently reliable, trading coverage for lower error through abstention, referral, or extra evidence gathering.
See all 287 papers