Box-conditional Hypothesis Testing for Out-of-Distribution Object Detection
Organizations: School of Games, Hongik University
Abstract
Detecting out-of-distribution (OOD) objects is essential for ensuring the reliability and safety of object detectors in real-world deployment. We treat the decision on a detected box as a hypothesis test with the null hypothesis that the box covers an object of an in-distribution (ID) class. The null distribution changes with the size of the box and its contact with the image border, and the detector observes both. The test can therefore rank a box only against ID boxes of the same size and border contact, which gives a distribution-free bound on the false-alarm rate for each of them and uncovers the separation that one null distribution hides. The power of the test also rests on the layer it reads. Layers built only for the ID classes have no reason to keep what sets an unknown object apart, so the test reads the features before them. We propose PLUM, a training-free score that pools the last backbone stage inside the box and ranks its distance to the ID classes among ID boxes of the same size and border contact. On the standard benchmark, PLUM reaches the highest AUROC among the compared methods without external models, with 91.5 and 94.5 on PASCAL-VOC and 95.1 and 97.5 on BDD100K. Code: https://github.com/DDOZMI/PLUM
Figures & tables
| ID: PASCAL-VOC | ID: BDD100K | ||||||||
| MS-COCO | OpenImages | MS-COCO | OpenImages | ||||||
| Method | Retrain? | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC |
| MSP [ 16 ] | 72.04 | 83.20 | 74.57 | 81.44 | 80.77 | 75.85 | 79.04 | 77.10 | |
| ODIN [ 25 ] | 57.73 | 83.25 | 59.47 | 82.64 | 62.80 | 74.43 | 59.45 | 76.20 | |
| Energy [ 29 ] | 57.40 | 83.55 | 59.70 | 82.64 | 60.03 | 77.47 | 55.95 | 79.23 | |
| Mahalanobis [ 23 ] | 96.88 | 57.98 | 97.56 | 55.40 | 56.97 | 88.09 | 59.45 | 87.06 | |
| PASCAL-VOC | BDD100K | |||||
|---|---|---|---|---|---|---|
| Null distribution | AUROC | FPR95 | False alarms | AUROC | FPR95 | False alarms |
| One for all boxes | 74.2 | 80.2 | 0.05–20.1 | 94.4 | 38.8 | 0.0–36.6 |
| Size level | 91.8 | 40.7 | 2.5–6.3 | 96.2 | 19.3 | 3.4–36.2 |
| Size level border | 91.8 | 41.1 | 0.8–7.4 | 95.9 | 22.0 | 2.6–5.3 |
| PLUM, with line | 93.0 | 36.7 | 0.8–7.5 | 96.3 | 19.1 | 2.6–5.8 |
| PASCAL-VOC | BDD100K | |||||||
| AUROC | FPR95 | AUROC | FPR95 | |||||
| Layer | Raw | Leveled | Raw | Leveled | Raw | Leveled | Raw | Leveled |
| Faster R-CNN | ||||||||
| res2 | 72.5 | 86.8 | 82.6 | 49.3 | 53.5 | 89.6 | 92.5 | 51.6 |
| res3 | 66.9 | 88.4 | 93.6 | 47.3 | 51.8 | 92.3 | 93.7 | 38.9 |
| res4 | 63.5 | 89.7 | 91.8 | 45.7 | 78.2 | 94.9 | 74.9 | 30.0 |
| Significant | ||||||
|---|---|---|---|---|---|---|
| Detector | ID | Strata | One null | PLUM | One null | PLUM |
| Faster R-CNN | PASCAL-VOC | 8 | 27.0 | 3.0 | 3 | 1 |
| BDD100K | 8 | 49.1 | 5.2 | 3 | 0 | |
| Deformable DETR | PASCAL-VOC | 8 | 21.0 | 0.1 | 2 | 0 |
| BDD100K | 7 | 22.7 | 2.1 | 2 | 1 | |
| PASCAL-VOC | BDD100K | |||||||
|---|---|---|---|---|---|---|---|---|
| MS-COCO | OpenImages | MS-COCO | OpenImages | |||||
| Level | Inner | Border | Inner | Border | Inner | Border | Inner | Border |
| 2 | 84.8 | – | 92.8 | – | 95.9 | 90.5 | 97.9 | 96.1 |
| 3 | 90.8 | – | 94.5 | – | 96.8 | 94.8 | 97.4 | – |
| 4 | 94.2 | 88.6 | 95.4 | 90.6 | 96.8 | 96.8 | 98.0 | 98.0 |
| 5 | 92.9 | 92.7 | 95.1 | 94.6 | 93.8 | 91.0 | 97.9 | 96.9 |
| Item | Setting |
|---|---|
| GPU | NVIDIA GeForce RTX 4090 |
| Python | 3.12 |
| PyTorch | 2.14 with CUDA 13.2 |
| detectron2 | 0.6, built from source |
| Faster R-CNN | ResNet-50 FPN, VOS checkpoints |
| Deformable DETR | ResNet-50, SIREN checkpoints |
| ID: PASCAL-VOC | ID: BDD100K | ||||||||
| MS-COCO | OpenImages | MS-COCO | OpenImages | ||||||
| Method | Retrain? | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC |
| Mahalanobis [ 23 ] | 97.36 | 50.44 | 97.84 | 49.12 | 70.97 | 79.89 | 67.48 | 83.42 | |
| KNN [ 42 ] | 94.66 | 59.51 | 94.53 | 57.05 | 54.08 | 87.61 | 46.90 | 91.58 | |
| Gram [ 40 ] ‡ | 94.16 | 43.97 | 95.29 | 38.81 | 73.81 | 60.13 | 71.56 | 57.14 | |
| SAFE [ 49 ] † | 50.38 | 77.25 | 7.32 | 97.48 | 39.35 | 85.55 | 21.81 | 94.30 | |
| PASCAL-VOC | BDD100K | |||||
|---|---|---|---|---|---|---|
| Null distribution | AUROC | FPR95 | False alarms | AUROC | FPR95 | False alarms |
| One for all boxes | 80.5 | 75.4 | 0.2–15.3 | 95.8 | 24.6 | 2.0–19.7 |
| Size level | 92.4 | 38.0 | 3.4–6.6 | 95.9 | 20.5 | 3.9–19.6 |
| Size level border | 92.5 | 38.0 | 3.7–4.8 | 94.8 | 25.5 | 3.4–7.9 |
| PLUM, with line | 92.9 | 34.4 | 3.5–4.8 | 94.4 | 28.3 | 2.6–6.3 |
| ID: PASCAL-VOC | ID: BDD100K | |||||||||||||||
| MS-COCO | OpenImages | MS-COCO | OpenImages | |||||||||||||
| AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | |||||||||
| Layer | Raw | Leveled | Raw | Leveled | Raw | Leveled | Raw | Leveled | Raw | Leveled | Raw | Leveled | Raw | Leveled | Raw | Leveled |
| Faster R-CNN | ||||||||||||||||
| res2 | 66.3 | 80.4 | 88.9 | 65.2 | 78.7 | 93.3 | 76.3 | 33.3 | 52.9 | 87.4 | 92.4 | 57.2 | 54.1 | 91.8 | 92.6 | 46.0 |
| res3 | 61.8 | 82.8 | 93.8 | 65.0 | 72.0 | 94.0 | 93.5 | 29.6 | 50.4 | 90.3 | 93.7 | 47.8 | 53.3 | 94.4 | 93.7 | 30.0 |
| ID: PASCAL-VOC | ID: BDD100K | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Level | Contact | ID | Reference | 1% | 5% | 10% | ID | Reference | 1% | 5% | 10% |
| 2 | Inner | 3,210 | 2,585 | 1.09 | 5.58 | 11.59 | 180,169 | 44,333 | 0.95 | 5.08 | 10.11 |
| 2 | Border | 241 | 217 | 0.41 | 0.83 | 2.90 | 4,031 | 1,065 | 0.55 | 4.07 | 9.15 |
| 3 | Inner | 3,299 | 2,424 | 0.61 | 5.24 | 11.28 | 12,972 | 3,414 | 0.78 | 4.39 | 9.71 |
| 3 | Border | 544 | 362 | 1.47 | 7.54 | 10.11 | 2,039 | 561 | 1.28 | 4.32 | 11.23 |
| 4 | Inner | 3,581 | 2,813 | 0.92 | 4.41 | 8.91 | 4,511 | 1,136 | 1.35 | 4.66 | 10.71 |
| Detector | ID | Strata | Below | Above | Below | Above | Below | Above |
|---|---|---|---|---|---|---|---|---|
| Faster R-CNN | PASCAL-VOC | 8 | 2 | 0 | 1 | 0 | 1 | 0 |
| BDD100K | 8 | 1 | 0 | 1 | 0 | 1 | 0 | |
| Deformable DETR | PASCAL-VOC | 8 | 5 | 0 | 4 | 0 | 5 | 0 |
| BDD100K | 7 | 0 | 0 | 1 | 0 | 1 | 0 | |
| Detector | ID | One null | Size level | Size level border | PLUM | Resampled |
|---|---|---|---|---|---|---|
| Faster R-CNN | PASCAL-VOC | 27.0 | 1.8 | 3.4 | 3.0 | 5.2 |
| BDD100K | 49.1 | 50.7 | 5.5 | 5.2 | 8.5 | |
| Deformable DETR | PASCAL-VOC | 21.0 | 3.2 | 0.0 | 0.1 | 1.4 |
| BDD100K | 22.7 | 22.7 | 8.2 | 2.1 | 4.4 |
| Image | Geometry | Placement | |||||||||
| VOC | BDD | VOC | VOC | BDD | VOC | BDD | |||||
| Method | COCO | OI | COCO | OI | COCO | COCO | OI | COCO | OI | ||
| MSP | 2.2 | 0.3 | 6.4 | 5.1 | 2.1 | 12.9 | 46.0 | 1.1 | 1.2 | 0.6 | 0.7 |
| ODIN | 5.7 | 2.8 | 10.7 | 11.3 | 0.9 | 13.0 | 60.7 | 1.2 | 1.1 | 1.4 | 1.6 |
| Energy | 5.2 | 0.1 | 13.7 | 15.4 | 1.2 | 12.6 | 52.9 | 1.6 | 1.3 | 0.3 | 0.7 |
OOD Detection
Out-of-distribution (OOD) detection identifies inputs unlike a model’s training data so systems can flag uncertainty, reject inputs, or adapt safely.
See all 239 papers