Calibrated Alarm Thresholds for Compact Fire and Smoke Detectors
Organizations: California, USA
Abstract
Compact object detectors such as YOLOv8n are a common choice for camera-based fire and smoke detection on low-power devices. These detectors are usually evaluated with box-level metrics such as mean average precision, and their outputs are turned into decisions with a fixed confidence threshold such as 0.25 or 0.5. A fire monitoring system, however, makes one decision per image: raise an alarm or not. In this paper we evaluate two publicly available compact detectors on the D-Fire test set at the image level. We compare three simple rules for turning box detections into one alarm score per image, and we use a split conformal procedure to choose the alarm threshold from held-out images so that the expected fraction of missed fire and smoke images stays below a chosen target. We find that the same fixed threshold leads to very different miss rates for the two detectors, 17.3 and 8.4 percent at a threshold of 0.5, while the calibrated threshold meets miss-rate targets of 10, 5 and 1 percent for both detectors and all three rules. At a 5 percent target, the false alarm rate on images without fire or smoke stays between 1.2 and 3.0 percent. We also find that smoke-only images stay below the target even when the overall target is met, and we describe the main failure cases.
Figures & tables
| Metric | YOLOv8n | YOLOv5s |
|---|---|---|
| Precision | 0.685 | 0.678 |
| Recall | 0.724 | 0.781 |
| mAP@50 | 0.746 | 0.771 |
| mAP@50–95 | 0.424 | 0.429 |
| AP@50, fire | 0.684 | 0.720 |
| AP@50, smoke | 0.807 | 0.822 |
| Model | Rule | AUC | ||||
|---|---|---|---|---|---|---|
| DR | FAR | DR | FAR | |||
| YOLOv8n | Max | 0.993 | 0.957 | 0.024 | 0.827 | 0.007 |
| Noisy-OR | 0.994 | 0.972 | 0.032 | 0.924 | 0.011 | |
| Area-gated | 0.990 | 0.942 | 0.023 | 0.815 | 0.006 | |
| YOLOv5s | Max | 0.995 | 0.977 | 0.029 | 0.916 | 0.009 |
| Noisy-OR | 0.996 | 0.987 | 0.038 | 0.958 | 0.014 | |
| Model | Rule | ||||||
|---|---|---|---|---|---|---|---|
| Miss | FAR | Miss | FAR | Miss | FAR | ||
| YOLOv8n | Max | 0.098 | 0.013 | 0.048 | 0.023 | 0.009 | 0.092 |
| Noisy-OR | 0.099 | 0.010 | 0.048 | 0.019 | 0.009 | 0.084 | |
| Area-gated | 0.099 | 0.015 | 0.049 | 0.030 | 0.009 | 0.191 | |
| YOLOv5s | Max | 0.099 | 0.008 | 0.049 | 0.014 | 0.009 | 0.069 |
| Noisy-OR | 0.099 | 0.005 | 0.049 | 0.012 | 0.009 | 0.067 | |
Object Detection
Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.
See all 673 papers