Object detectors often produce over-confident predictions for objects outside their training categories, leading to so-called out-of-distribution (OoD) hallucinations. Existing approaches for detecting or mitigating such hallucinations typically either construct scoring functions directly over learned object detector representations or modify the object detector itself to suppress hallucination emergence. However, the latent priors implicitly encoded in these representations remain largely unexplored and have not been explicitly decoded for OoD detection. To uncover and exploit these latent priors, we propose Structured Prior Knowledge (SPK), a hallucination-oriented framework that explicitly elicits OoD-relevant priors from pretrained object detectors. Specifically, SPK leverages in-distribution data and hallucination-inducing samples as diagnostic supervision to elicit part-level semantic concepts underlying object detector decision-making, rather than using them merely for rejection or object detector adaptation. The elicited semantic priors are further integrated with geometric and contextual priors to form a compact five-dimensional SPK representation for OoD detection. Extensive experiments across diverse object detector architectures and multiple OoD benchmarks demonstrate that SPK achieves state-of-the-art OoD detection. Our findings reveal that pretrained object detectors already encode substantially richer latent knowledge than is typically exploited for OoD detection. More importantly, this knowledge can be explicitly elicited and organized into a compact, structured, and interpretable knowledge space for prediction reliability analysis. This suggests a promising proactive route for improving object detector reliability by explicitly uncovering and leveraging latent priors. Code and data are available at: https://gricad-gitlab.univ-grenoble-alpes.fr/dnn-safety/spk
Figures & tables
Figure 1: The proposed SPK framework, a proactive OoD hallucination mitigation framework, further reduces OoD-induced hallucinations beyond previous state-of-the-art methods ( Wu et al. 2026 ) , achieving additional improvements in challenging high-performance regimes.
Figure 2: Overview of the proposed SPK framework. SPK elicits semantic, geometric, and contextual priors from a pretrained object detector and organizes them into a compact five-dimensional representation for OoD hallucination detection.
Method
YOLO
Faster R-CNN
RT-DETR
PASCAL-VOC
BDD-100K
PASCAL-VOC
BDD-100K
PASCAL-VOC
BDD-100K
Near-OoD
Far-OoD
Near-OoD
Far-OoD
Near-OoD
Far-OoD
Near-OoD
Far-OoD
Near-OoD
Far-OoD
Near-OoD
Far-OoD
MSP
67.48
67.18
72.93
74.12
68.36
78.69
77.46
74.41
70.44
67.81
77.46
74.41
EBO
90.49
90.84
87.22
87.06
60.62
56.21
94.83
94.37
98.03
96.69
94.83
94.37
MLS
89.88
90.08
86.47
87.06
59.62
57.89
92.55
91.08
92.84
89.75
92.55
91.08
SCALE
80.67
80.92
77.44
70.59
92.34
80.70
86.35
84.98
81.12
76.92
86.35
84.98
Table 1: Comparison of OoD detection performance using FPR95 across different detector architectures trained on PASCAL-VOC and BDD-100K. Lower is better.
Model
Method
VOC (N/F)
BDD (N/F)
YOLO
Original
946 / 440
701 / 666
Proximal-OoD
134 / 60
80 / 47
SPK
135 / 52
69 / 5
Faster R-CNN
Original
2150 / 1335
2576 / 1634
Proximal-OoD
710 / 253
207 / 167
SPK
299 / 140
60 / 25
Table 2: OoD detection counts (Near-OoD/Far-OoD) across different detector architectures. Lower is better.
Ldice
Lsuppress
Lgroup
VOC
BDD
Average
Near / Far
Near / Far
FPR95 ↓
✗
✓
✓
25.80 / 21.30
17.85 / 18.26
20.80
✓
✓
✗
22.10 / 16.40
15.29 / 15.97
17.44
✓
✗
✓
20.50 / 15.80
14.18 / 12.93
15.85
✓
✓
✓
14.25 / 11.84
9.86 / 0.70
9.16
Table 3: Ablation study of the SPK loss components on YOLO trained on PASCAL-VOC and BDD-100K. Results are reported as Near-OoD / Far-OoD FPR95. The average is computed over all four results. Lower is better.
Prior components
VOC
BDD
Average
Near / Far
Near / Far
FPR95 ↓
Semantic
15.43 / 13.28
30.37 / 25.58
21.17
Semantic + Geometric
13.23 / 11.50
20.88 / 3.84
12.36
Semantic + Geometric + Contextual
14.25 / 11.84
9.86 / 0.70
9.16
Table 4: Ablation study of different prior components on YOLO trained on PASCAL-VOC and BDD-100K. Each dataset column reports Near-OoD / Far-OoD FPR95. The average is computed over all four results. Lower is better.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Automated part-level annotation pipeline. Given an RoI crop and its object category, OWLv2 grounds the corresponding GPT-5-generated concept vocabulary into part bounding boxes. Each box prompts SAM 2 to produce a refined pixel-level part mask. Masks satisfying the object-mask coverage threshold are projected into detector-RoI coordinates and rasterized as binary 7×7 targets. Concepts without a retained mask receive an all-zero target.
Figure 4: Part-Level Concept Annotation Example. We use a bird RoI crop to illustrate the step-by-step annotation process for the concepts wing , head , beak , and torso . Step 1: OWLv2 grounds each concept within the detector RoI and generates a bounding-box proposal (a). Step 2: SAM 2 refines each proposal into a pixel-level part mask, which is retained only if at least 70% of its pixels overlap with the corresponding SAM 2 object mask (b). Step 3: Each retained mask is projected into detector-RoI coordinates and converted into a binary 7×7 supervision target (c). Step 4: The binary target is overlaid on the RoI crop for visualization (d).
Table 5: Quality of the automated part-level annotations. We report the overall concept-mIoU, [email protected], number of covered concepts, and per-class concept-mIoU. All values except concept coverage are percentages. Best results are shown in bold.
Detector
Feature source
Channels
RoI feature Fi
YOLO
Detect neck
128+256+512
896×7×7
RT-DETR
Hybrid-encoder neck
256+256+256
768×7×7
Faster R-CNN
ResNet-FPN
256
256×7×7
Appendix
Table 6: RoI feature sources and dimensions for each detector. YOLO and RT-DETR concatenate RoI-aligned features from three feature scales, whereas Faster R-CNN uses the Detectron2 box_pooler .
Semantic Elicitation Head
Hyperparameter
Value
Architecture
Hidden channels
256
Dropout
0.1
Normalization
GroupNorm
Activation
GELU
Residual blocks
2
Output head
1×1 conv → num_concepts
Appendix
Table 7: Hyperparameters of the Semantic Elicitation Head.
Detector
Feature source
Channels
Embedding dimension
YOLO
Backbone L6 (stride 16 )
256
mean+std:512
RT-DETR
HGBlock L9 backbone (stride 32 )
2048
mean+std:4096
Faster R-CNN
ResNet-FPN P2–P5
4×256
mean+std:2048
Appendix
Table 8: Image-level embedding sources and dimensions for each detector. We globally pool each selected feature map by its spatial mean and standard deviation, then concatenate the resulting statistics. YOLO and RT-DETR use their deepest selected backbone stage, whereas Faster R-CNN aggregates all four ResNet-FPN levels.
Figure 5: Qualitative visualization of part-level semantic responses. The learned concept maps are well aligned with the corresponding regions, demonstrating that the semantic elicitation head successfully decodes spatially grounded semantic evidence from detector RoI features.
Data split
Accuracy (%)
ID training set
96.5
Proximal OoD
81.0
Background
77.0
Appendix
Table 9: Semantic-head group-classification accuracy. Computed by applying argmax to concept activations for YOLO on PASCAL-VOC.
Figure 6: Distributions of the learned semantic group responses. Samples from different data sources predominantly activate their corresponding semantic groups, validating the effectiveness of the proposed group objective.
Figure 7: UMAP visualization of representation spaces for detections from the dog , sheep , and cat categories. The visualizations are obtained using YOLO trained on PASCAL-VOC. We compare detector classification logits (left) with the proposed SPK representations (right), using the same ID-validation, Near-OoD, and Far-OoD samples. SPK yields a more structured representation space with clearer distributional differences between ID and OoD samples.
Method
YOLO
Faster R-CNN
RT-DETR
PASCAL-VOC
BDD-100K
PASCAL-VOC
BDD-100K
PASCAL-VOC
BDD-100K
Near-OoD
Far-OoD
Near-OoD
Far-OoD
Near-OoD
Far-OoD
Near-OoD
Far-OoD
Near-OoD
Far-OoD
Near-OoD
Far-OoD
MSP
81.24
79.47
77.63
75.12
78.71
73.84
72.48
76.39
79.16
78.72
74.31
75.06
EBO
60.73
57.92
65.41
62.86
82.58
86.14
52.07
50.83
40.12
43.31
49.68
52.74
MLS
59.41
61.26
63.78
65.02
84.76
83.58
54.82
59.46
56.37
59.74
54.91
57.43
SCALE
69.84
71.73
72.46
79.31
57.49
69.72
64.18
67.82
69.37
75.16
66.42
65.73
Appendix
Table 10: AUROC comparison across detector architectures on PASCAL-VOC and BDD-100K. Higher is better.
Figure 8: Qualitative SPK OoD verdicts for OoD-induced hallucination. Results are shown for YOLO on PASCAL-VOC. Near-OoD examples (left) exhibit strong proximal-concept responses, whereas Far-OoD examples (right) are dominated by background-concept responses. Each example shows the predicted bounding box, detector confidence, and five SPK prior components. Green bands represent class-specific normal ranges estimated from ID training data, red bands indicate values outside these ranges, and circular markers denote the observed component values. All examples are correctly rejected as OoD.
Ldice
Lsuppress
Lgroup
Faster R-CNN
RT-DETR
PASCAL-VOC
BDD-100K
PASCAL-VOC
BDD-100K
Near-OoD
Far-OoD
Near-OoD
Far-OoD
Near-OoD
Far-OoD
Near-OoD
Far-OoD
✗
✓
✓
27.87
19.98
18.36
18.74
32.31
27.08
25.35
29.14
✓
✓
✗
24.42
15.42
14.15
16.11
27.86
22.53
20.71
24.36
✓
✗
✓
22.19
14.73
11.34
13.27
24.86
21.64
18.15
20.81
✓
✓
✓
13.92
10.52
2.31
1.52
15.48
17.32
11.42
9.25
Appendix
Table 11: Ablation of SPK loss components. Evaluated across Faster R-CNN and RT-DETR on PASCAL-VOC and BDD-100K. Lower FPR95 is better.
Prior components
Faster R-CNN
RT-DETR
PASCAL-VOC
BDD-100K
PASCAL-VOC
BDD-100K
Near-OoD
Far-OoD
Near-OoD
Far-OoD
Near-OoD
Far-OoD
Near-OoD
Far-OoD
Semantic
18.17
18.91
23.92
25.21
18.34
29.17
33.36
34.75
Semantic + Geometric
15.78
16.43
14.52
15.71
15.85
25.35
26.51
26.82
Semantic + Geometric + Contextual
13.92
10.52
2.31
1.52
15.48
17.32
11.42
9.25
Appendix
Table 12: Ablation of SPK prior components. Evaluated across Faster R-CNN and RT-DETR on PASCAL-VOC and BDD-100K. Lower FPR95 is better.
Method
ID: PASCAL-VOC
ID: BDD-100K
OoD: MS-COCO
OoD: OpenImages
OoD: MS-COCO
OoD: OpenImages
FPR95 ↓
AUROC ↑
FPR95 ↓
AUROC ↑
FPR95 ↓
AUROC ↑
FPR95 ↓
AUROC ↑
MDS ( Lee et al. 2018b )
97.39
50.28
97.88
49.08
70.86
76.83
71.43
77.98
Gram matrices ( Sastry and Oore 2020 )
94.16
43.97
95.29
38.81
73.81
60.13
71.56
57.14
KNN ( Sun et al. 2022 )
91.80
62.15
91.36
59.64
64.75
80.90
61.13
79.64
CSI ( Tack et al. 2020 )
84.00
55.07
79.16
51.37
70.27
77.93
71.30
76.42
Appendix
Table 13: Comparison with competitive OoD detection methods for Deformable-DETR. Results are reported on PASCAL-VOC and BDD-100K as ID datasets, with MS-COCO and OpenImages as OoD datasets. Higher AUROC and lower FPR95 indicate better OoD detection performance. Methods marked with † employ an external DINO ViT encoder to extract additional visual representations for OoD detection, rather than relying solely on Deformable-DETR features. SPK (DINO ViT) is included to provide a fair comparison with UNO-Adapter, as both methods operate under this setting. The best results are highlighted in bold.
Method
MS-COCO
OpenImages
AUROC ↑
FPR95 ↓
AUROC ↑
FPR95 ↓
CSI ( Tack et al. 2020 )
82.95
57.41
81.83
59.91
GAN-Synthesis ( Lee et al. 2018a )
82.67
59.97
83.67
60.93
VOS ( Du et al. 2022b )
85.23
51.33
88.70
47.53
SIREN ( Du et al. 2022a )
85.36
64.68
82.78
68.53
TIB ( Wu and Deng 2023 )
90.36
41.55
88.09
47.19
Appendix
Table 14: Comparison with competitive OoD detection methods for Faster R-CNN. Results are reported on PASCAL-VOC as the ID dataset, with MS COCO and OpenImages as OoD datasets. Higher AUROC and lower FPR95 indicate better OoD detection performance. Methods marked with † employ an external DINO ViT encoder to extract additional visual representations for OoD detection, rather than relying solely on Faster R-CNN features. SPK (DINO ViT) is included to provide a fair comparison with UNO-Adapter, as both methods operate under this setting. The best results are highlighted in bold.
Component
Cost (ms)
Original inference
10.15
SPK inference *
12.65
Semantic prior elicitation
0.17
Isolation Forest
0.05
Appendix
Table 15: Per-image runtime of the complete SPK inference pipeline. Reported for a YOLO model pretrained on PASCAL-VOC. The complete SPK pipeline introduces an additional 2.72 ms latency per image, corresponding to a 26.8% runtime overhead over the original detector inference. * SPK inference obtains detection outputs, image-level contextual embeddings, and RoI features within the same forward pass.
Reliable detection of out-of-distribution (OOD) samples is crucial for the safe deployment of machine learning models. Neural networks often produce overconfident predictions for inputs that deviate from their training data, leading to significant degradation in performance. While many OOD detection methods focus on the final output layer, they neglect the rich hierarchical information present in intermediate network layers. This paper introduces a novel approach that leverages sparse autoencoders (SAEs) to learn interpretable features from these intermediate activations. We find that in-distribution (ID) and OOD data activate distinct sets of these sparse features. We propose a new OOD score derived from the cosine similarity between the sparse feature activations of a test sample and the mean activations of ID classes. Our post-hoc detection method not only achieves state-of-the-art performance on standard OOD detection benchmarks, but yields interpretable insights into how distribution shift affects learned representations.
Ayush Karmacharya, Luke Luschwitz, Lucia Romero +2
Object detectors deployed in safety-critical environments can fail silently, e.g. missing pedestrians, workers, or other safety-critical objects without emitting any warning. Traditional Out Of Distribution (OOD) detection methods focus on identifying unfamiliar inputs, but do not directly predict functional failures of the detector itself. We introduce Knowledge Guided Failure Prediction (KGFP), a representation-based monitoring framework that treats missed safety-critical detections as anomalies to be detected at runtime. KGFP measures semantic misalignment between internal object detector features and visual foundation model embeddings using a dual-encoder architecture with an angular distance metric. A key property is that when either the detector is operating outside its competence or the visual foundation model itself encounters novel inputs, the two embeddings diverge, producing a high-angle signal that reliably flags unsafe images. We compare our novel KGFS method to baseline OOD detection methods. On COCO person detection, applying KGFP as a selective-prediction gate raises person recall among accepted images from 64.3% to 84.5% at 5% False Positive Rate (FPR), and maintains strong performance across six COCO-O visual domains, outperforming OOD baselines by large margins. Our code, models, and features are published at https://gitlab.cc-asp.fraunhofer.de/iosb_public/KGFP.
Jakob Paul Zimmermann, Gerrit Holzbach, David Lerch
Out-of-Distribution (OOD) data poses a significant threat to machine learning models, often leading to model failure during deployment. All existing OOD detection methods are post-hoc, relying on evaluation metrics such as accuracy and AUC-ROC during inference to indirectly assess the model's response to OOD data by measuring deviations. In contrast to existing approaches, the proposed work shifts the paradigm from OOD detection to OOD prediction by proposing a pre-hoc anticipatory framework called PLSP for OOD prediction. We make several key contributions: (a) a dataset-independent metric called the CREDibility Score (CREDS) is proposed for OOD prediction; (b) credibility curves are introduced to study the maximum credibility a model can attain; and (c) credibility heat maps (and volume under surface) are introduced to characterize pre-hoc model behavior across different datasets. This work provides a novel perspective on signal processing under distributional shifts. Experiments across multiple datasets demonstrate that the proposed metric serves as a valuable measure for improving the robustness of machine learning models toward OOD prediction.
University of Wisconsin Madison, Madison, WI, USA · IMC University of Applied Sciences, Krems, Austria · Indian Institute of Technology Roorkee, Roorkee, India