Self-Aware Object Detection via Degradation Manifolds

Stefan Becker; Simon Weiss; Wolfgang Hübner; Michael Arens

cs.CVFeb 20, 2026

Self-Aware Object Detection via Degradation Manifolds

Authors: Stefan Becker, Simon Weiss, Wolfgang Hübner, Michael Arens

Organizations: Fraunhofer Institute of Optronics, System Technologies and Image Exploitation IOSB Gutleuthausstraße 1, 76275 Ettlingen

Abstract

Object detectors achieve strong performance under nominal imaging conditions but can fail silently when exposed to blur, noise, compression, adverse weather, or resolution changes. In safety-critical settings, it is therefore insufficient to produce predictions without assessing whether the input remains within the detector's nominal operating regime. We refer to this capability as self-aware object detection. We introduce a degradation-aware self-awareness framework based on degradation manifolds, which explicitly structure a detector's feature space according to image degradation rather than semantic content. Our method augments a standard detection backbone with a lightweight embedding head trained via multi-layer contrastive learning. Images sharing the same degradation composition are pulled together, while differing degradation configurations are pushed apart, yielding a geometrically organized representation that captures degradation type and severity without requiring degradation labels or explicit density modeling. To anchor the learned geometry, we estimate a pristine prototype from clean training embeddings, defining a nominal operating point in representation space. Self-awareness emerges as geometric deviation from this reference, providing an intrinsic, image-level signal of degradation-induced shift that is independent of detection confidence. Extensive experiments on synthetic corruption benchmarks, cross-dataset zero-shot transfer, and natural weather-induced distribution shifts demonstrate strong pristine-degraded separability, consistent behavior across multiple detector architectures, and robust generalization under semantic shift. These results suggest that degradation-aware representation geometry provides a practical and detector-agnostic foundation.

Figures & tables

Panel of Figure 1 — Figure 1: Proposed degradation-manifold framework for self-awareness. Multi-layer backbone features are fused via $1{\times}1$ projections and attention pooling and mapped into a normalized embedding space trained with contrastive degradation compositions. A pristine prototype, computed from clean training images, anchors the manifold to nominal operating conditions. At inference, cosine distance to this prototype yields the image-level degradation score $S_{\mathrm{deg}}(\mathbf{x})$ .

Panel of Figure 2 — Figure 2: Detection performance ([email protected] and [email protected]) of different detection models under increasing corruption severity on the COCO val2017 dataset. Here the results of YOLOv9-m Wang et al. (2024b) , YOLOv10-m Wang et al. (2024a) , YOLOv11-m Jocher and Qiu (2024) , and the transformer-based RT-DETR-l Zhao et al. (2024) are shown.

Panel of Figure 3 — Figure 3: Relative detection performance drop of YOLOv10-m Wang et al. (2024a) on COCO val2017 under increasing native severity levels ([email protected]:0.95). Each cell shows the percentage drop relative to the clean val2017 baseline (pristine COCO images, no added degradations). Brighter cells indicate larger drops. Left: degradations from Michaelis et al. (2019) . Right: degradations from Agnolucci et al. (2024) .

Panel of Figure 4 — Figure 4: t-SNE visualization Hinton and Roweis (2003) of the learned degradation manifold under two corruption taxonomies. ( Top ) Degradations from Michaelis et al. (2019) based on Hendrycks and Dietterich (2019) . ( Bottom ) Degradations from Agnolucci et al. (2024) . Each color denotes a specific degradation type, while marker styles indicate the corresponding degradation group. For visualization, 100 COCO Lin et al. (2014) samples are corrupted at severity level 5. Distinct clusters emerge for different degradation types. For training, only the degradations from Agnolucci et al. (2024) were used.

Figure 5: t-SNE visualization Hinton and Roweis (2003) of the learned degradation manifold. Samples are drawn from multiple datasets (COCO Lin et al. (2014) , BDD Yu et al. (2020) , KITTI Geiger et al. (2012) , DETRAC Wen et al. (2020) , UAVDT Du et al. (2018) , and FLIR (VIS) FLIR (2022) (V2) ). Degradations are clearly separated in embedding space, while pristine images form a compact cluster across datasets, indicating content-independence of the learned representation. Notably, BDD and KITTI images were not used during training, demonstrating cross-dataset generalization of the degradation manifold and that the embedding organizes images according to degradation characteristics rather than semantic content.

	Severity levels AUROC $\uparrow$
Model	1	2	3	4	5
Image Quality Assessment (IQA)
MANIQA Yang et al. (2022) (score-based)	52.88	54.49	55.27	56.67	61.56
MANIQA Yang et al. (2022) (embedding-based; prototype anchored)	65.87	71.55	73.42	76.97	77.30
QualiCLIP Agnolucci et al. (2025) (score-based)	53.17	56.77	66.28	70.60	72.53
QualiCLIP Agnolucci et al. (2025) (embedding-based; prototype anchored)	45.36	41.53	46.77	49.79	56.16

Table 1: AUROC scores of detecting degradation-induced covariate shift for synthetically corrupted variants of the COCO dataset. Best results are highlighted as first , second , and third .

Degradation Manifold (Ours)
	Severity levels AUROC $\uparrow$
Dataset	1	2	3	4	5
VISDRONE Zhu et al. (2021)	83.66	89.15	92.08	95.85	96.79
KITTI Geiger et al. (2012)	94.19	96.37	97.16	97.25	98.52
DETRAC Wen et al. (2020)	94.65	97.05	98.87	99.69	99.81
FLIR (VIS) FLIR (2022) (V2)	82.83	91.39	92.72	95.11	96.66

Table 2: AUROC scores of detecting OoD covariate shift using our degradation manifold across different datasets.

Degradation Manifold (Ours)
	Severity levels AUROC $\uparrow$
Datasets	1	2	3	4	5
COCO Lin et al. (2014) & VISDRONE Zhu et al. (2021)	83.54	89.91	90.03	95.06	95.95
COCO Lin et al. (2014) & KITTI Geiger et al. (2012)	89.27	92.44	93.86	96.40	97.21
COCO Lin et al. (2014) & DETRAC Wen et al. (2020)	92.26	93.29	94.92	98.62	98.97
COCO Lin et al. (2014) & FLIR (VIS) FLIR (2022) (V2)	82.45	88.92	89.32	94.32	95.97

Table 3: AUROC for pristine–degraded separability under mixed-dataset evaluation. Pristine samples are drawn from combined datasets, while degraded samples originate from a degraded version of the datasets. COCO is subsampled for class-balanced comparison. Results demonstrate stable degradation-aware separability under simultaneous content and degradation variation.

Panel of Table 4 — Table 4: Natural weather-induced distribution shifts on BDD Yu et al. (2020) and STF Bijelic et al. (2020) . AUROC measures separability between nominal-weather (pristine) (ID) and adverse-weather (OoD) subsets. Training with synthetic weather corruptions improves transfer to real-world heavy weather conditions.

Degradation Manifold (Ours)
	Severity levels AUROC $\uparrow$
Variant	1	2	3	4	5
Full model (multi-layer + attention + hard neg.)	88.64	89.70	89.75	95.28	97.14
Last backbone layer only	81.87	85.13	87.56	89.13	92.75
Global average pooling (no attention)	85.51	88.44	89.49	95.01	96.30
Without hard negative mining	86.00	87.17	87.58	90.41	91.54

Table 5: Ablation study of the proposed degradation manifold using a YOLOv10-m backbone. We analyze the impact of multi-layer readout, attention-based pooling, and hard negative mining on AUROC across severity levels. The full configuration consistently achieves the strongest separability.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Examples from Seeing Through Fog (STF) Bijelic et al. (2020) used for the natural weather-shift evaluation. We construct an ID split from clear/nominal scenes and OoD splits from adverse-weather conditions ( Dense Fog , Heavy Snow , and Heavy Rain after filtering).

Panel of Figure 7 — Figure 7: Examples from BDD100K Yu et al. (2020) illustrating the weather-based ID/OoD construction. Left: ID example from the clear subset. Middle: ambiguous rainy example with weak cues (e.g., wet road only), which is excluded by manual refinement. Right: retained OoD example with strong visible rain cues ( heavy rain ).

Panel of Figure 8 — Figure 8: t-SNE visualization Hinton and Roweis (2003) of the learned degradation manifold under the corruption taxonomy of Michaelis et al. (2019) , based on Hendrycks and Dietterich (2019) , shown for three severity levels (1, 3, and 5). Each color denotes a specific degradation type, while marker styles indicate the corresponding degradation group. For visualization, 100 COCO Lin et al. (2014) samples are corrupted at the respective severity level. At low severity (level 1), clusters begin to form but remain partially overlapping. With increasing severity (levels 3 and 5), degradation types become progressively more separated, indicating that the embedding geometry preserves a meaningful ordering with respect to degradation strength. Note that training used degradation compositions from Agnolucci et al. (2024) , demonstrating zero-shot generalization of the learned manifold.

Panel of Figure 9 — Figure 9: t-SNE visualization Hinton and Roweis (2003) of the learned degradation manifold under the corruption taxonomy of Agnolucci et al. (2024) , shown for three severity levels (1, 3, and 5). Each color denotes a specific degradation type, while marker styles indicate the corresponding degradation group. For visualization, 100 COCO Lin et al. (2014) samples are corrupted at the respective severity level. At low severity (level 1), clusters begin to form but remain partially overlapping. With increasing severity (levels 3 and 5), degradation types become progressively more separated, indicating that the embedding geometry preserves a meaningful ordering with respect to degradation strength.

			Severity levels AUROC $\uparrow$
	Detector	Model	1	2	3	4	5
Top- $3$ Detection Uncertainties	FasterRCNN DMM	Trace	51.22	53.49	54.84	57.75	62.91
		Determinant	48.63	47.15	47.18	47.42	51.31
		Entropy (Regression)	50.71	52.55	53.85	57.77	62.40
		Confidence Score	53.51	59.04	60.50	67.91	71.23
		Entropy (Classification)	55.82	59.30	63.76	70.17	74.93

Appendix

Table 6: AUROC scores for Top-

3

detection uncertainties (as proposed from Oksuz et al. (2023) ) across using the probabilistic detector of Harakeh and Waslander (2021) on synthetically corrupted versions of the COCO dataset.

	Severity Levels AUROC $\uparrow$
Degradation Function Agnolucci et al. (2024)	1	2	3	4	5
’Gaussian Blur’	48.15	61.56	85.20	99.84	99.89
’Lens Blur’	98.02	98.45	98.90	99.20	99.50
’Motion Blur’	48.93	93.43	99.62	99.92	99.95
’White Noise’	98.38	99.28	99.60	99.75	99.85
’White Noise in Color Component’	85.23	98.72	99.36	99.70	99.85

Appendix

Table 7: AUROC for pristine–degraded separability under individual synthetic degradations on COCO. Degradations follow the IQA-style definitions of Agnolucci et al. (2024) .

	Severity Levels AUROC $\uparrow$
Degradation Function Hendrycks and Dietterich (2019)	1	2	3	4	5
’Gaussian Noise’	99.91	99.94	99.93	99.94	99.92
’Shot Noise’	99.63	99.74	99.83	99.81	99.85
’Impulse Noise’	99.96	99.94	99.92	99.91	99.90
’Speckle Noise’	99.70	99.72	99.73	99.72	99.71
’Defocus Blur’	99.86	99.87	99.88	99.87	99.86

Appendix

Table 8: AUROC for pristine–degraded separability under individual synthetic degradations on COCO. Degradations follow the robustness benchmarks of Hendrycks and Dietterich (2019) and Michaelis et al. (2019) .

Panel of Figure 10 — Figure 10: Distribution of $S_{\mathrm{deg}}(\mathbf{x})$ scores for ID and OoD samples at different corruption severity levels, using degradation manifold approach with a YOLOv10-m backbone.

Panel of Figure 11 — Figure 11: Detection performance of YOLOv10-m Wang et al. (2024a) under different training objectives on the COCO-mini person subset Samet et al. (2020) . Detection-only training preserves baseline accuracy. Degradation-only training significantly disrupts semantic detection features. Joint multi-task optimization mitigates this effect while introducing a moderate performance trade-off.

Panel of Figure 12 — Figure 12: Overview of the applied image degradation model. Randomly assembled degradation compositions $C$ are applied sequentially to synthesize diverse degradation patterns. Each degradation composition contains up to $N_{\text{deg}}$ degradations sampled from pre-defined degradation groups (e.g., the degradation groups of Agnolucci et al. (2024) ).

Panel of Figure 13 — Figure 13: Training pair construction for degradation-manifold learning. Two pristine images are degraded using the same sampled composition, forming a positive pair in embedding space ( $z^{1},z^{2}$ ). To construct hard negatives, we apply an additional half-size crop before resizing to the detector input resolution. All degraded views are resized to the detector’s default input size prior to encoding. The objective pulls embeddings of equally degraded full-resolution images together while pushing them away from their resolution-perturbed counterparts, encouraging sensitivity to fidelity changes beyond semantic similarity.

Object Detection

Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.

See all 624 papers

Task-Sensitive Geometry of Representation Transfer for Object Detection under Image Degradation
Oct 3, 2026Van Vung PhamObject Detection Feature-Based Knowledge Distillation
Computer Science Department, Sam Houston State University, Huntsville, Texas, USA
Degradation-Consistent Paired Training for Robust AI-Generated Image Detection
Apr 11, 2026Zongyou Yang, Yinghan Hou, Xiaokun YangAI-Generated Image Detection Image Corruption Robustness
Department of Earth Science and Engineering, Imperial College London, London, United Kingdom · School of Electronic Information, Nanchang Institute of Technology, Nanchang, China
FR-DETR: Frequency and Recurrent Feature Refinement for Robust Object Detection under Adverse Weather
Jun 29, 2026Tuan-Duc Nguyen, Duc-Trong LeFrequency-Domain Feature Learning Detection Transformer
FPT Software AI Center · VNU University of Engineering and Technology

Oct 3, 2026J/K move · Enter open · S save

Task-Sensitive Geometry of Representation Transfer for Object Detection under Image Degradation

Van Vung Pham

Computer Science Department, Sam Houston State University, Huntsville, Texas, USA

Object Detection Feature-Based Knowledge Distillation

Open paper AskPDF arXiv

Object detection under image degradation can benefit from clean-image supervision, but aggregate gains do not imply that transferred representation changes are uniformly useful. We study how clean task knowledge affects degraded-image representations and whether local responses to structured representation directions can be characterized geometrically. Using paired clean and Gaussian-degraded BDD100K images, we show that clean-teacher distillation improves observed detection accuracy while producing heterogeneous object-level transfer. We isolate a representation component complementary to direct clean-teacher alignment and map it into the distilled student space through an orthogonal bridge. Controlled interventions rescue 13.22% of objects lost under the distilled representation, versus 4.30% under norm-matched random perturbations, with very low harm on preserved objects. We introduce task-sensitive geometry, a gradient-derived channel-space geometry constructed from normalized detection-loss gradients. On a reserved cohort, mapped-complement orientation within this frozen geometry is positively associated with local intervention-response magnitude after controlling for intervention magnitude (partial Spearman

ρ

= 0.242, 95% CI [0.108, 0.359]). The relationship eplicates on independent data (

ρ

= 0.180) and with RT-DETR-L (

ρ

= 0.227), but not for the direct clean-teacher residual family, and it weakens for large interventions. Routing rules and specialized distillation objectives based on these signals do not yield statistically reliable gains over CLEANKD. These results support a local, direction-family-dependent task-sensitive geometry while showing that converting such structure into improved global training remains an open problem.

Abstract

Object Detection

Explore similar work

Task-Sensitive Geometry of Representation Transfer for Object Detection under Image Degradation

Degradation-Consistent Paired Training for Robust AI-Generated Image Detection

FR-DETR: Frequency and Recurrent Feature Refinement for Robust Object Detection under Adverse Weather