Not All Confusion Is Equal: A Source-Aware Uncertainty Diagnosis for Fine-Grained Aircraft Detection

Hai Huang; Helmut Mayer

cs.CVSep 24, 2026

Not All Confusion Is Equal: A Source-Aware Uncertainty Diagnosis for Fine-Grained Aircraft Detection

Authors: Hai Huang, Helmut Mayer

Organizations: Chair of Visual Computing, Institute for Applied Computer Science, Universität der Bundeswehr München, Germany

Abstract

Fine-grained object detectors are commonly evaluated with confusion matrices, which show where the model is confused but not why, nor whether the confusion can be reduced. We argue that confusion can be attributed to distinct, separable sources, each quantitatively measurable, turning a passive measurement into actionable guidance. We present $A^2E^2$ , a diagnostic tool that decomposes the sources of confusion along two axes, $\{$ aleatoric, epistemic $\} \times \{$ within-class, between-class $\}$ , giving a $2\times2$ taxonomy that enumerates the source types. Each quadrant is measured by its own quantity, computed in one of three places (input geometry, output-space disagreement, and the bias-parameter posterior), so the two epistemic sources are separated by construction rather than by an empirical correlation. On fine-grained aircraft detection, the four quadrants become four named sources with their own remedy verdict: affinity (geometric similarity, irreducible from size alone), heterogeneity (geometrically heterogeneous sub-variants, pointing to re-labeling rather than more data), contested (an insufficiently trained but learnable boundary, improvable), and collapsed (a class starved of data, reducible). After attributing the confusion to a specific reducible source, we apply a targeted intervention and verify experimentally that it reduces the diagnosed source specifically while leaving the irreducible sources unchanged. $A^2E^2$ thus turns confusion measurement into a concrete, validatable and actionable "diagnosis" in which the same off-diagonal mass can carry opposite causes and opposite remedies. We also state this framework's limits, including which sources are only partially identifiable on this specific dataset and why.

Figures & tables

Figure 1: Overview of $A^{2}E^{2}$ : the four confusion sources as matrices over the ten classes. Within-class sources occupy the diagonal only, between-class sources the (symmetric) upper triangle. Affinity (between-ale) is the OBB-size Bhattacharyya overlap BC ( Eq. 2 ); contested (between-epi) is the per-pair mutual information MI ( Eq. 4 ); collapsed (within-epi) is the bias-posterior variance $\sigma^{2}_{b}$ ( Eq. 5 ); heterogeneity (within-ale) is the size dispersion (CV) over a noise floor, split per diagonal cell into body/wing (collapsed classes shown but confounded by scarcity, cf. Fig. 3 ). Aleatoric sources in orange, epistemic in purple. Red dots mark MI pairs involving ARJ21 or Boeing777 (zero top-1 predictions at baseline), computed on a restricted population ( Sec. 3.3 ) and not read as contested.

Case	metric	value	Attributed source	Remedy verdict
ARJ21 (class)	$\sigma^{2}_{b}$	rank 1/10	collapsed (within-epi)	add class data (verified, T1)
A220–A350 (pair)	MI	$0.387$	contested (between-epi)	add boundary data (supported, directional, T2)
A330–Boeing787 (pair)	BC	$0.945$	affinity (between-ale)	irreducible from size; more data will not help
A330 (class)	excess-CV	$+0.011$ body	heterogeneity (within-ale)	more data will not help; re-label (initial test null)

Table 1: Four study cases with remedy verdict covering the four quadrants

Metric	Predicted	Observed	Verdict
val AP	$\uparrow$	$0.000\rightarrow 0.333$	$\checkmark$ strong
test AP	$\uparrow$	$0.000\rightarrow 0.139$	$\checkmark$ (test harder than val)
$\sigma^{2}_{b}$	$\downarrow$	$-12.5\%$	$\checkmark$ targeted narrowing
retention (test)	$\uparrow$	$0.000\rightarrow 0.072$	$\checkmark$ targeted revival
$C_{\textrm{ale}}$ ref.	no change	$\approx-0.04$ (uniform)	no targeted response
BC	no change	unchanged	geometry is data-independent

Table 2: Verification of T1 (ARJ21 collapsed-epistemic intervention): pre-registered predictions vs. observed outcomes. All four predictions on ARJ21 are confirmed; the aleatoric-reference rows show no matching targeted response. The untreated collapsed control (Boeing777) is discussed in the text. See Sec. 4.5 .

Metric	Predicted	Observed	Verdict
A220–A350 MI	$\downarrow$	$0.387\rightarrow 0.183$ ( $-53\%$ )	$\checkmark$ strong
A350 val AP	$\uparrow$	$0.698\rightarrow 0.788$ ( $+0.090$ )	$\checkmark$
A350 test AP	$\uparrow$	$0.582\rightarrow 0.619$ ( $+0.037$ )	$\checkmark$
BC (overlap)	unchanged	replicated from baseline	$\checkmark$ by construction
A220 val AP	— (not predicted)	$+0.027$	neutral-to-positive
A220 test AP	— (not predicted)	$-0.041$	opposite sign from val

Table 3: Verification of T2 (A220–A350 contested-epistemic intervention): pre-registered predictions vs. observed outcomes. The main effect (MI drop, A350 improvement, BC unchanged) is confirmed on both splits; A220’s val/test divergence was not predicted and is a caveat discussed in the text.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Per-class physical size geometry (body length vs. wingspan, meters, GSD-corrected) on the development set, with Gaussian $1\sigma/2\sigma$ contours. Narrow- and wide-body aircraft are well separated (BC $\approx 0$ ).

Measure (source)	ARJ21	Reading
$\sigma^{2}_{b}$ (collapsed, Laplace bias posterior)	$6.48\times 10^{-3}$	rank 1 of 10
retention ratio (collapsed, raw counts)	$0.000$	tied-lowest
MI (“other”–ARJ21) (contested, output space)	$0.084$	low-to-moderate

Appendix

Table 4: Single-class cross-validation on ARJ21, the rarest class. Two measures computed along different paths (Laplace posterior and prediction counts) both read collapsed-extreme, while the output-space contested measure reads low; see Appendix 0.C.3 .

Figure 3: Heterogeneity metric: class-conditional size CV against the noise floor (dashed, mean CV of the reference classes A321/A350, $\approx 0.062$ ; A350 is drawn as multi-variant by catalogue, but its instances here are predominantly one variant, Sec. 3.3 ). Left : body length; the multi-variant classes B737/A330/A220/B787 exceed the floor, consistent with their sub-variants; single-variant A321 (gray) sits at it. Right : wingspan; the same classes sit near the floor, since their variants share a wingspan. Collapsed classes (orange, ARJ21/B777) are excluded from anchoring: their dispersion is confounded by scarcity and, for B747/B777 wingspan, by OBB artifacts. See Sec. 5.1 .

Figure 4: Boeing777 body length shows a genuine bimodal structure whose peak spacing ( $\approx 11$ m) matches the 777-200/-300 fuselage difference ( $10.2$ m). However, Boeing777 is a collapsed class, so this within-class signal is confounded with data scarcity and we do not use it as a heterogeneity anchor; it is shown here as a rejected/contaminated case.

Explore similar work

Jun 15, 2026cs.CV

Hierarchical Fine-Grained Aerial Object Detection

Fine-grained aerial object detection, driven by the intrinsic granularity of real-world object categories, is crucial for advanced scene understanding in remote sensing. Existing methods largely inherit the paradigm of coarse-grained object detection, relying solely on single-label supervision and thus struggling to distinguish model-level categories with subtle structural differences. However, for each specific model (e.g., Boeing 787), structured prior knowledge such as attributes and hierarchies offers discriminative semantics across multiple granularities. Motivated by this, we present ExpertDet, a scheme that incorporates expert-informed cues to enhance fine-grained aerial object detection. Specifically, we design Vision-aware Masked Attribute Modeling (VMAM), which aligns attribute semantics with visual structures by reconstructing randomly masked attributes from visual cues, enabling the detector to capture subtle structural distinctions. We further propose Hierarchical Visual Instance Promotion (HierVIP), which builds a visual prototype tree based on hierarchical relations and imposes taxonomy-aware constraints to preserve cross-level semantic continuity while enhancing category discrimination. Moreover, we curate a new fine-grained object detection benchmark for Precise recognition of model-specific Ships and Planes from aerial imagery, PSP, covering 106 ship classes and 30 airplane models, respectively, featuring the most extensive collection of model-specific categories among existing aerial object detection datasets to date. We benchmark state-of-the-art object detection algorithms on the PSP benchmark. Extensive evaluation demonstrates that ExpertDet consistently outperforms other fine-grained competitors across hierarchy levels. The dataset, benchmark, and code are available at https://nnnnerd.github.io/PSP-Benchmark/.

Yan Zhang, Fang Xu, Wen Yang +1

School of Computer Science, Wuhan University, Wuhan, 430072, China · School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China · School of Electronic Information, Wuhan University, Wuhan, 430072, China

Aerial Dataset Fine-Grained Perception Scene Reasoning

Ask PDF

Jun 5, 2026cs.CV

Differences in Detection: Explainability Where it Matters

We propose Differences in Detection (DnD), an intuitive method to compare two object detection models. Based on the same matching algorithm, it complements the standard metrics of mean Average Precision (

mAP

) and TIDE error analysis with the ability to compare two models directly. More specifically, we calculate the intersection of ground truth labels that are recognized by both models, followed by the corresponding difference sets and the complement set of ground truth labels that are missed by both models. The resulting comparison is more direct and intuitive than a comparison of independent summary statistics. It reveals individual and shared mistakes and becomes particularly interesting when combined with error types. In this case, the differences in detection errors can be analyzed naturally in a standard confusion matrix. While valuable in itself, we believe that one of the best applications of DnD is to guide explainability methods such as ODAM towards metric-relevant examples, grounded in structured subsets. The code for our method is available here: https://github.com/JohannesTheo/differences-in-detection

Johannes Theodoridis, Johannes Maucher, Andreas Schilling

University of T ubingen · Institute for Applied AI · Hochschule der Medien Stuttgart

Detection Performance Open-Vocabulary Object Detection Explainability Ground Truth

Ask PDF

Jul 2, 2026cs.CV

Diagnosing Aerial-View Object Detectors with Foundational Image Generative Models

Recent advances in large-scale image generative models enable photorealistic scene synthesis with controllable attributes. Beyond data augmentation, their potential as diagnostic tools for trained vision systems remains unexplored in the aerial and remote sensing domains. We introduce a synthetic diagnostic framework for aerial-view vehicle detection that combines text-guided generation, attribute-controlled editing, and automated attribute verification to construct a controllable synthetic testbed. This enables fine-grained evaluation of pretrained detectors under diverse scene types and environmental conditions that are difficult to isolate in real datasets. Across three detection architectures and three real aerial datasets, synthetic scene-wise performance trends closely match real-world weaknesses. Guided by these diagnostics, targeted supplementation with small real datasets from the identified weak categories yields improvements of up to 13% AP50 while requiring substantially fewer additional samples than non-targeted augmentation. Our results show that controlled synthetic probing can predict real-domain performance gaps and guide efficient data collection. The proposed diagnostic framework is modular and can incorporate alternative generative or vision-language models as capabilities evolve. Our code and datasets are available here: https://humansensinglab.github.io/AVODDiag/

Stanislav Panev, Minhyek Jeon, Vaishnavi Khindkar +5

Carnegie Mellon University, USA · DEVCOM Army Research Laboratory, USA · Florida State University, USA

Aerial Dataset Synthetic Image Detection Diagnostic Framework Generative Models

Ask PDF