In this work, we tackle the problem of Open World Object Detection (OWOD). This challenging scenario requires the detector to incrementally learn to classify known objects without forgetting while identifying unknown objects without supervision. Previous OWOD methods have enhanced the unknown discovery process and employed memory replay to mitigate catastrophic forgetting. However, since existing methods heavily rely on the detector's known class predictions for detecting unknown objects, they struggle to effectively learn and recognize unknown object representations. Moreover, while memory replay mitigates forgetting of old classes, it often sacrifices the knowledge of newly learned classes. To resolve these limitations, we propose DEUS (Detecting Unknowns via energy-based Separation), a novel framework that addresses the challenges of Open World Object Detection. DEUS consists of Equiangular Tight Frame (ETF)-Subspace Unknown Separation (EUS) and an Energy-based Known Distinction (EKD) loss. EUS leverages ETF-based geometric properties to create orthogonal subspaces, enabling cleaner separation between known and unknown object representations. Unlike prior energy-based approaches that consider only the known space, EUS utilizes energies from both spaces to better capture distinct patterns of unknown objects. Furthermore, EKD loss enforces the separation between previous and current classifiers, thus minimizing knowledge interference between previous and newly learned classes during memory replay. We thoroughly validate DEUS on OWOD benchmarks, demonstrating outstanding performance improvements in unknown detection while maintaining competitive known class performance.
Figures & tables
Figure 1: Motivation and objectives of DEUS. (a) PCA visualization of proposal features: the baseline (top) entangles known, unknown, and background, while DEUS (bottom) achieves clear separation. (b) Objectives of DEUS: separating known from unknown proposals while distinguishing previous and current known classes.
Figure 2: Overview of DEUS. ETF-Subspace Unknown Separation (EUS) utilizes a Simplex ETF to construct known and unknown spaces, where each space serves as an energy module to compute space energy scores, and Energy-based Known Distinction (EKD) loss is applied during the memory replay phase, where the classification branch is split into two sub-classifiers to calculate energy scores for previous and current tasks.
Task IDs
Task 1
Task 2
Task 3
Task 4
Method
Current mAP
U-Rec
H-Score
Previous mAP
Current mAP
Known mAP
U-Rec
H-Score
Previous mAP
Current mAP
Known mAP
U-Rec
H-Score
Previous mAP
Current mAP
Known mAP
ORE [ 8 ]
56.0
4.9
9.0
52.7
26.0
39.4
2.9
5.4
38.2
12.7
29.7
3.9
6.9
29.6
12.4
25.3
OW-DETR [ 7 ]
59.2
7.5
13.3
53.6
33.5
42.9
6.2
10.8
38.3
15.8
30.8
5.7
9.6
31.4
17.1
27.8
CAT [ 12 ]
60.0
23.7
34.0
55.5
32.7
44.1
19.1
26.7
42.8
18.7
34.8
24.4
28.7
34.4
16.6
29.9
PROB † [ 26 ]
66.4
28.3
39.7
62.6
39.2
50.9
26.4
34.8
49.6
33.5
44.2
29.3
35.2
44.0
26.5
39.7
OrthogonalDet † [ 21 ]
65.1
36.3
46.6
58.2
44.2
51.2
30.2
38.0
50.9
40.1
47.3
28.7
35.7
49.1
31.5
44.7
Table 1: Experimental results on M-OWODB (top) and S-OWODB (bottom). Results are reported in terms of mean average precision (mAP) for known classes, unknown class recall (U-Rec), and harmonic score (H-Score). The best performance is highlighted in bold, with the second-best performance underlined. † denotes reproduced results after correcting M-OWODB annotation duplication bug identified in [ 24 ] , which may differ from the originally reported numbers.
Task IDs
Task 1
Task 2
Task 3
Task 4
Method
Known mAP
U-Rec
H-Score
Known mAP
U-Rec
H-Score
Known mAP
U-Rec
H-Score
Known mAP
OrthogonalDet [ 21 ]
65.4
23.7
34.8
62.3
8.9
15.6
62.4
9.3
16.2
64.2
DEUS (Ours)
67.7
58.1
62.5
65.6
28.1
39.4
66.0
29.7
40.9
68.3
Table 2: Experimental results on RS-OWODB. The best performance is highlighted in bold.
Task IDs
Task 1
Task 2
Task 3
Task 4
EUS
EKD
Current mAP
U-Rec
H-Score
Previous mAP
Current mAP
Known mAP
U-Rec
H-Score
Previous mAP
Current mAP
Known mAP
U-Rec
H-Score
Previous mAP
Current mAP
Known mAP
66.0
36.8
47.2
58.8
45.3
52.0
29.0
37.3
52.7
43.5
49.7
30.3
37.6
49.2
31.3
44.7
✓
66.0
36.8
47.2
59.2
45.9
52.6
40.0
45.4
53.6
43.6
50.3
38.9
43.9
50.3
32.7
45.9
✓
66.2
65.1
65.6
58.8
45.0
51.9
63.8
57.2
52.6
42.4
49.2
69.0
57.5
48.1
29.7
43.5
✓
✓
66.2
65.1
65.6
61.0
45.7
53.3
66.2
59.0
53.4
43.3
50.1
69.0
58.0
50.5
32.8
46.0
Table 3: Ablation study of DEUS on M-OWODB. EUS and EKD represent ETF-Subspace Unknown Separation and Energy-based Known Distinction loss, respectively. EUS aims to improve the detection performance for unknown objects, while EKD is designed to enhance the performance for known objects. The best performance is highlighted in bold, with the second-best performance underlined.
Figure 3: (a) Per-class subspace score comparison between known (green) and unknown (red) for proposals matched with ground-truth. (b) Energy scores of proposals from each task computed by each task head (w/o vs. w/ EKD). Parentheses show changes with EKD.
Figure 4: Qualitative comparison between OrthogonalDet and DEUS. Purple boxes indicate unknown predictions, while other colored boxes indicate known class predictions. The model was trained on Task 1, where cow is known and giraffe is unknown.
Open-world object detection (OWOD) requires incrementally detecting known categories while reliably identifying unknown objects. Existing methods primarily focus on improving unknown recall, yet overlook interpretability, often leading to known-unknown confusion and reduced prediction reliability. This paper aims to make the entire OWOD framework interpretable, enabling the detector to truly "knowing the unknown". To this end, we propose a concept-driven InterPretable OWOD framework(IPOW) by introducing a Concept Decomposition Model (CDM) for OWOD, which explicitly decomposes the coupled RoI features in Faster R-CNN into discriminative, shared, and background concepts. Discriminative concepts identify the most discriminative features to enlarge the distances between known categories, while shared and background concepts, due to their strong generalization ability, can be readily transferred to detect unknown categories. Leveraging the interpretable framework, we identify that known-unknown confusion arises when unknown objects fall into the discriminative space of known classes. To address this, we propose Concept-Guided Rectification (CGR) to further resolve such confusion. Extensive experiments show that IPOW significantly improves unknown recall while mitigating confusion, and provides concept-level interpretability for both known and unknown predictions.
Xueqiang Lv, Shizhou Zhang, Yinghui Xing +3
School of Computer Science, Northwestern Polytechnical Unviersity, Xi’an, China · Huawei Technologies Ltd.
Real-world object detection operates under ambiguous supervision, where unlabeled regions may correspond to missing annotations of known objects or genuinely unknown categories. These challenges have been addressed separately in Sparsely Annotated Object Detection (SAOD) and Open-World Object Detection (OWOD). In practice, their co-occurrence remains an open problem. To address this problem, we introduce Sparsely Annotated Open-World Object Detection (SA-OWOD), a new task that jointly considers sparse supervision and the presence of unseen categories. We propose Dual-Perspective Object Discovery (DPOD), a unified framework that jointly models unlabeled known and unknown instances via two complementary mechanisms. The Known Target Recovery Module (KTRM) recovers supervision for unlabeled known instances and explicitly regularizes the feature space to separate known and unknown representations. Complementarily, the Dual-Disagreement Target Generator (DDTG) identifies reliable unknown candidates through cross-view semantic inconsistency. By integrating these modules, DPOD resolves contradictory supervision signals caused by ambiguous unlabeled regions. As a result, it prevents misclassification between known and unknown objects and stabilizes the decision boundaries. Experimental results on sparsely annotated open-world benchmarks demonstrate that the proposed method outperforms existing open-world detection methods, particularly in detecting unknown objects.
Open-World Object Detection (OWOD) requires detectors to identify previously unseen objects as unknown and incrementally incorporate them into the set of known categories, while preserving previously acquired knowledge. Existing frameworks rely heavily on exemplar replay to mitigate catastrophic forgetting, but in some real applications, storing raw data conflicts with data access restrictions and leads to data exposure risks, while incurring significant memory overhead. In this paper, we propose REAL-OW, a novel rehearsal-free framework that decouples incremental knowledge through a collaborative adapter architecture based on Low-Rank Adaptation (LoRA). Specifically, we deploy General Adapters (GAs) in the backbone to enable the significance-aware refinement of cross-task universal representations, while Specific Adapters (SAs) in the decoder provide orthogonal storage for task-specific expertise. To resolve representation drift in objectness modeling under rehearsal-free constraints, we introduce Dual-Stage Objectness Modeling (DSOM), which alternates between feature aggregation and boundary consolidation to stabilize objectness distributions while maintaining the separation between known and unknown categories. Furthermore, DSOM is supported by a Calibrated Gaussian Negative Log-Likelihood (CG-NLL) distance tailored for the dispersed feature distributions inherent in rehearsal-free settings. Extensive evaluations demonstrate that REAL-OW achieves state-of-the-art performance, surpassing existing exemplar replay methods in both detection precision and unknown discovery. Our approach establishes a new baseline for rehearsal-free OWOD.