DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection

Siheng Wang; Yanshu Li; Bohan Hu; Zhengdao Li; Haibo Zhan; Linshan Li; Weiming Liu; Ruizhi Qian

cs.CVApr 3, 2026

DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection

Authors: Siheng Wang, Yanshu Li, Bohan Hu, Zhengdao Li, Haibo Zhan, Linshan Li, Weiming Liu, Ruizhi Qian,

Organizations: Jiangsu University · MBZUAI · Brown University · Nanyang Technological University ·

Abstract

Open-vocabulary object detection (OVOD) enables models to recognize objects beyond predefined categories, but existing approaches remain limited in practical deployment. On the one hand, multimodal designs often incur substantial computational overhead due to their reliance on text encoders at inference time. On the other hand, tightly coupled training objectives introduce a trade-off between closed-set detection accuracy and open-world generalization. Thus, we propose Decoupled Cognition DETR (DeCo-DETR), a vision-centric framework that addresses these challenges through a unified decoupling paradigm. Instead of depending on online text encoding, DeCo-DETR constructs a hierarchical semantic prototype space from region-level descriptions generated by pre-trained LVLMs and aligned via CLIP, enabling efficient and reusable semantic representation. Building upon this representation, the framework further disentangles semantic reasoning from localization through a decoupled training strategy, which separates alignment and detection into parallel optimization streams. Extensive experiments on standard OVOD benchmarks demonstrate that DeCo-DETR achieves competitive zero-shot detection performance while significantly improving inference efficiency. These results highlight the effectiveness of decoupling semantic cognition from detection, offering a practical direction for scalable OVOD systems.

Figures & tables

Figure 1: Three-stage pipeline of DeCo-DETR. (a) DHCP constructs a hierarchical prototype memory from region-level descriptions via LLaVA generation and CLIP-based filtering, capturing both coarse and fine-grained semantics. (b) Hi-Know DPA projects detector queries into the shared embedding space and enhances them through prototype aggregation for efficient open-set knowledge transfer. (c) PD-DuGi decouples localization and semantic alignment into two optimization streams, mitigating task interference. At inference, the prototype pool provides semantic priors, and the decoupled decoder jointly predicts bounding boxes and category semantics without a text encoder.

Benchmark	Method	$\mathbf{AP_{50}^{novel}}$	$\mathbf{AP_{50}^{base}}$	$\mathbf{AP_{50}}$
V-OVD	ViLD ( Gu et al., 2021 )	29.4	52.6	48.9
	OADP ( Wang et al., 2023b )	30.0	53.3	47.2
	DK-DETR ( Li et al., 2023 )	32.3	61.1	53.6
	BARON ( Wu et al., 2023b )	33.1	54.8	49.1
	LBP ( Li et al., 2024a )	37.8	58.7	53.2
	OC-OVD ( Rasheed et al., 2022 )	36.6	54.0	49.4

Table 1: OV-COCO comparison (

AP_{50}

) across a wide range of widely used OVOD methods.

Method	$\mathbf{AP_{r}}$	$\mathbf{AP_{c}}$	$\mathbf{AP_{f}}$	$\mathbf{AP}$
DetPro ( Du et al., 2022 )	20.8	27.8	32.4	28.4
VLDet ( Lin et al., 2022 )	21.7	29.8	34.3	30.1
OC-OVD ( Rasheed et al., 2022 )	21.1	25.0	29.1	25.9
OADP ( Wang et al., 2023b )	21.9	28.4	32.0	28.7
CORA ( Wu et al., 2023c )	22.2	32.0	40.2	33.5
BARON ( Wu et al., 2023b )	23.2	29.3	32.5	29.5

Table 2: OV-LVIS comparison (

AP

) across a wide range of widely used OVOD methods.

Figure 2: Qualitative comparison between DeCo-DETR and the baseline. DeCo-DETR shows stronger open-vocabulary generalization by accurately localizing and recognizing novel categories.

Method	Latency (ms/img)			GFLOPs			Params (M)
	R50	ViT	Swin	R50	ViT	Swin	R50	ViT	Swin
Deformable DETR ( Zhu et al., 2020b )	120	210	220	220	320	325	41	87	95
DetPro ( Du et al., 2022 )	140	250	260	240	340	345	45	91	100
UP-DETR ( Dai et al., 2021 )	115	205	215	215	315	320	40	85	92
DeCo-DETR (Ours)	135	240	250	235	335	340	44	90	97

Table 3: Inference latency, GFLOPs, and parameter size across three backbone architectures.

Configuration	$\mathbf{AP^{novel}_{50}}$	$\mathbf{AP^{novel}_{50}}$	$\mathbf{AP_{50}}$
1. Baseline only	30.4	52.6	46.8
2. + Hierarchical DHCP	36.6	54.0	49.4
3. + PD-DuGi (Gradient Isolation)	37.5	55.1	50.5
4. + Cosine $\lambda(t)$ (Full Model)	41.3	55.5	51.0

Table 4: Main ablation studies.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Caption	Category Prior	Image Label
V-OVD
C-OVD	✓
G-OVD		✓
WS-OVD	✓	✓	✓

Appendix

Table 5: Summary of OVD benchmarks. “Caption”: in-domain captions like COCO-Captions. “Category Prior”: human priors on novel categories. “Image Label”: image-level category labels.

Model	$\mathbf{AP_{50}^{novel}}$	$\mathbf{AP_{50}^{base}}$
LLaVA-1.5 7B	30.1	52.1
LLaVA-1.5 13B	38.2	55.5
LLaVA-NEXT 7B	32.1	53.3
LLaVA-NEXT 13B	38.6	55.8
Qwen2.5-VL 7B	33.1	53.9
Qwen2.5-VL 32B	38.9	55.9

Appendix

Table 6: Comparison of performance of different models on the OV-COCO dataset

Configuration	Queries ( $N$ )	Fine Units ( $M_{2}$ )	$\mathbf{AP^{novel}_{50}}$	$\mathbf{AP^{base}_{50}}$	Latency (ms)
DeCo-DETR	300	4800	36.5	53.8	125
DeCo-DETR	1000	4800	39.1	54.5	130
DeCo-DETR	2000	4800	41.3	55.5	135
DeCo-DETR	2000	0 (Coarse only)	30.8	54.1	131
DeCo-DETR	2000	9600 (Double)	41.5	55.6	142

Appendix

Table 7: Ablation study on the number of Decoder Queries (

N

) and Fine-grained Prototypes (

M_{2}

). The default setting is highlighted in bold.

Method	Backbone	$\mathbf{AP^{novel}_{50}}$	Text Enc.	Latency (ms)	FPS
Fusion-based:
Grounding DINO-T	Swin-T	42.1	BERT-Base	$\sim$ 280	3.5
VL-PLM	ResNet-50	32.3	RoBERTa	$\sim$ 210	4.7
Distillation/Decoupled:
DetPro	ResNet-50	29.4	-	250	4.0
CAKE	ResNet-50	38.2	-	145	6.9

Appendix

Table 8: Comparison of inference efficiency and open-vocabulary detection performance on COCO. DeCo-DETR achieves the best trade-off between accuracy and speed.

Open-Vocabulary Object Detection

Open-vocabulary object detection lets models locate and label objects using text categories beyond their fixed training labels.

See all 87 papers

The Detector Teaches Itself: Lightweight Self-Supervised Adaptation for Open-Vocabulary Object Detection
May 5, 2026Yazhe Wan, Changjae OhOpen-Vocabulary Object Detection VLM Adaptation
Queen Mary University of London · Queen Mary University of London, London, UK
RT-DETR-World: Transferring Rich LLM Semantics to Real-Time Open-Vocabulary Detection
Oct 7, 2026Yupeng Zhang, Ziyi Zhao, Juntao Cheng +4Open-Vocabulary Object Detection Real-Time Object Detection
Tianjin University. · Shenzhen University of Advanced Technology. · Shanghai Jiaotong University. +1
VL-DINO: Leveraging CLIP Vision-Language Knowledge for Open-Vocabulary Object Detectio
Jun 10, 2026Hao Zhang, Qinran Lin, Linqi Song +1Open-Vocabulary Object Detection Detection Transformer
Chongqing University, Chongqing, China · City University of Hong Kong, Hong Kong, China

May 5, 2026J/K move · Enter open · S save

The Detector Teaches Itself: Lightweight Self-Supervised Adaptation for Open-Vocabulary Object Detection

Yazhe Wan, Changjae Oh

Queen Mary University of London · Queen Mary University of London, London, UK

Open-Vocabulary Object Detection VLM Adaptation Vision-Language Alignment

Open paper AskPDF arXiv

Open-vocabulary object detection aims to recognize objects from an open set of categories, which leverages vision-language models (VLMs) pre-trained on large-scale image-text data. The cooperative paradigm combines an object detector with a VLM to achieve zero-shot recognition of novel objects. However, VLMs pre-trained on full images often struggle to capture local object details, limiting their effectiveness when applied to region-level detection. We present Decoupled Adaptivity Training (DAT), a self-supervised fine-tuning approach to improve VLMs for cooperative model-based object detection. Given a cooperative model consists of a closed-set detector and a VLM, we first construct a region-aware pseudo-labeled dataset using a pre-trained closed-set object detector, in which regions corresponding to novel objects may be present but remain unlabeled or mislabeled. We then fine-tune the visual backbone of the VLM in a decoupled manner, which enhances local feature alignment while preserving global semantic knowledge via weight interpolation. DAT is a plug-and-play module that requires no inference overhead and fine-tunes less than 0.8M parameters. Experiments on the COCO and LVIS datasets show that DAT consistently improves detection performance on both novel and known categories, establishing a new state of the art in cooperative open-vocabulary detection.

Abstract

Open-Vocabulary Object Detection

Explore similar work

The Detector Teaches Itself: Lightweight Self-Supervised Adaptation for Open-Vocabulary Object Detection

RT-DETR-World: Transferring Rich LLM Semantics to Real-Time Open-Vocabulary Detection

VL-DINO: Leveraging CLIP Vision-Language Knowledge for Open-Vocabulary Object Detectio