OV-DEIM: Real-time DETR-Style Open-Vocabulary Object Detection with GridSynthetic Augmentation

Leilei Wang; Longfei Liu; Xi Shen; Xuanlong Yu; Ying Tiffany He; Fei Richard Yu; Yingyi Chen

cs.CVMar 7, 2026

OV-DEIM: Real-time DETR-Style Open-Vocabulary Object Detection with GridSynthetic Augmentation

Authors: Leilei Wang, Longfei Liu, Xi Shen, Xuanlong Yu, Ying Tiffany He, Fei Richard Yu, Yingyi Chen

Organizations: Intellindust AI Lab · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, China · College of Computer Science and Software Engineering, Shenzhen University, China · School of Information Technology, Carleton University, Canada ·

Abstract

Real-time open-vocabulary object detection (OVOD) is essential for practical deployment in dynamic environments, where models must recognize a large and evolving set of categories under strict latency constraints. Current real-time OVOD methods are predominantly built upon YOLO-style models. In contrast, real-time DETR-based methods still lag behind in terms of inference latency, model lightweightness, and overall performance. In this work, we present OV-DEIM, an end-to-end DETR-style open-vocabulary detector built upon the recent DEIMv2 framework with integrated vision-language modeling for efficient open-vocabulary inference. We further introduce a simple query supplement strategy that improves Fixed AP without compromising inference speed. Beyond architectural improvements, we introduce GridSynthetic, a simple yet effective data augmentation strategy that composes multiple training samples into structured image grids. By exposing the model to richer object co-occurrence patterns and spatial layouts within a single forward pass, GridSynthetic mitigates the negative impact of noisy localization signals on the classification loss and improves semantic discrimination, particularly for rare categories. Extensive experiments demonstrate that OV-DEIM achieves state-of-the-art performance on open-vocabulary detection benchmarks, delivering superior efficiency and notable improvements on challenging rare categories. Code and pretrained models are available at https://github.com/wleilei/OV-DEIM.

Figures & tables

Figure 1: Architecture of the OV-DEIM Framework. Given an image $\bm{I}$ , the backbone and hybrid encoder extract flattened multi-scale visual features $\bm{F_{I}}$ , which are processed by the vision-text alignment head (CLS) to produce similarity scores and into the bounding box regression head to predict object locations. Given text prompts, the text encoder maps them into textual embeddings $\bm{F_{T}}$ , which are used to guide query selection and vision-text alignment. Through the text-aware query selection, the top-ranked queries $Q_{\text{top}}$ are fed into the decoder for iterative refinement, while $Q_{\text{dn}}$ are used for the denoising loss [ 17 ] and $Q_{\text{sup}}$ serve as additional queries for computing Fixed AP [ 6 ] . Each query comprises both a feature embedding and location information.

Figure 2: Effectiveness of GridSynthetic. The EMA-smoothed GIoU loss curves show that GridSynthetic consistently achieves the lowest loss throughout training, alleviating the difficulty of localization and leading to more sufficient classification supervision and cleaner semantic alignment.

Dataset	Texts	Samples	Images	Annotations
Objects365V1 [ 28 ]	365	609k	609k	9,621k
GQA [ 15 ]	387k	621k	46k	3,681k
Flickr30k [ 23 ]	94k	149k	30k	641k

Table 1: Pretraining Datasets. “Texts”: unique text descriptions; “Samples”: differently annotated images; “Images”: distinct images; “Annotations”: total object-level annotations.

Method	Backbone	Params	Pre-trained Data	FPS	AP/AP ${}^{\text{Fixed}}$	AP r /AP r ${}^{\text{Fixed}}$	AP c /AP c ${}^{\text{Fixed}}$	AP f / AP f ${}^{\text{Fixed}}$
GLIP-T [ 18 ]	Swin-T	232M	OG	-	-/24.9	-/17.7	-/19.5	-/31.0
GLIP-T [ 18 ]	Swin-T	232M	OG,Cap4M	-	-/26.0	-/20.8	-/21.4	-/31.0
GLIPv2-T [ 41 ]	Swin-T	232M	OG,Cap4M	-	-/29.0	-/-	-/-	-/-
GDINO-T [ 20 ]	Swin-T	172M	OG	-	-/25.6	-/14.4	-/19.6	-/32.2
GDINO-T [ 20 ]	Swin-T	172M	OG,Cap4M	-	-/27.4	-/18.1	-/23.3	-/32.7
G1.5-Edge [ 26 ]	EfficientViT-L1	-	G-20M	-	-/33.5	-/28.0	-/34.3	-/33.9

Table 2: Zero-shot Evaluation on LVIS [ 12 ] . AP denotes the standard Average Precision, whereas AP

{}^{\text{Fixed}}

corresponds to Fixed AP, both evaluated on the LVIS minival split. Inference speed (FPS) is measured on an NVIDIA T4 GPU using TensorRT. Following RT-DETR [ 43 ] , FPS marked with * includes the post-processing overhead of NMS, averaged over the 5,000 images in the minival set. OG indicates Objects365v1 [ 28 ] and GoldG [ 16 ] , Cap4M is a collection of 4M image-text pairs from GLIP [ 18 ] , and G-20M represents Grounding-20M [ 26 ] .

Method	Backbone	Params	AP	AP 50	AP 75
Zero-shot transfer
YOLO-Worldv1-S [ 5 ]	YOLOv8-S	13M	37.6	52.3	40.7
YOLO-Worldv1-M [ 5 ]	YOLOv8-M	29M	42.8	58.3	46.4
YOLO-Worldv1-L [ 5 ]	YOLOv8-L	48M	44.4	59.8	48.3
Linear probing
YOLOEv8-S [ 33 ]	YOLOv8-S	12M	35.6	51.5	38.9

Table 3: Zero-shot Evaluation on COCO [ 19 ] . The proposed model achieves better zero-shot performance on COCO compared to YOLO-Worldv1-S [ 5 ] . It also consistently outperforms the linear probing version of YOLOE [ 33 ] in zero-shot evaluation.

Figure 3: Visualizations of Zero-shot Inference on LVIS [ 12 ] . We employ the pretrained OV-DEIM-L model and perform inference with the LVIS vocabulary containing 1,203 categories, only using a confidence threshold of 0.5.

Ablation on GridSynthetic
Data Augmentation	AP	AP r	AP c	AP f
Base w/ GS (full)	33.1	29.3	33.5	33.4
(w/o CSS)	32.5	30.2	32.2	33.1
(w/o CSS & Context Cues)	30.6	29.3	31.3	30.3
General Data Augmentations
Base	32.4	29.3	32.6	32.7

Table 4: Ablation Study on Data Augmentations and GridSynthetic. GS denotes GridSynthetic, CSS represents complex scene simulation, and CopyPaste 4/16 indicates pasting 4 or 16 objects per sample. Base augmentations (color, affine, flip, mosaic) are applied with probability 0.75, while additional augmentations (MixUp and/or GridSynthetic) are applied with probabilities adjusted per variant, e.g. , 0.25 for Base w/ MixUp, 0.125 each for Base w/ GS & MixUp. All experiments are conducted using OV-DEIM-L and trained on the OG dataset.

Extra Queries	AP ${}^{\text{Fixed}}$	AP r ${}^{\text{Fixed}}$	AP c ${}^{\text{Fixed}}$	AP f ${}^{\text{Fixed}}$
0	33.1	30.3	33.0	33.7
100	33.7	30.8	33.6	34.4
200	34.2	30.8	34.1	34.9
300	34.5	31.0	34.6	35.2
400	34.8	31.4	34.8	35.4
500	34.9	31.5	35.0	35.6

Table 5: Ablation Study on Query Supplement Trick. Beyond the default 300 decoder queries, we introduce additional candidate queries from the encoder to generate more bounding box predictions. This simple strategy increases the number of detection candidates per image and proves effective under the Fixed AP [ 6 ] .

Open-Vocabulary Object Detection

Open-vocabulary object detection lets models locate and label objects using text categories beyond their fixed training labels.

See all 89 papers

DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection
Apr 3, 2026Siheng Wang, Yanshu Li, Bohan Hu +12Open-Vocabulary Object Detection Detection Transformer
Jiangsu University · MBZUAI · Brown University +5
RT-DETR-World: Transferring Rich LLM Semantics to Real-Time Open-Vocabulary Detection
Oct 7, 2026Yupeng Zhang, Ziyi Zhao, Juntao Cheng +4Open-Vocabulary Object Detection Real-Time Object Detection
Tianjin University. · Shenzhen University of Advanced Technology. · Shanghai Jiaotong University. +1
ExDet: Open-Domain Open-Vocabulary Detection with Cross-modal Extrapolation and Rectification
Jun 8, 2026Yupeng Zhang, Yuzhong Feng, Ruize Han +3Open-Vocabulary Object Detection Object Detection
College of Intelligence and Computing, Tianjin University, Tianjin, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China · School of Artificial Intelligence, Nanchang University, Jiangxi, China

Apr 3, 2026J/K move · Enter open · S save

DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection

Siheng Wang, Yanshu Li, Bohan Hu, Zhengdao Li+11

Jiangsu University · MBZUAI · Brown University+5

Open-Vocabulary Object Detection Detection Transformer Efficient Object Detection

Open paper AskPDF arXiv

Open-vocabulary object detection (OVOD) enables models to recognize objects beyond predefined categories, but existing approaches remain limited in practical deployment. On the one hand, multimodal designs often incur substantial computational overhead due to their reliance on text encoders at inference time. On the other hand, tightly coupled training objectives introduce a trade-off between closed-set detection accuracy and open-world generalization. Thus, we propose Decoupled Cognition DETR (DeCo-DETR), a vision-centric framework that addresses these challenges through a unified decoupling paradigm. Instead of depending on online text encoding, DeCo-DETR constructs a hierarchical semantic prototype space from region-level descriptions generated by pre-trained LVLMs and aligned via CLIP, enabling efficient and reusable semantic representation. Building upon this representation, the framework further disentangles semantic reasoning from localization through a decoupled training strategy, which separates alignment and detection into parallel optimization streams. Extensive experiments on standard OVOD benchmarks demonstrate that DeCo-DETR achieves competitive zero-shot detection performance while significantly improving inference efficiency. These results highlight the effectiveness of decoupling semantic cognition from detection, offering a practical direction for scalable OVOD systems.

Abstract

Open-Vocabulary Object Detection

Explore similar work

DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection

RT-DETR-World: Transferring Rich LLM Semantics to Real-Time Open-Vocabulary Detection

ExDet: Open-Domain Open-Vocabulary Detection with Cross-modal Extrapolation and Rectification