PET-DINO: Unifying Visual Cues into Grounding DINO with Prompt-Enriched Training

Weifu Fu; Jinyang Li; Bin-Bin Gao; Jialin Li; Yuhuan Lin; Hanqiu Deng; Wenbing Tao; Yong Liu

cs.CVApr 1, 2026

PET-DINO: Unifying Visual Cues into Grounding DINO with Prompt-Enriched Training

Authors: Weifu Fu, Jinyang Li, Bin-Bin Gao, Jialin Li, Yuhuan Lin, Hanqiu Deng, Wenbing Tao, Yong Liu,

Organizations: YouTu Lab, Tencent · Huazhong University of Science and Technology · Kling Team, Kuaishou Technology

Abstract

Open-Set Object Detection (OSOD) enables recognition of novel categories beyond fixed classes but faces challenges in aligning text representations with complex visual concepts and the scarcity of image-text pairs for rare categories. This results in suboptimal performance in specialized domains or with complex objects. Recent visual-prompted methods partially address these issues but often involve complex multi-modal designs and multi-stage optimizations, prolonging the development cycle. Additionally, effective training strategies for data-driven OSOD models remain largely unexplored. To address these challenges, we propose PET-DINO, a universal detector supporting both text and visual prompts. Our Alignment-Friendly Visual Prompt Generation (AFVPG) module builds upon an advanced text-prompted detector, addressing the limitations of text representation guidance and reducing the development cycle. We introduce two prompt-enriched training strategies: Intra-Batch Parallel Prompting (IBP) at the iteration level and Dynamic Memory-Driven Prompting (DMD) at the overall training level. These strategies enable simultaneous modeling of multiple prompt routes, facilitating parallel alignment with diverse real-world usage scenarios. Comprehensive experiments demonstrate that PET-DINO exhibits competitive zero-shot object detection capabilities across various prompt-based detection protocols. These strengths can be attributed to inheritance-based philosophy and prompt-enriched training strategies, which play a critical role in building an effective generic object detector. Project page: https://fuweifuvtoo.github.io/pet-dino.

Figures & tables

Figure 1: Overall architecture of PET-DINO. Input coordinates undergo a Visual Prompt Generation process, interacting with enhanced image features to obtain visual prompts. The text encoder creates text embeddings, which interact with image features in the Feature Enhancer module to produce text prompts. Both types of prompts guide the Query Selection Module, offering location priors for initial queries. These queries are then refined through decoder layers to predict objects and classifications.

Figure 2: Intra-Batch Parallel Prompting Diagram. Image and their target coordinates within a batch are processed to generate image-level visual prompts. We incorporate visual prompts from other samples as additional prompts for the current image, aggregating those from the same category to form class-level visual prompts.

Method	Open	Prompt	Backbone	VLM sup.	Data	Training Data	COCO	LVIS	ODinW35
Method	Source	Type	Backbone	VLM sup.	Size	Training Data	COCO	LVIS	ODinW35
T-Rex2 [ 10 ]	N	Visual-I	Swin-T	CLIP	3.1M	O365, OpenImages, HierText, CrowdHuman, SA-1B	56.6	59.3	37.7
CP-DETR-T [ 1 ]	N	Visual-I	Swin-T	CLIP	3.3M	O365, OpenImages, GoldG, V3Det	61.8	64.1	41.0
PET-DINO	Y	Visual-I	Swin-T	-	0.6M	O365	64.0	61.8	38.8
PET-DINO	Y	Visual-I	Swin-T	-	2.5M	O365, OpenImages, V3Det	64.3	64.5	48.3
T-Rex2 [ 10 ]	N	Visual-I	Swin-L	CLIP	3.1M	O365, OpenImages, HierText, CrowdHuman, SA-1B	58.5	62.5	39.7

Table 1: Comparison for zero-shot interactive object detection through visual prompts. Fixed AP is reported on LVIS minival set. Bold and underline denote the best and second-best results in each column, respectively. * means the training set of the evaluation dataset was used for pre-training with text prompts, so this model is not in a zero-shot setting.

Method	Open	Prompt	Backbone	VLM sup.	Data	Training Data	COCO	LVIS	ODinW35
Method	Source	Type	Backbone	VLM sup.	Size	Training Data	COCO	LVIS	ODinW35
T-Rex2 [ 10 ]	N	Visual-G	Swin-T	CLIP	3.1M	O365, OpenImages, HierText, CrowdHuman, SA-1B	38.8	37.4	23.6
YOLOE-v11-S [ 27 ]	Y	Visual-G	YOLO-v11-S	MobileCLIP	1.4M	O365, GoldG	-	26.2	-
PET-DINO	Y	Visual-G	Swin-T	-	0.6M	O365	40.3	29.6	20.4
PET-DINO	Y	Visual-G	Swin-T	-	2.5M	O365, OpenImages, V3Det	38.4	31.5	25.5
T-Rex2 [ 10 ]	N	Visual-G	Swin-L	CLIP	3.1M	O365, OpenImages, HierText, CrowdHuman, SA-1B	46.5	47.6	27.8

Table 2: Comparison for zero-shot generic object detection through visual prompts. Fixed AP is reported on LVIS minival set. Bold and underline denote the best and second-best results in each column, respectively.

Method	Prompt	Backbone	Training Data	COCO	LVIS	ODinW35
Method	Type	Backbone	Training Data	COCO	LVIS	ODinW35
Grounding DINO [ 20 ]	Text	Swin-T	O365, GoldG, Cap4M	48.4	27.4	22.3
MM-Grounding-DINO [ 36 ]	Text	Swin-T	O365, GoldG, V3Det	50.6	40.5	21.4
PET-DINO	Text	Swin-T	O365 †	49.8 ↓0.8	37.8 ↓2.7	20.6 ↓0.8
Grounding DINO [ 20 ]	Text	Swin-L	O365, OpenImages, GoldG	52.5	33.9	26.1
MM-Grounding-DINO [ 36 ]	Text	Swin-L	O365V2, OpenImage, GoldG	53.0	36.7	23.9

Table 3: Comparison for zero-shot generic object detection through text prompts. Fixed AP is reported on LVIS minival set. † means that a small number of iterations on Object365 are not for dedicated text-prompt training, but specifically to prevent the degradation of the inherited text-prompt performance.

Method	Prompt	Backbone	Training Data	COCO
Method	Type	Backbone	Training Data	COCO
X-Decoder T [ 39 ]	Text	Focal-T	COCO+CC3M+…	43.6
OpenSeed T [ 34 ]	Text	Swin-T	COCO+O365	52.0
OpenSeed L [ 34 ]	Text	Swin-L	COCO+O365	58.2
MI Grounding-S [ 35 ]	Image	ViT-L	COCO+LVIS	54.7
DINOv [ 14 ]	Visual-G	Swin-T	COCO+SA-1B	45.2

Table 4: In-domain evaluation on COCO. We evaluate all models on COCO val set with the box AP reported.

T-Rex2 VP encoder	AFVPG	COCO Text	COCO Visual-I	COCO Visual-G
✓		49.6	59.2	37.6
	✓	49.8	64.0 ↑4.8	40.3 ↑2.7

Table 5: Ablation study of visual prompt generation. AFVPG indicates Alignment-Friendly Visual Prompt Generation and yields consistent improvements on both Visual-I and Visual-G protocols.

Row	AFVPG	IBP	DMD	COCO Text	COCO Visual-I	COCO Visual-G
0				50.6	-	-
1	✓			49.7	67.0	12.5
2	✓		✓	49.8	63.5	24.7
3	✓	✓		49.6	63.2	37.2
4	✓	✓	✓	49.8	64.0	40.3

Table 6: Ablation study of PET-DINO. IBP and DMD denote Intra-Batch Parallel prompting and Dynamic Memory-Driven prompting, respectively.

(from scratch)	-	O365	62.1	32.7
Method	Training Data		COCO	COCO
Method	MM-Grounding-DINO	PET-DINO	visual-I	visual-G
PET-DINO-T	-	O365	62.1	32.7
PET-DINO-T	O365, GoldG, V3Det	O365	64.0	40.3
(from pre-trained)	O365, GoldG, V3Det	O365	64.0	40.3

Table 7: With or without inheriting the pre-trained model. All models are trained with Swin-T.

Panel of Figure 3 — Figure 3: Comparison between training from scratch and inheriting the pre-trained model. The models are trained on O365 and evaluated on COCO val set.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Dynamic Memory-Driven Prompting Diagram. During each iteration, the Visual Cues Bank updates its stored prompts with visual prompts used by PET-DINO, while PET-DINO utilizes the enriched prompts from the bank to improve training.

Figure 2: Feature correlation analysis between visual prompts and instance-level image features showing the impact of AFVPG.

Figure 3: t-SNE visualization of visual prompt features showing the impact of IBP and DMD.

Figure 4: Zero-shot detection visualizations of PET-DINO on interactive visual prompt-based detection in single -category dense object scenarios.

Figure 5: Zero-shot detection visualizations of PET-DINO on interactive visual prompt-based detection in multi -category dense object scenarios.

Figure 6: Zero-shot detection visualizations of PET-DINO on cross-image exemplar visual prompt-based detection. Exemplars are shown above, and prediction outputs are shown below.

Figure 7: Zero-shot detection visualizations of PET-DINO with class-level generic visual prompts. The visual prompt embeddings are pre-extracted from the training set.

Figure 8: Zero-shot detection visualizations of PET-DINO with text prompts. The category names from the dataset are utilized as textual inputs for prompt generation.

Object Detection

Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.

See all 624 papers

Apr 16, 2026

DETR-ViP: Detection Transformer with Robust Discriminative Visual Prompts

Visual prompted object detection enables interactive and flexible definition of target categories, thereby facilitating open-vocabulary detection. Since visual prompts are derived directly from image features, they often outperform text prompts in recognizing rare categories. Nevertheless, research on visual prompted detection has been largely overlooked, and it is typically treated as a byproduct of training text prompted detectors, which hinders its development. To fully unlock the potential of visual-prompted detection, we investigate the reasons why its performance is suboptimal and reveal that the underlying issue lies in the absence of global discriminability in visual prompts. Motivated by these observations, we propose DETR-ViP, a robust object detection framework that yields class-distinguishable visual prompts. On top of basic image-text contrastive learning, DETR-ViP incorporates global prompt integration and visual-textual prompt relation distillation to learn more discriminative prompt representations. In addition, DETR-ViP employs a selective fusion strategy that ensures stable and robust detection. Extensive experiments on COCO, LVIS, ODinW, and Roboflow100 demonstrate that DETR-ViP achieves substantially higher performance in visual prompt detection compared to other state-of-the-art counterparts. A series of ablation studies and analyses further validate the effectiveness of the proposed improvements and shed light on the underlying reasons for the enhanced detection capability of visual prompts.

Bo Qian, Dahu Shi, Xing Wei

School of Software Engineering, Xi’an Jiaotong University · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University · Work done during the internship at Hikrobot Co., Ltd. +2

Open-Vocabulary Object Detection Detection Transformer Visual Prompt Learning

Ask PDF

May 27, 2026

LV-OSD: Language-Vision-Complementary Open-Set Object Detection

Object detection is an important task in computer vision, which aims to detect the objects of interest. through the given category list or query images. In this work, we propose a new problem of language-visual-complementary open-set object detection (LV-OSD), i.e., using the flexible text-based and/or image-based prompts to specify the desired object categories. This setting is more common and practical in real-world applications. For this purpose, we design a dual-branch detection framework, LVDor, which can simultaneously accept both text and image prompts. Specifically, we first build the Multi-modal Prompts (MPr) containing various text descriptions and image samples for each category. Subsequently, to bridge the semantic gap among the input image, text prompts, and image prompts, we design a Target-guided Prompt Dynamic Weighting (TPDW) module. Guided by the prior information of the target image, this module dynamically produces the text and image prompts that best align with the target semantics, achieving precise alignment and effectively reducing the discrepancy between the two modalities, thereby accommodating the LV-OSD setting. We also propose a simple Prompt Random Masking (PRM) mechanism during training to simulate the arbitrary combination of text and/or image prompts in testing. Extensive experimental results verify our problem formulation's reasonability and our method's effectiveness. Prompts and code will be released publicly.

Yupeng Zhang, Ruize Han, Wei Feng +2

College of Intelligence and Computing, Tianjin University, Tianjin 300350, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen 518107, Guangdong, China

Open-Vocabulary Object Detection Object Detection Vision-Language Alignment Visual Prompt Learning Multimodal Object Detection

Ask PDF

Mar 24, 2026

DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection

Multi-Modal LLMs (MLLMs) demonstrate strong visual grounding capabilities on popular object detection benchmarks like OdinW-13 and RefCOCO. However, state-of-the-art models still struggle to generalize to out-of-distribution classes, tasks and imaging modalities not typically found in their pre-training. While in-context prompting is a common strategy to improve performance across diverse tasks, we find that it often yields lower detection accuracy than prompting with class names alone. This suggests that current MLLMs cannot yet effectively leverage few-shot visual examples and rich textual descriptions for object detection. Since frontier MLLMs are typically only accessible via APIs, and state-of-the-art open-weights models are prohibitively expensive to fine-tune on consumer-grade hardware, we instead explore black-box prompt optimization for few-shot object detection. To this end, we propose Detection Prompt Optimization (DetPO), a gradient-free test-time optimization approach that refines text-only prompts by maximizing detection accuracy on few-shot visual training examples while calibrating prediction confidence. Our proposed approach yields consistent improvements across generalist MLLMs on Roboflow20-VL and LVIS, outperforming prior black-box approaches by up to 9.7 mAP. Our code and optimized prompts are available at https://ggare-cmu.github.io/DetPO/

Gautam Rajendrakumar Gare, Neehar Peri, Matvei Popov +3

Carnegie Mellon University · Roboflow

Large Vision-Language Models Few-Shot Object Detection Few-Shot Prompting Object Detection Prompt Optimization Multimodal Object Detection

Ask PDF

Abstract

Object Detection

Explore similar work

DETR-ViP: Detection Transformer with Robust Discriminative Visual Prompts

LV-OSD: Language-Vision-Complementary Open-Set Object Detection

DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection