PET-DINO: Unifying Visual Cues into Grounding DINO with Prompt-Enriched Training

Weifu Fu; Jinyang Li; Bin-Bin Gao; Jialin Li; Yuhuan Lin; Hanqiu Deng; Wenbing Tao; Yong Liu

cs.CVApr 1, 2026

PET-DINO: Unifying Visual Cues into Grounding DINO with Prompt-Enriched Training

Authors: Weifu Fu, Jinyang Li, Bin-Bin Gao, Jialin Li, Yuhuan Lin, Hanqiu Deng, Wenbing Tao, Yong Liu,

Organizations: YouTu Lab, Tencent · Huazhong University of Science and Technology · Kling Team, Kuaishou Technology

Abstract

Open-Set Object Detection (OSOD) enables recognition of novel categories beyond fixed classes but faces challenges in aligning text representations with complex visual concepts and the scarcity of image-text pairs for rare categories. This results in suboptimal performance in specialized domains or with complex objects. Recent visual-prompted methods partially address these issues but often involve complex multi-modal designs and multi-stage optimizations, prolonging the development cycle. Additionally, effective training strategies for data-driven OSOD models remain largely unexplored. To address these challenges, we propose PET-DINO, a universal detector supporting both text and visual prompts. Our Alignment-Friendly Visual Prompt Generation (AFVPG) module builds upon an advanced text-prompted detector, addressing the limitations of text representation guidance and reducing the development cycle. We introduce two prompt-enriched training strategies: Intra-Batch Parallel Prompting (IBP) at the iteration level and Dynamic Memory-Driven Prompting (DMD) at the overall training level. These strategies enable simultaneous modeling of multiple prompt routes, facilitating parallel alignment with diverse real-world usage scenarios. Comprehensive experiments demonstrate that PET-DINO exhibits competitive zero-shot object detection capabilities across various prompt-based detection protocols. These strengths can be attributed to inheritance-based philosophy and prompt-enriched training strategies, which play a critical role in building an effective generic object detector. Project page: https://fuweifuvtoo.github.io/pet-dino.

Figures & tables

Figure 1: Overall architecture of PET-DINO. Input coordinates undergo a Visual Prompt Generation process, interacting with enhanced image features to obtain visual prompts. The text encoder creates text embeddings, which interact with image features in the Feature Enhancer module to produce text prompts. Both types of prompts guide the Query Selection Module, offering location priors for initial queries. These queries are then refined through decoder layers to predict objects and classifications.

Figure 2: Intra-Batch Parallel Prompting Diagram. Image and their target coordinates within a batch are processed to generate image-level visual prompts. We incorporate visual prompts from other samples as additional prompts for the current image, aggregating those from the same category to form class-level visual prompts.

Method	Open	Prompt	Backbone	VLM sup.	Data	Training Data	COCO	LVIS	ODinW35
Method	Source	Type	Backbone	VLM sup.	Size	Training Data	COCO	LVIS	ODinW35
T-Rex2 [ 10 ]	N	Visual-I	Swin-T	CLIP	3.1M	O365, OpenImages, HierText, CrowdHuman, SA-1B	56.6	59.3	37.7
CP-DETR-T [ 1 ]	N	Visual-I	Swin-T	CLIP	3.3M	O365, OpenImages, GoldG, V3Det	61.8	64.1	41.0
PET-DINO	Y	Visual-I	Swin-T	-	0.6M	O365	64.0	61.8	38.8
PET-DINO	Y	Visual-I	Swin-T	-	2.5M	O365, OpenImages, V3Det	64.3	64.5	48.3
T-Rex2 [ 10 ]	N	Visual-I	Swin-L	CLIP	3.1M	O365, OpenImages, HierText, CrowdHuman, SA-1B	58.5	62.5	39.7

Table 1: Comparison for zero-shot interactive object detection through visual prompts. Fixed AP is reported on LVIS minival set. Bold and underline denote the best and second-best results in each column, respectively. * means the training set of the evaluation dataset was used for pre-training with text prompts, so this model is not in a zero-shot setting.

Method	Open	Prompt	Backbone	VLM sup.	Data	Training Data	COCO	LVIS	ODinW35
Method	Source	Type	Backbone	VLM sup.	Size	Training Data	COCO	LVIS	ODinW35
T-Rex2 [ 10 ]	N	Visual-G	Swin-T	CLIP	3.1M	O365, OpenImages, HierText, CrowdHuman, SA-1B	38.8	37.4	23.6
YOLOE-v11-S [ 27 ]	Y	Visual-G	YOLO-v11-S	MobileCLIP	1.4M	O365, GoldG	-	26.2	-
PET-DINO	Y	Visual-G	Swin-T	-	0.6M	O365	40.3	29.6	20.4
PET-DINO	Y	Visual-G	Swin-T	-	2.5M	O365, OpenImages, V3Det	38.4	31.5	25.5
T-Rex2 [ 10 ]	N	Visual-G	Swin-L	CLIP	3.1M	O365, OpenImages, HierText, CrowdHuman, SA-1B	46.5	47.6	27.8

Table 2: Comparison for zero-shot generic object detection through visual prompts. Fixed AP is reported on LVIS minival set. Bold and underline denote the best and second-best results in each column, respectively.

Method	Prompt	Backbone	Training Data	COCO	LVIS	ODinW35
Method	Type	Backbone	Training Data	COCO	LVIS	ODinW35
Grounding DINO [ 20 ]	Text	Swin-T	O365, GoldG, Cap4M	48.4	27.4	22.3
MM-Grounding-DINO [ 36 ]	Text	Swin-T	O365, GoldG, V3Det	50.6	40.5	21.4
PET-DINO	Text	Swin-T	O365 †	49.8 ↓0.8	37.8 ↓2.7	20.6 ↓0.8
Grounding DINO [ 20 ]	Text	Swin-L	O365, OpenImages, GoldG	52.5	33.9	26.1
MM-Grounding-DINO [ 36 ]	Text	Swin-L	O365V2, OpenImage, GoldG	53.0	36.7	23.9

Table 3: Comparison for zero-shot generic object detection through text prompts. Fixed AP is reported on LVIS minival set. † means that a small number of iterations on Object365 are not for dedicated text-prompt training, but specifically to prevent the degradation of the inherited text-prompt performance.

Method	Prompt	Backbone	Training Data	COCO
Method	Type	Backbone	Training Data	COCO
X-Decoder T [ 39 ]	Text	Focal-T	COCO+CC3M+…	43.6
OpenSeed T [ 34 ]	Text	Swin-T	COCO+O365	52.0
OpenSeed L [ 34 ]	Text	Swin-L	COCO+O365	58.2
MI Grounding-S [ 35 ]	Image	ViT-L	COCO+LVIS	54.7
DINOv [ 14 ]	Visual-G	Swin-T	COCO+SA-1B	45.2

Table 4: In-domain evaluation on COCO. We evaluate all models on COCO val set with the box AP reported.

T-Rex2 VP encoder	AFVPG	COCO Text	COCO Visual-I	COCO Visual-G
✓		49.6	59.2	37.6
	✓	49.8	64.0 ↑4.8	40.3 ↑2.7

Table 5: Ablation study of visual prompt generation. AFVPG indicates Alignment-Friendly Visual Prompt Generation and yields consistent improvements on both Visual-I and Visual-G protocols.

Row	AFVPG	IBP	DMD	COCO Text	COCO Visual-I	COCO Visual-G
0				50.6	-	-
1	✓			49.7	67.0	12.5
2	✓		✓	49.8	63.5	24.7
3	✓	✓		49.6	63.2	37.2
4	✓	✓	✓	49.8	64.0	40.3

Table 6: Ablation study of PET-DINO. IBP and DMD denote Intra-Batch Parallel prompting and Dynamic Memory-Driven prompting, respectively.

(from scratch)	-	O365	62.1	32.7
Method	Training Data		COCO	COCO
Method	MM-Grounding-DINO	PET-DINO	visual-I	visual-G
PET-DINO-T	-	O365	62.1	32.7
PET-DINO-T	O365, GoldG, V3Det	O365	64.0	40.3
(from pre-trained)	O365, GoldG, V3Det	O365	64.0	40.3

Table 7: With or without inheriting the pre-trained model. All models are trained with Swin-T.

Panel of Figure 3 — Figure 3: Comparison between training from scratch and inheriting the pre-trained model. The models are trained on O365 and evaluated on COCO val set.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Dynamic Memory-Driven Prompting Diagram. During each iteration, the Visual Cues Bank updates its stored prompts with visual prompts used by PET-DINO, while PET-DINO utilizes the enriched prompts from the bank to improve training.

Figure 2: Feature correlation analysis between visual prompts and instance-level image features showing the impact of AFVPG.

Figure 3: t-SNE visualization of visual prompt features showing the impact of IBP and DMD.

Figure 4: Zero-shot detection visualizations of PET-DINO on interactive visual prompt-based detection in single -category dense object scenarios.

Figure 5: Zero-shot detection visualizations of PET-DINO on interactive visual prompt-based detection in multi -category dense object scenarios.

Figure 6: Zero-shot detection visualizations of PET-DINO on cross-image exemplar visual prompt-based detection. Exemplars are shown above, and prediction outputs are shown below.

Figure 7: Zero-shot detection visualizations of PET-DINO with class-level generic visual prompts. The visual prompt embeddings are pre-extracted from the training set.

Figure 8: Zero-shot detection visualizations of PET-DINO with text prompts. The category names from the dataset are utilized as textual inputs for prompt generation.

Object Detection

Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.

See all 624 papers

DETR-ViP: Detection Transformer with Robust Discriminative Visual Prompts
Apr 16, 2026Bo Qian, Dahu Shi, Xing WeiOpen-Vocabulary Object Detection Detection Transformer
School of Software Engineering, Xi’an Jiaotong University · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University · Work done during the internship at Hikrobot Co., Ltd. +2
LV-OSD: Language-Vision-Complementary Open-Set Object Detection
May 27, 2026Yupeng Zhang, Ruize Han, Wei Feng +2Open-Vocabulary Object Detection Object Detection
College of Intelligence and Computing, Tianjin University, Tianjin 300350, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen 518107, Guangdong, China
DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection
Mar 24, 2026Gautam Rajendrakumar Gare, Neehar Peri, Matvei Popov +3Large Vision-Language Models Few-Shot Object Detection
Carnegie Mellon University · Roboflow

Apr 16, 2026J/K move · Enter open · S save

DETR-ViP: Detection Transformer with Robust Discriminative Visual Prompts

Bo Qian, Dahu Shi, Xing Wei

School of Software Engineering, Xi’an Jiaotong University · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University · Work done during the internship at Hikrobot Co., Ltd.+2

Open-Vocabulary Object Detection Detection Transformer Visual Prompt Learning

Open paper AskPDF arXiv

Visual prompted object detection enables interactive and flexible definition of target categories, thereby facilitating open-vocabulary detection. Since visual prompts are derived directly from image features, they often outperform text prompts in recognizing rare categories. Nevertheless, research on visual prompted detection has been largely overlooked, and it is typically treated as a byproduct of training text prompted detectors, which hinders its development. To fully unlock the potential of visual-prompted detection, we investigate the reasons why its performance is suboptimal and reveal that the underlying issue lies in the absence of global discriminability in visual prompts. Motivated by these observations, we propose DETR-ViP, a robust object detection framework that yields class-distinguishable visual prompts. On top of basic image-text contrastive learning, DETR-ViP incorporates global prompt integration and visual-textual prompt relation distillation to learn more discriminative prompt representations. In addition, DETR-ViP employs a selective fusion strategy that ensures stable and robust detection. Extensive experiments on COCO, LVIS, ODinW, and Roboflow100 demonstrate that DETR-ViP achieves substantially higher performance in visual prompt detection compared to other state-of-the-art counterparts. A series of ablation studies and analyses further validate the effectiveness of the proposed improvements and shed light on the underlying reasons for the enhanced detection capability of visual prompts.

Abstract

Object Detection

Explore similar work

DETR-ViP: Detection Transformer with Robust Discriminative Visual Prompts

LV-OSD: Language-Vision-Complementary Open-Set Object Detection

DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection