Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Grounding

Zilin Du; Haoxin Li; Jianfei Yu; Boyang Li

cs.CVDec 1, 2024

Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Grounding

Authors: Zilin Du, Haoxin Li, Jianfei Yu, Boyang Li

Organizations: Nanyang Technological University Singapore · Nanjing University of Science and Technology China

Abstract

Visual grounding aims to localize the image regions based on a textual query. Given the difficulty of large-scale data curation, we investigate how to effectively learn visual grounding under data-scarce settings in this paper. To address the data scarcity, we propose a novel framework, POBF (Paint Outside the Box and Filter). POBF synthesizes images by inpainting outside the box, tackling a label misalignment issue encountered in previous works. Furthermore, POBF leverages an innovative filtering scheme to select the most effective training data. This scheme combines a hardness score and an overfitting score, balanced by a penalty term. Extensive experiments across four benchmark datasets demonstrate that POBF consistently improves performance, achieving an average gain of 5.83% over the real-data-only method and outperforming leading baselines by 2.29%-3.85% in accuracy. Additionally, we validate the robustness and generalizability of POBF across various generative models, training data sizes, and model architectures.

Figures & tables

Figure 1: Examples of generated images from different methods. X-Paste Zhao et al. (2023) synthesizes unnatural images. Gen2Det Suri et al. (2024) exhibits the label misalignment issue. GeoDiffusion Chen et al. (2024c) either fails to generate the correct object or introduces excessive artifacts. In contrast, our method synthesizes high-quality images with accurately aligned bounding boxes.

Figure 2: Overview of our proposed framework POBF, which consists of four steps: data generation, teacher training, data filtering, and student training. The real data sample (image, caption, box) is denoted as $(I,T,B)$ . $T^{\prime}$ and $I^{\prime}_{j}$ denote the generated caption and image, respectively. $\mathcal{G}_{T}$ and $\mathcal{G}_{I}$ are generative models. $\mathcal{T}$ and $\mathcal{S}$ refer to the teacher and student model, respectively.

Figure 3: Illustration of the inputs used to compute each score for a generated image. Here, $B$ and $\hat{B}$ denote the ground truth and predicted bounding box, respectively.

Dataset	Images	Texts	Objects
RefCOCO Yu et al. (2016)	19,994	142,210	50,000
RefCOCO+ Yu et al. (2016)	19,992	141,564	49,856
RefCOCOg Mao et al. (2016) ; Nagaraja et al. (2016)	25,799	95,010	49,822
ReferItGame Kazemzadeh et al. (2014)	19,894	130,525	96,654

Table 1: Statistics of visual grounding datasets.

Method	Dataset		RefCOCO		RefCOCO+		RefCOCOg	ReferIt	Avg
Method	Real	Syn	TestA	TestB	TestA	TestB	Test	Test	Avg
Results on One Sampled Dataset
Real	✓		44.26	34.34	23.58	17.70	24.87	21.87	27.78
GeoDiffusion Chen et al. (2024c)	✓	✓	47.22	34.90	28.01	18.83	25.39	24.20	29.76 +1.98
Gen2Det Suri et al. (2024)	✓	✓	46.61	35.19	28.85	20.65	26.10	25.00	30.40 +2.62
X-Paste Zhao et al. (2023)	✓	✓	47.85	39.01	27.59	21.25	27.08	25.16	31.32 +3.54

Table 2: Performance comparison with leading baselines under the data-scarce visual grounding setting. All methods employ the proposed filtering scheme.

Training Data

Filtering

RefCOCO

RefCOCO+

RefCOCOg

ReferIt

Avg

Real

\rm Syn_{img}

\rm Syn_{txt}

S_{1}

S_{2}

P

TestA

TestB

TestA

TestB

Test

the Effectiveness of the Synthetic Data

✓

44.26

34.34

23.58

17.70

24.87

21.87

27.78

✓

49.97

39.16

29.01

20.56

26.67

22.49

31.31 +3.53

✓

47.00

36.19

25.36

18.74

26.33

22.13

29.29 +1.51

the Effectiveness of the Filtering Scheme

Table 3: Ablations on the synthetic data and filtering scheme.

\rm Syn_{img}

and

\rm Syn_{txt}

denote synthetic images and texts, respectively.

Filter	RefCOCO		RefCOCO+		Avg
Filter	TestA	TestB	TestA	TestB	Avg
Random teacher	48.52	38.84	27.47	20.27	33.76
No teacher	48.45	38.68	28.40	20.29	33.96 +0.20
CLIP teacher Chen et al. (2024a)	48.84	39.75	28.07	20.35	34.25 +0.49
Moderate-Loss Xia et al. (2022)	48.86	39.17	27.01	21.17	34.05 +0.29
Moderate-DS Xia et al. (2022)	49.61	40.47	28.21	19.84	34.31 +0.55

Table 4: Results obtained by isolating scores without generated text. Results with a gray background indicate performance lower than the Random teacher baseline.

Figure 4: Scatter plot illustrating the relationship between unnormalized hardness score $S_{1}$ and overfitting score $S_{2}$ across different datasets. Each point presents a generated sample.

	RefCOCO+		RefCOCOg	Avg
Method	TestA	TestB	Test	Avg
Real	23.6%	17.7%	24.9%	22.1%
Ours-top1	31.4%	21.3%	30.0%	27.6%
Ours-top3	30.9%	21.1%	28.9%	27.0%
Ours-top5	29.8%	20.4%	28.3%	26.2%

Table 5: Performance of different filtering weight configurations on RefCOCO+ and RefCOCOg. Ours-top

k

refers to the best

k

filtering-weight configurations selected on the validation set.

	RefCOCO+		RefCOCOg	Avg
Method	TestA	TestB	Test	Avg
Real	23.58	17.70	24.87	22.05
POBF (Ours)	31.37	21.25	29.98	27.53 +5.48
Replace the Image Captioner with $\dots$
LLAVA Liu et al. (2024)	31.09	21.54	28.57	27.07 +5.02
ViT-GPT2 2	29.95	21.22	29.90	27.02 +4.97

Table 6: Performance comparison across generative models.

	RefCOCO+		RefCOCOg	Avg
Method	TestA	TestB	Test	Avg
0.5% Training Data
Real	16.42	10.07	15.43	13.97
POBF (Ours)	21.41	15.87	20.28	19.18 +5.22
2.0% Training Data
Real	33.04	25.03	33.28	30.45

Table 7: Performance comparison across training data sizes.

	RefCOCO+		RefCOCOg	Avg
Method	TestA	TestB	Test	Avg
TransVG Deng et al. (2021) (#Paras = 149M)
Real	23.58	17.70	24.87	22.05
POBF (Ours)	31.37	21.25	29.98	27.53 +5.48
Dynamic-MDTRE Shi et al. (2023) (#Paras = 160M)
Real	23.82	18.55	29.07	23.81

Table 8: Performance comparison across model architectures. #Paras denotes the number of trainable model parameters.

	RefCOCO+		RefCOCOg	Avg
Teacher	TestA	TestB	Test	Avg
No	29.10	20.10	28.31	25.83
TransVG	31.37	21.25	29.98	27.53
QRNet	31.65	21.18	30.42	27.75
Dynamic-MDTRE	32.17	22.08	30.97	28.41

Table 9: Robustness analysis with different teacher architectures. The student model is fixed as TransVG.

Method	RefCOCO TestA			RefCOCO TestB			RefCOCOg
Method	SMALL	MEDIUM	LARGE	SMALL	MEDIUM	LARGE	SMALL	MEDIUM	LARGE
Real-only	40.23	44.57	47.98	30.57	34.61	37.84	21.71	24.82	28.08
POBF	46.52	53.89	57.90	34.08	41.13	47.67	25.30	29.97	34.67
Gain	+6.29	+9.32	+9.92	+3.51	+6.52	+9.83	+3.59	+5.15	+6.59

Table 10: Performance stratified by target bounding-box size. For each dataset, samples are divided into SMALL, MEDIUM, and LARGE groups according to the relative bounding-box area.

Method	RefCOCO		RefCOCO+		RefCOCOg	ReferIt	Avg
Method	TestA	TestB	TestA	TestB	Test	Test	Avg
Real	44.26	34.34	23.58	17.70	24.87	21.87	27.78
Color Jitter	45.12	34.71	24.75	18.03	25.11	22.36	28.35 +0.57
Random Erasing	44.82	34.13	24.22	17.96	24.94	22.13	28.03 +0.25
POBF (Ours)	52.77	40.96	31.37	21.25	29.98	25.28	33.61 +5.83

Table 11: Comparison with conventional data augmentation methods under the data-scarce visual grounding setting.

Figure 5: Qualitative examples illustrating the effectiveness of the two proposed scores. The green score indicates the synthetic image with the higher corresponding score.

Vision-Language Grounding

Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.

See all 418 papers

IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models
Jul 17, 2026Xiuyuan Zhu, Ke Lu, Hao Wu +4Cross-Modal Knowledge Distillation Multimodal Large Language Models
University of Chinese Academy of Sciences, Beijing, China · State Key Laboratory of Communication Content Cognition, Beijing, China · Peng Cheng Laboratory, Shenzhen, Guangdong, China +1
Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding
Jun 15, 2026Yifan Wang, Peiming Li, Shiyu Li +5Efficient VLM Inference Large Vision-Language Models
1Tencent BAC · School of Electronic and Computer Engineering, Peking University · School of Mathematics and Statistics, University of Glasgow +1
Pro-Bench: Prompt-Robust Open-Vocabulary Visual Grounding Across Real-World Heterogeneous Environments
Sep 22, 2026Linus Nwankwo, Muslim Alaran, Christian Rauch +2Vision-Language Grounding Visual Grounding
Chair of Cyber-Physical Systems, Montanuniversität Leoben, Austria. · Department of Design Engineering and Mathematics, Middlesex University, London, UK.

Jul 17, 2026J/K move · Enter open · S save

IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models

Xiuyuan Zhu, Ke Lu, Hao Wu, Siwen Jiao+3

University of Chinese Academy of Sciences, Beijing, China · State Key Laboratory of Communication Content Cognition, Beijing, China · Peng Cheng Laboratory, Shenzhen, Guangdong, China+1

Cross-Modal Knowledge Distillation Multimodal Large Language Models Vision-Language Grounding Visual Grounding Learning Using Privileged Information

Open paper AskPDF arXiv

Visual grounding with multimodal large language models is commonly formulated as autoregressive coordinate generation, where a model outputs bounding-box coordinates as text given an image and a referring-expression prompt. While this interface is simple and compatible with instruction following, it introduces a mismatch between training and evaluation: training optimizes token-level likelihood over coordinate strings, whereas grounding quality is measured by geometric overlap. We propose IoU-PD, an IoU-aware privileged distillation method for coordinate-generating multimodal large language models. IoU-PD uses ground-truth boxes not only as coordinate targets, but also as privileged training-time guidance. During training, the student receives the original image and prompt, while a frozen teacher receives a box-marked image and an augmented prompt that indicates the marked region. The student is trained with a supervised fine-tuning anchor and a privileged distillation loss whose token weights reflect both geometric importance and teacher reliability. At inference time, IoU-PD requires no box overlay, privileged hint, teacher branch, or additional prediction module. Experiments on standard referring-expression grounding benchmarks show consistent region-level improvements over strong coordinate-generating baselines, demonstrating that ground-truth boxes can provide useful privileged guidance beyond serving as coordinate labels. Project page: https://xyzzzh.github.io/IoU-PD/

Abstract

Vision-Language Grounding

Explore similar work

IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models

Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding

Pro-Bench: Prompt-Robust Open-Vocabulary Visual Grounding Across Real-World Heterogeneous Environments