Heuristic-inspired Reasoning Priors Facilitate Data-Efficient Referring Object Detection

Xu Zhang; Zhe Chen; Jing Zhang; Dacheng Tao

cs.CVMar 25, 2026

Heuristic-inspired Reasoning Priors Facilitate Data-Efficient Referring Object Detection

Authors: Xu Zhang, Zhe Chen, Jing Zhang, Dacheng Tao

Organizations: The University of Sydney, Australia · La Trobe University, Australia · Wuhan University, China · Nanyang Technological University, Singapore

Abstract

Most referring object detection (ROD) models, especially the modern grounding detectors, are designed for data-rich conditions, yet many practical deployments, such as robotics, augmented reality, and other specialized domains, would face severe label scarcity. In such regimes, end-to-end grounding detectors need to learn spatial and semantic structure from scratch, wasting precious samples. We ask a simple question: Can explicit reasoning priors help models learn more efficiently when data is scarce? To explore this, we first introduce a Data-efficient Referring Object Detection (De-ROD) task, which is a benchmark protocol for measuring ROD performance in low-data and few-shot settings. We then propose the HeROD (Heuristic-inspired ROD), a lightweight, model-agnostic framework that injects explicit, heuristic-inspired spatial and semantic reasoning priors, which are interpretable signals derived based on the referring phrase, into 3 stages of a modern DETR-style pipeline: proposal ranking, prediction fusion, and Hungarian matching. By biasing both training and inference toward plausible candidates, these priors promise to improve label efficiency and convergence performance. On RefCOCO, RefCOCO+, and RefCOCOg, HeROD consistently outperforms strong grounding baselines in scarce-label regimes. More broadly, our results suggest that integrating simple, interpretable reasoning priors provides a practical and extensible path toward better data-efficient vision-language understanding.

Figures & tables

Figure 1: Conceptual illustration. (a) Without priors, grounding detectors trained with limited labels often ignore spatial/semantic cues (e.g., “on the left”). (b) HeROD injects heuristic-inspired reasoning priors (spatial + semantic), guiding the detector toward correct localization and improving data-efficient learning.

Figure 2: HeROD pipeline. Text and image features are encoded, and spatial/semantic reasoning priors are derived from the phrase and image. These priors are injected into reference generation, final prediction, and training loss, guiding the detector toward plausible regions for more data-efficient learning.

RefCOCO
	0.1% Data			0.2% Data			0.5% Data			1% Data			2% Data			5% Data
Method	val	testA	testB	val	testA	testB	val	testA	testB	val	testA	testB	val	testA	testB	val	testA	testB
UNINEXT	17.75	22.03	14.27	21.53	23.23	16.80	24.05	27.24	18.61	31.04	34.35	27.77	46.24	49.67	39.73	54.30	57.24	51.40
HeROD-U (ours)	25.60	33.20	18.47	31.59	36.45	25.26	35.75	41.24	32.11	47.89	53.51	40.12	59.45	65.21	52.99	65.33	70.53	58.74
$\Delta$	+7.85	+11.17	+4.20	+10.06	+13.22	+8.46	+11.70	+14.00	+13.5	+16.85	+19.16	+12.35	+13.21	+15.54	+13.26	+11.03	+13.29	+7.34
Grounding DINO	57.93	65.64	50.26	59.17	67.72	50.72	60.97	69.40	52.05	63.66	72.41	56.70	66.60	74.69	58.37	79.32	84.37	73.50

Table 1: Top-1 accuracy comparison on the RefCOCO/+/g datasets for the low-data ROD.

\Delta

denotes the gain of HeROD over its baseline.

Method	Support Data	Support Set Training			2k Finetune			1k Finetune			0.5k Finetune
Method	Support Data	val	testA	testB	val	testA	testB	val	testA	testB	val	testA	testB
Grounding DINO	2k	61.18	70.66	52.23	67.23	72.30	63.85	62.98	69.56	58.43	62.08	68.84	56.84
HeROD-G (ours)	2k	61.18	70.66	52.23	78.17 +10.94	80.50 +8.20	75.62 +11.77	76.90 +13.92	80.15 +10.59	74.31 +15.88	75.08 +13.00	78.75 +9.91	71.03 +14.19
Grounding DINO	1k	59.28	68.62	50.52	64.74	69.91	61.02	61.81	67.93	56.86	60.65	67.74	54.78
HeROD-G (ours)	1k	59.28	68.62	50.52	76.81 +12.07	79.51 +9.60	75.39 +14.37	75.56 +13.75	78.27 +10.34	73.54 +16.68	72.85 +12.20	76.91 +9.17	70.30 +15.52

Table 2: Top-1 accuracy comparison on the RefCOCO dataset for the few-shot ROD.

Reference Gen	Final Prediction	Learning Proc	Performance
w/o $\mathcal{H}$	Original	w/o $\mathcal{H}$	63.66
w/o $\mathcal{H}$	Heuristic (static)	w/ $\mathcal{H}$	64.78
w/o $\mathcal{H}$	Heuristic (adaptive)	w/ $\mathcal{H}$	71.33
w/ $\mathcal{H}$	Heuristic (adaptive)	w/ $\mathcal{H}$	77.91

Table 3: Ablation studies on the RefCOCO dataset for HeROD.

\mathcal{H}

represents the heuristics. ‘Reference Gen’ and ‘Learning Proc’ represent Object Reference Generation and Learning Process.

Reference Generation		Final Prediction		Performance
$H_{s}$	$H_{v}$	$H_{s}$	$H_{v}$	Performance
✓	✓	✓		76.36
✓	✓		✓	74.93
✓		✓	✓	76.02
	✓	✓	✓	72.43
✓	✓	✓	✓	77.91

Table 4: Ablation studies on the RefCOCO dataset for

H_{s}

H_{v}

in different stages of HeROD.

Method	Fine-tuning	RefCOCO			RefCOCO+			RefCOCOg
Method	Fine-tuning	val	testA	testB	val	testA	testB	val	test
MAttNet [ 50 ]	w/	76.65	81.14	69.99	65.33	71.62	56.02	66.58	67.27
VGTR [ 9 ]	w/	79.20	82.32	73.78	63.91	70.09	56.51	65.73	67.23
TransVG [ 7 ]	w/	81.02	82.72	78.35	64.82	70.70	56.94	68.67	67.73
VILLA-L [ 10 ]	w/	82.39	87.48	74.84	76.17	81.54	66.84	76.18	76.71
RefTR [ 22 ]	w/	85.65	88.73	81.16	77.55	82.26	68.99	79.25	80.01

Table 5: Top-1 accuracy comparison for the data-rich setting. * represents the reproduced results using only the ROD datasets with the official code.

Fusion of CLIPSeg Embedding	Performance
fuse w/o HeROD pipeline	63.99
fuse w HeROD pipeline	77.91

Table 6: Impact of CLIPSeg embedding on the RefCOCO dataset.

$H_{s}$	$H_{v}$	Performance
✓	✓	77.91
✓	x	73.53
x	x	63.66

Table 7: Overall impact of different heuristic-inspired reasoning priors.

$L_{conf}:L_{cls}$	Performance
0.5	77.18
1.0	77.91
2.0	77.53

Table 8: Ablation studies on loss weights selection.

method	0.1% data	2% data
Grounding DINO	60.57	68.64
HeROD	74.83	82.42

Table 9: Results on the cleaned RefCOCO datasets.

Figure 3: Visualization of four Spatial Heuristic-inspired Scoring Maps: left-related, top-related, right-related, and bottom-related (listed from the first column to the second column, from the first row to the second row). Brighter colors are used to indicate higher values, while darker colors represent lower values.

Figure 4: Visualization of performance comparison across varying data volumes between HeROD-U and the baseline UNINEXT.

Figure 5: Visualization of performance comparison across varying data volumes between HeROD-G and the baseline Grounding DINO.

Figure 6: Predictions illustration of the pre-trained Grounding DINO. The ground truth bounding boxes are indicated by dashed red lines. It suggests that the vanilla grounding detection may overlook the contextual information in the text descriptions without heavy finetuning.

Figure 7: Qualitative examples. The top row shows correct prediction results by HeROD, the bottom row shows incorrect prediction results by the baseline. The responding referring expressions are illustrated below the figure.

Method	RefCOCO			RefCOCO+			RefCOCOg
Method	val	testA	testB	val	testA	testB	val	test
Global-Local CLIP [ 51 ]	26.70	24.99	26.48	28.22	26.54	27.86	33.02	33.12
TAS [ 39 ]	39.91	42.85	35.85	43.99	50.58	36.44	47.68	47.41
IteRPrimE* [ 42 ]	40.20	46.50	33.90	44.20	51.60	35.30	46.00	45.80
HeROD-G (ours)	78.21	80.16	74.30	71.52	76.86	63.86	72.52	72.39

Table 10: Referring image segmentation results. * denotes that the paper’s reported results were kept to one decimal place, and we directly replicated them here.

Vision-Language Grounding

Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.

See all 418 papers

Where Grounding Accuracy Lives on the IoU Curve: Label-Free Inference-Time Boundary Refinement
Aug 20, 2026Bo MaVision-Language Grounding Visual Grounding
Auckland University of Technology
STORM: End-to-End Referring Multi-Object Tracking in Videos
Apr 12, 2026Zijia Lu, Jingru Yi, Jue Wang +4Referring MOT Multi-Object Tracking
Amazon
Generalized Small Object Detection:A Point-Prompted Paradigm and Benchmark
Apr 3, 2026Haoran Zhu, Wen Yang, Guangyou Yang +5Small Object Detection Object Detection
School of Electronic Information, Wuhan University, Wuhan, 430072 China. · Environmental Computational Science and Earth Observation Laboratory, EPFL, Sion, Switzerland. · School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China.

Aug 20, 2026J/K move · Enter open · S save

Where Grounding Accuracy Lives on the IoU Curve: Label-Free Inference-Time Boundary Refinement

Bo Ma

Auckland University of Technology

Vision-Language Grounding Visual Grounding Referring Expression Comprehension

Open paper AskPDF arXiv

Vision--language models can identify the correct referent while returning an imprecise bounding box. We study whether a frozen direct-answer model can use its own prediction to allocate one additional localized observation without accessing target annotations at inference. Label-free precision refinement (LFPR) routes predicted-small regions to a higher-resolution pass, re-grounds the expression inside a context crop, admits a candidate only under fixed geometric guards, and returns a fixed coordinate-wise midpoint. We report results across three evidence tiers. On 31,921 retrospective Ref-L4 expressions, LFPR raises mAcc

_{0.5:0.95}

from 72.947% to 76.013% ([email protected] 88.531%

\to

89.725%, [email protected] 55.788%

\to

61.142%). A frozen transfer to 30,969 RefCOCO/RefCOCO+/RefCOCOg expressions improves every dataset at [email protected], mAcc, and mean IoU (pooled mAcc

+0.645

, [email protected]

+0.817

), while [email protected] is unchanged overall: routing alone gains

+1.162

points there, but crop, guards, and fusion give back

-1.192

, offsetting rather than showing no strict-IoU effect. A prospective, image-disjoint Flickr30K Entities evaluation improves every endpoint (mAcc

+0.973

, [email protected]

+1.022

), more strongly under a single-box variant (mAcc

+2.575

, [email protected]

+3.689

). The same operator applied to two released grounding specialists improves every endpoint ([email protected]

+1.569

+6.716

for EGM-4B/8B) at roughly twice the latency, composing with specialist training rather than replacing it. A genuine unguarded control (guard removed from the same candidates) underperforms the incumbent on every metric, showing the guard is load-bearing. Together, these results show that referent selection and boundary precision are partially separable, with different components moving opposing regions of the IoU curve -- behavior a single threshold cannot reveal.

Abstract

Vision-Language Grounding

Explore similar work

Where Grounding Accuracy Lives on the IoU Curve: Label-Free Inference-Time Boundary Refinement

STORM: End-to-End Referring Multi-Object Tracking in Videos

Generalized Small Object Detection:A Point-Prompted Paradigm and Benchmark