Jag: Direct Box Prediction for Efficient Visual Grounding

Xiuyuan Zhu; Ke Lu; Hao Wu; Zijin Du; Dongming Zhang; Jian Xue

cs.CVOct 9, 2026

Jag: Direct Box Prediction for Efficient Visual Grounding

Authors: Xiuyuan Zhu, Ke Lu, Hao Wu, Zijin Du, Dongming Zhang, Jian Xue

Organizations: University of Chinese Academy of Sciences, Beijing, China · State Key Laboratory of Communication Content Cognition, Beijing, China · Peng Cheng Laboratory, Shenzhen, Guangdong, China

Abstract

Visual grounding combines image--language understanding with geometric prediction, yet autoregressive multimodal models often express a box as a coordinate sequence. Sequential decoding adds computation to the multimodal processing needed for localization. Adapting these representations to box geometry offers a route to direct prediction. We present Jag, which adapts a compact pretrained multimodal model for direct continuous box prediction. A lightweight regression head reads the last valid input-token state, conditioned on the image and referring expression, and predicts all four box coordinates together. Geometric supervision jointly adapts the language backbone, visual merger, and regression head, allowing localization losses to shape both the multimodal representations and the coordinate mapping. Inference retains image--language processing and returns the box in one forward pass, without coordinate generation or text parsing. Across benchmarks, Jag consistently improves grounding accuracy over its base model across the observed target sizes and expression groups. Component ablation show that jointly adapting the representations and head substantially improves accuracy over training the head alone. Against existing specialized grounding models, it achieves competitive accuracy with lower single-request latency and GPU memory use. These results show how supervision tailored to the geometric answer can specialize a compact pretrained multimodal model for accurate grounding and efficient direct prediction. Project page: https://xyzzzh.github.io/Jag/

Figures & tables

Method	State source	Box output
NExT-Chat	Generated trigger	Continuous
LocateAnything	Box-aligned blocks	Discrete
Jag	Existing input state	Continuous

Table 1: Box prediction in closely related models ( Zhang et al., 2024 ; Wang et al., 2026 ) . NExT-Chat and Jag use MLP heads for continuous box regression. LocateAnything predicts discrete coordinates in parallel within each box block.

Figure 1: Jag adapts a compact multimodal model for accurate and efficient visual grounding. It retains multimodal input processing and replaces sequential coordinate generation with continuous box prediction in one forward pass. A lightweight regression head reads the last valid input state; geometric supervision jointly adapts the head, language backbone, and visual merger for localization. The resulting task adaptation improves grounding accuracy over the base model, while the direct output reduces inference latency relative to autoregressive localization.

Model	RefCOCO		RefCOCO+		RefCOCOg
Model	testA	testB	testA	testB	test
Base	84.27	74.72	76.53	62.57	77.96
Jag	93.90	88.36	90.69	80.08	87.72
$\Delta$ vs. Base	+9.63	+13.64	+14.16	+17.51	+9.76
NExT-Chat	89.66	77.04	83.76	66.19	79.28
LocateAnything	93.23	89.26	88.00	79.57	88.54

Table 2: Grounding accuracy across five test splits in [email protected] (%). The shaded row highlights Jag;

\Delta

reports its improvement over Base in accuracy points. Bold denotes the highest model accuracy in each column.

Figure 2: Jag achieves the lowest end-to-end latency among the compared models and uses less GPU memory than NExT-Chat, LocateAnything, and Hi-Token. Annotations report its speedup over each reference model; the shaded region denotes combinations with both higher latency and higher memory use than Jag.

Figure 3: Joint adaptation of the language backbone, visual merger, and head outperforms head-only training. Panels compare readout and training variants using [email protected]. Gray points are configuration controls; orange dashed lines show the AR reference.

Figure 4: Increasing the training budget improves Jag on all three datasets. Blue curves show [email protected]; gray dashed lines mark the accuracy of Base. The example and update budgets vary together.

Figure 5: Gains over Base occur across the observed target sizes and expression groups. Paired points show [email protected] and counts refer to expressions. Area is normalized GT box area; spatial tags are lexical proxies. Intervals and empty groups are reported in the appendix.

Figure 6: Individual gains and failures in the Base comparison. Four RefCOCOg examples cover shared success, success only for Jag, success only for Base, and shared failure at IoU 0.5. Cases are selected near each outcome’s median prediction pair.

Dataset	AR	Parallel discrete	Jag
RefCOCO	90.02	73.52	89.57
	$\pm\,0.12$	$\pm\,1.88$	$\pm\,0.07$
RefCOCO+	84.57	68.13	83.22
	$\pm\,0.31$	$\pm\,1.26$	$\pm\,0.17$
RefCOCOg	86.41	68.04	85.12
	$\pm\,0.01$	$\pm\,1.45$	$\pm\,0.40$

Table 3: Output-formulation ablations: [email protected] mean

\pm

sample standard deviation across multiple random seeds. Parallel discrete uses four independent categorical heads.

Metric	RefCOCO	RefCOCO+	RefCOCOg
AR	90.93	85.82	88.48
Jag	91.28	85.80	87.72
$\Delta$	+0.34	-0.02	-0.76
95% CI	[-0.25, +0.93]	[-0.73, +0.70]	[-1.41, -0.13]

Table 4: Full-data comparison with the fine-tuned autoregressive model (AR), in [email protected] (%). Differences and paired 95% intervals are Jag minus AR.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	RefCOCO	RefCOCO+	RefCOCOg
Base	79.74	70.10	77.96
Jag	91.28	85.80	87.72
$\Delta$	+11.53	+15.70	+9.76
95% CI	[+10.62, +12.41]	[+14.67, +16.77]	[+8.86, +10.73]

Appendix

Table 5: Pooled Base comparison in [email protected] (%).

\Delta

is Jag minus Base, with paired image-cluster 95% intervals.

RefCOCO
Split	$N$	Base	Jag	$\Delta$
testA	5,657	84.27	93.90	+9.63
testB	5,095	74.72	88.36	+13.64
RefCOCO+
testA	5,726	76.53	90.69	+14.16
testB	4,889	62.57	80.08	+17.51

Appendix

Table 6: Split accuracies in [email protected] (%) and sample counts for the Base comparison.

\Delta

denotes the absolute accuracy gain.

Outcome	RefCOCO	RefCOCO+	RefCOCOg
Both	8,345	7,063	7,149
Jag only	1,469	2,045	1,274
Base only	229	378	337
Neither	709	1,129	842

Appendix

Table 7: Joint outcomes for the original Base comparison at IoU 0.5. The four outcomes are mutually exclusive and cover each benchmark.

Dataset	Group	$N$	Base	Jag	$\Delta$	95% CI
RefCOCO	Area $<1\%$	0	NA	NA	NA	NA
RefCOCO	Area 1–<10%	2,665	74.90	87.02	+12.12	[+10.00, +14.20]
RefCOCO	Area $\geq$ 10%	8,087	81.34	92.68	+11.34	[+10.31, +12.41]
RefCOCO	$\leq$ 5 words	9,214	81.41	92.51	+11.10	[+10.18, +12.03]
RefCOCO	6–10 words	1,423	70.91	84.82	+13.91	[+11.19, +16.65]
RefCOCO	$\geq$ 11 words	115	55.65	72.17	+16.52	[+8.21, +25.58]

Appendix

Table 8: Base-comparison groups: [email protected] (%), absolute gain over Base, and paired 95% intervals. Empty groups are retained as NA.

Configuration	RefCOCO	RefCOCO+	RefCOCOg
Jag	89.62	83.10	84.84
Text mean	42.76	35.37	70.60
Input box marker	89.47	82.68	85.70
Joint from start	89.15	82.67	85.25
Frozen backbone	25.99	20.35	20.18

Appendix

Table 9: Readout and adaptation accuracies in [email protected] (%), underlying Figure 3 . Training uses the component-ablation budget.

Dataset	AR	Jag
RefCOCO	91.69	91.31
RefCOCO+	87.33	85.77
RefCOCOg	89.12	88.28

Appendix

Table 10: Model-scale ablation in [email protected] (%) using Qwen3.5-2B and the component-ablation budget. The comparison varies model size within the same backbone family.

RefCOCO ( $N=10,752$ )
Model	@.5	@.75	@.9	mAcc
Base	79.74	66.29	43.64	62.63
Jag	91.28	79.06	50.36	72.80
AR	90.93	80.49	61.03	77.04
RefCOCO+ ( $N=10,615$ )
Base	70.10	58.06	38.51	54.86

Appendix

Table 11: Supplemental localization accuracy (%). Columns @.5, @.75, and @.9 denote IoU thresholds. mAcc averages ten thresholds from 0.50 to 0.95 at intervals of 0.05. Each dataset pools all its test expressions; Base denotes Qwen3.5-0.8B; AR denotes its fine-tuned counterpart.

Model	Mean (ms)	P95 (ms)	Rate (samples/s)	Memory (GiB)
Jag	65.97	68.82	15.16	2.523
Base	1293.01	1496.69	0.77	2.246
NExT-Chat	120.01	141.00	8.33	15.986
LocateAnything	239.05	384.76	4.18	9.660
Hi-Token	616.90	635.55	1.62	7.910

Appendix

Table 12: Single-request model-comparison measurements supporting Figure 2 . Latency covers complete end-to-end prediction; rate denotes throughput and memory denotes sampled peak GPU process occupancy.

Metric	AR	Jag
B1 mean (ms)	605.80	68.05
B1 P50 (ms)	604.26	67.37
B1 P95 (ms)	623.98	74.28
B1 (expressions/s)	1.651	14.695
B8 (expressions/s)	9.954	35.244

Appendix

Table 13: Output-formulation runtime ablation. B1 and B8 denote batch sizes one and eight; latency is in ms and throughput in expressions per second.

Visual Grounding

Visual grounding connects language to the image regions, objects, or pixels that support a description, question, or action.

See all 153 papers

IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models
Jul 17, 2026Xiuyuan Zhu, Ke Lu, Hao Wu +4Cross-Modal Knowledge Distillation Multimodal Large Language Models
University of Chinese Academy of Sciences, Beijing, China · State Key Laboratory of Communication Content Cognition, Beijing, China · Peng Cheng Laboratory, Shenzhen, Guangdong, China +1
Groundbench: Multi-Resolution Polygon Grounding Exposes the Geometry Gap in Vision-Language Models
Aug 18, 2026Zhonghan Bian, Zhenran Wang, Jinsong Li +1Visual Grounding VLM Evaluation
Equal contribution.
LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding
May 26, 2026Shihao Wang, Shilong Liu, Yuanguo Kuang +10Efficient VLM Inference Vision-Language Grounding
The Hong Kong Polytechnic University · NVIDIA · Princeton University +2

Jul 17, 2026J/K move · Enter open · S save

IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models

Xiuyuan Zhu, Ke Lu, Hao Wu, Siwen Jiao+3

University of Chinese Academy of Sciences, Beijing, China · State Key Laboratory of Communication Content Cognition, Beijing, China · Peng Cheng Laboratory, Shenzhen, Guangdong, China+1

Cross-Modal Knowledge Distillation Multimodal Large Language Models Vision-Language Grounding Visual Grounding Learning Using Privileged Information

Open paper AskPDF arXiv

Visual grounding with multimodal large language models is commonly formulated as autoregressive coordinate generation, where a model outputs bounding-box coordinates as text given an image and a referring-expression prompt. While this interface is simple and compatible with instruction following, it introduces a mismatch between training and evaluation: training optimizes token-level likelihood over coordinate strings, whereas grounding quality is measured by geometric overlap. We propose IoU-PD, an IoU-aware privileged distillation method for coordinate-generating multimodal large language models. IoU-PD uses ground-truth boxes not only as coordinate targets, but also as privileged training-time guidance. During training, the student receives the original image and prompt, while a frozen teacher receives a box-marked image and an augmented prompt that indicates the marked region. The student is trained with a supervised fine-tuning anchor and a privileged distillation loss whose token weights reflect both geometric importance and teacher reliability. At inference time, IoU-PD requires no box overlay, privileged hint, teacher branch, or additional prediction module. Experiments on standard referring-expression grounding benchmarks show consistent region-level improvements over strong coordinate-generating baselines, demonstrating that ground-truth boxes can provide useful privileged guidance beyond serving as coordinate labels. Project page: https://xyzzzh.github.io/IoU-PD/

Abstract

Visual Grounding

Explore similar work

IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models

Groundbench: Multi-Resolution Polygon Grounding Exposes the Geometry Gap in Vision-Language Models

LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding