Jag: Direct Box Prediction for Efficient Visual Grounding
Authors: Xiuyuan Zhu, Ke Lu, Hao Wu, Zijin Du, Dongming Zhang, Jian Xue
Organizations: University of Chinese Academy of Sciences, Beijing, China · State Key Laboratory of Communication Content Cognition, Beijing, China · Peng Cheng Laboratory, Shenzhen, Guangdong, China
Visual grounding combines image--language understanding with geometric prediction, yet autoregressive multimodal models often express a box as a coordinate sequence. Sequential decoding adds computation to the multimodal processing needed for localization. Adapting these representations to box geometry offers a route to direct prediction. We present Jag, which adapts a compact pretrained multimodal model for direct continuous box prediction. A lightweight regression head reads the last valid input-token state, conditioned on the image and referring expression, and predicts all four box coordinates together. Geometric supervision jointly adapts the language backbone, visual merger, and regression head, allowing localization losses to shape both the multimodal representations and the coordinate mapping. Inference retains image--language processing and returns the box in one forward pass, without coordinate generation or text parsing. Across benchmarks, Jag consistently improves grounding accuracy over its base model across the observed target sizes and expression groups. Component ablation show that jointly adapting the representations and head substantially improves accuracy over training the head alone. Against existing specialized grounding models, it achieves competitive accuracy with lower single-request latency and GPU memory use. These results show how supervision tailored to the geometric answer can specialize a compact pretrained multimodal model for accurate grounding and efficient direct prediction. Project page: https://xyzzzh.github.io/Jag/
Figures & tables
Method
State source
Box output
NExT-Chat
Generated trigger
Continuous
LocateAnything
Box-aligned blocks
Discrete
Jag
Existing input state
Continuous
Table 1: Box prediction in closely related models ( Zhang et al., 2024 ; Wang et al., 2026 ) . NExT-Chat and Jag use MLP heads for continuous box regression. LocateAnything predicts discrete coordinates in parallel within each box block.
Figure 1: Jag adapts a compact multimodal model for accurate and efficient visual grounding. It retains multimodal input processing and replaces sequential coordinate generation with continuous box prediction in one forward pass. A lightweight regression head reads the last valid input state; geometric supervision jointly adapts the head, language backbone, and visual merger for localization. The resulting task adaptation improves grounding accuracy over the base model, while the direct output reduces inference latency relative to autoregressive localization.
Model
RefCOCO
RefCOCO+
RefCOCOg
testA
testB
testA
testB
test
Base
84.27
74.72
76.53
62.57
77.96
Jag
93.90
88.36
90.69
80.08
87.72
Δ vs. Base
+9.63
+13.64
+14.16
+17.51
+9.76
NExT-Chat
89.66
77.04
83.76
66.19
79.28
LocateAnything
93.23
89.26
88.00
79.57
88.54
Table 2: Grounding accuracy across five test splits in [email protected] (%). The shaded row highlights Jag; Δ reports its improvement over Base in accuracy points. Bold denotes the highest model accuracy in each column.
Figure 2: Jag achieves the lowest end-to-end latency among the compared models and uses less GPU memory than NExT-Chat, LocateAnything, and Hi-Token. Annotations report its speedup over each reference model; the shaded region denotes combinations with both higher latency and higher memory use than Jag.
Figure 3: Joint adaptation of the language backbone, visual merger, and head outperforms head-only training. Panels compare readout and training variants using [email protected]. Gray points are configuration controls; orange dashed lines show the AR reference.
Figure 4: Increasing the training budget improves Jag on all three datasets. Blue curves show [email protected]; gray dashed lines mark the accuracy of Base. The example and update budgets vary together.
Figure 5: Gains over Base occur across the observed target sizes and expression groups. Paired points show [email protected] and counts refer to expressions. Area is normalized GT box area; spatial tags are lexical proxies. Intervals and empty groups are reported in the appendix.
Figure 6: Individual gains and failures in the Base comparison. Four RefCOCOg examples cover shared success, success only for Jag, success only for Base, and shared failure at IoU 0.5. Cases are selected near each outcome’s median prediction pair.
Dataset
AR
Parallel discrete
Jag
RefCOCO
90.02
73.52
89.57
±0.12
±1.88
±0.07
RefCOCO+
84.57
68.13
83.22
±0.31
±1.26
±0.17
RefCOCOg
86.41
68.04
85.12
±0.01
±1.45
±0.40
Table 3: Output-formulation ablations: [email protected] mean ± sample standard deviation across multiple random seeds. Parallel discrete uses four independent categorical heads.
Metric
RefCOCO
RefCOCO+
RefCOCOg
AR
90.93
85.82
88.48
Jag
91.28
85.80
87.72
Δ
+0.34
-0.02
-0.76
95% CI
[-0.25, +0.93]
[-0.73, +0.70]
[-1.41, -0.13]
Table 4: Full-data comparison with the fine-tuned autoregressive model (AR), in [email protected] (%). Differences and paired 95% intervals are Jag minus AR.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
RefCOCO
RefCOCO+
RefCOCOg
Base
79.74
70.10
77.96
Jag
91.28
85.80
87.72
Δ
+11.53
+15.70
+9.76
95% CI
[+10.62, +12.41]
[+14.67, +16.77]
[+8.86, +10.73]
Appendix
Table 5: Pooled Base comparison in [email protected] (%). Δ is Jag minus Base, with paired image-cluster 95% intervals.
Split
N
Base
Jag
Δ
RefCOCO
testA
5,657
84.27
93.90
+9.63
testB
5,095
74.72
88.36
+13.64
RefCOCO+
testA
5,726
76.53
90.69
+14.16
testB
4,889
62.57
80.08
+17.51
Appendix
Table 6: Split accuracies in [email protected] (%) and sample counts for the Base comparison. Δ denotes the absolute accuracy gain.
Outcome
RefCOCO
RefCOCO+
RefCOCOg
Both
8,345
7,063
7,149
Jag only
1,469
2,045
1,274
Base only
229
378
337
Neither
709
1,129
842
Appendix
Table 7: Joint outcomes for the original Base comparison at IoU 0.5. The four outcomes are mutually exclusive and cover each benchmark.
Dataset
Group
N
Base
Jag
Δ
95% CI
RefCOCO
Area <1%
0
NA
NA
NA
NA
RefCOCO
Area 1–<10%
2,665
74.90
87.02
+12.12
[+10.00, +14.20]
RefCOCO
Area ≥ 10%
8,087
81.34
92.68
+11.34
[+10.31, +12.41]
RefCOCO
≤ 5 words
9,214
81.41
92.51
+11.10
[+10.18, +12.03]
RefCOCO
6–10 words
1,423
70.91
84.82
+13.91
[+11.19, +16.65]
RefCOCO
≥ 11 words
115
55.65
72.17
+16.52
[+8.21, +25.58]
Appendix
Table 8: Base-comparison groups: [email protected] (%), absolute gain over Base, and paired 95% intervals. Empty groups are retained as NA.
Configuration
RefCOCO
RefCOCO+
RefCOCOg
Jag
89.62
83.10
84.84
Text mean
42.76
35.37
70.60
Input box marker
89.47
82.68
85.70
Joint from start
89.15
82.67
85.25
Frozen backbone
25.99
20.35
20.18
Appendix
Table 9: Readout and adaptation accuracies in [email protected] (%), underlying Figure 3 . Training uses the component-ablation budget.
Dataset
AR
Jag
RefCOCO
91.69
91.31
RefCOCO+
87.33
85.77
RefCOCOg
89.12
88.28
Appendix
Table 10: Model-scale ablation in [email protected] (%) using Qwen3.5-2B and the component-ablation budget. The comparison varies model size within the same backbone family.
Model
@.5
@.75
@.9
mAcc
RefCOCO ( N=10,752 )
Base
79.74
66.29
43.64
62.63
Jag
91.28
79.06
50.36
72.80
AR
90.93
80.49
61.03
77.04
RefCOCO+ ( N=10,615 )
Base
70.10
58.06
38.51
54.86
Appendix
Table 11: Supplemental localization accuracy (%). Columns @.5, @.75, and @.9 denote IoU thresholds. mAcc averages ten thresholds from 0.50 to 0.95 at intervals of 0.05. Each dataset pools all its test expressions; Base denotes Qwen3.5-0.8B; AR denotes its fine-tuned counterpart.
Visual grounding with multimodal large language models is commonly formulated as autoregressive coordinate generation, where a model outputs bounding-box coordinates as text given an image and a referring-expression prompt. While this interface is simple and compatible with instruction following, it introduces a mismatch between training and evaluation: training optimizes token-level likelihood over coordinate strings, whereas grounding quality is measured by geometric overlap. We propose IoU-PD, an IoU-aware privileged distillation method for coordinate-generating multimodal large language models. IoU-PD uses ground-truth boxes not only as coordinate targets, but also as privileged training-time guidance. During training, the student receives the original image and prompt, while a frozen teacher receives a box-marked image and an augmented prompt that indicates the marked region. The student is trained with a supervised fine-tuning anchor and a privileged distillation loss whose token weights reflect both geometric importance and teacher reliability. At inference time, IoU-PD requires no box overlay, privileged hint, teacher branch, or additional prediction module. Experiments on standard referring-expression grounding benchmarks show consistent region-level improvements over strong coordinate-generating baselines, demonstrating that ground-truth boxes can provide useful privileged guidance beyond serving as coordinate labels. Project page: https://xyzzzh.github.io/IoU-PD/
Xiuyuan Zhu, Ke Lu, Hao Wu +4
University of Chinese Academy of Sciences, Beijing, China · State Key Laboratory of Communication Content Cognition, Beijing, China · Peng Cheng Laboratory, Shenzhen, Guangdong, China +1
Bounding-box scores on RefCOCO-family grounding leave little room to distinguish frontier vision-language systems, yet boxes discard object shape. We introduce GroundingBench, a matched benchmark that re-targets the same 1,500 image-expression-referent triples to exact-N polygons at five vertex budgets. A fixed-denominator harness separately audits filled-region intersection over union (IoU) and legal-polygon completion. The strongest tested configuration reaches 88.2 box IoU and 97.1 accuracy at IoU >= .5 ([email protected]), versus 57.7 and 69.2 for direct polygons; because these headline scores use different references, we also compare direct polygons with predicted boxes rasterised against the same contour target, obtaining 57.7 versus 57.3 when pooled. Performance is non-monotone in N and collapses at the densest budget, where legality failures compound residual geometric error. Qwen's thinking-setting contrast is the largest tested input-preserving configuration difference; under frozen templates, false spatial cues are more damaging than false colour cues, and target preference can remain high while contour tracing is poor. Alternate masks and a continuous-area scorer preserve the principal ordering. GroundingBench therefore measures an operational output-geometry gap spanning localisation, boundary construction, serialisation, and topology, rather than latent boundary perception alone.
Vision-language models (VLMs) commonly formulate visual grounding and detection as a coordinate-token generation problem, serializing each 2D box into multiple 1D tokens that are learned and decoded largely independently. This token-by-token decoding mismatches the coupled structure of box geometry and creates a practical inference bottleneck due to strictly sequential generation. We introduce LocateAnything, a unified generative grounding and detection framework based on Parallel Box Decoding (PBD). By decoding geometric elements such as bounding boxes and points as atomic units in a single step, LocateAnything preserves intra-box geometric coherence and unlocks substantial parallelism. We show that PBD improves both decoding throughput and localization accuracy. We further develop a scalable data engine and curate LocateAnything-Data, a large-scale dataset with more than 138 million training samples, substantially increasing data diversity for high-precision localization. Extensive evaluations show that LocateAnything advances the speed-accuracy frontier, achieving significantly higher decoding throughput while improving high-IoU localization quality across diverse benchmarks. The results highlight the complementary benefits of Parallel Box Decoding and large-scale training data in enabling efficient and precise unified visual grounding and detection.
Shihao Wang, Shilong Liu, Yuanguo Kuang +10
The Hong Kong Polytechnic University · NVIDIA · Princeton University +2