LOCUS: Locate Once, Check Until Settled. Draft-and-Verify Visual Grounding with a Box Head and Coordinate-Token Verification

Muhammad Talha

cs.CVOct 26, 2026

LOCUS: Locate Once, Check Until Settled. Draft-and-Verify Visual Grounding with a Box Head and Coordinate-Token Verification

Authors: Muhammad Talha

Organizations: Independent Researcher

Abstract

Vision-language models (VLMs) ground a referring expression in one of two ways. They decode the box as coordinate tokens, which is precise but takes one sequential step per token, or they regress it from a hidden state with a small head, which takes one step but loses accuracy at high IoU. The two fail differently: a head usually finds the right object but places its edges loosely, while coordinate tokens place edges precisely. We therefore use the regression head as a speculative drafter for the model's own coordinate tokens. Locus writes the head's box as coordinate tokens and checks them with the same backbone by Jacobi iteration over a shared image prefix. The fixed point is the model's greedy coordinate-token answer, reached in about 5.60 sequential passes on average instead of 18.46 decoding steps; head and verifier are one set of weights, trained jointly. On the 8 RefCOCO, RefCOCO+ and RefCOCOg splits, verification raises Locus-2B's [email protected] from 54.87 for its own head to 59.13, and the one-pass head is ahead at [email protected] of every published model with at most 3B parameters except Florence-2-L. Against Jag, the strongest open regression-head model, at its own size and training budget, Locus ties at [email protected] and, with typical acceptance, gains +4.52 points at [email protected], with paired significance tests. Exact answers come 1.5x faster than vLLM greedy-decoding the same coordinate tokens with the same weights; a pure regression head remains faster still.

Figures & tables

Panel of Figure 2 — Figure 2: Architecture and inference of Locus, illustrated with Locus-2B on the RefCOCO val expression “right elephant” (dashed white: ground truth). (a) The image and a fixed prefix are prefilled once. Each question then runs as a short branch over the cached prefix (attention mask on the right), so all box and verifier branches of an image share one prefill. The head reads the final state of the learned [BOX] token; the verifier branch is the backbone’s own grounding prompt, read by its LM head. Head and verifier are trained jointly. (b) The head’s box is rendered as the coordinate tokens the model would write, and the model checks all of them in one parallel pass, keeping the longest prefix it agrees with (green) and correcting the next token (orange). Repeating this until the draft stops changing yields the model’s greedy answer, here in 3 passes instead of 19 decoding steps. Blue tokens are those that verification changed.

Specialist grounding models
			RefCOCO			RefCOCO+			RefCOCOg		Mean
Model	Params	Output	val	testA	testB	val	testA	testB	val	test	all 8	5 test
MDETR [ 14 ]	–	detector	86.8	89.6	81.4	79.5	84.1	70.6	81.6	80.9	81.8	81.3
PolyFormer [ 19 ]	–	polygon tokens	90.4	92.9	87.2	85.0	89.8	78.0	85.8	85.9	86.9	86.8
UNINEXT [ 29 ]	0.74B	detector	92.6	94.3	91.5	85.2	89.6	79.8	88.7	89.4	88.9	88.9
Grounding DINO [ 20 ]	–	detector	90.6	93.2	88.2	88.2	89.0	75.9	86.1	87.0	87.3	86.7

Table 1: [email protected] against published grounding models. Rows above “Ours” are the numbers reported by each paper (source tables are listed in our repository); several papers report only test splits, so the last column averages the 5 test splits, which every row reports. Our rows (tinted) are measured in our harness. Bold: best, underlined: second best per column, among models with at most 3B parameters. Training data differ: most models train on the union of the RefCOCO-family training sets, which contains 46.9% of the evaluation targets ( Section 5.4 ); we exclude every evaluation image.

Model	Output	[email protected] $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$
Jag-0.8B	head	88.16	76.51	48.76
LocateAnything-3B †	box tokens	88.68	80.82	65.55
Hi-Token-3B †	digits	87.24	76.93	54.75
NExT-Chat-7B	decoder	80.14	53.42	12.48
	head (1 pass)	90.01	79.79	54.87
	exact (5.60)	89.30	80.31	59.13

Table 2: Grounding accuracy, mean over the 8 val and test splits of RefCOCO, RefCOCO+ and RefCOCOg, all run in our harness on the same rows. Baselines use their released weights and model-card prompts at the resolution their processor uses ( † native; NExT-Chat a

336^{2}

padded square; Jag

512^{2}

pixels); Locus runs at its

640^{2}

training budget. Locus-0.8B uses Jag’s backbone and training budget. Passes: mean sequential branch passes per question, including the greedy steps of the few questions that do not converge within 8 Jacobi passes. Our rows are tinted.

Figure 3: Accuracy at each IoU threshold

t

, mean over the 8 splits, for the models of Table 2 . Locus’s heads are dashed and its verified answers solid. The gap between a head and its verified answer opens above

t\approx 0.8

; LocateAnything-3B, trained on far more grounding data, is ahead at high thresholds.

	Objects365		COCO		All
Model	@0.5 $\uparrow$	@0.9 $\uparrow$	@0.5 $\uparrow$	@0.9 $\uparrow$	@0.5 $\uparrow$	@0.9 $\uparrow$
Jag-0.8B	35.8	7.4	89.2	40.0	70.0	33.9
Qwen3-VL-2B (0-shot)	74.5	32.7	88.6	57.1	83.9	50.2
Hi-Token-3B †	24.7	1.1	88.0	44.9	65.0	35.9
LocateAnything-3B †	78.0	49.5	90.2	64.4	86.2	60.5
Locus-0.8B head	39.3	9.2	89.6	43.7	71.2	32.9

Table 3: Ref-L4 test: Objects365 rows (images no model was trained on), COCO rows (only images outside every RefCOCO/+/g training split), and all 31,921 rows. Each model runs as in its RefCOCO evaluation, with no Ref-L4 tuning. Columns are [email protected] and [email protected]. † Native resolution.

Figure 4: Margin over Jag-0.8B on all rows and on clean rows, pooled over the 8 splits, with 95% image-bootstrap intervals. Every margin grows on the rows neither model was trained on.

System	Engine	ms $\downarrow$	Steps	@0.5 $\uparrow$	@0.9 $\uparrow$
One RTX A6000, batch 1, end to end
Jag-0.8B	eager	53.1	0.0	91.5	48.8
Jag-0.8B	graphs	20.3	0.0	91.5	48.8
Qwen3-VL-2B tokens	HF	1484.6	41.8	88.5	59.6
Qwen3-VL-2B tokens	vLLM	297.2	41.8	88.5	59.6
coord. only	vLLM	150.9	18.3	–	–

Table 4: Latency and accuracy, all on one RTX A6000, batch 1, 256 RefCOCO testA questions, end to end from the image file to the parsed box. Inputs are

512^{2}

pixels except † native resolution and ‡ a

336^{2}

padded square, as each model’s own processor requires. Engine: eager or CUDA graphs (PyTorch), HF (Hugging Face generate, eager), own (the authors’ released inference code, eager) or vLLM (CUDA graphs). “Coord. only”: the verifier’s prompt with the JSON preamble already written, as in Locus’s verification; with Locus-2B’s weights the greedy answer is Locus-2B exact’s, whose accuracy is shown (–: not run on the accuracy rows). Steps: sequential forward passes after the image prefill (Locus: the head branch plus verifier passes; token models: generated tokens; –: not exposed by the model’s generate). [email protected] and [email protected]: first 3000 rows of RefCOCO val, testA and testB, for Locus from runs at the timed

512^{2}

pixels. Bold: best, underlined: second best; our rows are tinted.

Figure 5: [email protected] against measured latency (RTX A6000, batch 1, 256 RefCOCO testA questions, every system with CUDA graphs). Each Locus line runs from the one-pass head through typical acceptance (

\alpha=0.5

0.7

; 2B only for 0.5) to exact verification. Accuracy is on the first 3000 rows of RefCOCO val, testA and testB, where the coordinate-token baseline was run, with Locus at the timed

512^{2}

pixels. Qwen3-VL-2B decodes its full JSON answer (41.8 tokens on average) in vLLM.

per question at $Q=8$
	Jag-0.8B	Locus-2B
$Q$	head	head	typical	exact
1	20.2	43.7	79.8	94.5
2	37.4	44.5	94.0	117.0
4	71.6	46.7	123.3	163.8
8	127.3	53.1	197.1	287.3

Table 5: Latency (ms, end to end, batch of one image) with

Q

questions about the same image, on 128 RefCOCO testA images that have at least 8 expressions; same GPU and engineering as Table 4 . Jag has no shared prefill and is served the faster of

Q

graphed requests and one batch of

Q

. Typical:

\alpha=0.5

Qwen3.5-0.8B, 36k steps (2 epochs)
	Head		Verified (exact)
Variant	@0.5 $\uparrow$	@0.9 $\uparrow$	@0.5 $\uparrow$	@0.9 $\uparrow$	Passes $\downarrow$
Joint aLoRA	87.36	33.58	86.36	49.79	7.98
Dual LoRA	88.06	45.72	86.81	52.52	7.22
+ merger LoRA	87.87	46.13	86.65	52.72	7.30
Dual full FT (Locus)	87.97	47.03	87.05	53.10	6.68

Table 6: Ablations on adapter placement, adapted modules, and joint versus separate training. Mean over the 8 splits. All variants but separate train the head and the verifier jointly, as one model; tinted rows are the configurations we report as Locus.

\ddagger

Evaluated on a different inference path; pass counts are only comparable within a path.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Jag-0.8B and Locus-2B (verified) on RefCOCO and RefCOCO+ test questions. The first three columns are the first questions in split order where Locus’s IoU exceeds 0.9 and Jag’s is below 0.6; the last is the first where Locus fails (IoU $<0.4$ ) and Jag succeeds (IoU $>0.8$ ). To skip typos and slang in the crowd-written expressions, only expressions whose every word occurs at least 100 times in the training expressions are eligible. Crops are centred on the three boxes.

Figure 7: Jag-0.8B and Locus-2B (verified) on openly licensed photos from Wikimedia Commons, at the same $512^{2}$ -pixel budget. The 12 expressions were written before either model was run, and every result is shown; there is no ground truth, so we make no accuracy claim. The two models mostly agree; Locus’s boxes are tighter for “woman in the teal shirt” and “stack of red plastic crates”, they disagree on “floor lamp on the right”, and both return the whole person for “her blue headscarf”. Images (left to right, top to bottom, two expressions each): Fruit Market, Old Kathmandu by Sundar1 (CC BY-SA 3.0); Amsterdam cyclists by Fons Heijnsbroek (CC0); Person cooks in a cozy wooden kitchen by Shixart1985 (CC BY 2.0); Dogs playing 001 by Sardaka (CC BY-SA 4.0); Living room with 1960s objects by Kotivalo (CC BY-SA 4.0); Farmer manually harvesting rice in Raichur, Karnataka by Nanditha Gogate, WELL Labs (CC BY-SA 4.0).

Model	Output	[email protected] $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$
Qwen3-VL-2B, zero-shot	tokens	88.53	79.52	59.56
Qwen3-VL-4B, zero-shot	tokens	90.60	81.44	61.92
Jag-0.8B	head	91.50	78.81	48.83
Locus-0.8B	head	91.46	78.57	47.46
	verified	90.30	78.84	53.97
Locus-2B	head	93.09	82.58	55.81

Appendix

Table 7: Coordinate-token decoding without grounding fine-tuning, on the first 3000 rows of RefCOCO val, testA and testB (mean of the three), with every system scored on exactly these rows. The Qwen models’ pretraining includes RefCOCO training data.

Compared	Path	Rows	Differ	$\geq$ 10	Passes
ver. vs. greedy	graphed, bf16	10,834	35.4	2.9	5.41
ver. vs. greedy	eager, bf16	505	38.6	1.4	5.66
ver. vs. greedy	eager, fp32	505	0.0	0.0	4.39
greedy vs. greedy	batch 2/1	505	48.9	2.8	–
greedy vs. greedy	eager/graphed	505	51.3	3.6	–
greedy vs. greedy	bf16/fp32	505	56.6	5.0	–

Appendix

Table 8: Exactness in floating point (Locus-2B, RefCOCO val; the 505-row runs are its first questions). Differ: % of boxes with any coordinate different;

\geq

10: % whose largest coordinate change of at least 10 on the 0–1000 grid. Passes: mean verifier passes, against 18.44 greedy steps.

		$\Delta$ vs Jag-0.8B, clean rows (95% CI)
Model	Output	[email protected]	[email protected]	[email protected]
Locus-2B	head	$+2.95$	$+5.90$	$+11.03$
		$[+2.50,+3.46]$	$[+5.15,+6.68]$	$[+9.98,+12.13]$
	verified	$+2.35$	$+6.65$	$+15.66$
		$[+1.87,+2.89]$	$[+5.88,+7.48]$	$[+14.58,+16.80]$
Locus-0.8B	head	$+0.50$	$+1.08$	$+3.58$

Appendix

Table 9: The comparison restricted to evaluation rows whose target object is in no RefCOCO-family training split (46.9% of rows are excluded). Pooled over the 8 splits, row-weighted; CI from a bootstrap over images.

Acceptance	[email protected] $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	Passes $\downarrow$
exact ( $\alpha=1$ )	89.30	80.31	59.13	5.60
$\alpha=0.7$	89.44	80.37	59.24	4.04
$\alpha=0.5$	89.60	80.49	58.95	3.43
$\alpha=0.3$	89.77	80.53	58.39	2.64
head only	90.01	79.79	54.87	0

Appendix

Table 10: Effect of the typical-acceptance threshold

\alpha

(Locus-2B, mean over the 8 splits).

Vision-Language Grounding

Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.

See all 418 papers

Aug 4, 2026

Hi-Token: Hierarchical Coordinate Tokenization for Generative Visual Grounding

Generative Vision-Language Models (VLMs) express bounding boxes as coordinate sequences, whose token identities often leave axis and place value implicit. We study how explicitly encoding these roles improves visual grounding. Hi-Token assigns a distinct token to each combination of axis, decimal place, and digit value, representing a box with 12 tokens drawn from 60 types. This design exposes coarse-to-fine coordinate structure and increases token reuse relative to atomic location tokens while retaining the existing VLM architecture. Hi-GAR complements this representation with a geometry-based reward for Group Relative Policy Optimization (GRPO), combining box overlap, coordinate accuracy at multiple scales, strict-IoU bonuses, and validity gating. Matched training comparisons show that Hi-Token improves localization throughout the evaluated IoU range, with gains across three VLM backbones. Hi-Token also outperforms an additionally tuned atomic-token baseline on the RefCOCO family before reinforcement learning. Hi-GAR further improves localization and is used only during training. The resulting Hi-R1 model achieves higher values than strong specialist baselines on most reported metrics. Token-frequency and geometric diagnostics characterize the representation and reward stages. These results establish explicit axis and place-value encoding as an effective design for generative visual grounding. Project page: https://xyzzzh.github.io/Hi-Token/

Xiuyuan Zhu, Ke Lu, Kun Dong +6

University of Chinese Academy of Sciences, Beijing, China · State Key Laboratory of Communication Content Cognition, Beijing, China · Peng Cheng Laboratory, Shenzhen, Guangdong, China +1

Vision-Language Grounding

Ask PDF

May 26, 2026

LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding

Vision-language models (VLMs) commonly formulate visual grounding and detection as a coordinate-token generation problem, serializing each 2D box into multiple 1D tokens that are learned and decoded largely independently. This token-by-token decoding mismatches the coupled structure of box geometry and creates a practical inference bottleneck due to strictly sequential generation. We introduce LocateAnything, a unified generative grounding and detection framework based on Parallel Box Decoding (PBD). By decoding geometric elements such as bounding boxes and points as atomic units in a single step, LocateAnything preserves intra-box geometric coherence and unlocks substantial parallelism. We show that PBD improves both decoding throughput and localization accuracy. We further develop a scalable data engine and curate LocateAnything-Data, a large-scale dataset with more than 138 million training samples, substantially increasing data diversity for high-precision localization. Extensive evaluations show that LocateAnything advances the speed-accuracy frontier, achieving significantly higher decoding throughput while improving high-IoU localization quality across diverse benchmarks. The results highlight the complementary benefits of Parallel Box Decoding and large-scale training data in enabling efficient and precise unified visual grounding and detection.

Shihao Wang, Shilong Liu, Yuanguo Kuang +10

The Hong Kong Polytechnic University · NVIDIA · Princeton University +2

Efficient VLM Inference Vision-Language Grounding Parallel Decoding Object Detection Visual Grounding

Ask PDF

Mar 30, 2026

MolmoPoint: Better Pointing for VLMs with Grounding Tokens

Grounding has become a fundamental capability of vision-language models (VLMs). Most existing VLMs point by generating coordinates as part of their text output, which requires learning a complicated coordinate system and results in a high token count. Instead, we propose a more intuitive pointing mechanism that directly selects the visual tokens that contain the target concept. Our model generates a special pointing token that cross-attends to the input image or video tokens and selects the appropriate one. To make this model more fine-grained, we follow these pointing tokens with an additional special token that selects a fine-grained subpatch within the initially selected region, and then a third token that specifies a location within that subpatch. We further show that performance improves by generating points sequentially in a consistent order, encoding the relative position of the previously selected point, and including a special no-more-points class when selecting visual tokens. Using this method, we set a new state-of-the-art on image pointing (70.7% on PointBench), set a new state-of-the-art among fully open models on GUI pointing (61.1% on ScreenSpotPro), and improve video pointing (59.1% human preference win rate vs. a text coordinate baseline) and tracking (+6.3% gain on Molmo2Track). We additionally show that our method achieves much higher sample efficiency and discuss the qualitative differences that emerge from this design change.

Christopher Clark, Yue Yang, Jae Sung Park +8

Allen Institute for AI · University of Washington

Vision-Language Grounding Large Vision-Language Models

Ask PDF

Abstract

Vision-Language Grounding

Explore similar work

Hi-Token: Hierarchical Coordinate Tokenization for Generative Visual Grounding

LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding

MolmoPoint: Better Pointing for VLMs with Grounding Tokens