GroundAnything: Reconciling Parallel Decoding with Precise Visual Grounding at Flash Speed

Qize Yu; Lianrui Fan; Bowen Ping; Xini Ding; Zetian Song; Junbo Niu; Kaixuan Wang; Tianxing Chen

cs.CVSep 30, 2026

GroundAnything: Reconciling Parallel Decoding with Precise Visual Grounding at Flash Speed

Authors: Qize Yu, Lianrui Fan, Bowen Ping, Xini Ding, Zetian Song, Junbo Niu, Kaixuan Wang, Tianxing Chen, +14 more

Organizations: XPeng Inc. · Peking University · The University of Hong Kong · National University of Singapore · HKUST (GZ) · University of California, Berkeley · Princeton University

Abstract

Autoregressive (AR) grounding models serialize spatial predictions, introducing sequential latency and imposing a causal order on output tokens. We view grounding as visual evidence extraction: objects, locations, and spatial relations are jointly constrained by the image and query, yet their dependencies do not imply an intrinsic left-to-right generation order. This distinction makes bidirectional diffusion a natural fit, allowing spatial hypotheses to emerge in parallel and be jointly refined through iterative denoising. We introduce GroundAnything, a 4B-parameter grounding foundation model that reconciles fast parallel decoding with precise localization through blockwise denoising. Training combines grounding pretraining from public datasets and dedicated data engines, direct AR-to-diffusion conversion with joint AR and diffusion objectives, supervised fine-tuning, and GRPO-based reinforcement post-training. Across 30 grounding benchmarks, our autoregressive variant, GroundAnything-VLM, establishes a new overall state of the art among similarly sized models at 72.42%, remaining competitive with GPT-6 Astra (71.35%). With entropy-guided decoding, GroundAnything also surpasses the prior state of the art at this scale, averaging 61.75% versus 53.32% for the fast MTP-based LocateAnything model. We further explore decoding strategies, showing that an optional self-speculative mode achieves a $4.51\times$ speedup over the AR counterpart with a 0.74 percentage-point drop in COCO F1mIoU. Infrastructure experiments show that progressive inference optimizations translate parallel decoding into practical speedups. These support efficient visual grounding in latency-sensitive real-world systems.

Figures & tables

Figure 1 : GroundAnything architecture. Clean multimodal inputs and response-corrupted text views share one visual encoding and jointly supervise causal prediction and masked denoising. One noisy view is illustrated, with its uncorrupted prompt copy omitted for clarity. The bottom-left panel illustrates parallel denoising, and the grounding example includes two minions and one human and an absent car category. Architecture specifications and parameter counts are provided in Section 6.2 .

Figure 2 : Data engine. Field-level teacher fusion and task-specific validation guide expert iteration, with targeted observations resolving uncertain annotations.

Figure 3 : Conversion attention mask. Rows/columns are queries/keys. Purple denotes within-block bidirectionality, light green the permitted clean prefix, and green clean-stream causality. Other entries are blocked. The symbols

v

and

p

condense visual and prompt tokens. One noisy view with two-token blocks is shown; noisy prompt copies are omitted. Training uses

B=32

Figure 4 : Training and inference pipeline. SFT and causal GRPO update shared parameters reused for blockwise diffusion inference. The schematic illustrates token-level supervision and grounding/OCR reward aggregation; the final advantage includes the additional group normalization in Equation 9 . Rollouts and unmasking states are illustrative.

Figure 5 : Self-speculative decoding with shared model weights (

B=4

). Linear decoding separates drafting from causal verification:

d_{1}

and

d_{2}

are accepted, while the mismatched

d_{3}

is replaced by the AR correction

r

. Quadratic decoding compares

a_{i}

with

d_{i+1}

a_{0}=d_{1}

a_{1}=d_{2}

, and

a_{2}\neq d_{3}

, thereby committing

[d_{0},d_{1},d_{2}]

and reusing Row 2 as the next still-unverified proposal. Purple denotes bidirectional proposals and green causal verification. Matrix rows and columns denote queries and keys, respectively; the cached prefix is omitted.

M

denotes [MASK] ,

s

the last accepted seed, and

p_{ij}

proposal token

j

from row

i

. One NFE per quadratic round applies after initialization.

Figure 6 : Grounding across task groups. GroundAnything-VLM, GroundAnything, and leading baselines across box, point, text, and exemplar-conditioned tasks. Complete results: Section 10 .

Closed-set Specialized Detectors
3 6 8 16 17 18 21 Model	Common	Long-tailed	Dense & Tiny		Referring grounding
	COCO	LVIS	Dense200	VisDrone	RefCOCOg val	RefCOCOg test	RefCOCO avg
DINO-R50 * ( Zhang et al., 2022 )	55.60	–	–	–	–	–	–
DETR-R50 * ( Carion et al., 2020 )	48.30	–	–	–	–	–	–
Open-set Specialized Detectors
GroundingDINO ( Liu et al., 2024 )	60.56	52.61	24.92	34.47	49.77	50.43	45.15

Table 1 : Detection and referring grounding (F1mIoU). External rows follow Jiang et al. (2026) . Full results appear in Sections 10.1 , 10.2 and 10.3 .

Open-set Specialized Detectors
3 5 16 17 18 24 Model	Robot and spatial pointing			GUI grounding
	RefSpatial (avg)	RefSpatial Unseen	RoboSpatial Context	ScreenSpot-Pro	ScreenSpot-V2	OSWorld-G
GroundingDINO ( Liu et al., 2024 )	14.25	4.33	4.92	N/A *	N/A *	N/A *
Vision-Language Models (<10B)
JEDI * ( Xie et al., 2026 )	–	–	–	36.10	88.60	–
UI-R1 * ( Lu et al., 2026 )	–	–	–	17.80	85.40	–

Table 2 : Robot, spatial, and GUI grounding (accuracy). External RefSpatial and JEDI/UI-R1 scores follow Jiang et al. (2026) ; GUI-Owl scores follow Wang et al. (2026) . Full results appear in Sections 10.5 and 10.7 .

Closed-set Specialized Detectors
3 6 14 15 16 19 Model	OCR				Layout grounding		Visual prompting
	HierText	ICDAR2015	TotalText	SROIE	DocLayNet	M6Doc	FSC147	Dense200
DocLayout-YOLO * ( Zhao et al., 2024 )	–	–	–	–	81.10	–	–	–
PaddleOCRv5 * ( Cui et al., 2025 )	30.50	25.60	25.70	58.60	–	–	–	–
Vision-Language Models (<10B)
Qwen3-VL-4B ( Bai et al., 2025a )	23.48	28.41	38.35	40.41	40.81	24.73	18.90	2.49

Table 3 : OCR, layout, and visual prompting (F1mIoU). External rows follow Jiang et al. (2026) . SenseNova-Vision uses published HierText/ICDAR2015 scores ( Han et al., 2026 ) and locally evaluated TotalText/SROIE scores. Full results appear in Sections 10.6 , 10.8 and 10.9 .

Open-set Specialized Detectors
3 5 14 15 16 19 Model	Referring object pointing		Common / long-tailed		Dense / tiny
	RefCOCOg val	RefCOCOg test	COCO	LVIS	Dense200	VisDrone
GroundingDINO ( Liu et al., 2024 )	49.34	49.97	70.41	55.07	32.93	39.45
Vision-Language Models (<10B)
Qwen3-VL-4B ( Bai et al., 2025a )	76.43	77.64	65.33	55.08	21.72	23.50
Qwen3.5-9B ( Qwen Team, 2026a )	77.59	77.85	72.21	64.00	65.35	44.73

Table 4 : Object pointing (F1@Point). External rows follow Jiang et al. (2026) , where Molmo denotes Molmo-7B-D. Full results appear in Section 10.4 .

Figure 7 : Self-speculative schedules. TPF and TPS across block sizes; dashed lines denote GroundAnything-VLM.

Figure 8 : Entropy-threshold sensitivity. TPF, TPS, and COCO F1mIoU at

B=32

with sub-block size 4. Dashed lines denote GroundAnything-VLM.

Figure 9 : Decoding strategy and block size. Linear self-speculation and entropy guidance (

\tau=0.8

, sub-block size 4) compared on COCO. Dashed lines denote GroundAnything-VLM.

Figure 10 : Acceptance at matched candidate length. One-step DLM and causal MTP share prefixes and a frozen AR verifier. Accepted tokens exclude verifier corrections and bonuses.

Figure 12 : Refinement and available decode time. (a,b) Varying refinement depth at

K=16

. (c) Committed tokens under equal deadlines on long-output prefixes, using validation-selected policies.

Figure 13 : Draft diagnostics before verification. Comparisons at

K=8

; both DLMs use two refinement steps. Valid-box misses count unmatched proposals, not missed ground-truth instances. GUI denotes ScreenSpot-Pro.

Figure 14 : Progressive inference optimization. Time per output token (

\downarrow

) and cumulative speedup relative to each decoding mode’s native PyTorch implementation.

Figure 16 : Generation cost versus predicted object count. Average generation time for GroundAnything-VLM and GroundAnything, with output-token counts across box-count ranges.

Task	Instruction
Category grounding	Locate all the instances that match the following categories: [CATEGORIES].
Referring grounding	Locate the target referred to by the following description: [EXPRESSION].
Category pointing	Point to: [CATEGORIES].
Referring pointing	Point to the target referred to by the following description: [EXPRESSION].
GUI grounding	Point to the UI element to click for the following instruction: [INSTRUCTION].
OCR	OCR task detect all the text in box format.

Table 6: Task-specific input instructions. Category names and referring expressions are preserved in the response identifiers.

Token(s)	Function
<\|object_ref_start\|> , <\|object_ref_end\|>	Delimit a semantic identifier
<\|box_start\|> , <\|box_end\|>	Delimit a geometric payload
<0> – <999>	Atomic quantized coordinates
</c>	Separate requested categories
<\|vision_start\|> , <\|vision_end\|>	Delimit visual input
<\|image_pad\|>	Image placeholder in the input template

Table 7: Grounding and multimodal token conventions.

Component	Configuration
Vision encoder	27 layers; width 1024; FFN width 4096; 12 heads; QKV width 1536; patch size 14
Projector	LayerNorm(1024), $2\times 2$ aggregation, bias-free linear layers $4096\to 4096\to 2560$ with GELU, then RMSNorm(2560); normalization $\epsilon=10^{-5}$
Language backbone	36 layers; width 2560; FFN width 9728; 32 query heads and 8 KV heads; head dimension 128
Language numerics	SiLU; attention dropout 0; RMSNorm $\epsilon=10^{-6}$ ; RoPE base $5{,}000{,}000$
Positional capacity	262,144 positions

Table 8: Shared backbone architecture. Positional capacity is independent of the training sequence lengths.

Training phase	GroundAnything-VLM	GroundAnything
Base VLM Training (Pretrain 1)	Stages 1–3; causal cross-entropy	Stages 1–3; causal cross-entropy
Coordinate Alignment (Pretrain 2 / SFT)	Stage 4; assistant-only spatial cross-entropy	Stage 4; assistant-only spatial cross-entropy
AR-to-Diffusion conversion	—	$0.5\mathcal{L}_{\mathrm{MDM}}+0.5\mathcal{L}_{\mathrm{AR}}$ ; shared weights
Reinforcement learning (RL)	Causal GRPO	Causal GRPO; shared AR/diffusion weights

Table 10: Training pipelines. GroundAnything has four phases; GroundAnything-VLM has three and proceeds directly from coordinate alignment to RL.

	Base VLM Training (Pretrain 1)			Pretrain 2
Setting	Stage 1	Stage 2	Stage 3	Stage 4
Trainable modules	Projector	All	All	All
Language peak LR	Frozen	$10^{-5}$	$10^{-5}$	$10^{-5}$
Vision peak LR	Frozen	$10^{-6}$	$10^{-6}$	$10^{-6}$
Projector peak LR	$5\times 10^{-5}$	$10^{-6}$	$10^{-6}$	$10^{-6}$
Per-device batch / accumulation	6 / 1	2 / 1	2 / 1	2 / 1

Table 11: Configured pretraining and alignment recipe. Base VLM Training (Pretrain 1) comprises Stages 1–3; Coordinate Alignment (Pretrain 2 / SFT) is Stage 4. Context and packing lengths count complete multimodal sequences.

Setting	Phase I	Phase II
GPUs	256	256
Per-device / global batch	32 / 8192	32 / 8192
Clean sequence length	4096	4096
Trainable modules	Projector and language	All
Language / projector peak LR	$2\times 10^{-5}$ / $2\times 10^{-5}$	$2\times 10^{-5}$ / $2\times 10^{-5}$
Vision peak LR	Frozen	$2\times 10^{-6}$

Table 12: GroundAnything conversion configuration.

Setting	GroundAnything	GroundAnything-VLM
GPUs	256	256
Per-device response batch / accumulation	1 / 1	1 / 1
Responses per prompt	8	8
Prompt / response batch	32 / 256	32 / 256
Updates per rollout	1	1
Learning rate	$2\times 10^{-6}$	$5\times 10^{-7}$

Table 13: Variant-specific GRPO settings. Global batches count generated responses, not distinct prompts.

Branch	Parse status	Grade
Word/line	Complete, valid structure with no invalid instances	1.00
	Incomplete structure with reliably parsed instances	0.60
	Only some valid instances can be recovered	0.30
	No reliable parse	0.00
Complementary	Strictly valid structure with no invalid content	1.00
references	Complete main structure with extraneous content	0.45

Table 14: OCR format grades used for squared reward gating.

Closed-set Specialized Detectors
3 7 9 26 27 28 36 Model	Zero-shot	IoU 0.50			IoU 0.95			mIoU
		R	P	F1	R	P	F1	R	P	F1
DINO-R50 * ( Zhang et al., 2022 )	NO	62.60	76.50	68.80	17.80	25.80	21.10	50.00	62.40	55.60
DETR-R50 * ( Carion et al., 2020 )	NO	59.60	73.90	65.90	10.60	19.00	13.60	42.90	55.30	48.30
DyHead-R50 * ( Dai et al., 2021 )	NO	58.10	76.60	66.10	11.90	20.60	15.00	44.80	60.10	51.30
Open-set Specialized Detectors

Table 15 : COCO. Complete box-grounding metrics. Model-name stars denote external scores from Table 2 of Jiang et al. (2026) .

Open-set Specialized Detectors
3 5 22 23 24 32 Model	Zero-shot	IoU 0.50			IoU 0.95			mIoU
		R	P	F1	R	P	F1	R	P	F1
GroundingDINO ( Liu et al., 2024 )	YES	52.64	82.61	64.30	23.34	26.55	24.84	44.30	59.58	52.61
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	YES	58.50	73.11	64.99	17.76	20.04	18.83	44.02	46.58	46.74
LocateAnything Fast ( Wang et al., 2026 )	YES	48.06	61.58	53.98	20.37	25.12	22.50	38.43	48.55	42.90

Table 16 : LVIS. Complete box-grounding metrics. The starred SEED1.5-VL scores are from Table 3 of Jiang et al. (2026) .

Open-set Specialized Detectors
3 5 22 23 24 32 Model	IoU 0.50			IoU 0.95			mIoU
	R	P	F1	R	P	F1	R	P	F1
GroundingDINO ( Liu et al., 2024 )	22.43	36.30	27.73	10.92	18.38	13.70	20.16	32.62	24.92
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	70.61	75.13	72.80	8.66	9.17	8.91	51.79	54.88	53.29
LocateAnything Fast ( Wang et al., 2026 )	22.45	26.12	24.15	9.16	10.50	9.78	19.30	22.31	20.70

Table 17 : Dense200. Complete box-grounding metrics. Local BAGEL and DeepSeek runs did not reproduce the reference results, so only available external scores are reported for these models. The starred BAGEL scores are taken from Table 1 of Han et al. (2026) . No matching external result is available for full DeepSeek-VL2; Small and Tiny checkpoint results are not substituted. The starred DeepSeek-VL2-Small and SEED1.5-VL scores are from Table 4 of Jiang et al. (2026) , also reported in Table 2 of Wang et al. (2026) .

Open-set Specialized Detectors
3 5 22 23 24 32 Model	IoU 0.50			IoU 0.95			mIoU
	R	P	F1	R	P	F1	R	P	F1
GroundingDINO ( Liu et al., 2024 )	32.38	87.09	47.21	2.81	7.42	4.08	23.65	63.54	34.47
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	42.47	52.33	46.89	1.07	1.27	1.16	24.78	30.12	27.19
LocateAnything Fast ( Wang et al., 2026 )	13.58	15.44	14.45	0.85	0.97	0.90	9.27	10.44	9.82

Table 18 : VisDrone. Complete box-grounding metrics. Local BAGEL and DeepSeek runs did not reproduce the reference results, so only available external scores are reported for these models. The starred BAGEL scores are taken from Table 1 of Han et al. (2026) . No matching external result is available for full DeepSeek-VL2; Small and Tiny checkpoint results are not substituted. The starred DeepSeek-VL2-Small and SEED1.5-VL scores are from Table 4 of Jiang et al. (2026) , also reported in Table 2 of Wang et al. (2026) .

Open-set Specialized Detectors
3 5 22 23 24 32 Model	RefCOCOg val			RefCOCOg test
	F1@.50	F1@.95	F1mIoU	F1@.50	F1@.95	F1mIoU
GroundingDINO ( Liu et al., 2024 )	58.37	22.47	49.77	58.52	24.38	50.43
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	87.01	35.23	73.90	87.36	36.51	74.76
LocateAnything Fast ( Wang et al., 2026 )	87.99	39.39	75.30	88.34	42.08	76.50

Table 19 : RefCOCOg validation and test. Complete recorded F1 metrics at IoU 0.50, 0.95, and mIoU. The starred BAGEL scores are taken from Table 1 of Han et al. (2026) , because local runs did not reproduce the reported performance. The starred SEED1.5-VL scores are from Table 5 of Jiang et al. (2026) .

Open-set Specialized Detectors
3 5 22 23 24 31 Model	RefCOCO			RefCOCOg			RefCOCOplus
	F1@.50	F1@.95	F1mIoU	F1@.50	F1@.95	F1mIoU	F1@.50	F1@.95	F1mIoU
GroundingDINO ( Liu et al., 2024 )	51.19	23.28	44.33	57.99	23.17	49.69	49.20	20.89	41.42
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	84.37	32.85	71.43	84.74	34.74	72.34	76.92	29.53	64.10
LocateAnything Fast ( Wang et al., 2026 )	92.48	43.95	80.61	88.47	40.69	76.17	84.43	39.53	73.08

Table 20 : RefCOCO family. The three datasets are reported separately; their F1mIoU arithmetic mean is RefCOCO avg in the main text.

3 5 23 24 25 33 Model	RefCOCOg val	RefCOCOg test
	F1@Point	F1@Point
Open-set Specialized Detectors
GroundingDINO ( Liu et al., 2024 )	49.34	49.97
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	84.96	85.32
LocateAnything Fast ( Wang et al., 2026 )	73.84	74.94

Table 21 : Referring object pointing. Starred BAGEL scores are retained despite unreliable support for the unified pointing protocol. Kimi-K3, both MiMo variants, and both DeepSeek variants are N/A under that protocol. These outcomes do not establish intrinsic pointing capability. The starred Molmo and SEED1.5-VL scores are from Table 7 of Jiang et al. (2026) ; its Molmo checkpoint is Molmo-7B-D.

Open-set Specialized Detectors
3 5 23 24 25 33 Model	COCO			LVIS
	R@Point	P@Point	F1@Point	R@Point	P@Point	F1@Point
GroundingDINO ( Liu et al., 2024 )	68.92	71.97	70.41	44.91	71.15	55.07
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	77.81	81.77	79.74	63.46	78.14	70.04
LocateAnything Fast ( Wang et al., 2026 )	68.99	74.26	71.53	55.81	70.05	62.13

Table 22 : Object pointing on COCO and LVIS. Starred BAGEL scores are retained despite unreliable support for the unified pointing protocol. Kimi-K3, both MiMo variants, and both DeepSeek variants are N/A under that protocol. These outcomes do not establish intrinsic pointing capability. The starred Molmo and SEED1.5-VL scores are from Table 7 of Jiang et al. (2026) ; its Molmo checkpoint is Molmo-7B-D.

Open-set Specialized Detectors
3 5 23 24 25 33 Model	Dense200			VisDrone
	R@Point	P@Point	F1@Point	R@Point	P@Point	F1@Point
GroundingDINO ( Liu et al., 2024 )	22.00	65.45	32.93	27.20	71.76	39.45
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	75.59	77.76	76.66	47.81	56.92	51.97
LocateAnything Fast ( Wang et al., 2026 )	64.63	66.65	65.63	55.22	57.55	56.36

Table 23 : Object pointing on Dense200 and VisDrone. Starred BAGEL scores are retained despite unreliable support for the unified pointing protocol. Kimi-K3, both MiMo variants, and both DeepSeek variants are N/A under that protocol. These outcomes do not establish intrinsic pointing capability. The starred Molmo and SEED1.5-VL scores are from Table 7 of Jiang et al. (2026) ; its Molmo checkpoint is Molmo-7B-D.

Open-set Specialized Detectors
3 5 24 25 26 37 Model	Point-in-mask accuracy
	RefSpatial Location	RefSpatial Placement	RefSpatial Unseen	RoboSpatial Context
GroundingDINO ( Liu et al., 2024 )	26.50	2.00	4.33	4.92
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	51.00	52.50	37.01	59.02
LocateAnything Fast ( Wang et al., 2026 )	53.00	19.00	16.88	15.57

Table 24 : Robot and spatial pointing. Point-in-mask accuracy is reported for each dataset. The starred RefSpatial baselines follow Table 11 of Jiang et al. (2026) ; RoboRefer uses the setting without a depth prior. Values retain the precision recorded in our evaluation tables.

Closed-set Specialized Detectors
3 5 7 24 25 26 34 Model	HierText					ICDAR2015
	F1@.50	F1@.75	F1@.95	F1mIoU	Parse err.	F1@.50	F1@.75	F1@.95	F1mIoU	Parse err.
PaddleOCRv5 * ( Cui et al., 2025 )	45.20	–	3.40	30.50	–	38.20	–	1.20	25.60	–
Open-set Specialized Detectors
GroundingDINO ( Liu et al., 2024 )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *
Vision-Language Models (<10B)

Table 25 : OCR on HierText and ICDAR2015. Each dataset reports four loose-match F1 measures and parse-error rate. GroundingDINO is N/A because OCR is unsupported. Kimi-K3 and both DeepSeek variants are N/A because their outputs do not satisfy the evaluation protocol; this does not establish a lack of OCR capability. SenseNova-Vision uses the HierText and ICDAR2015 F1mIoU scores of 31.20 and 49.50 reported in Table 1 of Han et al. (2026) , because its local evaluation prompts could not be aligned. Other metrics for these datasets are unavailable; TotalText and SROIE use local results. The starred PaddleOCRv5 and SEED1.5-VL scores use the BBOX results in Table 10 of Jiang et al. (2026) .

Closed-set Specialized Detectors
3 5 7 24 25 26 34 Model	TotalText					SROIE
	F1@.50	F1@.75	F1@.95	F1mIoU	Parse err.	F1@.50	F1@.75	F1@.95	F1mIoU	Parse err.
PaddleOCRv5 * ( Cui et al., 2025 )	40.20	–	0.70	25.70	–	77.70	–	5.60	58.60	–
Open-set Specialized Detectors
GroundingDINO ( Liu et al., 2024 )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *
Vision-Language Models (<10B)

Table 26 : OCR on TotalText and SROIE. Each dataset reports four loose-match F1 measures and parse-error rate. GroundingDINO is N/A because OCR is unsupported. Kimi-K3 and both DeepSeek variants are N/A because their outputs do not satisfy the evaluation protocol; this does not establish a lack of OCR capability. The starred PaddleOCRv5 and SEED1.5-VL scores use the BBOX results in Table 10 of Jiang et al. (2026) .

Open-set Specialized Detectors
3 5 25 26 27 35 Model	Dev		Creative		CAD		Sci		Office		OS		Overall
	Text	Icon	Text	Icon	Text	Icon	Text	Icon	Text	Icon	Text	Icon	Action acc.	Parse err.
GroundingDINO ( Liu et al., 2024 )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	61.04	9.66	53.03	12.59	23.35	9.38	57.64	26.36	65.54	24.53	42.06	13.48	36.75	–
LocateAnything Fast ( Wang et al., 2026 )	68.18	44.83	59.60	32.87	58.38	35.94	71.53	53.64	75.14	56.60	51.40	37.08	56.04	–

Table 27 : ScreenSpot-Pro. Action accuracy is broken down by domain and target type, followed by overall action accuracy and parse-error rate. GroundingDINO is N/A because GUI grounding is unsupported. BAGEL is N/A because its outputs do not satisfy the unified protocol. The starred JEDI, UI-R1, and UI-TARS scores are from Table 8 of Jiang et al. (2026) ; GUI-Owl-32B scores are from Table 3 of Wang et al. (2026) .

Open-set Specialized Detectors
3 5 25 26 27 34 Model	ScreenSpot-V2								OSWorld-G
	Mobile text	Mobile icon	Desktop text	Desktop icon	Web text	Web icon	Action acc.	Parse err.	Exact acc.	Parse err.
GroundingDINO ( Liu et al., 2024 )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	96.90	82.46	97.94	80.71	89.74	76.35	88.29	–	46.10	–
LocateAnything Fast ( Wang et al., 2026 )	94.48	81.04	93.81	86.43	88.89	83.25	88.44	–	59.93	–

Table 28 : ScreenSpot-V2 and OSWorld-G. ScreenSpot-V2 includes text/icon results for mobile, desktop, and web environments, overall action accuracy, and parse-error rate; OSWorld-G reports exact accuracy and parse-error rate. GroundingDINO is N/A because GUI grounding is unsupported. BAGEL is N/A because its outputs do not satisfy the unified protocol. The starred ScreenSpot-V2 scores are from Table 8 of Jiang et al. (2026) .

Closed-set Specialized Detectors
3 5 7 24 25 26 34 Model	IoU 0.50			IoU 0.95			mIoU
	R	P	F1	R	P	F1	R	P	F1
DocLayout-YOLO * ( Zhao et al., 2024 )	–	–	91.20	–	–	52.10	–	–	81.10
Open-set Specialized Detectors
GroundingDINO ( Liu et al., 2024 )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *
Vision-Language Models (<10B)

Table 29 : DocLayNet. Complete box-grounding metrics. GroundingDINO lacks a compatible document-region interface; it, Kimi-K3, and both DeepSeek variants are N/A. Starred MiMo and BAGEL scores retain observations under suspected output-protocol incompatibility and are not formal capability measurements. The starred DocLayout-YOLO and SEED1.5-VL scores are from Table 9 of Jiang et al. (2026) .

Open-set Specialized Detectors
3 5 22 23 24 32 Model	IoU 0.50			IoU 0.95			mIoU
	R	P	F1	R	P	F1	R	P	F1
GroundingDINO ( Liu et al., 2024 )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	73.09	78.27	75.59	18.16	18.78	18.46	53.36	56.64	54.95
LocateAnything Fast ( Wang et al., 2026 )	59.82	61.54	60.67	18.06	18.50	18.28	46.39	47.69	47.03

Table 30 : M6Doc. Complete box-grounding metrics. GroundingDINO lacks a compatible document-region interface; it, Kimi-K3, and both DeepSeek variants are N/A. Starred MiMo and BAGEL scores retain observations under suspected output-protocol incompatibility and are not formal capability measurements. The starred SEED1.5-VL scores are from Table 9 of Jiang et al. (2026) .

Open-set Specialized Detectors
3 5 22 23 24 31 Model	IoU 0.50			IoU 0.95			mIoU
	R	P	F1	R	P	F1	R	P	F1
GroundingDINO ( Liu et al., 2024 )	7.94 *	20.16 *	11.39 *	1.56 *	4.94 *	2.37 *	6.34 *	16.01 *	9.08 *
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	78.97	77.92	78.44	9.13	9.01	9.07	57.56	56.75	57.15
LocateAnything Fast ( Wang et al., 2026 )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *

Table 31 : FSC147 visual prompting. Complete box-grounding metrics. LocateAnything variants lack a supported visual-prompt interface; both DeepSeek variants have incompatible output protocols. Their entries are N/A. Starred GroundingDINO scores come from an unsupported visual-prompt task interface; starred MiMo and BAGEL scores are retained observations under suspected output-protocol incompatibility.

Open-set Specialized Detectors
3 5 22 23 24 31 Model	IoU 0.50			IoU 0.95			mIoU
	R	P	F1	R	P	F1	R	P	F1
GroundingDINO ( Liu et al., 2024 )	2.03 *	9.98 *	3.38 *	1.35 *	5.64 *	2.18 *	1.90 *	9.00 *	3.14 *
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	73.05	74.81	73.92	10.89	11.12	11.00	54.93	56.08	55.50
LocateAnything Fast ( Wang et al., 2026 )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *

Table 32 : Dense200 visual prompting. Complete box-grounding metrics. LocateAnything variants lack a supported visual-prompt interface; both DeepSeek variants have incompatible output protocols. Their entries are N/A. Kimi-K3 is N/A because the required prompt format is unsupported. Starred GroundingDINO scores come from an unsupported visual-prompt task interface; starred MiMo and BAGEL scores are retained observations under suspected output-protocol incompatibility.

Figure 17 : Multi-scale object grounding in a cluttered indoor scene.

Figure 18 : Dense instance grounding of fruit decorations.

Figure 19 : Dense grounding of repeated bricks.

Figure 20 : Separating overlapping birds with deceptive silhouettes.

Figure 21 : Grounding a pair specified by a visual relationship.

Figure 22 : Grounding a spatially specified, partially visible person.

Figure 23 : Point grounding within an occluded person’s visible mask.

Figure 24 : Dense point grounding of overlapping balloons.

Figure 25 : Dense point grounding of vehicles in a night scene.

Figure 26 : GUI target grounding in a multi-window desktop.

Figure 27 : OCR of artistic lettering embedded in illustration.

Figure 28 : OCR of highly stylized poster lettering.

Figure 29 : OCR of outdoor signs with perspective distortion.

Figure 30 : OCR of a low-contrast receipt with aligned fields.

Figure 31 : Exemplar-guided grounding of repeated rounded objects.

Figure 32 : Exemplar-guided grounding of traffic cones.

Explore similar work

May 26, 2026cs.CV

LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding

Vision-language models (VLMs) commonly formulate visual grounding and detection as a coordinate-token generation problem, serializing each 2D box into multiple 1D tokens that are learned and decoded largely independently. This token-by-token decoding mismatches the coupled structure of box geometry and creates a practical inference bottleneck due to strictly sequential generation. We introduce LocateAnything, a unified generative grounding and detection framework based on Parallel Box Decoding (PBD). By decoding geometric elements such as bounding boxes and points as atomic units in a single step, LocateAnything preserves intra-box geometric coherence and unlocks substantial parallelism. We show that PBD improves both decoding throughput and localization accuracy. We further develop a scalable data engine and curate LocateAnything-Data, a large-scale dataset with more than 138 million training samples, substantially increasing data diversity for high-precision localization. Extensive evaluations show that LocateAnything advances the speed-accuracy frontier, achieving significantly higher decoding throughput while improving high-IoU localization quality across diverse benchmarks. The results highlight the complementary benefits of Parallel Box Decoding and large-scale training data in enabling efficient and precise unified visual grounding and detection.

Shihao Wang, Shilong Liu, Yuanguo Kuang +10

The Hong Kong Polytechnic University · NVIDIA · Princeton University +2

Vision-Language Model Grounding Weak Visual Grounding Open-Vocabulary Object Detection Constrained Decoding

Ask PDF

May 10, 2026cs.CV

What Happens Before Decoding? Prefill Determines GUI Grounding in VLMs

Existing training-free approaches for GUI grounding often rely on multiple inference runs, such as iterative cropping or candidate aggregation, to identify target elements. Despite this additional computation, each forward pass still independently interprets the instruction and parses the visual layout, without enabling progressive interaction among visual tokens. In this paper, we study what happens during GUI grounding in Vision-Language Models (VLMs) and identify a previously overlooked bottleneck. We show that grounding follows a two-stage paradigm: the prefill stage determines candidate UI elements, while the decoding stage subsequently refines the final coordinates. This asymmetry establishes prefill as the critical step, as errors in candidate selection cannot be effectively corrected during decoding. Based on this observation, we propose Re-Prefill, a training-free method that revisits inference by introducing an attention-guided second prefill stage to refine target selection. Specifically, visual tokens that consistently receive high attention from the query position, i.e., the final token, across layers are extracted as a preliminary target hypothesis and appended to the input, together with the instruction hidden states, enabling the model to deeply re-think its decision before coordinate generation. Experiments across four VLMs and five benchmarks, including ScreenSpot-Pro, ScreenSpot-V2, OSWorld-G, UI-Vision, and MMBench-GUI, demonstrate consistent improvements without additional training, with gains of up to 4.3% on ScreenSpot-Pro. Code will be available at https://github.com/linjiaping1/Re-Prefill.

Jiaping Lin, Fei Shen, Junzhe Li +4

1Guangming Laboratory · 2National University of Singapore · 3Peking University +2

Attention-Guided Training Prefill Graphical User Interface Grounding Training-Free Decoding

Ask PDF

May 28, 2026cs.CV

Grounded 3D-Aware Spatial Vision-Language Modeling

We present GR3D, a spatial vision language model equipped with three complementary grounding capabilities--explicit 2D grounding, implicit 2D grounding, and monocular 3D grounding--within a single framework. GR3D introduces an implicit grounding mechanism that identifies entity mentions during generation and inserts the corresponding region tokens into the text stream, allowing the model to reference visual evidence on the fly when producing spatial chain-of-thought responses. In parallel, a region-prompted monocular 3D grounding design predicts 3D bounding boxes in the camera view from grounded region queries, supported by intrinsic-aware normalization and dense geometric supervision. Together, these grounding capabilities enable GR3D to decompose complex spatial understanding problems into grounded 2D perception followed by 3D inference. GR3D achieves consistent improvements across grounded and non-grounded spatial benchmarks, demonstrating grounding as an effective inductive bias for strengthening spatial understanding in VLMs. These grounding capabilities collectively enhance general spatial understanding beyond the grounding task itself.

An-Chieh Cheng, Yang Fu, Yatai Ji +12

UCSD · NVIDIA · MIT

3D Visual Grounding Spatial Grounding 3D Perception Chain-of-Thought Reasoning

Ask PDF