GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives

Qize Yu; Lianrui Fan; Boyu Chen; Jiaqi Liang; Xini Ding; Yue Chen; Zetian Song; Yuran Wang

cs.CVSep 30, 2026

GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives

Authors: Qize Yu, Lianrui Fan, Boyu Chen, Jiaqi Liang, Xini Ding, Yue Chen, Zetian Song, Yuran Wang, +18 more

Organizations: XPeng Inc. · Peking University · The University of Hong Kong · Tsinghua University · National University of Singapore · HKUST (GZ) · University of California, Berkeley

Abstract

Precise grounding matters. It specifies which object is the target and where that object is, even in clutter and for tiny objects, and it has to be fast enough for closed-loop control. Yet vision-language-action (VLA) and world-action models (WAMs) take perception from general-purpose vision-language and video-generation backbones, which still fail in these settings. We introduce GroundingPI, a 4B grounding foundation model that generates points and boxes as quantized coordinates in a shared vocabulary. Training combines multimodal and spatial pretraining, supervised fine-tuning, and reinforcement learning with GRPO, using supervision from public datasets and dedicated data engines. Against 44 baselines across 34 grounding benchmarks spanning 11 perceptual capabilities, GroundingPI establishes a new state of the art, averaging 73.68%, above the larger GPT-6 Astra (71.54%). As a downstream visual backbone, GroundingPI improves performance on robotic manipulation and autonomous driving. On RoboTwin 2.0, it outperforms every mainstream backbone we evaluate in all four out-of-distribution settings, by up to 24.8% relative to the strongest backbone. On RoboCasa-GR1, GroundingPI trained with 50% of the demonstrations outperforms those baselines trained with 75%. On nuScenes, used as the visual backbone, GroundingPI attains an average open-loop L2 error of 0.296 m. We systematically analyze GroundingPI's pretraining in scale and data composition. Downstream autonomous driving and robotic manipulation improve as the pretraining is scaled. Analyzing the data recipe across these 11 perceptual capabilities shows dense grounding's substantial benefits for both, and OCR's potential as a catalyst for perceptual learning. These results support grounding as a perceptual foundation, and dedicated perceptual pretraining as a promising direction for foundation models of physical intelligence.

Figures & tables

Figure 1 : GroundingPI architecture. The grounding example includes two minions and one human and an absent car category ( None ). Architecture specifications and parameter counts are provided in Section 7.2 .

Figure 2 : Data engine. Multi-teacher fusion and task-specific validation guide expert iteration, with targeted observations resolving uncertain labels.

Figure 3 : Overview of GroundingPI’s two-stage training pipeline. The first stage uses supervised fine-tuning (SFT) to learn structured grounding outputs with token-level cross-entropy. The second stage applies group relative policy optimization (GRPO) with task-specific rewards that balance grounding completeness and localization, and assess OCR text–geometry consistency and output reliability. Illustrative rollouts highlight common prediction errors and alternative OCR granularities.

Figure 4 : Grounding across perceptual capabilities. GroundingPI and leading baselines across box, point, text, and exemplar-conditioned tasks.

Closed-set Specialized Detectors
3 6 8 15 16 19 Model	Common	Long-tailed	Dense & Tiny		Referring grounding
	COCO	LVIS	Dense200	VisDrone	HumanRef	RefCOCOg val	RefCOCOg test	RefCOCO avg
DINO-R50 * ( Zhang et al., 2022 )	55.60	–	–	–	–	–	–	–
DETR-R50 * ( Carion et al., 2020 )	48.30	–	–	–	–	–	–	–
Open-set Specialized Detectors
GroundingDINO ( Liu et al., 2024 )	60.56	52.61	24.92	34.47	46.13	49.77	50.43	45.15

Table 1 : Detection and referring grounding (F1mIoU). External rows are from ( Jiang et al., 2026 ) . Full results appear in Appendices 12.1 – 12.3 .

Open-set Specialized Detectors
3 5 15 16 22 Model	Robot and spatial pointing			GUI grounding
	RefSpatial (avg)	RefSpatial Unseen	RoboSpatial Context	ScreenSpot-Pro	ScreenSpot-V2	OSWorld-G
GroundingDINO ( Liu et al., 2024 )	14.25	4.33	4.92	N/A *	N/A *	N/A *
Vision-Language Models (<10B)
JEDI * ( Xie et al., 2026 )	–	–	–	36.10	88.60	–
UI-R1 * ( Lu et al., 2026 )	–	–	–	17.80	85.40	–

Table 2 : Robot, spatial, and GUI grounding (accuracy). External RefSpatial and JEDI/UI-R1 scores are from ( Jiang et al., 2026 ) , respectively; GUI-Owl scores are from ( Wang et al., 2026a ) . Full results appear in Sections 12.5 and 12.7 .

Closed-set Specialized Detectors
3 6 13 14 17 Model	OCR				Layout grounding
	HierText	ICDAR2015	TotalText	SROIE	DocLayNet	M6Doc
DocLayout-YOLO * ( Zhao et al., 2024 )	–	–	–	–	81.10	–
PaddleOCRv5 * ( Cui et al., 2025 )	30.50	25.60	25.70	58.60	–	–
Vision-Language Models (<10B)
Qwen3-VL-4B ( Bai et al., 2025a )	23.48	28.41	38.35	40.41	40.81	24.73

Table 3 : OCR and document layout (F1mIoU). External rows are from ( Jiang et al., 2026 ) . SenseNova-Vision uses published HierText/ICDAR2015 scores ( Han et al., 2026 ) and locally evaluated TotalText/SROIE scores. Full results appear in Sections 12.6 and 12.8 .

Open-set Specialized Detectors
3 5 13 14 17 Model	Referring object pointing			Common / long-tailed		Dense / tiny
	HumanRef	RefCOCOg val	RefCOCOg test	COCO	LVIS	Dense200	VisDrone
GroundingDINO ( Liu et al., 2024 )	46.85	49.34	49.97	70.41	55.07	32.93	39.45
Vision-Language Models (<10B)
Qwen3-VL-4B ( Bai et al., 2025a )	66.89	76.43	77.64	65.33	55.08	21.72	23.50
Qwen3.5-9B ( Qwen Team, 2026a )	78.21	77.59	77.85	72.21	64.00	65.35	44.73

Table 4 : Object pointing (F1@Point). External rows are from ( Jiang et al., 2026 ) , where Molmo denotes Molmo-7B-D. Full results appear in Section 12.4 .

Figure 5 : Transfer to physical intelligence. Manipulation success rates and reciprocal nuScenes open-loop L2 errors (higher is better in all panels). Manipulation comparisons fix the action expert, action data, and training budget within each benchmark.

Figure 6 : Perception before action. A conceptual progression toward a foundation with native grounding capabilities. Segment sizes are illustrative and do not represent measured parameter or data proportions.

Figure 7 : Action-data efficiency on RoboCasa-GR1 ID. Success rate at four demonstration fractions with the remaining action-learning setup fixed.

Figure 8 : Grounding and action scaling. Grounding Avg and RoboCasa-GR1 ID/OOD success as grounding-training exposure increases under a fixed downstream recipe.

Figure 9 : Which perceptual supervision transfers? Included task groups and downstream results. Leave-one-out differences compare with all six groups; – denotes an unavailable ablation.

Figure 10 : Grounding-model ablations. Effects of coordinate representation, visual encoder, and Stage III reinforcement post-training on grounding Avg.

Figure 11 : Generation cost versus predicted object count. GroundingPI generation time and output-token count across box-count ranges.

4	COCO			Dense200
Model	Boxes/img	Tokens/img	Tokens/box	Boxes/img	Tokens/img	Tokens/box
SEED1.5-VL ( Guo et al., 2025 )	4.2	631.0	148.8	73.1	5446.3	74.5
GroundingPI	6.0	45.6	7.6	87.2	444.7	5.1

Table 5 : Output token efficiency. SEED1.5-VL values are from ( Jiang et al., 2026 ) . This cross-model comparison measures serialization rather than matched latency.

Token or delimiter	Role
<0> – <999>	One atomic token per quantized coordinate
</c>	Separator between categories in the input query
<\|object_ref_start\|> , <\|object_ref_end\|>	Delimit a semantic label or transcription
<\|box_start\|> , <\|box_end\|>	Delimit a box, point, or ordered-point payload
None	Ordinary text indicating an absent queried target
<\|im_end\|>	End of the assistant response

Table 6: Tokens and delimiters in GroundingPI’s input/output protocol.

Task	User prompt	Output
Grounding / dense grounding	Locate all the instances that match the following categories: cat1 </c> cat2 </c> ….	Boxes
Referring	Locate the target referred to by the following description: phrase .	Boxes
Object pointing	Point to: cat1 </c> cat2 </c> ….	Points
Referring pointing	Point to the target referred to by the following description: phrase .	Points
GUI grounding	Point to the UI element to click for the following instruction: instruction .	Point
OCR	OCR task detect all the text in box format.	Text + boxes

Table 7: Canonical prompts and output geometry. Italic fields are replaced by query-specific text or coordinates.

Component	Configuration
Vision encoder	27 layers; hidden width 1024; FFN width 4096; 12 attention heads; QKV hidden width 1536; patch size 14
Spatial aggregation	$2\times 2$ neighboring patches; four 1024-dimensional features form one 4096-dimensional feature
Projector	LayerNorm(1024), spatial aggregation, two bias-free linear layers ( $4096\to 4096\to 2560$ ) with GELU, and RMSNorm(2560); normalization $\epsilon=10^{-5}$
Language backbone	36 full-attention layers; hidden width 2560; FFN width 9728; 32 query heads and 8 KV heads; head dimension 128
Language numerics	SiLU; attention dropout 0; RMSNorm $\epsilon=10^{-6}$ ; RoPE base $5{,}000{,}000$ ; no sliding window
Positional capacity	262,144 positions

Table 8: GroundingPI architecture and visual processing. Positional capacity is distinct from the training sequence limit.

	Base VLM training (Pretrain 1)			Pretrain 2 / SFT
Setting	Stage 1: Projector alignment	Stage 2: Joint multimodal pretraining	Stage 3: General visual/video understanding	Stage 4: Coordinate alignment
Trainable modules	Projector only	All	All	All
Per-device batch	6	2	2	2
Global batch	768	256	256	256
Context / packing length	8192 / 8192	8192 / 8192	32768 / 32768	8192 / 8192
Language peak LR	Frozen	$10^{-5}$	$10^{-5}$	$10^{-5}$

Table 10: Training configuration for Base VLM training (Pretrain 1, Stages 1–3) and coordinate alignment (Pretrain 2 / SFT, Stage 4). Each column uses 128 GPUs; language-side updates include both vocabulary matrices.

Setting	Value
GPUs / precision / sharding	256 / BF16 / ZeRO-2
Responses per prompt	8
Per-device response batch / accumulation	1 / 1
Distinct-prompt / response batch	32 / 256
Updates per rollout	1
Trainable modules	Language parameters, including embedding and output head

Table 11: GroundingPI reinforcement post-training configuration.

Component	Weight
Structured-format validity	0.10
Agreement between predicted and reference counts	0.10
Mean detection F1 at IoU $0.50$ , $0.75$ , and $0.95$	0.50
Matched-box IoU	0.25
Compliance with the output ordering convention	0.05
Oversized-box penalty	$-0.07$

Table 12: Components of the strict grounding reward; the weighted sum is clipped to

[0,1]

Annotation structure	Global views $D$	Content score $Q_{g}$
With granularity conflict	$0.90V_{\mathrm{hi}}+0.10V_{\mathrm{lo}}$	$0.35D+0.55\Gamma$
Without granularity conflict	$0.75V_{\mathrm{hi}}+0.25V_{\mathrm{lo}}$	$0.50D+0.40\Gamma$

Table 13: Aggregation of complete word/line views and local granularity groups.

Component	Shared configuration
Backbone conditions	8 normalized-depth hidden states
Condition interface	64 tokens per depth, width 1024
Action topology	16 atomic blocks: 8 cross-attention and 8 self-attention blocks, interleaved from cross-attention
Residual / attention width	1024; 16 heads; head dimension 64
Feed-forward width	4096
Planning tokens	32

Table 14: Unified action architecture used for every manipulation foundation.

Item	Configuration
Optimization length	40,000 updates
Hardware / precision	40 accelerators; bfloat16; DeepSpeed ZeRO-2 ( Rajbhandari et al., 2020 )
Batch size	32 per device; 1280 global
Optimizer	AdamW ( Loshchilov & Hutter, 2019 ) , $\beta_{1}=0.95$ , $\beta_{2}=0.999$ , $\epsilon=10^{-8}$
Weight decay / clipping	$10^{-5}$ ; gradient norm 1.0
Learning rates	$10^{-4}$ for the action expert and interface; $10^{-5}$ for adapted backbone parameters

Table 15: Matched downstream training budget for the manipulation comparison.

Closed-set Specialized Detectors
3 7 9 26 27 35 Model	Zero-shot	IoU 0.50			IoU 0.95			mIoU
		R	P	F1	R	P	F1	R	P	F1
DINO-R50 * ( Zhang et al., 2022 )	NO	62.60	76.50	68.80	17.80	25.80	21.10	50.00	62.40	55.60
DETR-R50 * ( Carion et al., 2020 )	NO	59.60	73.90	65.90	10.60	19.00	13.60	42.90	55.30	48.30
DyHead-R50 * ( Dai et al., 2021 )	NO	58.10	76.60	66.10	11.90	20.60	15.00	44.80	60.10	51.30
Open-set Specialized Detectors

Table 19 : COCO. Complete box-grounding metrics. Model-name stars denote external scores from Table 2 of Jiang et al. (2026) .

Open-set Specialized Detectors
3 5 22 23 31 Model	Zero-shot	IoU 0.50			IoU 0.95			mIoU
		R	P	F1	R	P	F1	R	P	F1
GroundingDINO ( Liu et al., 2024 )	YES	52.64	82.61	64.30	23.34	26.55	24.84	44.30	59.58	52.61
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	YES	58.50	73.11	64.99	17.76	20.04	18.83	44.02	46.58	46.74
LocateAnything Fast ( Wang et al., 2026a )	YES	48.06	61.58	53.98	20.37	25.12	22.50	38.43	48.55	42.90

Table 20 : LVIS. Complete box-grounding metrics. The starred SEED1.5-VL scores are from Table 3 of Jiang et al. (2026) .

Open-set Specialized Detectors
3 5 22 23 31 Model	IoU 0.50			IoU 0.95			mIoU
	R	P	F1	R	P	F1	R	P	F1
GroundingDINO ( Liu et al., 2024 )	22.43	36.30	27.73	10.92	18.38	13.70	20.16	32.62	24.92
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	70.61	75.13	72.80	8.66	9.17	8.91	51.79	54.88	53.29
LocateAnything Fast ( Wang et al., 2026a )	22.45	26.12	24.15	9.16	10.50	9.78	19.30	22.31	20.70

Table 21 : Dense200. Complete box-grounding metrics. Local BAGEL and DeepSeek runs did not reproduce the reference results, so only available external scores are reported for these models. The starred BAGEL scores are taken from Table 1 of Han et al. (2026) . No matching external result is available for full DeepSeek-VL2; Small and Tiny checkpoint results are not substituted. The starred DeepSeek-VL2-Small and SEED1.5-VL scores are from Table 4 of Jiang et al. (2026) , also reported in Table 2 of Wang et al. (2026a) .

Open-set Specialized Detectors
3 5 22 23 31 Model	IoU 0.50			IoU 0.95			mIoU
	R	P	F1	R	P	F1	R	P	F1
GroundingDINO ( Liu et al., 2024 )	32.38	87.09	47.21	2.81	7.42	4.08	23.65	63.54	34.47
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	42.47	52.33	46.89	1.07	1.27	1.16	24.78	30.12	27.19
LocateAnything Fast ( Wang et al., 2026a )	13.58	15.44	14.45	0.85	0.97	0.90	9.27	10.44	9.82

Table 22 : VisDrone. Complete box-grounding metrics. Local BAGEL and DeepSeek runs did not reproduce the reference results, so only available external scores are reported for these models. The starred BAGEL scores are taken from Table 1 of Han et al. (2026) . No matching external result is available for full DeepSeek-VL2; Small and Tiny checkpoint results are not substituted. The starred DeepSeek-VL2-Small and SEED1.5-VL scores are from Table 4 of Jiang et al. (2026) , also reported in Table 2 of Wang et al. (2026a) .

Open-set Specialized Detectors
3 5 22 23 31 Model	IoU 0.50			IoU 0.95			mIoU
	R	P	F1	R	P	F1	R	P	F1
GroundingDINO ( Liu et al., 2024 )	54.80	46.81	50.49	37.38	31.26	34.05	50.32	42.59	46.13
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	85.91	84.95	85.43	65.72	65.09	65.40	80.33	79.42	79.87
LocateAnything Fast ( Wang et al., 2026a )	68.01	71.34	69.64	53.25	55.15	54.19	62.15	64.72	63.41

Table 23 : HumanRef. Complete box-grounding metrics. The starred BAGEL scores are taken from Table 1 of Han et al. (2026) , because local runs did not reproduce the reported performance. The starred SEED1.5-VL scores are from Table 5 of Jiang et al. (2026) .

Open-set Specialized Detectors
3 5 22 23 31 Model	RefCOCOg val			RefCOCOg test
	F1@.50	F1@.95	F1mIoU	F1@.50	F1@.95	F1mIoU
GroundingDINO ( Liu et al., 2024 )	58.37	22.47	49.77	58.52	24.38	50.43
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	87.01	35.23	73.90	87.36	36.51	74.76
LocateAnything Fast ( Wang et al., 2026a )	87.99	39.39	75.30	88.34	42.08	76.50

Table 24 : RefCOCOg validation and test. Complete recorded F1 metrics at IoU 0.50, 0.95, and mIoU. The starred BAGEL scores are taken from Table 1 of Han et al. (2026) , because local runs did not reproduce the reported performance. The starred SEED1.5-VL scores are from Table 5 of Jiang et al. (2026) .

Open-set Specialized Detectors
3 5 22 23 30 Model	RefCOCO			RefCOCOg			RefCOCOplus
	F1@.50	F1@.95	F1mIoU	F1@.50	F1@.95	F1mIoU	F1@.50	F1@.95	F1mIoU
GroundingDINO ( Liu et al., 2024 )	51.19	23.28	44.33	57.99	23.17	49.69	49.20	20.89	41.42
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	84.37	32.85	71.43	84.74	34.74	72.34	76.92	29.53	64.10
LocateAnything Fast ( Wang et al., 2026a )	92.48	43.95	80.61	88.47	40.69	76.17	84.43	39.53	73.08

Table 25 : RefCOCO family. The three datasets are reported separately; their F1mIoU arithmetic mean is RefCOCO avg in the main text.

Open-set Specialized Detectors
3 5 23 24 32 Model	HumanRef			RefCOCOg val	RefCOCOg test
	R@Point	P@Point	F1@Point	F1@Point	F1@Point
GroundingDINO ( Liu et al., 2024 )	50.22	43.90	46.85	49.34	49.97
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	84.05	82.76	83.40	84.96	85.32
LocateAnything Fast ( Wang et al., 2026a )	66.10	68.28	67.18	73.84	74.94

Table 26 : Referring object pointing. Starred BAGEL scores are retained despite unreliable support for the unified pointing protocol. Kimi-K3, both MiMo variants, and both DeepSeek variants are N/A under that protocol. These outcomes do not establish intrinsic pointing capability. The starred Molmo and SEED1.5-VL scores are from Table 7 of Jiang et al. (2026) ; its Molmo checkpoint is Molmo-7B-D.

Open-set Specialized Detectors
3 5 23 24 32 Model	COCO			LVIS
	R@Point	P@Point	F1@Point	R@Point	P@Point	F1@Point
GroundingDINO ( Liu et al., 2024 )	68.92	71.97	70.41	44.91	71.15	55.07
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	77.81	81.77	79.74	63.46	78.14	70.04
LocateAnything Fast ( Wang et al., 2026a )	68.99	74.26	71.53	55.81	70.05	62.13

Table 27 : Object pointing on COCO and LVIS. Starred BAGEL scores are retained despite unreliable support for the unified pointing protocol. Kimi-K3, both MiMo variants, and both DeepSeek variants are N/A under that protocol. These outcomes do not establish intrinsic pointing capability. The starred Molmo and SEED1.5-VL scores are from Table 7 of Jiang et al. (2026) ; its Molmo checkpoint is Molmo-7B-D.

Open-set Specialized Detectors
3 5 23 24 32 Model	Dense200			VisDrone
	R@Point	P@Point	F1@Point	R@Point	P@Point	F1@Point
GroundingDINO ( Liu et al., 2024 )	22.00	65.45	32.93	27.20	71.76	39.45
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	75.59	77.76	76.66	47.81	56.92	51.97
LocateAnything Fast ( Wang et al., 2026a )	64.63	66.65	65.63	55.22	57.55	56.36

Table 28 : Object pointing on Dense200 and VisDrone. Starred BAGEL scores are retained despite unreliable support for the unified pointing protocol. Kimi-K3, both MiMo variants, and both DeepSeek variants are N/A under that protocol. These outcomes do not establish intrinsic pointing capability. The starred Molmo and SEED1.5-VL scores are from Table 7 of Jiang et al. (2026) ; its Molmo checkpoint is Molmo-7B-D.

Open-set Specialized Detectors
3 5 24 25 36 Model	Point-in-mask accuracy
	RefSpatial Location	RefSpatial Placement	RefSpatial Unseen	RoboSpatial Context
GroundingDINO ( Liu et al., 2024 )	26.50	2.00	4.33	4.92
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	51.00	52.50	37.01	59.02
LocateAnything Fast ( Wang et al., 2026a )	53.00	19.00	16.88	15.57

Table 29 : Robot and spatial pointing. Point-in-mask accuracy is reported for each dataset. The starred RefSpatial baselines follow Table 11 of Jiang et al. (2026) ; RoboRefer uses the setting without a depth prior. Values retain the precision recorded in our evaluation tables.

Closed-set Specialized Detectors
3 5 7 24 25 33 Model	HierText					ICDAR2015
	F1@.50	F1@.75	F1@.95	F1mIoU	Parse err.	F1@.50	F1@.75	F1@.95	F1mIoU	Parse err.
PaddleOCRv5 * ( Cui et al., 2025 )	45.20	–	3.40	30.50	–	38.20	–	1.20	25.60	–
Open-set Specialized Detectors
GroundingDINO ( Liu et al., 2024 )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *
Vision-Language Models (<10B)

Table 30 : OCR on HierText and ICDAR2015. Each dataset reports four loose-match F1 measures and parse-error rate. GroundingDINO is N/A because OCR is unsupported. Kimi-K3 and both DeepSeek variants are N/A because their outputs do not satisfy the evaluation protocol; this does not establish a lack of OCR capability. SenseNova-Vision uses the HierText and ICDAR2015 F1mIoU scores of 31.20 and 49.50 reported in Table 1 of Han et al. (2026) , because its local evaluation prompts could not be aligned. Other metrics for these datasets are unavailable; TotalText and SROIE use local results. The starred PaddleOCRv5 and SEED1.5-VL scores use the BBOX results in Table 10 of Jiang et al. (2026) .

Closed-set Specialized Detectors
3 5 7 24 25 33 Model	TotalText					SROIE
	F1@.50	F1@.75	F1@.95	F1mIoU	Parse err.	F1@.50	F1@.75	F1@.95	F1mIoU	Parse err.
PaddleOCRv5 * ( Cui et al., 2025 )	40.20	–	0.70	25.70	–	77.70	–	5.60	58.60	–
Open-set Specialized Detectors
GroundingDINO ( Liu et al., 2024 )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *
Vision-Language Models (<10B)

Table 31 : OCR on TotalText and SROIE. Each dataset reports four loose-match F1 measures and parse-error rate. GroundingDINO is N/A because OCR is unsupported. Kimi-K3 and both DeepSeek variants are N/A because their outputs do not satisfy the evaluation protocol; this does not establish a lack of OCR capability. The starred PaddleOCRv5 and SEED1.5-VL scores use the BBOX results in Table 10 of Jiang et al. (2026) .

Open-set Specialized Detectors
3 5 25 26 34 Model	Dev		Creative		CAD		Sci		Office		OS		Overall
	Text	Icon	Text	Icon	Text	Icon	Text	Icon	Text	Icon	Text	Icon	Action acc.	Parse err.
GroundingDINO ( Liu et al., 2024 )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	61.04	9.66	53.03	12.59	23.35	9.38	57.64	26.36	65.54	24.53	42.06	13.48	36.75	–
LocateAnything Fast ( Wang et al., 2026a )	68.18	44.83	59.60	32.87	58.38	35.94	71.53	53.64	75.14	56.60	51.40	37.08	56.04	–

Table 32 : ScreenSpot-Pro. Action accuracy is broken down by domain and target type, followed by overall action accuracy and parse-error rate. GroundingDINO is N/A because GUI grounding is unsupported. BAGEL is N/A because its outputs do not satisfy the unified protocol. The starred JEDI, UI-R1, and UI-TARS scores are from Table 8 of Jiang et al. (2026) ; GUI-Owl-32B scores are from Table 3 of Wang et al. (2026a) .

Open-set Specialized Detectors
3 5 25 26 33 Model	ScreenSpot-V2								OSWorld-G
	Mobile text	Mobile icon	Desktop text	Desktop icon	Web text	Web icon	Action acc.	Parse err.	Exact acc.	Parse err.
GroundingDINO ( Liu et al., 2024 )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	96.90	82.46	97.94	80.71	89.74	76.35	88.29	–	46.10	–
LocateAnything Fast ( Wang et al., 2026a )	94.48	81.04	93.81	86.43	88.89	83.25	88.44	–	59.93	–

Table 33 : ScreenSpot-V2 and OSWorld-G. ScreenSpot-V2 includes text/icon results for mobile, desktop, and web environments, overall action accuracy, and parse-error rate; OSWorld-G reports exact accuracy and parse-error rate. GroundingDINO is N/A because GUI grounding is unsupported. BAGEL is N/A because its outputs do not satisfy the unified protocol. The starred ScreenSpot-V2 scores are from Table 8 of Jiang et al. (2026) .

Closed-set Specialized Detectors
3 5 7 24 25 33 Model	IoU 0.50			IoU 0.95			mIoU
	R	P	F1	R	P	F1	R	P	F1
DocLayout-YOLO * ( Zhao et al., 2024 )	–	–	91.20	–	–	52.10	–	–	81.10
Open-set Specialized Detectors
GroundingDINO ( Liu et al., 2024 )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *
Vision-Language Models (<10B)

Table 34 : DocLayNet. Complete box-grounding metrics. GroundingDINO lacks a compatible document-region interface; it, Kimi-K3, and both DeepSeek variants are N/A. Starred MiMo and BAGEL scores retain observations under suspected output-protocol incompatibility and are not formal capability measurements. The starred DocLayout-YOLO and SEED1.5-VL scores are from Table 9 of Jiang et al. (2026) .

Open-set Specialized Detectors
3 5 22 23 31 Model	IoU 0.50			IoU 0.95			mIoU
	R	P	F1	R	P	F1	R	P	F1
GroundingDINO ( Liu et al., 2024 )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	73.09	78.27	75.59	18.16	18.78	18.46	53.36	56.64	54.95
LocateAnything Fast ( Wang et al., 2026a )	59.82	61.54	60.67	18.06	18.50	18.28	46.39	47.69	47.03

Table 35 : M6Doc. Complete box-grounding metrics. GroundingDINO lacks a compatible document-region interface; it, Kimi-K3, and both DeepSeek variants are N/A. Starred MiMo and BAGEL scores retain observations under suspected output-protocol incompatibility and are not formal capability measurements. The starred SEED1.5-VL scores are from Table 9 of Jiang et al. (2026) .

Open-set Specialized Detectors
3 5 22 23 30 Model	IoU 0.50			IoU 0.95			mIoU
	R	P	F1	R	P	F1	R	P	F1
GroundingDINO ( Liu et al., 2024 )	7.94 *	20.16 *	11.39 *	1.56 *	4.94 *	2.37 *	6.34 *	16.01 *	9.08 *
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	78.97	77.92	78.44	9.13	9.01	9.07	57.56	56.75	57.15
LocateAnything Fast ( Wang et al., 2026a )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *

Table 36 : FSC147 visual prompting. Complete box-grounding metrics. LocateAnything variants lack a supported visual-prompt interface; both DeepSeek variants have incompatible output protocols. Their entries are N/A. Starred GroundingDINO scores come from an unsupported visual-prompt task interface; starred MiMo and BAGEL scores are retained observations under suspected output-protocol incompatibility.

Open-set Specialized Detectors
3 5 22 23 30 Model	IoU 0.50			IoU 0.95			mIoU
	R	P	F1	R	P	F1	R	P	F1
GroundingDINO ( Liu et al., 2024 )	2.03 *	9.98 *	3.38 *	1.35 *	5.64 *	2.18 *	1.90 *	9.00 *	3.14 *
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	73.05	74.81	73.92	10.89	11.12	11.00	54.93	56.08	55.50
LocateAnything Fast ( Wang et al., 2026a )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *

Table 37 : Dense200 visual prompting. Complete box-grounding metrics. LocateAnything variants lack a supported visual-prompt interface; both DeepSeek variants have incompatible output protocols. Their entries are N/A. Kimi-K3 is N/A because the required prompt format is unsupported. Starred GroundingDINO scores come from an unsupported visual-prompt task interface; starred MiMo and BAGEL scores are retained observations under suspected output-protocol incompatibility.

Open-set Specialized Detectors
3 5 22 23 30 Model	IoU 0.50			IoU 0.95			mIoU
	R	P	F1	R	P	F1	R	P	F1
GroundingDINO ( Liu et al., 2024 )	26.84 *	27.30 *	27.07 *	13.69 *	13.52 *	13.60 *	23.62 *	23.83 *	23.73 *
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	78.69	67.17	72.48	20.38	18.49	19.39	61.99	53.44	57.40
LocateAnything Fast ( Wang et al., 2026a )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *

Table 38 : COCO visual prompting. Complete box-grounding metrics. LocateAnything variants lack a supported visual-prompt interface; both DeepSeek variants have incompatible output protocols. Their entries are N/A. Kimi-K3 is N/A because the required prompt format is unsupported. Starred GroundingDINO scores come from an unsupported visual-prompt task interface; starred MiMo and BAGEL scores are retained observations under suspected output-protocol incompatibility.

Open-set Specialized Detectors
3 5 22 23 30 Model	IoU 0.50			IoU 0.95			mIoU
	R	P	F1	R	P	F1	R	P	F1
GroundingDINO ( Liu et al., 2024 )	18.16 *	18.90 *	18.52 *	9.85 *	9.99 *	9.92 *	15.91 *	16.38 *	16.14 *
Vision-Language Models (<10B)
Rex-Omni ( Jiang et al., 2026 )	69.31	61.02	64.90	17.42	15.42	16.36	52.74	46.39	49.36
LocateAnything Fast ( Wang et al., 2026a )	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *	N/A *

Table 39 : LVIS visual prompting. Complete box-grounding metrics. LocateAnything variants lack a supported visual-prompt interface; both DeepSeek variants have incompatible output protocols. Their entries are N/A. Kimi-K3 is N/A because the required prompt format is unsupported. Starred GroundingDINO scores come from an unsupported visual-prompt task interface; starred MiMo and BAGEL scores are retained observations under suspected output-protocol incompatibility.

Figure 12 : Multi-scale grounding in a crowded classroom scene.

Figure 13 : Dense fruit grounding under partial occlusion.

Figure 14 : Grounding a person through an occluded spatial relationship.

Figure 15 : Point grounding on the visible surface of a sofa.

Figure 16 : Dense point grounding in an aerial urban scene.

Figure 17 : GUI target grounding of a small side-panel control.

Figure 18 : OCR of layered artistic typography and small print.

Figure 19 : OCR of slanted display text and mixed lettering styles.

Figure 20 : OCR of multilingual packaging and numerical fields.

Figure 21 : Exemplar-guided grounding of densely stacked pipe openings.

Explore similar work

Grounded Action Model: 3D Grounding as a Foundation for Robotics
Sep 20, 2026Gehao Zhang, Weikai Huang, Shailesh Shailesh +33D Visual Grounding
Northwestern University · University of Washington · National University of Singapore +1
World Pilot: Steering Vision-Language-Action Models with World-Action Priors
Jun 10, 2026Zefu Lin, Rongxu Cui, Junjia Xu +4Efficient World-Action Model World Models
Institute of Automation, Chinese Academy of Sciences (CASIA) · Nanjing University · Beihang University
RynnBrain 1.1: Towards More Capable and Generalizable Embodied Foundation Model
Jul 20, 2026Kehan Li, Bohan Hou, Minghao Zhu +283D Visual Grounding Wireless Foundation Models
1DAMO Academy, Alibaba Group

Sep 20, 2026, cs.ROJ/K move · Enter open · S save

Grounded Action Model: 3D Grounding as a Foundation for Robotics

Gehao Zhang, Weikai Huang, Shailesh Shailesh, Yiyan Peng+2

Northwestern University · University of Washington · National University of Singapore+1

3D Visual Grounding

Open paper AskPDF arXiv

Manipulation policies must know which objects matter and where they are, yet the pretrained backbones that current robot foundation models build on, from language in vision-language-action models (VLAs) to video generation in world-action models (WAMs), do not directly require this metric grounding, leaving it to be learned implicitly from robot demonstrations. We propose Grounded Action Models (GAMs), a new paradigm of robot foundation models built with 3D grounding. GAM can be conditioned using language, points, or box prompts, which are first transformed into a shared object-centric representation of the selected objects. This representation captures target-focused visual features and metric object geometry, which is mixed with robot state history through a multi-stream transformer to predict action chunks. Although GAMs can be run autonomously, they can also serve as a low-level controller that a high-level planner controls using its various input modalities, allowing for long-horizon and memory-dependent manipulation. On RoboTwin 2.0, GAM achieves an average success rate of 55.3% across 50 tasks (vs. 52.0% for Spatial Forcing), including 47.6% under scene randomization (vs. 30.4% for Abot-M0), with its action policy trained only on clean-scene demonstrations. On LIBERO-PRO, it achieves a state-of-the-art average success rate of 61% (vs. 53% for

π_{0.5}

) across 16 perturbation settings, with the largest gains when targets are relocated or newly designated. On two real robots, GAM retains 17/20 successes under visual shift on a bimanual YAM versus 4/20 for

π_{0.5}

, while its composition with a Molmo2 planner on a Franka achieves 64.7% ID and 49.8% OOD step completion on long-horizon and memory-dependent tasks.

Abstract

Explore similar work

Grounded Action Model: 3D Grounding as a Foundation for Robotics

World Pilot: Steering Vision-Language-Action Models with World-Action Priors

RynnBrain 1.1: Towards More Capable and Generalizable Embodied Foundation Model