Robots collaborating with humans must convert natural language goals into actionable, physically grounded decisions. For example, executing a command such as "go two meters to the right of the fridge" requires grounding semantic references, spatial relations, and metric constraints within a 3D scene. While recent vision language models (VLMs) demonstrate strong semantic grounding capabilities, they are not explicitly designed to reason about metric constraints in physically defined spaces. In this work, we empirically demonstrate that state-of-the-art VLM-based grounding approaches struggle with complex metric-semantic language queries. To address this limitation, we propose MAPG (Multi-Agent Probabilistic Grounding), an agentic framework that decomposes language queries into structured subcomponents and queries a VLM to ground each component. MAPG then probabilistically composes these grounded outputs to produce metrically consistent, actionable decisions in 3D space. We evaluate MAPG on the HM-EQA benchmark and show consistent performance improvements over strong baselines. Furthermore, we introduce a new benchmark, MAPG-Bench, specifically designed to evaluate metric-semantic goal grounding, addressing a gap in existing language grounding evaluations. We also present a real-world robot demonstration showing that MAPG transfers beyond simulation when a structured scene representation is available.
Figures & tables
Figure 1
Fig. 1 : When a human specifies an open world metric-semantic query, referring to a location in the world, MAPG decomposes the query into subparts as learned probability distributions and composes them for grounding.
Fig. 2 : MAPG overview. (a) The agent observes a 3 D environment and receives a natural-language spatial query. (b) Agentic spatial reasoning layer: The Orchestrator decomposes the query into anchor, relation, and metric components and grounds the anchor instance using multi-view evidence and the 3 D scene graph. (c) MAPG composes probabilistic kernels, producing a continuous target-location PDF in the global frame that can be used as an actionable navigation or object selection goal.
Fig. 3 : MAPG system overview. Egocentric observations ot are fused into a 3 D scene graph Γt of labeled objects with poses and bounding boxes. The Orchestrator parses a language query q into composable symbolic predicates (or,r,d) . The Grounding Agent resolves the referent or against Γt , Spatial Agent constructs analytic kernels κ=κs⋅κm⋅κd and composes them into a goal distribution P(x) over free space. The Verifier checks coherence and triggers corrective retries when needed. Finally, the Planner selects a navigation action at , closing the loop as new observations update Γt . Our contributions:*
Fig. 4 : MAPG grounding for the query “Where is 2m to the right of the fridge?” Semantic grounding identifies the fridge, the metric kernel models the 2m offset, and the directional spatial kernel captures the predicate “right of.” Their analytical composition yields a planner-ready goal distribution over feasible target locations.
Fig. 5 : Distribution of query categories (left) and anchor object labels (right) in MAPG-Bench.
Method
d Err. O-O ↓
d Err. O-W ↓
Yaw Err. ↓
Pitch Err. ↓
Obj. Sel. ↑
Common Sense ↑
TSR ↑
Anchor Pick SR ↑
Avg. Traj. Len ↓
( m )
( m )
( \SIUnitSymbolDegree )
( \SIUnitSymbolDegree )
(Acc.)
(Acc.)
(%)
(%)
( m )
Open-Sourced Specialist
SRGPT
0.5
N/A
N/A
N/A
0.36
N/A
N/A
N/A
N/A
GraphEQA
N/A
5.82
13.5
27.9
0.34
0.14
0.78
0.48
1.32
Ours
MAPG (Gemini 3 Pro)
0.58
0.08
13.8
6.2
0.30
0.14
0.46
0.46
14.6
TABLE I: Comparison of MAPG and base model performance on MAPG-Bench . Proprietary generalist models were benchmarked; they exhibit inconsistent performance and do not reliably produce actionable 3 D goals across tasks, so we omitted them for space (available on our website). ∗ represents best overall model (highlighted among Methods). Metrics with ↑ are higher-is-better; ↓ are lower-is-better. d Err. (O-O/O-W) measures goal localization precision relative to the anchor and in the global frame, while yaw/pitch errors measure directional consistency of the predicted offset (e.g., left/right/front/behind). Obj. Sel. evaluates whether the predicted target satisfies the query (i.e., the agent selects a goal consistent with the intended spatial relation), Anchor Pick SR measures anchor instance disambiguation under partial observability. TSR captures end-to-end reliability of completing the instruction, and Avg. Traj. Len measures behavioral efficiency.
Benchmarking on HM-EQA Question Answering
Method
TSR ↑
Lτ ( m ) ↓
Explore-EQA
0.52
38.1
Explore-EQA-GPT4o
0.46
6.3
Explore-EQA-Llama4-Mav
0.44
10.4
Explore-EQA-Gemini-2.5Pro
0.54
12.3
GraphEQA-Gemini-2.5Pro
0.67
7.41
TABLE II : EQA question-answering accuracy comparison. + indicates our implementation of the particular baseline. While MAPG achieves higher TSR, we also have longer Trajectory length, consistent with our system, which carries out more explorations before deciding on the target object.
Method
Object selection SR ↑
GraphEQA (base)
0.34
MAPG (CoT w/o spatial reasoner)
0.20
MAPG (explicit spatial reasoner)
0.42
TABLE III : Replacing the explicit spatial reasoner with CoT degrades object selection success rate, while the full MAPG variant achieves the best performance.
Method
Object selection SR ↑
GraphEQA (base)
0.30
MAPG (CoT- w/o spatial reasoner)
0.30
MAPG (explicit spatial reasoner)
0.50
TABLE IV : Occlusion highlights the limits of CoT-only reasoning, while the full MAPG variant performs best.
Fig. 6 : Real-world MAPG grounding example. Using observations collected from a Robotis AI Worker robot, we construct an offline scene graph of a physical indoor environment. Three of those image frames are demonstrated here, with the query “Where is 1 meter to the right of the trash can close to the bicycle?” MAPG infers a spatial probability distribution over candidate goal locations (right). The anchor object (red) and predicted goal location (green) illustrate metrically consistent semantic grounding beyond simulation. The grounded area was 1 meter to the right of the blue trash can, which is also the general region of the bicycle.