Physico-Geospatial Grounded Scene Interpretation for Mobile Robotics

Nicolas Schuler; Janik Kurtz; Lea Dewald; Marcel Sauber; Félicia Teferle; Jürgen Graf

cs.CVSep 6, 2026

Physico-Geospatial Grounded Scene Interpretation for Mobile Robotics

Authors: Nicolas Schuler, Janik Kurtz, Lea Dewald, Marcel Sauber, Félicia Teferle, Jürgen Graf

Organizations: Department of Computer Science Trier University of Applied Sciences Trier, Germany · Department of Engineering University of Luxembourg Kirchberg, Luxembourg

Abstract

Recent advancements in deep learning allow robotic agents to interact with dynamic and unstructured environments. Of special interest is the integration of physico-geospatial world knowledge into such systems, either by using physics-aware machine learning models, knowledge graphs to model relationships or spatio-temporal and logical reasoning. In the present work, we introduce an approach to augment the output of pre-trained, unmodified VLMs used for scene interpretation by integrating semantic descriptions, OpenStreetMap building data and street information with positional, temporal and metric information obtained from our sensory systems, fusing this information using LLMs. We apply this concept to an outdoor recording within a university campus, achieving an F1-Score of 0.83 in the task of grounding buildings and 0.64 for path surface grounding on our pilot evaluation set. The results demonstrate the conceptual capability of the proposed solution to deliver physico-geospatial grounded natural language descriptions. Code and results are available at https://datahub.rz.rptu.de/hstr-csrl-public/publications/physico-geospatial-grounded-scene-interpretation

Figures & tables

Fig. 1: Different cognitive systems used within our laboratory . Unitree Go1 (top left), vehicle (bottom left), Unitree G1 (top and bottom mid), wheelchair (top right) and tripod (bottom right) with custom modular sensory system setups. Images partially taken from [ 41 ] .

Element	Comparison	Subset Accuracy	F1-Score
Buildings	OSM vs GT	$0.56^{+0.08}_{-0.08}$	$0.83^{+0.04}_{-0.04}$
	OSM vs Detection	$0.66^{+0.08}_{-0.08}$	$0.87^{+0.04}_{-0.04}$
	GT vs Detection	$0.76^{+0.06}_{-0.07}$	$0.92^{+0.03}_{-0.03}$
Paths	OSM vs GT	$0.40^{+0.08}_{-0.08}$	$0.64^{+0.06}_{-0.06}$
	OSM vs Detection	$0.53^{+0.08}_{-0.08}$	$0.69^{+0.06}_{-0.06}$
	GT vs Detection	$0.80^{+0.07}_{-0.07}$	$0.90^{+0.04}_{-0.04}$

TABLE I: Accuracy and F1-Score of the proposed solutions . For building and path identification as well as the utilized custom re-identification model. Values are given with 95% confidence intervals, indicated by superscript and subscript notation.

Error Type	Llama4:128x17b	Llama4:16x17b
Hallucination	$0.00^{+0.03}_{-0.00}$	$0.04^{+0.04}_{-0.02}$
Wrong Entities	$0.13^{+0.07}_{-0.05}$	$0.15^{+0.07}_{-0.05}$
Wrong Relations	$0.02^{+0.04}_{-0.01}$	$0.03^{+0.04}_{-0.02}$
Input Description Left Out	$0.04^{+0.04}_{-0.02}$	$0.01^{+0.03}_{-0.01}$
Not Included All Objects	$0.02^{+0.04}_{-0.01}$	$0.09^{+0.06}_{-0.04}$
Not Included All Buildings	$0.09^{+0.06}_{-0.04}$	$0.12^{+0.06}_{-0.04}$

TABLE II: LLM Error Statistic . Error prevalence of the evaluated categories. Values are given with 95% confidence intervals, indicated by superscript and subscript notation.

Attribute	Model	Cohen’s $\kappa$	Agreement
Hallucination	Llama4:128x17b	$0.000\,000$	$0.971\,831$
	Llama4:16x17b	$0.585\,401$	$0.978\,31$
Wrong Entities	Llama4:128x17b	$0.585\,081$	$0.880\,282$
	Llama4:16x17b	$0.876\,950$	$0.964\,789$
Wrong Relations	Llama4:128x17b	$0.486\,438$	$0.971\,831$
	Llama4:16x17b	$0.585\,401$	$0.971\,831$

TABLE III: Annotation Reliability as inter-rater agreement measured by Cohen’s

\kappa

[ 50 ] and ‘Agreement’ as a ratio. Annotations of augmented descriptions compared to the baseline.

Fig. 2: Exemplary image from the evaluation data . Buildings O, N and L detected and identified by the pipeline, manually highlighted for clarity.

Robotic Perception

Robotic perception enables robots to interpret scenes, objects, motion, and contact from cameras and other sensors to guide action.

See all 321 papers

Language as a Sensor: Calibrated Spatial Belief Estimation in 3D Scenes from Natural Language
Jun 7, 2026Aryan Naveen, Jason Xinyu Liu, Luca Carlone +1Vision-Language Grounding Semantic Mapping
MIT Laboratory for Information & Decision Systems · MIT Computer Science & Artificial Intelligence Laboratory
Meanings and Measurements: Multi-Agent Probabilistic Grounding for Vision-Language Navigation
Mar 19, 2026Swagat Padhan, Lakshya Jain, Bhavya Minesh Shah +3Visual Grounding for Robotics Spatial Grounding for VLA Models
Arizona State University · Haverford College
ReSiReg: Towards Spatially Consistent Semantics in Language-Conditioned Robotic Tasks
Jun 17, 2026Simon Schwaiger, David Seyser, Alessandro Scherl +2Vision-Language Models Vision-Language Grounding
Graz University of Technology, Institute of Software Engineering and Artificial Intelligence · University of Applied Sciences Technikum Wien, Department of Industrial Engineering · 1Graz University of Technology, Institute of Software Engineering and Artificial Intelligence +2

Jun 7, 2026J/K move · Enter open · S save

Language as a Sensor: Calibrated Spatial Belief Estimation in 3D Scenes from Natural Language

Aryan Naveen, Jason Xinyu Liu, Luca Carlone, Andreea Bobu

MIT Laboratory for Information & Decision Systems · MIT Computer Science & Artificial Intelligence Laboratory

Vision-Language Grounding Semantic Mapping Multimodal Sensor Fusion Multimodal Understanding 3D Semantic Mapping

Open paper AskPDF arXiv

Robots deployed in human-centric environments routinely receive natural-language descriptions of spatial information ("I left my backpack on the table") that reference parts of the world beyond their perceptual field of view. Traditional metric-semantic mapping ignores this signal, while off-the-shelf multimodal models remain limited in 3D spatial reasoning and are not directly amenable to fusion with other sensor modalities. To convert language observations into a calibrated spatial distribution, we train a Language Sensor Model (LSM) that maps each utterance and its scene-graph context to a multimodal distribution, with mixture weights encoding referential ambiguity (e.g., which table) and component covariances encoding spatial uncertainty (e.g., where on the table the target lies). We then introduce VL-Map (Vision-Language Metric-Semantic Mapping), a probabilistic framework that treats these language predictions as stochastic observations and fuses them with onboard perception within a unified belief map. On the VLA-3D benchmark as well as on a real-world mobile robot, LSM is the only language predictor whose covariance estimates remain within the calibrated regime; fused into VL-Map, it leads to more accurate predictions of the target object location (about 70% more probability mass on the true target compared to the strongest foundation-model baseline). Project website: https://aryan-naveen.github.io/language-as-a-sensor/

Abstract

Robotic Perception

Explore similar work

Language as a Sensor: Calibrated Spatial Belief Estimation in 3D Scenes from Natural Language

Meanings and Measurements: Multi-Agent Probabilistic Grounding for Vision-Language Navigation

ReSiReg: Towards Spatially Consistent Semantics in Language-Conditioned Robotic Tasks