PIVOT: Physically Informed Vision-Language Off-Road Traversability for Field Robot Navigation

Aoran Jiao; Wenda Zhao; Hshmat Sahak; Timothy D. Barfoot

cs.ROSep 17, 2026

PIVOT: Physically Informed Vision-Language Off-Road Traversability for Field Robot Navigation

Authors: Aoran Jiao, Wenda Zhao, Hshmat Sahak, Timothy D. Barfoot

Abstract

Terrain assessment is a critical capability for off-road mobile robots, enabling safe and reliable navigation through unstructured and geometrically complex environments. Conventional geometry-based terrain assessment is fast to compute but often overly conservative in unstructured environments. We present PIVOT: a Physically Informed Vision-Language Off-Road Traversability navigation system that augments conventional geometry-based planning with vision-language-model (VLM)-based semantic reasoning for field robots. To physically ground this assessment, we quantify how strongly the VLM's predicted traversal energy cost, robot vibration, and wheel slip correlate with real-world measurements and introduce a unified traversability score that weights each modality by its prediction-measurement correlation. For efficiency, we design a two-level navigation architecture that retains geometry-based planning as the nominal mode and invokes semantic replanning only when that mode fails to find a path. Across five repeated closed-loop trials on a mixed-terrain route totalling around $6.4$ km, the proposed system increases overall autonomy from $59.6\%$ to $97.0\%$ , reduces human interventions from $11$ to $3$ , and increases the mean distance between interventions (MDBI) from $69.2$ m to $412.9$ m compared with geometry-only navigation. These results demonstrate that physically grounded VLM-based terrain assessment can substantially extend autonomous navigation beyond the limitations of geometry alone, while preserving efficient geometric planning as the nominal mode.

Figures & tables

Fig. 1: Overview of the proposed PIVOT system. PIVOT augments geometry-based planning with VLM-based semantic terrain assessment, enabling traversal through semantically traversable terrain while retaining conservative behavior around non-traversable obstacles. A video demonstrating the comprehensive closed-loop navigation experiments is available at https://video_placeholder.

Fig. 2: The proposed two-level navigation architecture, consisting of nominal geometry-based planning and semantic fallback planning, with human intervention serving as an external recovery mechanism. The C-BIT ∗ planner first plans on the geometry-based costmap. If no valid plan is found, the system performs VLM-based terrain assessment and replans on a semantic costmap; persistent failure leads to human intervention. Dashed transitions indicate a return to nominal planning after recovery or when a new waypoint is provided.

Fig. 3: Sample LiDAR inputs used for semantic terrain assessment. The cyan highlighting denotes the region of interest for terrain assessment and tells the VLM where the robot is expected to traverse next.

Panel of Figure 4 — Fig. 4: GPT-5 prompt used for physically informed terrain assessment. The prompt specifies the robot platform, coordinate convention, and region of interest, and requests the VLM (GPT-5) to return three separate scores, each in the range $[0,1]$ , for the expected traversal power consumption, robot vibration, and wheel slip.

Fig. 5: Diagram of the spatial window and its corresponding temporal interval.

Fig. 6: Offline physical grounding of the VLM prediction score against normalized electrical power, vibration, and wheel slip. Higher GPT scores indicate greater expected physical difficulty. Blue lines show linear fits, with

R^{2}=0.74

0.67

, and

0.57

, respectively.

Fig. 7: Proposed traversability score $\tau$ versus the reference score $\tau^{\mathrm{ref}}$ , where each point is colored by its geometry-based traversability score (green: low cost; red: high cost). The dark line shows a linear fit. For $\tau^{\mathrm{ref}}$ between approximately $0.3$ and $0.5$ , several yellow and red points correspond to geometry-based scores of about $0.6$ to $1.0$ , indicating overestimation of traversal difficulty. In contrast, the proposed traversability score $\tau$ remains consistent with the physical reference score $\tau^{\textrm{ref}}$ .

Fig. 8: Representative PIVOT-enabled traversal of the complete 1277 m experimental route. We repeated this route five times on three different days, totalling $6.4$ km . The center panel overlays the robot trajectory on a satellite map; stars mark VLM query locations, with colors distinguishing clear, caution, and no-go semantic assessments as indicated in the legend. Adjacent labels report the returned difficulty scores. Manual-driving locations are marked on the route. Numbered images show the corresponding robot–terrain interactions: images 2, 3, and 6 identify locations requiring manual intervention, whereas images 1, 4, and 5 show representative autonomously traversed terrain. A video summarizing these comprehensive closed-loop navigation experiments is available at https://video_placeholder.

Route	Distance (m)	Method	Autonomy (%)		Human interventions			Mean distance between interventions (MDBI) (m)	Time (min)
Route	Distance (m)	Method	Mean	Std	Total	TP	FP	Mean distance between interventions (MDBI) (m)	Mean	Std
		Geometry	85.2	$\pm$ 1.6	3	1	2	90.6	10.2	$\pm$ 0.4
Building Segment	319	PIVOT	96.3	$\pm$ 0.8	1	1	0	307.2	13.3	$\pm$ 1.6
		Geometry	12.3	$\pm$ 1.1	3	0	3	9.1	8.2	$\pm$ 0.4
Grass Segment	221	PIVOT	100.0	$\pm$ 0.0	0	0	0	—	13.8	$\pm$ 1.3
		Geometry	22.7	$\pm$ 1.3	5	2	3	16.2	12.9	$\pm$ 0.5

TABLE I: Online navigation performance over five repeated trials totalling

6.4

km. Autonomy rate and traversal time are reported as mean

\pm

standard deviation. Mean distance between interventions (MDBI) is the autonomous distance traveled per intervention. Each human intervention denotes one continuous operator takeover. TP (true positive) denotes a necessary stop due to genuinely non-traversable terrain; FP (false positive) denotes an unnecessary stop due to an overly conservative assessment of traversable terrain.

Off-Road Robot Navigation

Off-road robot navigation studies how ground robots perceive terrain, assess traversability, and plan safe routes through unstructured environments.

See all 47 papers

Which Terrain Is Better? Preference Learning with VLM Prototypes for Off-Road Traversability Ranking
Sep 20, 2026Ji-Hoon Hwang, Jisung Bae, E-In Son +3Visual Navigation Off-Road Robot Navigation
Distilling Global Traversability Priors for Image-based Affordance Prediction in Off-road Environments
Jul 20, 2026Matthew Sivaprakasam, Samuel Triest, Micah Nye +5Off-Road Robot Navigation Embodied Navigation
Robotics Institute, Carnegie Mellon University · Field AI
Learning Traversability for Long Horizon Off-Road Navigation
Jul 26, 2026Kasi Viswanath, Jason M. Gregory, Shaunak Kolhe +1Off-Road Robot Navigation Geospatial Representation Learning
J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77840, USA · DEVCOM Army Research laboratory, Adelphi, MD, USA

Sep 20, 2026J/K move · Enter open · S save

Which Terrain Is Better? Preference Learning with VLM Prototypes for Off-Road Traversability Ranking

Ji-Hoon Hwang, Jisung Bae, E-In Son, Dong-Wook Kim+2

Visual Navigation Off-Road Robot Navigation Robot Navigation VLMs for Robotics Preference Learning Robot Traversability Estimation

Open paper AskPDF arXiv

In vision-based off-road navigation, a robot needs to know not only which obstacles to avoid but also which terrain is better. The first is handled by freespace detection or semantic segmentation. The second is usually answered with a traversability score, but no universal ground truth exists for such a score, so perception falls back on a predefined value per semantic class or a freespace confidence. These scores say what a region is, not which region a robot should prefer. We therefore formulate this preference as visual traversability ranking, an ordering of visible terrain that can be supervised by comparisons between two regions. Standard annotations do not label preference, but they imply its direction. We present TravPro, which converts these annotations into ordered region pairs and fits a small readout on frozen vision--language model (VLM) patch tokens to these pairs. The tokens are clustered once into a fixed prototype bank, and the readout learns a preference score per prototype. The readout is then applied to every patch and serves as a teacher that turns sparse comparisons into dense preference pseudo-labels without pixel-wise annotation. An RGB student distills these maps into a dense terrain-preference map together with a non-ground mask that excludes obstacles and background from the ranking. On five unseen domains, TravPro reaches a mean pairwise accuracy of 0.915 against 0.783 for the strongest baseline, producing an ordering sensitive to surface condition that a per-class value cannot represent. The same VLM and the same supervision yield no such ordering when the VLM is prompted and the supervision is used as dense targets; what matters is how they are used.

Abstract

Off-Road Robot Navigation

Explore similar work

Which Terrain Is Better? Preference Learning with VLM Prototypes for Off-Road Traversability Ranking

Distilling Global Traversability Priors for Image-based Affordance Prediction in Off-road Environments

Learning Traversability for Long Horizon Off-Road Navigation