Traversability prediction is a core component of autonomous navigation, yet most vision-based methods operate frame-wise and produce temporally inconsistent predictions. We present SAM-VTP (Video Traversability Prediction), which fine-tunes SAM2's video memory pathway for dense, prompt-free traversability segmentation, conditioning each frame on a persistent memory of prior observations. Rather than inferring terrain geometry from appearance, we measure it: slope, roughness and height are computed from depth and fused into the image features. We also introduce an offline continuous supervision pipeline that generates dense per-pixel traversability targets in [0,1] from depth-derived geometry and semantic priors, requiring no robot deployment and no per-pixel annotation, capturing conditional feasibility that binary labels cannot represent. On TartanGround, a large-scale synthetic ground-robot dataset, SAM-VTP improves fuzzy Jaccard from 0.908 to 0.924 and binary IoU from 0.854 to 0.886 over SAM-TP, a frame-wise SAM2 baseline, against a seed-to-seed spread below 0.001, and cuts frame-to-frame flicker by 31%. In closed-loop trials at two field sites, driving the same planner and controller, the baseline's dropouts leave the planner without a feasible path 3.3x more often on a paved course and 6.9x on rough ground, while SAM-VTP runs at 21.4 FPS on the Orin Nano the robot carries.
Figures & tables
Fig. 1: Temporal comparison of SAM-TP (top), the frame-wise SAM2 traversability model released with GeNIE [ 1 ] and the closest published system to ours, against SAM-VTP (bottom) on consecutive frames from TartanGround ConstructionSite. At t2 SAM-TP loses roughly half of the traversable surface and recovers it at t3 , while SAM-VTP holds a coherent region across all three frames (fuzzy Jaccard, fJacc, 0.69 against 0.92 on this window). SAM-VTP’s output is also traversability-scaled, whereas SAM-TP’s is dominated by binary confidence.
Fig. 2: SAM-VTP: SAM2’s Hiera-T encoder and memory pathway. The four starred blocks are new or modified in this work: a depth-derived geometry encoder summed into the image features, a learned prompt replacing SAM2’s prompt encoder, a decoder emitting a single mask token, and a residual head refining the output at full resolution. Parameter counts are per block; the memory bank holds activations, not parameters.
Fig. 3: Continuous traversability supervision on TartanGround. Left to right: the semantic labels for one frame, the geometry-derived height above ground and local surface slope, and the final target y∈[0,1] . The target combines a platform-grounded semantic prior with continuous geometric modulation, and cannot be trivially derived from semantics or geometry alone.
Accuracy
Temporal
Throughput (FPS) ↑
Model
fJacc ↑
MAE ↓
ECE ↓
IoU ↑
Stab. ↓
Flick. ↓
RTX 5090
AGX Orin
Orin Nano
WayFASTER
0.791
0.066
0.039
0.740
0.0630
0.0264
456
52.8
26.8
WVN
0.760
0.079
0.048
0.741
0.0597
0.0246
19.5
– ‡
– ‡
SAM-TP
0.908
0.024
0.013
0.854
0.0387
0.0175
540*
44.9*
33.5*
SAM-VTP (ours)
0.924
0.018
0.010
0.886
0.0323
0.0120
296
30.1
21.4
SAM-TP (GeNIE) †
0.667
0.126
0.111
0.678
0.0908
0.0452
45.7
6.3
3.6
TABLE I: Accuracy, temporal consistency, and throughput on held-out TartanGround test trajectories. All models fine-tuned on TartanGround unless noted. Best in bold . fJacc is the fuzzy (Ruzicka) Jaccard index over continuous scores and is the primary metric; IoU is the binary Jaccard index at a per-model threshold calibrated on validation; ECE is the expected calibration error against the continuous targets. Stab. and Flick. are the mean per-sequence standard deviation of fJacc across frames and its mean absolute frame-to-frame change, both lower-is-better; basing them on fJacc rather than a thresholded score keeps them on the primary metric. Throughput uses each model’s own deployment resolution ( 512×512 except WayFASTER, which predicts a 100×100 BEV grid from 320×180 input; with targets interpolated to that grid and predictions upsampled back, limiting its attainable fidelity).
Component
Configuration
Mem.
Geo.
fJacc ↑
MAE ↓
ECE ↓
Flick. ↓
SAM-TP (frame-wise)
–
–
0.875
0.033
0.019
0.022
+ geometry
–
✓
0.900
0.026
0.014
0.019
SAM-VTP (memory)
✓
–
0.894
0.028
0.016
0.015
+ geometry (full)
✓
✓
0.914
0.022
0.013
0.013
Full, depth withheld
✓
–
0.853
0.042
0.031
0.020
TABLE II: Ablation on held-out TartanGround test trajectories, scored over 1000 sequence windows (24k frames). The first four rows form a 2×2 factorial over the two proposed components; SAM-TP + geometry is the fairness control, giving the frame-wise baseline the same depth-derived input as SAM-VTP. The last two rows withhold depth at inference and replace continuous targets with binary ones. Training budgets here are shorter than in Table I , so absolute scores are lower and the two tables are not directly comparable. Best in bold .
Fig. 4: Per-environment difference in fJacc between SAM-VTP and SAM-TP on the 22 held-out test trajectories, sorted by magnitude; bars are the mean over three seeds and error bars their standard deviation. SAM-VTP is ahead in 21 of 22 environments. The grey column gives SAM-TP’s own fJacc, showing that the largest gains fall where the frame-wise baseline is weakest ( r=−0.61 ) while environments it already scores above 0.95 gain least. OldScandinavia is clipped at the axis limit; its value is +0.141 .
Paved course
Rough terrain
Measure
TP
VTP
Ratio
TP
VTP
Ratio
No-path (%) ↓
32.4
9.9
0.31 ×
20.6
3.0
0.15 ×
Traversable frac. ↑
0.225
0.338
1.50 ×
0.076
0.216
2.86 ×
Flicker (region) ↓
0.129
0.030
0.23 ×
0.055
0.036
0.64 ×
Null predictions (%) ↓
7.9
0.9
0.12 ×
4.2
0.8
0.18 ×
Progress rate (m/s) ↑
0.141
0.182
1.29 ×
0.159
0.180
1.13 ×
TABLE III: Closed-loop navigation on a physical robot at two field sites: a paved course with mixed surfaces and vegetation (15 runs), and rough, undulating, overgrown ground (21 runs). The baseline is SAM-TP trained on the same data, objective and splits, with planner and controller shared and trials interleaved between models at both sites. SAM-TP takes RGB only and SAM-VTP takes RGB-D, so the table compares the two systems and does not separate memory from depth. Flicker is the mean absolute frame-to-frame change in predicted probability over the predicted region, the union of P>0.5 across the pair, as are traversable fraction and null predictions; both arms select P>0.5 independently on validation, and the ordering holds for any threshold in [0.1,0.8] . Every row separates completely on the course; on rough ground no-path, traversable fraction and flicker separate in all ten paired trials, a sign test at its floor ( p=0.002 ), while progress rate and distance overlap ( 8/10 and 7/10 pairs).
Fig. 5: Both models replayed over the same recorded frames in the deployed configuration (SAM-TP on RGB, SAM-VTP on RGB-D), so both see the same scene. Overlay opacity is the predicted probability; the white contour marks P>0.5 . The top two rows are the paved course, where SAM-TP breaks into disconnected fragments around clutter (2% against 32%) and again at the terrain and lighting transition at which its trials ended, where the planner was reporting no feasible path (2% against 28%); SAM-VTP holds a single coherent region in both. The bottom row is rough terrain, where SAM-TP under-segments a clearly defined path (6% against 25%) rather than fragmenting.
Visual traversability estimation is central to autonomous navigation, yet most approaches either rely on prompt-driven Vision-Language Model (VLM) or decouple traversability from trajectory planning, requiring separate planners with heavy mapping, manual tuning, and extended deployment time. We propose EmbodiedDiffusion, a diffusion-based framework that simultaneously predicts traversability maps and generates feasible trajectories from RGB images using planner-free synthetic supervision and embodiment conditioning for cross-platform transfer. The framework distills category-level traversability semantics from a VLM teacher into a lightweight student model during training, enabling prompt-free, real-time inference at deployment. A modular FiLM-based conditioning mechanism isolates embodiment-specific reasoning into a compact trainable subset of the network, allowing rapid adaptation to new robot platforms without retraining the visual backbone or the trajectory diffusion model. Across indoor environments with quadruped and aerial robots, EmbodiedDiffusion achieves 80-100% navigation success in the full-data regime with real-time inference (90 ms) and adapts to new platforms using only 10 min of visual data collection, demonstrating scalable, unified traversability reasoning and trajectory generation for heterogeneous robots.
Iana Zhura, Sausar Karaf, Faryal Batool +7
Intelligent Space Robotics Laboratory, Center for Digital Engineering, Skolkovo Institute of Science and Technology.
Traversability analysis in off-road settings remains a fundamental challenge for mobile robots. Key difficulties include constructing an accurate and expressive local map from multimodal sensor data, and using the map to design traversability rules that yield desirable navigation behavior. Importantly, this system must be resilient to the limited sensing regime brought about by complex environments and high speeds. In this paper, we present TravSUITE, a traversability system suitable for high-speed navigation in off-road environments. TravSUITE consists of two major components: 1) an efficient voxel mapper that leverages visual foundation models (VFMs) to build a rich geometric-semantic local map from streams of on-board sensor data, and 2) a unified neural network that jointly predicts traversability-relevant quantities in bird's eye view (BEV), including geometry, semantics, speed and cost. Our training strategy is entirely annotation-free and self-supervised, leveraging tasks such as map inpainting and inverse reinforcement learning (IRL) to learn both map representations and traversability. We perform a thorough ablation study and comparison to state-of-the-art approaches, and the results indicate that cost learning and auxiliary inpainting each contribute significantly to planning quality, and their combination is critical for achieving state-of-the-art performance in off-road path planning. We also design a simple risk adaptation mechanism to leverage our method's uncertainty estimates at deploy-time, and demonstrate that a combination of inpainting and risk estimation can result in 80% fewer navigation errors and 5% faster autonomous traversal speeds in real-world hardware experiments.
Samuel Triest, Amirreza Shaban, David D. Fan +2
Robotics Institute, Carnegie Mellon University · Field AI
Vision-based approaches have become the dominant paradigm for traversability estimation in unstructured outdoor environments, typically adapting vision foundation models (VFMs) via semantic segmentation supervision. However, this paradigm faces three fundamental challenges that undermine its reliability: the task-agnostic design of VFMs, the ambiguity of traversability annotations, and the discrepancy between semantic labels and physical safety. We propose Vision-to-Traversability Adaptation (ViTA), a framework that adapts VFMs for reliable traversability estimation, instantiated on SAM2. ViTA injects task-specific knowledge through learnable traversability prompts while preserving the VFM's cross-domain generalization. To handle annotation ambiguity, we introduce Perspective-Diversified Training, which estimates semantic uncertainty to suppress confident predictions at ambiguous boundaries. To bridge the semantic-traversability discrepancy, we distill geometric knowledge during training, enabling slope and elevation reasoning from RGB images alone at inference. The semantic and geometric outputs are fused into a continuous traversability score that reflects both semantic uncertainty and geometric risk. Evaluations across diverse domains, including challenging real-world off-road datasets, demonstrate that ViTA achieves state-of-the-art IoU and Precision with substantial false-positive reduction and strong cross-domain generalization.